2024-03-24T10:23:53.075 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T10:23:53.075 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/fsstress validater/valgrind} --archive /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619063 --owner scheduled_pdonnell@teuthology --name pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi -- /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619063/orig.config.yaml 2024-03-24T10:23:53.120 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T10:23:53.207 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619063 branch: wip-batrick-testing-20240323.003144-squid description: fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/fsstress validater/valgrind} email: pdonnell@redhat.com first_in_suite: false job_id: '7619063' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240323.003144-squid ceph: cephfs: max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mds beacon grace: 60 mds heartbeat grace: 60 osd heartbeat grace: 60 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds valgrind exit: true mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 mon osd crush smoke test: false osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd fast shutdown: false osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow request - SLOW_OPS - MON_DOWN sha1: 21b3bd672163092b4e542f772797fcba02d3d4de valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes watchdog: daemon_restart: normal ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: valgrind: - --tool=memcheck - --leak-check=full - --show-reachable=yes check-counter: counters: mds: - mds.exported - mds.imported install: ceph: debuginfo: true flavor: default sha1: 21b3bd672163092b4e542f772797fcba02d3d4de selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240323.003144-squid sha1: d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 4284 sha1: 21b3bd672163092b4e542f772797fcba02d3d4de sleep_before_teardown: 0 subset: 405/512 suite: fs suite_branch: wip-batrick-testing-20240323.003144-squid suite_path: /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi005.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFn+kiqlbs5zbTuBBHsHzEb/WYC+FI9IOgEygpTTmkkomwTk+KNf0/B2ea6hscWCvKN0mtgrm8ZyKJLkRdZ3nW8= smithi057.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMokMhSL3Mys8idkxCh6J+sEHRTJAMEjyaIKAU1fo3o+pBIjGzyT4BjzlGmoyU6eIQ8lbjbW+2u15K3TYKI+qsI= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - workunit: clients: all: - suites/fsstress.sh timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-24_04:56:01 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-24T10:23:53.207 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa; will attempt to use it 2024-03-24T10:23:53.208 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks 2024-03-24T10:23:53.208 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T10:23:53.210 INFO:teuthology.task.internal:Checking packages... 2024-03-24T10:23:53.233 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '21b3bd672163092b4e542f772797fcba02d3d4de' 2024-03-24T10:23:53.233 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T10:23:53.233 INFO:teuthology.packaging:ref: None 2024-03-24T10:23:53.233 INFO:teuthology.packaging:tag: None 2024-03-24T10:23:53.233 INFO:teuthology.packaging:branch: wip-batrick-testing-20240323.003144-squid 2024-03-24T10:23:53.233 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:23:53.233 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240323.003144-squid 2024-03-24T10:23:53.367 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1855.g21b3bd67 2024-03-24T10:23:53.369 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T10:23:53.375 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T10:23:53.375 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T10:23:53.389 INFO:teuthology.task.internal:Saving configuration 2024-03-24T10:23:53.406 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T10:23:53.412 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T10:23:53.437 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619063', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:18:53.323605', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFn+kiqlbs5zbTuBBHsHzEb/WYC+FI9IOgEygpTTmkkomwTk+KNf0/B2ea6hscWCvKN0mtgrm8ZyKJLkRdZ3nW8='} 2024-03-24T10:23:53.458 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619063', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:18:53.324931', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMokMhSL3Mys8idkxCh6J+sEHRTJAMEjyaIKAU1fo3o+pBIjGzyT4BjzlGmoyU6eIQ8lbjbW+2u15K3TYKI+qsI='} 2024-03-24T10:23:53.458 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T10:23:53.464 INFO:teuthology.task.internal:roles: ubuntu@smithi005.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-03-24T10:23:53.464 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-03-24T10:23:53.464 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T10:23:53.532 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f17a9e7a160>, signals=[15]) 2024-03-24T10:23:53.533 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T10:23:53.539 INFO:teuthology.task.internal:Opening connections... 2024-03-24T10:23:53.540 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi005.front.sepia.ceph.com 2024-03-24T10:23:53.541 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:23:53.643 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi057.front.sepia.ceph.com 2024-03-24T10:23:53.644 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:23:53.744 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T10:23:53.751 DEBUG:teuthology.orchestra.run.smithi005:> uname -m 2024-03-24T10:23:53.768 INFO:teuthology.orchestra.run.smithi005.stdout:x86_64 2024-03-24T10:23:53.769 DEBUG:teuthology.orchestra.run.smithi005:> cat /etc/os-release 2024-03-24T10:23:53.825 INFO:teuthology.orchestra.run.smithi005.stdout:NAME="CentOS Stream" 2024-03-24T10:23:53.825 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION="9" 2024-03-24T10:23:53.825 INFO:teuthology.orchestra.run.smithi005.stdout:ID="centos" 2024-03-24T10:23:53.825 INFO:teuthology.orchestra.run.smithi005.stdout:ID_LIKE="rhel fedora" 2024-03-24T10:23:53.826 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_ID="9" 2024-03-24T10:23:53.826 INFO:teuthology.orchestra.run.smithi005.stdout:PLATFORM_ID="platform:el9" 2024-03-24T10:23:53.826 INFO:teuthology.orchestra.run.smithi005.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T10:23:53.826 INFO:teuthology.orchestra.run.smithi005.stdout:ANSI_COLOR="0;31" 2024-03-24T10:23:53.826 INFO:teuthology.orchestra.run.smithi005.stdout:LOGO="fedora-logo-icon" 2024-03-24T10:23:53.826 INFO:teuthology.orchestra.run.smithi005.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T10:23:53.826 INFO:teuthology.orchestra.run.smithi005.stdout:HOME_URL="https://centos.org/" 2024-03-24T10:23:53.826 INFO:teuthology.orchestra.run.smithi005.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T10:23:53.826 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T10:23:53.826 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T10:23:53.827 INFO:teuthology.lock.ops:Updating smithi005.front.sepia.ceph.com on lock server 2024-03-24T10:23:53.855 DEBUG:teuthology.orchestra.run.smithi057:> uname -m 2024-03-24T10:23:53.871 INFO:teuthology.orchestra.run.smithi057.stdout:x86_64 2024-03-24T10:23:53.872 DEBUG:teuthology.orchestra.run.smithi057:> cat /etc/os-release 2024-03-24T10:23:53.928 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="CentOS Stream" 2024-03-24T10:23:53.928 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="9" 2024-03-24T10:23:53.929 INFO:teuthology.orchestra.run.smithi057.stdout:ID="centos" 2024-03-24T10:23:53.929 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE="rhel fedora" 2024-03-24T10:23:53.929 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="9" 2024-03-24T10:23:53.929 INFO:teuthology.orchestra.run.smithi057.stdout:PLATFORM_ID="platform:el9" 2024-03-24T10:23:53.929 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T10:23:53.929 INFO:teuthology.orchestra.run.smithi057.stdout:ANSI_COLOR="0;31" 2024-03-24T10:23:53.929 INFO:teuthology.orchestra.run.smithi057.stdout:LOGO="fedora-logo-icon" 2024-03-24T10:23:53.929 INFO:teuthology.orchestra.run.smithi057.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T10:23:53.929 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://centos.org/" 2024-03-24T10:23:53.929 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T10:23:53.929 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T10:23:53.929 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T10:23:53.929 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server 2024-03-24T10:23:54.050 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T10:23:54.059 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T10:23:54.065 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T10:23:54.065 DEBUG:teuthology.orchestra.run.smithi005:> test '!' -e /home/ubuntu/cephtest 2024-03-24T10:23:54.068 DEBUG:teuthology.orchestra.run.smithi057:> test '!' -e /home/ubuntu/cephtest 2024-03-24T10:23:54.084 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T10:23:54.090 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T10:23:54.090 DEBUG:teuthology.orchestra.run.smithi005:> test -z $(ls -A /var/lib/ceph) 2024-03-24T10:23:54.126 DEBUG:teuthology.orchestra.run.smithi057:> test -z $(ls -A /var/lib/ceph) 2024-03-24T10:23:54.151 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T10:23:54.207 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T10:23:54.221 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-24T10:23:54.222 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-24T10:23:54.222 DEBUG:teuthology.orchestra.run.smithi005:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T10:23:54.223 DEBUG:teuthology.orchestra.run.smithi057:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T10:23:54.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:23:54.239 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-03-24T10:23:54.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:23:54.240 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2024-03-24T10:23:54.297 INFO:teuthology.orchestra.run.smithi005.stdout:5.14.0-361.el9.x86_64 2024-03-24T10:23:54.297 INFO:teuthology.task.kernel:Running kernel on smithi005: 5.14.0-361.el9.x86_64 2024-03-24T10:23:54.298 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum install -y kernel 2024-03-24T10:23:54.300 INFO:teuthology.orchestra.run.smithi057.stdout:5.14.0-361.el9.x86_64 2024-03-24T10:23:54.300 INFO:teuthology.task.kernel:Running kernel on smithi057: 5.14.0-361.el9.x86_64 2024-03-24T10:23:54.300 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum install -y kernel 2024-03-24T10:23:56.359 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-24T10:23:56.664 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-24T10:23:57.917 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.0 MB 00:00 2024-03-24T10:23:58.064 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-9 - BaseOS 7.7 MB/s | 2.0 MB 00:00 2024-03-24T10:23:58.952 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-24T10:24:00.700 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-03-24T10:24:07.629 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - BaseOS 892 kB/s | 8.0 MB 00:09 2024-03-24T10:24:12.159 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - CRB 793 kB/s | 6.1 MB 00:07 2024-03-24T10:24:14.077 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 16 kB 00:00 2024-03-24T10:24:16.027 INFO:teuthology.orchestra.run.smithi057.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-24T10:24:21.010 INFO:teuthology.orchestra.run.smithi057.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-24T10:24:23.209 INFO:teuthology.orchestra.run.smithi057.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T10:24:23.209 INFO:teuthology.orchestra.run.smithi057.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T10:24:23.309 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-24T10:24:23.314 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-24T10:24:23.314 INFO:teuthology.orchestra.run.smithi057.stdout: Package Architecture Version Repository Size 2024-03-24T10:24:23.314 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-24T10:24:23.315 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-03-24T10:24:23.315 INFO:teuthology.orchestra.run.smithi057.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T10:24:23.315 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-03-24T10:24:23.315 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T10:24:23.315 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T10:24:23.315 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T10:24:23.315 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:24:23.315 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-03-24T10:24:23.315 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2024-03-24T10:24:23.315 INFO:teuthology.orchestra.run.smithi057.stdout:Install 4 Packages 2024-03-24T10:24:23.315 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:24:23.316 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 76 M 2024-03-24T10:24:23.316 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 123 M 2024-03-24T10:24:23.316 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-03-24T10:24:23.745 INFO:teuthology.orchestra.run.smithi057.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 272 kB/s | 44 kB 00:00 2024-03-24T10:24:24.195 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-24T10:24:28.313 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-24T10:24:29.780 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-03-24T10:24:31.109 INFO:teuthology.orchestra.run.smithi005.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-24T10:24:36.173 INFO:teuthology.orchestra.run.smithi005.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-03-24T10:24:38.296 INFO:teuthology.orchestra.run.smithi005.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T10:24:38.297 INFO:teuthology.orchestra.run.smithi005.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T10:24:38.389 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-24T10:24:38.397 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-24T10:24:38.397 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-03-24T10:24:38.397 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-24T10:24:38.397 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-24T10:24:38.398 INFO:teuthology.orchestra.run.smithi005.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T10:24:38.398 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-03-24T10:24:38.398 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T10:24:38.398 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T10:24:38.398 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T10:24:38.398 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:24:38.398 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-24T10:24:38.398 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-24T10:24:38.398 INFO:teuthology.orchestra.run.smithi005.stdout:Install 4 Packages 2024-03-24T10:24:38.398 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:24:38.399 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 76 M 2024-03-24T10:24:38.399 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 123 M 2024-03-24T10:24:38.399 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-24T10:24:38.633 INFO:teuthology.orchestra.run.smithi005.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 490 kB/s | 44 kB 00:00 2024-03-24T10:24:38.983 INFO:teuthology.orchestra.run.smithi005.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 34 MB/s | 15 MB 00:00 2024-03-24T10:24:39.391 INFO:teuthology.orchestra.run.smithi005.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 36 MB/s | 27 MB 00:00 2024-03-24T10:24:39.708 INFO:teuthology.orchestra.run.smithi005.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 29 MB/s | 33 MB 00:01 2024-03-24T10:24:39.709 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:24:39.709 INFO:teuthology.orchestra.run.smithi005.stdout:Total 58 MB/s | 76 MB 00:01 2024-03-24T10:24:40.403 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-24T10:24:40.857 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-24T10:24:40.857 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-24T10:24:41.912 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-24T10:24:41.912 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-24T10:24:44.508 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-24T10:24:45.068 INFO:teuthology.orchestra.run.smithi057.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 710 kB/s | 15 MB 00:21 2024-03-24T10:24:45.138 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T10:24:45.225 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:24:48.085 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:24:48.218 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:24:53.299 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:24:53.342 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:24:55.366 INFO:teuthology.orchestra.run.smithi057.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 889 kB/s | 27 MB 00:31 2024-03-24T10:24:58.459 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:25:01.116 INFO:teuthology.orchestra.run.smithi057.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 912 kB/s | 33 MB 00:37 2024-03-24T10:25:01.116 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:25:01.116 INFO:teuthology.orchestra.run.smithi057.stdout:Total 2.0 MB/s | 76 MB 00:37 2024-03-24T10:25:01.809 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-03-24T10:25:02.221 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-03-24T10:25:02.221 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-03-24T10:25:03.276 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-03-24T10:25:03.276 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-03-24T10:25:05.867 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-03-24T10:25:06.512 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T10:25:06.636 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:25:09.558 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:25:09.685 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:25:14.754 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:25:14.793 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:25:19.903 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:25:29.253 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:25:29.254 INFO:teuthology.orchestra.run.smithi005.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T10:25:29.254 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:25:29.299 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:25:29.690 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:25:29.691 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T10:25:29.691 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:25:29.691 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:25:30.088 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:25:30.088 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:25:30.088 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-24T10:25:30.089 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T10:25:30.089 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T10:25:30.089 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T10:25:30.089 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T10:25:30.089 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:25:30.089 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-24T10:25:30.425 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T10:25:30.529 INFO:teuthology.orchestra.run.smithi005.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T10:25:30.530 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T10:25:30.530 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T10:25:30.530 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T10:25:30.530 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T10:25:30.530 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-24T10:25:30.530 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi005.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T10:25:30.530 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T10:25:30.613 INFO:teuthology.orchestra.run.smithi005.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T10:25:30.614 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T10:25:30.614 DEBUG:teuthology.orchestra.run.smithi005:> sudo rpm -qi grub2-tools 2024-03-24T10:25:30.657 INFO:teuthology.orchestra.run.smithi005.stdout:Name : grub2-tools 2024-03-24T10:25:30.657 INFO:teuthology.orchestra.run.smithi005.stdout:Epoch : 1 2024-03-24T10:25:30.657 INFO:teuthology.orchestra.run.smithi005.stdout:Version : 2.06 2024-03-24T10:25:30.657 INFO:teuthology.orchestra.run.smithi005.stdout:Release : 68.el9 2024-03-24T10:25:30.658 INFO:teuthology.orchestra.run.smithi005.stdout:Architecture: x86_64 2024-03-24T10:25:30.658 INFO:teuthology.orchestra.run.smithi005.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T10:25:30.658 INFO:teuthology.orchestra.run.smithi005.stdout:Group : Unspecified 2024-03-24T10:25:30.658 INFO:teuthology.orchestra.run.smithi005.stdout:Size : 8264254 2024-03-24T10:25:30.658 INFO:teuthology.orchestra.run.smithi005.stdout:License : GPLv3+ 2024-03-24T10:25:30.658 INFO:teuthology.orchestra.run.smithi005.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T10:25:30.658 INFO:teuthology.orchestra.run.smithi005.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T10:25:30.658 INFO:teuthology.orchestra.run.smithi005.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T10:25:30.658 INFO:teuthology.orchestra.run.smithi005.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T10:25:30.658 INFO:teuthology.orchestra.run.smithi005.stdout:Packager : builder@centos.org 2024-03-24T10:25:30.658 INFO:teuthology.orchestra.run.smithi005.stdout:Vendor : CentOS 2024-03-24T10:25:30.658 INFO:teuthology.orchestra.run.smithi005.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T10:25:30.658 INFO:teuthology.orchestra.run.smithi005.stdout:Summary : Support tools for GRUB. 2024-03-24T10:25:30.659 INFO:teuthology.orchestra.run.smithi005.stdout:Description : 2024-03-24T10:25:30.659 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:25:30.659 INFO:teuthology.orchestra.run.smithi005.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T10:25:30.659 INFO:teuthology.orchestra.run.smithi005.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T10:25:30.659 INFO:teuthology.orchestra.run.smithi005.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T10:25:30.659 INFO:teuthology.orchestra.run.smithi005.stdout:hardware devices. 2024-03-24T10:25:30.659 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:25:30.659 INFO:teuthology.orchestra.run.smithi005.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T10:25:30.660 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T10:25:30.660 INFO:teuthology.task.kernel:Updating grub on smithi005 to boot 5.14.0-430.el9.x86_64 2024-03-24T10:25:30.660 DEBUG:teuthology.orchestra.run.smithi005:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T10:25:31.301 INFO:teuthology.orchestra.run.smithi005.stderr:Generating grub configuration file ... 2024-03-24T10:25:32.727 INFO:teuthology.orchestra.run.smithi005.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T10:25:32.760 INFO:teuthology.orchestra.run.smithi005.stderr:done 2024-03-24T10:25:32.762 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-03-24T10:25:32.817 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.CQe8C11ALi 2024-03-24T10:25:32.817 DEBUG:teuthology.orchestra.run.smithi005:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.CQe8C11ALi 2024-03-24T10:25:32.897 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.CQe8C11ALi 2024-03-24T10:25:33.039 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.CQe8C11ALi is 6KB 2024-03-24T10:25:33.088 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.CQe8C11ALi 2024-03-24T10:25:33.102 DEBUG:teuthology.orchestra.run.smithi005:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T10:25:33.171 INFO:teuthology.orchestra.run.smithi005.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T10:25:33.171 INFO:teuthology.orchestra.run.smithi005.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T10:25:33.171 INFO:teuthology.orchestra.run.smithi005.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T10:25:33.171 INFO:teuthology.orchestra.run.smithi005.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T10:25:33.172 DEBUG:teuthology.orchestra.run.smithi005:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T10:25:33.333 DEBUG:teuthology.orchestra.run.smithi005:> sudo shutdown -r now 2024-03-24T10:25:50.768 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:25:50.768 INFO:teuthology.orchestra.run.smithi057.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T10:25:50.768 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:25:50.813 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:25:51.227 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:25:51.228 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T10:25:51.228 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:25:51.228 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:25:51.653 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:25:51.653 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:25:51.654 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-03-24T10:25:51.654 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T10:25:51.654 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T10:25:51.654 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T10:25:51.654 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T10:25:51.654 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:25:51.654 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-24T10:25:51.979 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T10:25:52.080 INFO:teuthology.orchestra.run.smithi057.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T10:25:52.080 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T10:25:52.080 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T10:25:52.080 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T10:25:52.080 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T10:25:52.080 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-24T10:25:52.080 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi057.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T10:25:52.080 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T10:25:52.162 INFO:teuthology.orchestra.run.smithi057.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T10:25:52.163 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T10:25:52.163 DEBUG:teuthology.orchestra.run.smithi057:> sudo rpm -qi grub2-tools 2024-03-24T10:25:52.209 INFO:teuthology.orchestra.run.smithi057.stdout:Name : grub2-tools 2024-03-24T10:25:52.209 INFO:teuthology.orchestra.run.smithi057.stdout:Epoch : 1 2024-03-24T10:25:52.209 INFO:teuthology.orchestra.run.smithi057.stdout:Version : 2.06 2024-03-24T10:25:52.209 INFO:teuthology.orchestra.run.smithi057.stdout:Release : 68.el9 2024-03-24T10:25:52.209 INFO:teuthology.orchestra.run.smithi057.stdout:Architecture: x86_64 2024-03-24T10:25:52.209 INFO:teuthology.orchestra.run.smithi057.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T10:25:52.209 INFO:teuthology.orchestra.run.smithi057.stdout:Group : Unspecified 2024-03-24T10:25:52.209 INFO:teuthology.orchestra.run.smithi057.stdout:Size : 8264254 2024-03-24T10:25:52.209 INFO:teuthology.orchestra.run.smithi057.stdout:License : GPLv3+ 2024-03-24T10:25:52.209 INFO:teuthology.orchestra.run.smithi057.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T10:25:52.209 INFO:teuthology.orchestra.run.smithi057.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T10:25:52.209 INFO:teuthology.orchestra.run.smithi057.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T10:25:52.210 INFO:teuthology.orchestra.run.smithi057.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T10:25:52.210 INFO:teuthology.orchestra.run.smithi057.stdout:Packager : builder@centos.org 2024-03-24T10:25:52.210 INFO:teuthology.orchestra.run.smithi057.stdout:Vendor : CentOS 2024-03-24T10:25:52.210 INFO:teuthology.orchestra.run.smithi057.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T10:25:52.210 INFO:teuthology.orchestra.run.smithi057.stdout:Summary : Support tools for GRUB. 2024-03-24T10:25:52.210 INFO:teuthology.orchestra.run.smithi057.stdout:Description : 2024-03-24T10:25:52.210 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:25:52.210 INFO:teuthology.orchestra.run.smithi057.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T10:25:52.210 INFO:teuthology.orchestra.run.smithi057.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T10:25:52.210 INFO:teuthology.orchestra.run.smithi057.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T10:25:52.210 INFO:teuthology.orchestra.run.smithi057.stdout:hardware devices. 2024-03-24T10:25:52.210 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:25:52.210 INFO:teuthology.orchestra.run.smithi057.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T10:25:52.211 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T10:25:52.211 INFO:teuthology.task.kernel:Updating grub on smithi057 to boot 5.14.0-430.el9.x86_64 2024-03-24T10:25:52.211 DEBUG:teuthology.orchestra.run.smithi057:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T10:25:52.756 INFO:teuthology.orchestra.run.smithi057.stderr:Generating grub configuration file ... 2024-03-24T10:25:54.122 INFO:teuthology.orchestra.run.smithi057.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T10:25:54.155 INFO:teuthology.orchestra.run.smithi057.stderr:done 2024-03-24T10:25:54.158 DEBUG:teuthology.orchestra.run.smithi057:> mktemp 2024-03-24T10:25:54.173 INFO:teuthology.orchestra.run.smithi057.stdout:/tmp/tmp.23qw2EpgwL 2024-03-24T10:25:54.173 DEBUG:teuthology.orchestra.run.smithi057:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.23qw2EpgwL 2024-03-24T10:25:54.257 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0666 /tmp/tmp.23qw2EpgwL 2024-03-24T10:25:54.353 DEBUG:teuthology.orchestra.remote:smithi057:/tmp/tmp.23qw2EpgwL is 6KB 2024-03-24T10:25:54.374 DEBUG:teuthology.orchestra.run.smithi057:> rm -fr /tmp/tmp.23qw2EpgwL 2024-03-24T10:25:54.388 DEBUG:teuthology.orchestra.run.smithi057:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T10:25:54.466 INFO:teuthology.orchestra.run.smithi057.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T10:25:54.466 INFO:teuthology.orchestra.run.smithi057.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T10:25:54.466 INFO:teuthology.orchestra.run.smithi057.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T10:25:54.466 INFO:teuthology.orchestra.run.smithi057.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T10:25:54.468 DEBUG:teuthology.orchestra.run.smithi057:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T10:25:54.647 DEBUG:teuthology.orchestra.run.smithi057:> sudo shutdown -r now 2024-03-24T10:26:03.342 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-24T10:26:03.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-24T10:26:03.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:26:24.670 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-24T10:26:24.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-03-24T10:26:24.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:26:59.231 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.57 2024-03-24T10:27:03.349 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T10:27:08.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-03-24T10:27:08.235 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:27:12.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-24T10:27:12.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:27:12.679 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-03-24T10:27:13.077 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-24T10:27:13.077 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-24T10:27:13.077 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-03-24T10:27:13.091 INFO:teuthology.orchestra.run.smithi005.stdout:5.14.0-430.el9.x86_64 2024-03-24T10:27:13.091 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T10:27:13.092 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T10:27:13.092 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T10:27:14.092 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-24T10:27:14.093 DEBUG:teuthology.orchestra.run.smithi005:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T10:27:14.157 INFO:teuthology.orchestra.run.smithi005.stdout:ttyS1 2024-03-24T10:27:14.205 DEBUG:teuthology.parallel:result is None 2024-03-24T10:28:08.237 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T10:28:20.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-03-24T10:28:20.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:28:20.561 DEBUG:teuthology.orchestra.run.smithi057:> true 2024-03-24T10:28:20.931 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-03-24T10:28:20.931 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-24T10:28:20.931 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2024-03-24T10:28:20.945 INFO:teuthology.orchestra.run.smithi057.stdout:5.14.0-430.el9.x86_64 2024-03-24T10:28:20.946 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T10:28:20.946 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T10:28:20.946 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T10:28:21.947 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-24T10:28:21.947 DEBUG:teuthology.orchestra.run.smithi057:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T10:28:22.013 INFO:teuthology.orchestra.run.smithi057.stdout:ttyS1 2024-03-24T10:28:22.051 DEBUG:teuthology.parallel:result is None 2024-03-24T10:28:22.052 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T10:28:22.058 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T10:28:22.058 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T10:28:22.061 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T10:28:22.078 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T10:28:22.084 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T10:28:22.095 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T10:28:22.095 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T10:28:22.119 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T10:28:22.146 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T10:28:22.153 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T10:28:22.153 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T10:28:22.179 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T10:28:22.215 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T10:28:22.216 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T10:28:22.229 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T10:28:22.231 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T10:28:22.233 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T10:28:22.241 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T10:28:22.241 DEBUG:teuthology.orchestra.run.smithi005:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T10:28:22.275 DEBUG:teuthology.orchestra.run.smithi057:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T10:28:22.305 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T10:28:22.315 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T10:28:22.316 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T10:28:22.345 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T10:28:22.361 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T10:28:22.442 DEBUG:teuthology.orchestra.run.smithi005:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T10:28:22.495 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T10:28:22.578 DEBUG:teuthology.orchestra.run.smithi005:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T10:28:22.622 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:28:22.622 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T10:28:22.691 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T10:28:22.735 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T10:28:22.788 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T10:28:22.868 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T10:28:22.917 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:28:22.917 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T10:28:22.986 DEBUG:teuthology.orchestra.run.smithi005:> sudo service rsyslog restart 2024-03-24T10:28:22.988 DEBUG:teuthology.orchestra.run.smithi057:> sudo service rsyslog restart 2024-03-24T10:28:23.039 INFO:teuthology.orchestra.run.smithi005.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T10:28:23.073 INFO:teuthology.orchestra.run.smithi057.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T10:28:23.575 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T10:28:23.585 INFO:teuthology.task.internal:Starting timer... 2024-03-24T10:28:23.585 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T10:28:23.640 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T10:28:23.649 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-24T10:28:23.650 DEBUG:teuthology.orchestra.run.smithi005:> sudo service auditd rotate 2024-03-24T10:28:23.710 INFO:teuthology.orchestra.run.smithi005.stdout:Rotating logs: 2024-03-24T10:28:23.712 DEBUG:teuthology.orchestra.run.smithi057:> sudo service auditd rotate 2024-03-24T10:28:23.772 INFO:teuthology.orchestra.run.smithi057.stdout:Rotating logs: 2024-03-24T10:28:23.774 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T10:28:23.775 DEBUG:teuthology.orchestra.run.smithi005:> /usr/sbin/getenforce 2024-03-24T10:28:23.814 INFO:teuthology.orchestra.run.smithi005.stdout:Permissive 2024-03-24T10:28:23.815 DEBUG:teuthology.orchestra.run.smithi057:> /usr/sbin/getenforce 2024-03-24T10:28:23.836 INFO:teuthology.orchestra.run.smithi057.stdout:Permissive 2024-03-24T10:28:23.837 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi005.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi057.front.sepia.ceph.com': 'permissive'} 2024-03-24T10:28:23.837 DEBUG:teuthology.orchestra.run.smithi005:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T10:28:23.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:28:23.883 DEBUG:teuthology.orchestra.run.smithi057:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T10:28:23.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:28:23.913 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T10:28:23.913 DEBUG:teuthology.orchestra.run.smithi005:> sudo /usr/sbin/setenforce permissive 2024-03-24T10:28:23.957 DEBUG:teuthology.orchestra.run.smithi057:> sudo /usr/sbin/setenforce permissive 2024-03-24T10:28:23.997 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T10:28:24.010 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T10:28:24.028 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T10:28:24.029 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi005.front.sepia.ceph.com,smithi057.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T10:33:01.462 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi005.front.sepia.ceph.com'), Remote(name='ubuntu@smithi057.front.sepia.ceph.com')] 2024-03-24T10:33:01.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-24T10:33:01.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:33:01.597 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-03-24T10:33:01.674 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-24T10:33:01.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-03-24T10:33:01.674 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:33:01.783 DEBUG:teuthology.orchestra.run.smithi057:> true 2024-03-24T10:33:01.857 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi057.front.sepia.ceph.com' 2024-03-24T10:33:01.857 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T10:33:01.870 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T10:33:01.870 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T10:33:01.870 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T10:33:01.877 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T10:33:01.877 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T10:33:01.905 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T10:33:01.923 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T10:33:01.930 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T10:33:01.945 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T10:33:01.953 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: ntpd: command not found 2024-03-24T10:33:01.967 INFO:teuthology.orchestra.run.smithi005.stdout:506 Cannot talk to daemon 2024-03-24T10:33:01.975 INFO:teuthology.orchestra.run.smithi057.stderr:sudo: ntpd: command not found 2024-03-24T10:33:01.985 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T10:33:01.988 INFO:teuthology.orchestra.run.smithi057.stdout:506 Cannot talk to daemon 2024-03-24T10:33:02.000 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T10:33:02.006 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T10:33:02.021 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T10:33:02.055 INFO:teuthology.orchestra.run.smithi005.stderr:bash: line 1: ntpq: command not found 2024-03-24T10:33:02.059 INFO:teuthology.orchestra.run.smithi005.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T10:33:02.059 INFO:teuthology.orchestra.run.smithi005.stdout:=============================================================================== 2024-03-24T10:33:02.059 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:33:02.059 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:33:02.059 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:33:02.059 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:33:02.074 INFO:teuthology.orchestra.run.smithi057.stderr:bash: line 1: ntpq: command not found 2024-03-24T10:33:02.077 INFO:teuthology.orchestra.run.smithi057.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T10:33:02.077 INFO:teuthology.orchestra.run.smithi057.stdout:=============================================================================== 2024-03-24T10:33:02.078 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:33:02.078 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:33:02.078 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:33:02.078 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T10:33:02.078 INFO:teuthology.run_tasks:Running task install... 2024-03-24T10:33:02.093 DEBUG:teuthology.task.install:project ceph 2024-03-24T10:33:02.093 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'}} 2024-03-24T10:33:02.093 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'debuginfo': True, 'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'} 2024-03-24T10:33:02.093 INFO:teuthology.task.install:Using flavor: default 2024-03-24T10:33:02.100 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-24T10:33:02.100 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-24T10:33:02.101 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-24T10:33:02.101 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:33:02.103 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-24T10:33:02.104 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:33:02.348 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/ 2024-03-24T10:33:02.349 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1855.g21b3bd67 2024-03-24T10:33:02.373 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/ 2024-03-24T10:33:02.373 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1855.g21b3bd67 2024-03-24T10:33:02.514 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T10:33:02.514 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:33:02.514 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T10:33:02.540 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T10:33:02.540 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:33:02.541 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T10:33:02.546 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T10:33:02.546 DEBUG:teuthology.orchestra.run.smithi005:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/21b3bd672163092b4e542f772797fcba02d3d4de/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T10:33:02.568 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T10:33:02.568 DEBUG:teuthology.orchestra.run.smithi057:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/21b3bd672163092b4e542f772797fcba02d3d4de/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T10:33:02.618 DEBUG:teuthology.orchestra.run.smithi005:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T10:33:02.637 DEBUG:teuthology.orchestra.run.smithi057:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T10:33:02.700 DEBUG:teuthology.orchestra.run.smithi005:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T10:33:02.717 DEBUG:teuthology.orchestra.run.smithi057:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T10:33:02.768 INFO:teuthology.orchestra.run.smithi005.stdout:check_obsoletes = 1 2024-03-24T10:33:02.769 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum clean all 2024-03-24T10:33:02.784 INFO:teuthology.orchestra.run.smithi057.stdout:check_obsoletes = 1 2024-03-24T10:33:02.785 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum clean all 2024-03-24T10:33:03.010 INFO:teuthology.orchestra.run.smithi005.stdout:74 files removed 2024-03-24T10:33:03.037 INFO:teuthology.orchestra.run.smithi057.stdout:74 files removed 2024-03-24T10:33:03.037 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd ceph-base-debuginfo ceph-common-debuginfo ceph-immutable-object-cache-debuginfo ceph-radosgw-debuginfo ceph-test-debuginfo ceph-base-debuginfo ceph-mgr-debuginfo ceph-mds-debuginfo ceph-mon-debuginfo ceph-osd-debuginfo ceph-fuse-debuginfo librados-devel-debuginfo libcephfs2-debuginfo librados2-debuginfo librbd1-debuginfo python3-cephfs-debuginfo python3-rados-debuginfo python3-rbd-debuginfo python3-rgw-debuginfo rbd-fuse-debuginfo rbd-mirror-debuginfo rbd-nbd-debuginfo python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T10:33:03.062 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd ceph-base-debuginfo ceph-common-debuginfo ceph-immutable-object-cache-debuginfo ceph-radosgw-debuginfo ceph-test-debuginfo ceph-base-debuginfo ceph-mgr-debuginfo ceph-mds-debuginfo ceph-mon-debuginfo ceph-osd-debuginfo ceph-fuse-debuginfo librados-devel-debuginfo libcephfs2-debuginfo librados2-debuginfo librbd1-debuginfo python3-cephfs-debuginfo python3-rados-debuginfo python3-rbd-debuginfo python3-rgw-debuginfo rbd-fuse-debuginfo rbd-mirror-debuginfo rbd-nbd-debuginfo python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T10:33:03.821 INFO:teuthology.orchestra.run.smithi057.stdout:ceph packages for x86_64 183 kB/s | 83 kB 00:00 2024-03-24T10:33:03.831 INFO:teuthology.orchestra.run.smithi005.stdout:ceph packages for x86_64 172 kB/s | 83 kB 00:00 2024-03-24T10:33:04.305 INFO:teuthology.orchestra.run.smithi057.stdout:ceph noarch packages 23 kB/s | 10 kB 00:00 2024-03-24T10:33:04.306 INFO:teuthology.orchestra.run.smithi005.stdout:ceph noarch packages 24 kB/s | 10 kB 00:00 2024-03-24T10:33:04.681 INFO:teuthology.orchestra.run.smithi057.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-03-24T10:33:04.840 INFO:teuthology.orchestra.run.smithi005.stdout:ceph source packages 3.8 kB/s | 1.9 kB 00:00 2024-03-24T10:33:05.142 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-24T10:33:05.284 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-24T10:33:06.601 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-24T10:33:06.752 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-24T10:33:08.178 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - BaseOS 7.5 MB/s | 8.0 MB 00:01 2024-03-24T10:33:15.907 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - BaseOS 918 kB/s | 8.0 MB 00:08 2024-03-24T10:33:16.815 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 19 MB 00:07 2024-03-24T10:33:18.850 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-03-24T10:33:21.683 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.1 MB 00:01 2024-03-24T10:33:23.256 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-03-24T10:33:24.068 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.1 MB 00:01 2024-03-24T10:33:25.242 INFO:teuthology.orchestra.run.smithi005.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-24T10:33:25.516 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 16 kB 00:00 2024-03-24T10:33:27.205 INFO:teuthology.orchestra.run.smithi057.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-24T10:33:30.323 INFO:teuthology.orchestra.run.smithi005.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-03-24T10:33:32.304 INFO:teuthology.orchestra.run.smithi057.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-24T10:33:32.805 INFO:teuthology.orchestra.run.smithi005.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T10:33:32.806 INFO:teuthology.orchestra.run.smithi005.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T10:33:32.842 INFO:teuthology.orchestra.run.smithi005.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T10:33:32.844 INFO:teuthology.orchestra.run.smithi005.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T10:33:32.844 INFO:teuthology.orchestra.run.smithi005.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T10:33:32.846 INFO:teuthology.orchestra.run.smithi005.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T10:33:32.848 INFO:teuthology.orchestra.run.smithi005.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T10:33:32.851 INFO:teuthology.orchestra.run.smithi005.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T10:33:32.852 INFO:teuthology.orchestra.run.smithi005.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T10:33:32.944 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-24T10:33:32.966 INFO:teuthology.orchestra.run.smithi005.stdout:==================================================================================================== 2024-03-24T10:33:32.966 INFO:teuthology.orchestra.run.smithi005.stdout: Package Arch Version Repository Size 2024-03-24T10:33:32.966 INFO:teuthology.orchestra.run.smithi005.stdout:==================================================================================================== 2024-03-24T10:33:32.966 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-24T10:33:32.966 INFO:teuthology.orchestra.run.smithi005.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T10:33:32.967 INFO:teuthology.orchestra.run.smithi005.stdout: ceph x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.6 k 2024-03-24T10:33:32.967 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.3 M 2024-03-24T10:33:32.967 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 68 M 2024-03-24T10:33:32.967 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 437 M 2024-03-24T10:33:32.967 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 888 k 2024-03-24T10:33:32.967 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.0 M 2024-03-24T10:33:32.967 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 145 k 2024-03-24T10:33:32.967 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-24T10:33:32.967 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 53 M 2024-03-24T10:33:32.967 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-24T10:33:32.967 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 138 k 2024-03-24T10:33:32.967 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 1.8 M 2024-03-24T10:33:32.967 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 20 M 2024-03-24T10:33:32.968 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 7.4 M 2024-03-24T10:33:32.968 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-rook noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 49 k 2024-03-24T10:33:32.968 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 M 2024-03-24T10:33:32.968 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 287 M 2024-03-24T10:33:32.968 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-radosgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 9.8 M 2024-03-24T10:33:32.968 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 176 M 2024-03-24T10:33:32.968 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-test x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 47 M 2024-03-24T10:33:32.968 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 674 M 2024-03-24T10:33:32.968 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-volume noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 264 k 2024-03-24T10:33:32.968 INFO:teuthology.orchestra.run.smithi005.stdout: cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 748 k 2024-03-24T10:33:32.968 INFO:teuthology.orchestra.run.smithi005.stdout: cephfs-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 219 k 2024-03-24T10:33:32.969 INFO:teuthology.orchestra.run.smithi005.stdout: cephfs-shell x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 27 k 2024-03-24T10:33:32.969 INFO:teuthology.orchestra.run.smithi005.stdout: cephfs-top noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 18 k 2024-03-24T10:33:32.969 INFO:teuthology.orchestra.run.smithi005.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T10:33:32.969 INFO:teuthology.orchestra.run.smithi005.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T10:33:32.969 INFO:teuthology.orchestra.run.smithi005.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T10:33:32.969 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 32 k 2024-03-24T10:33:32.969 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 744 k 2024-03-24T10:33:32.969 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.4 M 2024-03-24T10:33:32.969 INFO:teuthology.orchestra.run.smithi005.stdout: librados-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 127 k 2024-03-24T10:33:32.969 INFO:teuthology.orchestra.run.smithi005.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 109 k 2024-03-24T10:33:32.969 INFO:teuthology.orchestra.run.smithi005.stdout: librados2-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 44 M 2024-03-24T10:33:32.969 INFO:teuthology.orchestra.run.smithi005.stdout: librbd1-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 52 M 2024-03-24T10:33:32.969 INFO:teuthology.orchestra.run.smithi005.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T10:33:32.970 INFO:teuthology.orchestra.run.smithi005.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T10:33:32.970 INFO:teuthology.orchestra.run.smithi005.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T10:33:32.970 INFO:teuthology.orchestra.run.smithi005.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T10:33:32.970 INFO:teuthology.orchestra.run.smithi005.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T10:33:32.970 INFO:teuthology.orchestra.run.smithi005.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T10:33:32.970 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cephfs x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 161 k 2024-03-24T10:33:32.970 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 615 k 2024-03-24T10:33:32.970 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rados x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 320 k 2024-03-24T10:33:32.970 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 770 k 2024-03-24T10:33:32.970 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 298 k 2024-03-24T10:33:32.970 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.1 M 2024-03-24T10:33:32.971 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 k 2024-03-24T10:33:32.971 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 337 k 2024-03-24T10:33:32.971 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 85 k 2024-03-24T10:33:32.971 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 509 k 2024-03-24T10:33:32.971 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T10:33:32.971 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 51 M 2024-03-24T10:33:32.971 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-nbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 170 k 2024-03-24T10:33:32.971 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.3 M 2024-03-24T10:33:32.971 INFO:teuthology.orchestra.run.smithi005.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T10:33:32.971 INFO:teuthology.orchestra.run.smithi005.stdout:Upgrading: 2024-03-24T10:33:32.971 INFO:teuthology.orchestra.run.smithi005.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T10:33:32.971 INFO:teuthology.orchestra.run.smithi005.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T10:33:32.971 INFO:teuthology.orchestra.run.smithi005.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T10:33:32.972 INFO:teuthology.orchestra.run.smithi005.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T10:33:32.972 INFO:teuthology.orchestra.run.smithi005.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T10:33:32.972 INFO:teuthology.orchestra.run.smithi005.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T10:33:32.972 INFO:teuthology.orchestra.run.smithi005.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T10:33:32.972 INFO:teuthology.orchestra.run.smithi005.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T10:33:32.972 INFO:teuthology.orchestra.run.smithi005.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T10:33:32.972 INFO:teuthology.orchestra.run.smithi005.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T10:33:32.972 INFO:teuthology.orchestra.run.smithi005.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T10:33:32.972 INFO:teuthology.orchestra.run.smithi005.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T10:33:32.972 INFO:teuthology.orchestra.run.smithi005.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T10:33:32.972 INFO:teuthology.orchestra.run.smithi005.stdout: librados2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.4 M 2024-03-24T10:33:32.972 INFO:teuthology.orchestra.run.smithi005.stdout: librbd1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T10:33:32.973 INFO:teuthology.orchestra.run.smithi005.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T10:33:32.973 INFO:teuthology.orchestra.run.smithi005.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T10:33:32.973 INFO:teuthology.orchestra.run.smithi005.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T10:33:32.973 INFO:teuthology.orchestra.run.smithi005.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T10:33:32.973 INFO:teuthology.orchestra.run.smithi005.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T10:33:32.973 INFO:teuthology.orchestra.run.smithi005.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T10:33:32.973 INFO:teuthology.orchestra.run.smithi005.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T10:33:32.973 INFO:teuthology.orchestra.run.smithi005.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T10:33:32.973 INFO:teuthology.orchestra.run.smithi005.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T10:33:32.973 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-03-24T10:33:32.973 INFO:teuthology.orchestra.run.smithi005.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T10:33:32.973 INFO:teuthology.orchestra.run.smithi005.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T10:33:32.974 INFO:teuthology.orchestra.run.smithi005.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T10:33:32.974 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 21 M 2024-03-24T10:33:32.974 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 22 M 2024-03-24T10:33:32.974 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 31 k 2024-03-24T10:33:32.974 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mds x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 2.4 M 2024-03-24T10:33:32.974 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 248 k 2024-03-24T10:33:32.974 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mon x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 4.7 M 2024-03-24T10:33:32.974 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-osd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 17 M 2024-03-24T10:33:32.974 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 15 k 2024-03-24T10:33:32.974 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-selinux x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 25 k 2024-03-24T10:33:32.974 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T10:33:32.974 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T10:33:32.975 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T10:33:32.975 INFO:teuthology.orchestra.run.smithi005.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T10:33:32.975 INFO:teuthology.orchestra.run.smithi005.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T10:33:32.975 INFO:teuthology.orchestra.run.smithi005.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T10:33:32.975 INFO:teuthology.orchestra.run.smithi005.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T10:33:32.975 INFO:teuthology.orchestra.run.smithi005.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T10:33:32.975 INFO:teuthology.orchestra.run.smithi005.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T10:33:32.975 INFO:teuthology.orchestra.run.smithi005.stdout: libcephsqlite x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 163 k 2024-03-24T10:33:32.975 INFO:teuthology.orchestra.run.smithi005.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T10:33:32.975 INFO:teuthology.orchestra.run.smithi005.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T10:33:32.975 INFO:teuthology.orchestra.run.smithi005.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T10:33:32.975 INFO:teuthology.orchestra.run.smithi005.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T10:33:32.975 INFO:teuthology.orchestra.run.smithi005.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T10:33:32.976 INFO:teuthology.orchestra.run.smithi005.stdout: libradosstriper1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 499 k 2024-03-24T10:33:32.976 INFO:teuthology.orchestra.run.smithi005.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T10:33:32.976 INFO:teuthology.orchestra.run.smithi005.stdout: librgw2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.1 M 2024-03-24T10:33:32.976 INFO:teuthology.orchestra.run.smithi005.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T10:33:32.976 INFO:teuthology.orchestra.run.smithi005.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T10:33:32.976 INFO:teuthology.orchestra.run.smithi005.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T10:33:32.976 INFO:teuthology.orchestra.run.smithi005.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T10:33:32.976 INFO:teuthology.orchestra.run.smithi005.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T10:33:32.976 INFO:teuthology.orchestra.run.smithi005.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T10:33:32.976 INFO:teuthology.orchestra.run.smithi005.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T10:33:32.976 INFO:teuthology.orchestra.run.smithi005.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T10:33:32.976 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T10:33:32.976 INFO:teuthology.orchestra.run.smithi005.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T10:33:32.976 INFO:teuthology.orchestra.run.smithi005.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T10:33:32.977 INFO:teuthology.orchestra.run.smithi005.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T10:33:32.977 INFO:teuthology.orchestra.run.smithi005.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T10:33:32.977 INFO:teuthology.orchestra.run.smithi005.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T10:33:32.977 INFO:teuthology.orchestra.run.smithi005.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T10:33:32.977 INFO:teuthology.orchestra.run.smithi005.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T10:33:32.977 INFO:teuthology.orchestra.run.smithi005.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T10:33:32.977 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T10:33:32.977 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-argparse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 45 k 2024-03-24T10:33:32.977 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 131 k 2024-03-24T10:33:32.977 INFO:teuthology.orchestra.run.smithi005.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T10:33:32.977 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T10:33:32.977 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T10:33:32.977 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T10:33:32.978 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T10:33:32.978 INFO:teuthology.orchestra.run.smithi005.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T10:33:32.978 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T10:33:32.978 INFO:teuthology.orchestra.run.smithi005.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T10:33:32.978 INFO:teuthology.orchestra.run.smithi005.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T10:33:32.978 INFO:teuthology.orchestra.run.smithi005.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T10:33:32.978 INFO:teuthology.orchestra.run.smithi005.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T10:33:32.978 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T10:33:32.978 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T10:33:32.978 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T10:33:32.978 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T10:33:32.978 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T10:33:32.978 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T10:33:32.979 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T10:33:32.979 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T10:33:32.979 INFO:teuthology.orchestra.run.smithi005.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T10:33:32.979 INFO:teuthology.orchestra.run.smithi005.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T10:33:32.979 INFO:teuthology.orchestra.run.smithi005.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T10:33:32.979 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T10:33:32.979 INFO:teuthology.orchestra.run.smithi005.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T10:33:32.979 INFO:teuthology.orchestra.run.smithi005.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T10:33:32.979 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T10:33:32.979 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T10:33:32.980 INFO:teuthology.orchestra.run.smithi005.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T10:33:32.980 INFO:teuthology.orchestra.run.smithi005.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T10:33:32.980 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T10:33:32.980 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T10:33:32.980 INFO:teuthology.orchestra.run.smithi005.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T10:33:32.980 INFO:teuthology.orchestra.run.smithi005.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T10:33:32.980 INFO:teuthology.orchestra.run.smithi005.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T10:33:32.980 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T10:33:32.980 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T10:33:32.980 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T10:33:32.980 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T10:33:32.980 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T10:33:32.981 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T10:33:32.981 INFO:teuthology.orchestra.run.smithi005.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T10:33:32.981 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T10:33:32.981 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T10:33:32.981 INFO:teuthology.orchestra.run.smithi005.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T10:33:32.981 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T10:33:32.981 INFO:teuthology.orchestra.run.smithi005.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T10:33:32.981 INFO:teuthology.orchestra.run.smithi005.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T10:33:32.981 INFO:teuthology.orchestra.run.smithi005.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T10:33:32.982 INFO:teuthology.orchestra.run.smithi005.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T10:33:32.982 INFO:teuthology.orchestra.run.smithi005.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T10:33:32.982 INFO:teuthology.orchestra.run.smithi005.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T10:33:32.982 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T10:33:32.982 INFO:teuthology.orchestra.run.smithi005.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T10:33:32.982 INFO:teuthology.orchestra.run.smithi005.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T10:33:32.982 INFO:teuthology.orchestra.run.smithi005.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T10:33:32.982 INFO:teuthology.orchestra.run.smithi005.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T10:33:32.982 INFO:teuthology.orchestra.run.smithi005.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T10:33:32.982 INFO:teuthology.orchestra.run.smithi005.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T10:33:32.982 INFO:teuthology.orchestra.run.smithi005.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T10:33:32.982 INFO:teuthology.orchestra.run.smithi005.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T10:33:32.982 INFO:teuthology.orchestra.run.smithi005.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T10:33:32.983 INFO:teuthology.orchestra.run.smithi005.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T10:33:32.983 INFO:teuthology.orchestra.run.smithi005.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T10:33:32.983 INFO:teuthology.orchestra.run.smithi005.stdout:Installing weak dependencies: 2024-03-24T10:33:32.983 INFO:teuthology.orchestra.run.smithi005.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T10:33:32.983 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:33:32.983 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-24T10:33:32.983 INFO:teuthology.orchestra.run.smithi005.stdout:==================================================================================================== 2024-03-24T10:33:32.983 INFO:teuthology.orchestra.run.smithi005.stdout:Install 172 Packages 2024-03-24T10:33:32.983 INFO:teuthology.orchestra.run.smithi005.stdout:Upgrade 24 Packages 2024-03-24T10:33:32.983 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:33:32.984 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 2.2 G 2024-03-24T10:33:32.984 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-24T10:33:33.741 INFO:teuthology.orchestra.run.smithi005.stdout:(1/196): ceph-19.0.0-1855.g21b3bd67.el9.x86_64. 18 kB/s | 6.6 kB 00:00 2024-03-24T10:33:34.774 INFO:teuthology.orchestra.run.smithi057.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T10:33:34.774 INFO:teuthology.orchestra.run.smithi057.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T10:33:34.805 INFO:teuthology.orchestra.run.smithi057.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T10:33:34.805 INFO:teuthology.orchestra.run.smithi057.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T10:33:34.806 INFO:teuthology.orchestra.run.smithi057.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T10:33:34.807 INFO:teuthology.orchestra.run.smithi057.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T10:33:34.809 INFO:teuthology.orchestra.run.smithi057.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T10:33:34.812 INFO:teuthology.orchestra.run.smithi057.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T10:33:34.813 INFO:teuthology.orchestra.run.smithi057.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T10:33:34.903 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2024-03-24T10:33:34.917 INFO:teuthology.orchestra.run.smithi057.stdout:==================================================================================================== 2024-03-24T10:33:34.917 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2024-03-24T10:33:34.918 INFO:teuthology.orchestra.run.smithi057.stdout:==================================================================================================== 2024-03-24T10:33:34.918 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2024-03-24T10:33:34.918 INFO:teuthology.orchestra.run.smithi057.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T10:33:34.918 INFO:teuthology.orchestra.run.smithi057.stdout: ceph x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.6 k 2024-03-24T10:33:34.918 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-base x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.3 M 2024-03-24T10:33:34.918 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 68 M 2024-03-24T10:33:34.918 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 437 M 2024-03-24T10:33:34.918 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 888 k 2024-03-24T10:33:34.918 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.0 M 2024-03-24T10:33:34.918 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 145 k 2024-03-24T10:33:34.918 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-24T10:33:34.918 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 53 M 2024-03-24T10:33:34.918 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-24T10:33:34.919 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 138 k 2024-03-24T10:33:34.919 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 1.8 M 2024-03-24T10:33:34.919 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 20 M 2024-03-24T10:33:34.919 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 7.4 M 2024-03-24T10:33:34.919 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-rook noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 49 k 2024-03-24T10:33:34.919 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 M 2024-03-24T10:33:34.919 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 287 M 2024-03-24T10:33:34.919 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-radosgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 9.8 M 2024-03-24T10:33:34.919 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 176 M 2024-03-24T10:33:34.919 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-test x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 47 M 2024-03-24T10:33:34.919 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 674 M 2024-03-24T10:33:34.919 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-volume noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 264 k 2024-03-24T10:33:34.919 INFO:teuthology.orchestra.run.smithi057.stdout: cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 748 k 2024-03-24T10:33:34.919 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 219 k 2024-03-24T10:33:34.920 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-shell x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 27 k 2024-03-24T10:33:34.920 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-top noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 18 k 2024-03-24T10:33:34.920 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T10:33:34.920 INFO:teuthology.orchestra.run.smithi057.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T10:33:34.920 INFO:teuthology.orchestra.run.smithi057.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T10:33:34.920 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 32 k 2024-03-24T10:33:34.920 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 744 k 2024-03-24T10:33:34.920 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.4 M 2024-03-24T10:33:34.920 INFO:teuthology.orchestra.run.smithi057.stdout: librados-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 127 k 2024-03-24T10:33:34.920 INFO:teuthology.orchestra.run.smithi057.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 109 k 2024-03-24T10:33:34.920 INFO:teuthology.orchestra.run.smithi057.stdout: librados2-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 44 M 2024-03-24T10:33:34.920 INFO:teuthology.orchestra.run.smithi057.stdout: librbd1-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 52 M 2024-03-24T10:33:34.921 INFO:teuthology.orchestra.run.smithi057.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T10:33:34.921 INFO:teuthology.orchestra.run.smithi057.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T10:33:34.921 INFO:teuthology.orchestra.run.smithi057.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T10:33:34.921 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T10:33:34.921 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T10:33:34.921 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T10:33:34.921 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cephfs x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 161 k 2024-03-24T10:33:34.921 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 615 k 2024-03-24T10:33:34.921 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rados x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 320 k 2024-03-24T10:33:34.921 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 770 k 2024-03-24T10:33:34.921 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 298 k 2024-03-24T10:33:34.921 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.1 M 2024-03-24T10:33:34.921 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 k 2024-03-24T10:33:34.921 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 337 k 2024-03-24T10:33:34.922 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 85 k 2024-03-24T10:33:34.922 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 509 k 2024-03-24T10:33:34.922 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T10:33:34.922 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 51 M 2024-03-24T10:33:34.922 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-nbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 170 k 2024-03-24T10:33:34.922 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.3 M 2024-03-24T10:33:34.923 INFO:teuthology.orchestra.run.smithi057.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T10:33:34.923 INFO:teuthology.orchestra.run.smithi057.stdout:Upgrading: 2024-03-24T10:33:34.923 INFO:teuthology.orchestra.run.smithi057.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T10:33:34.923 INFO:teuthology.orchestra.run.smithi057.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T10:33:34.923 INFO:teuthology.orchestra.run.smithi057.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T10:33:34.923 INFO:teuthology.orchestra.run.smithi057.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T10:33:34.923 INFO:teuthology.orchestra.run.smithi057.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T10:33:34.923 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T10:33:34.923 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T10:33:34.924 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T10:33:34.924 INFO:teuthology.orchestra.run.smithi057.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T10:33:34.924 INFO:teuthology.orchestra.run.smithi057.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T10:33:34.924 INFO:teuthology.orchestra.run.smithi057.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T10:33:34.924 INFO:teuthology.orchestra.run.smithi057.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T10:33:34.924 INFO:teuthology.orchestra.run.smithi057.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T10:33:34.924 INFO:teuthology.orchestra.run.smithi057.stdout: librados2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.4 M 2024-03-24T10:33:34.924 INFO:teuthology.orchestra.run.smithi057.stdout: librbd1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-24T10:33:34.924 INFO:teuthology.orchestra.run.smithi057.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T10:33:34.924 INFO:teuthology.orchestra.run.smithi057.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T10:33:34.925 INFO:teuthology.orchestra.run.smithi057.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T10:33:34.925 INFO:teuthology.orchestra.run.smithi057.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T10:33:34.925 INFO:teuthology.orchestra.run.smithi057.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T10:33:34.925 INFO:teuthology.orchestra.run.smithi057.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T10:33:34.925 INFO:teuthology.orchestra.run.smithi057.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T10:33:34.925 INFO:teuthology.orchestra.run.smithi057.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T10:33:34.925 INFO:teuthology.orchestra.run.smithi057.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T10:33:34.925 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2024-03-24T10:33:34.925 INFO:teuthology.orchestra.run.smithi057.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T10:33:34.925 INFO:teuthology.orchestra.run.smithi057.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T10:33:34.925 INFO:teuthology.orchestra.run.smithi057.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T10:33:34.925 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 21 M 2024-03-24T10:33:34.925 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-debuginfo x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 22 M 2024-03-24T10:33:34.926 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 31 k 2024-03-24T10:33:34.926 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mds x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 2.4 M 2024-03-24T10:33:34.926 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 248 k 2024-03-24T10:33:34.926 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mon x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 4.7 M 2024-03-24T10:33:34.926 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-osd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 17 M 2024-03-24T10:33:34.926 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 15 k 2024-03-24T10:33:34.926 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-selinux x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 25 k 2024-03-24T10:33:34.926 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T10:33:34.926 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T10:33:34.926 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T10:33:34.926 INFO:teuthology.orchestra.run.smithi057.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T10:33:34.927 INFO:teuthology.orchestra.run.smithi057.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T10:33:34.927 INFO:teuthology.orchestra.run.smithi057.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T10:33:34.927 INFO:teuthology.orchestra.run.smithi057.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T10:33:34.927 INFO:teuthology.orchestra.run.smithi057.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T10:33:34.927 INFO:teuthology.orchestra.run.smithi057.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T10:33:34.927 INFO:teuthology.orchestra.run.smithi057.stdout: libcephsqlite x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 163 k 2024-03-24T10:33:34.928 INFO:teuthology.orchestra.run.smithi057.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T10:33:34.928 INFO:teuthology.orchestra.run.smithi057.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T10:33:34.928 INFO:teuthology.orchestra.run.smithi057.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T10:33:34.928 INFO:teuthology.orchestra.run.smithi057.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T10:33:34.928 INFO:teuthology.orchestra.run.smithi057.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T10:33:34.928 INFO:teuthology.orchestra.run.smithi057.stdout: libradosstriper1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 499 k 2024-03-24T10:33:34.928 INFO:teuthology.orchestra.run.smithi057.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T10:33:34.928 INFO:teuthology.orchestra.run.smithi057.stdout: librgw2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.1 M 2024-03-24T10:33:34.928 INFO:teuthology.orchestra.run.smithi057.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T10:33:34.928 INFO:teuthology.orchestra.run.smithi057.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T10:33:34.928 INFO:teuthology.orchestra.run.smithi057.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T10:33:34.928 INFO:teuthology.orchestra.run.smithi057.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T10:33:34.928 INFO:teuthology.orchestra.run.smithi057.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T10:33:34.928 INFO:teuthology.orchestra.run.smithi057.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T10:33:34.929 INFO:teuthology.orchestra.run.smithi057.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T10:33:34.929 INFO:teuthology.orchestra.run.smithi057.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T10:33:34.929 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T10:33:34.929 INFO:teuthology.orchestra.run.smithi057.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T10:33:34.929 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T10:33:34.929 INFO:teuthology.orchestra.run.smithi057.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T10:33:34.929 INFO:teuthology.orchestra.run.smithi057.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T10:33:34.929 INFO:teuthology.orchestra.run.smithi057.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T10:33:34.929 INFO:teuthology.orchestra.run.smithi057.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T10:33:34.929 INFO:teuthology.orchestra.run.smithi057.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T10:33:34.929 INFO:teuthology.orchestra.run.smithi057.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T10:33:34.929 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T10:33:34.929 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-argparse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 45 k 2024-03-24T10:33:34.930 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 131 k 2024-03-24T10:33:34.930 INFO:teuthology.orchestra.run.smithi057.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T10:33:34.930 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T10:33:34.930 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T10:33:34.930 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T10:33:34.930 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T10:33:34.930 INFO:teuthology.orchestra.run.smithi057.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T10:33:34.930 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T10:33:34.930 INFO:teuthology.orchestra.run.smithi057.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T10:33:34.930 INFO:teuthology.orchestra.run.smithi057.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T10:33:34.930 INFO:teuthology.orchestra.run.smithi057.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T10:33:34.930 INFO:teuthology.orchestra.run.smithi057.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T10:33:34.931 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T10:33:34.931 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T10:33:34.931 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T10:33:34.931 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T10:33:34.931 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T10:33:34.931 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T10:33:34.931 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T10:33:34.931 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T10:33:34.931 INFO:teuthology.orchestra.run.smithi057.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T10:33:34.931 INFO:teuthology.orchestra.run.smithi057.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T10:33:34.931 INFO:teuthology.orchestra.run.smithi057.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T10:33:34.931 INFO:teuthology.orchestra.run.smithi057.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T10:33:34.931 INFO:teuthology.orchestra.run.smithi057.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T10:33:34.932 INFO:teuthology.orchestra.run.smithi057.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T10:33:34.932 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T10:33:34.932 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T10:33:34.932 INFO:teuthology.orchestra.run.smithi057.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T10:33:34.932 INFO:teuthology.orchestra.run.smithi057.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T10:33:34.932 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T10:33:34.932 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T10:33:34.932 INFO:teuthology.orchestra.run.smithi057.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T10:33:34.932 INFO:teuthology.orchestra.run.smithi057.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T10:33:34.932 INFO:teuthology.orchestra.run.smithi057.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T10:33:34.932 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T10:33:34.932 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T10:33:34.933 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T10:33:34.933 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T10:33:34.933 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T10:33:34.933 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T10:33:34.933 INFO:teuthology.orchestra.run.smithi057.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T10:33:34.933 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T10:33:34.933 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T10:33:34.933 INFO:teuthology.orchestra.run.smithi057.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T10:33:34.933 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T10:33:34.933 INFO:teuthology.orchestra.run.smithi057.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T10:33:34.933 INFO:teuthology.orchestra.run.smithi057.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T10:33:34.933 INFO:teuthology.orchestra.run.smithi057.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T10:33:34.933 INFO:teuthology.orchestra.run.smithi057.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T10:33:34.933 INFO:teuthology.orchestra.run.smithi057.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T10:33:34.934 INFO:teuthology.orchestra.run.smithi057.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T10:33:34.934 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T10:33:34.934 INFO:teuthology.orchestra.run.smithi057.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T10:33:34.934 INFO:teuthology.orchestra.run.smithi057.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T10:33:34.934 INFO:teuthology.orchestra.run.smithi057.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T10:33:34.934 INFO:teuthology.orchestra.run.smithi057.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T10:33:34.934 INFO:teuthology.orchestra.run.smithi057.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T10:33:34.934 INFO:teuthology.orchestra.run.smithi057.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T10:33:34.934 INFO:teuthology.orchestra.run.smithi057.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T10:33:34.934 INFO:teuthology.orchestra.run.smithi057.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T10:33:34.934 INFO:teuthology.orchestra.run.smithi057.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T10:33:34.934 INFO:teuthology.orchestra.run.smithi057.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T10:33:34.935 INFO:teuthology.orchestra.run.smithi057.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T10:33:34.935 INFO:teuthology.orchestra.run.smithi057.stdout:Installing weak dependencies: 2024-03-24T10:33:34.935 INFO:teuthology.orchestra.run.smithi057.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T10:33:34.935 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:33:34.935 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2024-03-24T10:33:34.935 INFO:teuthology.orchestra.run.smithi057.stdout:==================================================================================================== 2024-03-24T10:33:34.935 INFO:teuthology.orchestra.run.smithi057.stdout:Install 172 Packages 2024-03-24T10:33:34.935 INFO:teuthology.orchestra.run.smithi057.stdout:Upgrade 24 Packages 2024-03-24T10:33:34.935 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:33:34.936 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 2.2 G 2024-03-24T10:33:34.936 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2024-03-24T10:33:35.746 INFO:teuthology.orchestra.run.smithi057.stdout:(1/196): ceph-19.0.0-1855.g21b3bd67.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-03-24T10:33:38.316 INFO:teuthology.orchestra.run.smithi005.stdout:(2/196): ceph-base-19.0.0-1855.g21b3bd67.el9.x8 1.1 MB/s | 5.3 MB 00:04 2024-03-24T10:33:39.555 INFO:teuthology.orchestra.run.smithi057.stdout:(2/196): ceph-base-19.0.0-1855.g21b3bd67.el9.x8 1.3 MB/s | 5.3 MB 00:04 2024-03-24T10:33:54.006 INFO:teuthology.orchestra.run.smithi057.stdout:(3/196): ceph-common-19.0.0-1855.g21b3bd67.el9. 1.1 MB/s | 21 MB 00:18 2024-03-24T10:33:54.290 INFO:teuthology.orchestra.run.smithi005.stdout:(3/196): ceph-common-19.0.0-1855.g21b3bd67.el9. 1.0 MB/s | 21 MB 00:20 2024-03-24T10:34:00.914 INFO:teuthology.orchestra.run.smithi005.stdout:(4/196): ceph-base-debuginfo-19.0.0-1855.g21b3b 2.5 MB/s | 68 MB 00:27 2024-03-24T10:34:01.028 INFO:teuthology.orchestra.run.smithi057.stdout:(4/196): ceph-debuginfo-19.0.0-1855.g21b3bd67.e 3.2 MB/s | 22 MB 00:07 2024-03-24T10:34:01.231 INFO:teuthology.orchestra.run.smithi005.stdout:(5/196): ceph-fuse-19.0.0-1855.g21b3bd67.el9.x8 2.7 MB/s | 888 kB 00:00 2024-03-24T10:34:01.361 INFO:teuthology.orchestra.run.smithi057.stdout:(5/196): ceph-fuse-19.0.0-1855.g21b3bd67.el9.x8 2.6 MB/s | 888 kB 00:00 2024-03-24T10:34:02.849 INFO:teuthology.orchestra.run.smithi057.stdout:(6/196): ceph-fuse-debuginfo-19.0.0-1855.g21b3b 4.0 MB/s | 6.0 MB 00:01 2024-03-24T10:34:02.912 INFO:teuthology.orchestra.run.smithi057.stdout:(7/196): ceph-immutable-object-cache-19.0.0-185 2.1 MB/s | 145 kB 00:00 2024-03-24T10:34:03.173 INFO:teuthology.orchestra.run.smithi005.stdout:(6/196): ceph-fuse-debuginfo-19.0.0-1855.g21b3b 3.1 MB/s | 6.0 MB 00:01 2024-03-24T10:34:03.221 INFO:teuthology.orchestra.run.smithi057.stdout:(8/196): ceph-immutable-object-cache-debuginfo- 4.7 MB/s | 1.5 MB 00:00 2024-03-24T10:34:03.265 INFO:teuthology.orchestra.run.smithi005.stdout:(7/196): ceph-immutable-object-cache-19.0.0-185 1.5 MB/s | 145 kB 00:00 2024-03-24T10:34:03.813 INFO:teuthology.orchestra.run.smithi057.stdout:(9/196): ceph-mds-19.0.0-1855.g21b3bd67.el9.x86 4.0 MB/s | 2.4 MB 00:00 2024-03-24T10:34:03.848 INFO:teuthology.orchestra.run.smithi005.stdout:(8/196): ceph-immutable-object-cache-debuginfo- 2.5 MB/s | 1.5 MB 00:00 2024-03-24T10:34:04.658 INFO:teuthology.orchestra.run.smithi005.stdout:(9/196): ceph-mds-19.0.0-1855.g21b3bd67.el9.x86 2.9 MB/s | 2.4 MB 00:00 2024-03-24T10:34:10.557 INFO:teuthology.orchestra.run.smithi005.stdout:(10/196): ceph-debuginfo-19.0.0-1855.g21b3bd67. 1.4 MB/s | 22 MB 00:16 2024-03-24T10:34:11.465 INFO:teuthology.orchestra.run.smithi005.stdout:(11/196): ceph-mgr-19.0.0-1855.g21b3bd67.el9.x8 1.6 MB/s | 1.5 MB 00:00 2024-03-24T10:34:13.845 INFO:teuthology.orchestra.run.smithi057.stdout:(10/196): ceph-mds-debuginfo-19.0.0-1855.g21b3b 5.2 MB/s | 53 MB 00:10 2024-03-24T10:34:14.304 INFO:teuthology.orchestra.run.smithi057.stdout:(11/196): ceph-mgr-19.0.0-1855.g21b3bd67.el9.x8 3.3 MB/s | 1.5 MB 00:00 2024-03-24T10:34:19.731 INFO:teuthology.orchestra.run.smithi005.stdout:(12/196): ceph-mgr-debuginfo-19.0.0-1855.g21b3b 2.4 MB/s | 20 MB 00:08 2024-03-24T10:34:20.720 INFO:teuthology.orchestra.run.smithi057.stdout:(12/196): ceph-mgr-debuginfo-19.0.0-1855.g21b3b 3.1 MB/s | 20 MB 00:06 2024-03-24T10:34:21.273 INFO:teuthology.orchestra.run.smithi005.stdout:(13/196): ceph-mon-19.0.0-1855.g21b3bd67.el9.x8 3.0 MB/s | 4.7 MB 00:01 2024-03-24T10:34:22.995 INFO:teuthology.orchestra.run.smithi057.stdout:(13/196): ceph-mon-19.0.0-1855.g21b3bd67.el9.x8 2.1 MB/s | 4.7 MB 00:02 2024-03-24T10:34:27.697 INFO:teuthology.orchestra.run.smithi005.stdout:(14/196): ceph-mds-debuginfo-19.0.0-1855.g21b3b 2.3 MB/s | 53 MB 00:23 2024-03-24T10:34:33.161 INFO:teuthology.orchestra.run.smithi057.stdout:(14/196): ceph-base-debuginfo-19.0.0-1855.g21b3 1.2 MB/s | 68 MB 00:57 2024-03-24T10:34:52.012 INFO:teuthology.orchestra.run.smithi005.stdout:(15/196): ceph-osd-19.0.0-1855.g21b3bd67.el9.x8 708 kB/s | 17 MB 00:24 2024-03-24T10:35:09.657 INFO:teuthology.orchestra.run.smithi057.stdout:(15/196): ceph-osd-19.0.0-1855.g21b3bd67.el9.x8 471 kB/s | 17 MB 00:36 2024-03-24T10:36:00.118 INFO:teuthology.orchestra.run.smithi057.stdout:(16/196): ceph-mon-debuginfo-19.0.0-1855.g21b3b 1.0 MB/s | 99 MB 01:37 2024-03-24T10:36:03.911 INFO:teuthology.orchestra.run.smithi057.stdout:(17/196): ceph-radosgw-19.0.0-1855.g21b3bd67.el 2.6 MB/s | 9.8 MB 00:03 2024-03-24T10:36:11.560 INFO:teuthology.orchestra.run.smithi005.stdout:(16/196): ceph-mon-debuginfo-19.0.0-1855.g21b3b 915 kB/s | 99 MB 01:50 2024-03-24T10:36:16.994 INFO:teuthology.orchestra.run.smithi005.stdout:(17/196): ceph-radosgw-19.0.0-1855.g21b3bd67.el 1.8 MB/s | 9.8 MB 00:05 2024-03-24T10:36:50.621 INFO:teuthology.orchestra.run.smithi057.stdout:(18/196): ceph-radosgw-debuginfo-19.0.0-1855.g2 3.8 MB/s | 176 MB 00:46 2024-03-24T10:36:50.695 INFO:teuthology.orchestra.run.smithi057.stdout:(19/196): ceph-selinux-19.0.0-1855.g21b3bd67.el 337 kB/s | 25 kB 00:00 2024-03-24T10:37:00.037 INFO:teuthology.orchestra.run.smithi057.stdout:(20/196): ceph-osd-debuginfo-19.0.0-1855.g21b3b 2.6 MB/s | 287 MB 01:50 2024-03-24T10:37:03.803 INFO:teuthology.orchestra.run.smithi057.stdout:(21/196): ceph-common-debuginfo-19.0.0-1855.g21 2.1 MB/s | 437 MB 03:24 2024-03-24T10:37:04.278 INFO:teuthology.orchestra.run.smithi057.stdout:(22/196): ceph-test-19.0.0-1855.g21b3bd67.el9.x 3.5 MB/s | 47 MB 00:13 2024-03-24T10:37:04.303 INFO:teuthology.orchestra.run.smithi057.stdout:(23/196): cephfs-mirror-19.0.0-1855.g21b3bd67.e 438 kB/s | 219 kB 00:00 2024-03-24T10:37:04.354 INFO:teuthology.orchestra.run.smithi057.stdout:(24/196): cephfs-shell-19.0.0-1855.g21b3bd67.el 361 kB/s | 27 kB 00:00 2024-03-24T10:37:04.396 INFO:teuthology.orchestra.run.smithi057.stdout:(25/196): libcephfs-devel-19.0.0-1855.g21b3bd67 345 kB/s | 32 kB 00:00 2024-03-24T10:37:04.588 INFO:teuthology.orchestra.run.smithi057.stdout:(26/196): libcephfs2-19.0.0-1855.g21b3bd67.el9. 3.1 MB/s | 744 kB 00:00 2024-03-24T10:37:04.688 INFO:teuthology.orchestra.run.smithi057.stdout:(27/196): libcephsqlite-19.0.0-1855.g21b3bd67.e 1.6 MB/s | 163 kB 00:00 2024-03-24T10:37:04.783 INFO:teuthology.orchestra.run.smithi057.stdout:(28/196): librados-devel-19.0.0-1855.g21b3bd67. 1.3 MB/s | 127 kB 00:00 2024-03-24T10:37:04.880 INFO:teuthology.orchestra.run.smithi057.stdout:(29/196): librados-devel-debuginfo-19.0.0-1855. 1.1 MB/s | 109 kB 00:00 2024-03-24T10:37:05.897 INFO:teuthology.orchestra.run.smithi057.stdout:(30/196): libcephfs2-debuginfo-19.0.0-1855.g21b 3.6 MB/s | 5.4 MB 00:01 2024-03-24T10:37:06.081 INFO:teuthology.orchestra.run.smithi057.stdout:(31/196): libradosstriper1-19.0.0-1855.g21b3bd6 2.7 MB/s | 499 kB 00:00 2024-03-24T10:37:09.237 INFO:teuthology.orchestra.run.smithi005.stdout:(18/196): ceph-osd-debuginfo-19.0.0-1855.g21b3b 2.1 MB/s | 287 MB 02:17 2024-03-24T10:37:09.354 INFO:teuthology.orchestra.run.smithi005.stdout:(19/196): ceph-selinux-19.0.0-1855.g21b3bd67.el 216 kB/s | 25 kB 00:00 2024-03-24T10:37:12.463 INFO:teuthology.orchestra.run.smithi057.stdout:(32/196): librados2-debuginfo-19.0.0-1855.g21b3 5.9 MB/s | 44 MB 00:07 2024-03-24T10:37:13.139 INFO:teuthology.orchestra.run.smithi057.stdout:(33/196): librgw2-19.0.0-1855.g21b3bd67.el9.x86 7.6 MB/s | 5.1 MB 00:00 2024-03-24T10:37:13.172 INFO:teuthology.orchestra.run.smithi057.stdout:(34/196): python3-ceph-argparse-19.0.0-1855.g21 1.3 MB/s | 45 kB 00:00 2024-03-24T10:37:13.222 INFO:teuthology.orchestra.run.smithi057.stdout:(35/196): python3-ceph-common-19.0.0-1855.g21b3 2.6 MB/s | 131 kB 00:00 2024-03-24T10:37:13.273 INFO:teuthology.orchestra.run.smithi057.stdout:(36/196): python3-cephfs-19.0.0-1855.g21b3bd67. 3.2 MB/s | 161 kB 00:00 2024-03-24T10:37:13.364 INFO:teuthology.orchestra.run.smithi057.stdout:(37/196): python3-cephfs-debuginfo-19.0.0-1855. 6.6 MB/s | 615 kB 00:00 2024-03-24T10:37:13.423 INFO:teuthology.orchestra.run.smithi057.stdout:(38/196): python3-rados-19.0.0-1855.g21b3bd67.e 5.4 MB/s | 320 kB 00:00 2024-03-24T10:37:13.557 INFO:teuthology.orchestra.run.smithi057.stdout:(39/196): python3-rados-debuginfo-19.0.0-1855.g 5.6 MB/s | 770 kB 00:00 2024-03-24T10:37:13.615 INFO:teuthology.orchestra.run.smithi057.stdout:(40/196): python3-rbd-19.0.0-1855.g21b3bd67.el9 5.0 MB/s | 298 kB 00:00 2024-03-24T10:37:13.782 INFO:teuthology.orchestra.run.smithi057.stdout:(41/196): python3-rbd-debuginfo-19.0.0-1855.g21 6.7 MB/s | 1.1 MB 00:00 2024-03-24T10:37:13.833 INFO:teuthology.orchestra.run.smithi057.stdout:(42/196): python3-rgw-19.0.0-1855.g21b3bd67.el9 1.9 MB/s | 99 kB 00:00 2024-03-24T10:37:13.938 INFO:teuthology.orchestra.run.smithi057.stdout:(43/196): python3-rgw-debuginfo-19.0.0-1855.g21 3.1 MB/s | 337 kB 00:00 2024-03-24T10:37:13.983 INFO:teuthology.orchestra.run.smithi057.stdout:(44/196): rbd-fuse-19.0.0-1855.g21b3bd67.el9.x8 1.9 MB/s | 85 kB 00:00 2024-03-24T10:37:14.067 INFO:teuthology.orchestra.run.smithi057.stdout:(45/196): rbd-fuse-debuginfo-19.0.0-1855.g21b3b 6.0 MB/s | 509 kB 00:00 2024-03-24T10:37:14.501 INFO:teuthology.orchestra.run.smithi057.stdout:(46/196): rbd-mirror-19.0.0-1855.g21b3bd67.el9. 7.1 MB/s | 3.1 MB 00:00 2024-03-24T10:37:17.693 INFO:teuthology.orchestra.run.smithi057.stdout:(47/196): librbd1-debuginfo-19.0.0-1855.g21b3bd 4.5 MB/s | 52 MB 00:11 2024-03-24T10:37:17.826 INFO:teuthology.orchestra.run.smithi057.stdout:(48/196): rbd-nbd-19.0.0-1855.g21b3bd67.el9.x86 1.3 MB/s | 170 kB 00:00 2024-03-24T10:37:18.151 INFO:teuthology.orchestra.run.smithi057.stdout:(49/196): rbd-nbd-debuginfo-19.0.0-1855.g21b3bd 3.9 MB/s | 1.3 MB 00:00 2024-03-24T10:37:18.218 INFO:teuthology.orchestra.run.smithi057.stdout:(50/196): ceph-grafana-dashboards-19.0.0-1855.g 472 kB/s | 31 kB 00:00 2024-03-24T10:37:18.285 INFO:teuthology.orchestra.run.smithi057.stdout:(51/196): ceph-mgr-cephadm-19.0.0-1855.g21b3bd6 2.0 MB/s | 138 kB 00:00 2024-03-24T10:37:18.652 INFO:teuthology.orchestra.run.smithi057.stdout:(52/196): ceph-mgr-dashboard-19.0.0-1855.g21b3b 4.9 MB/s | 1.8 MB 00:00 2024-03-24T10:37:18.846 INFO:teuthology.orchestra.run.smithi005.stdout:(20/196): ceph-test-19.0.0-1855.g21b3bd67.el9.x 5.0 MB/s | 47 MB 00:09 2024-03-24T10:37:20.369 INFO:teuthology.orchestra.run.smithi057.stdout:(53/196): ceph-mgr-diskprediction-local-19.0.0- 4.3 MB/s | 7.4 MB 00:01 2024-03-24T10:37:20.469 INFO:teuthology.orchestra.run.smithi057.stdout:(54/196): ceph-mgr-modules-core-19.0.0-1855.g21 2.4 MB/s | 248 kB 00:00 2024-03-24T10:37:20.561 INFO:teuthology.orchestra.run.smithi057.stdout:(55/196): ceph-mgr-rook-19.0.0-1855.g21b3bd67.e 538 kB/s | 49 kB 00:00 2024-03-24T10:37:20.653 INFO:teuthology.orchestra.run.smithi057.stdout:(56/196): ceph-prometheus-alerts-19.0.0-1855.g2 167 kB/s | 15 kB 00:00 2024-03-24T10:37:20.745 INFO:teuthology.orchestra.run.smithi057.stdout:(57/196): ceph-volume-19.0.0-1855.g21b3bd67.el9 2.8 MB/s | 264 kB 00:00 2024-03-24T10:37:21.187 INFO:teuthology.orchestra.run.smithi057.stdout:(58/196): rbd-mirror-debuginfo-19.0.0-1855.g21b 7.7 MB/s | 51 MB 00:06 2024-03-24T10:37:21.221 INFO:teuthology.orchestra.run.smithi057.stdout:(59/196): cephadm-19.0.0-1855.g21b3bd67.el9.noa 1.5 MB/s | 748 kB 00:00 2024-03-24T10:37:21.238 INFO:teuthology.orchestra.run.smithi057.stdout:(60/196): cephfs-top-19.0.0-1855.g21b3bd67.el9. 366 kB/s | 18 kB 00:00 2024-03-24T10:37:21.388 INFO:teuthology.orchestra.run.smithi057.stdout:(61/196): automake-1.16.2-8.el9.noarch.rpm 3.9 MB/s | 662 kB 00:00 2024-03-24T10:37:21.430 INFO:teuthology.orchestra.run.smithi057.stdout:(62/196): bison-3.7.4-5.el9.x86_64.rpm 4.7 MB/s | 920 kB 00:00 2024-03-24T10:37:21.455 INFO:teuthology.orchestra.run.smithi057.stdout:(63/196): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-24T10:37:21.481 INFO:teuthology.orchestra.run.smithi057.stdout:(64/196): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-24T10:37:21.497 INFO:teuthology.orchestra.run.smithi057.stdout:(65/196): flexiblas-3.0.4-8.el9.x86_64.rpm 768 kB/s | 32 kB 00:00 2024-03-24T10:37:21.573 INFO:teuthology.orchestra.run.smithi057.stdout:(66/196): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-03-24T10:37:21.598 INFO:teuthology.orchestra.run.smithi057.stdout:(67/196): flexiblas-openblas-openmp-3.0.4-8.el9 173 kB/s | 17 kB 00:00 2024-03-24T10:37:21.623 INFO:teuthology.orchestra.run.smithi057.stdout:(68/196): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-03-24T10:37:21.649 INFO:teuthology.orchestra.run.smithi057.stdout:(69/196): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-24T10:37:21.675 INFO:teuthology.orchestra.run.smithi057.stdout:(70/196): librabbitmq-0.11.0-7.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2024-03-24T10:37:21.699 INFO:teuthology.orchestra.run.smithi057.stdout:(71/196): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-24T10:37:21.733 INFO:teuthology.orchestra.run.smithi057.stdout:(72/196): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 578 kB 00:00 2024-03-24T10:37:21.759 INFO:teuthology.orchestra.run.smithi057.stdout:(73/196): libzstd-devel-1.5.1-2.el9.x86_64.rpm 799 kB/s | 46 kB 00:00 2024-03-24T10:37:21.762 INFO:teuthology.orchestra.run.smithi005.stdout:(21/196): ceph-radosgw-debuginfo-19.0.0-1855.g2 2.7 MB/s | 176 MB 01:04 2024-03-24T10:37:21.786 INFO:teuthology.orchestra.run.smithi057.stdout:(74/196): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-24T10:37:21.809 INFO:teuthology.orchestra.run.smithi057.stdout:(75/196): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-24T10:37:21.835 INFO:teuthology.orchestra.run.smithi057.stdout:(76/196): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-24T10:37:21.860 INFO:teuthology.orchestra.run.smithi057.stdout:(77/196): openblas-0.3.21-2.el9.x86_64.rpm 684 kB/s | 34 kB 00:00 2024-03-24T10:37:21.954 INFO:teuthology.orchestra.run.smithi005.stdout:(22/196): cephfs-mirror-19.0.0-1855.g21b3bd67.e 1.1 MB/s | 219 kB 00:00 2024-03-24T10:37:21.960 INFO:teuthology.orchestra.run.smithi057.stdout:(78/196): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-24T10:37:22.010 INFO:teuthology.orchestra.run.smithi057.stdout:(79/196): postgresql-13.11-1.el9.x86_64.rpm 10 MB/s | 1.5 MB 00:00 2024-03-24T10:37:22.021 INFO:teuthology.orchestra.run.smithi005.stdout:(23/196): cephfs-shell-19.0.0-1855.g21b3bd67.el 407 kB/s | 27 kB 00:00 2024-03-24T10:37:22.036 INFO:teuthology.orchestra.run.smithi057.stdout:(80/196): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-24T10:37:22.061 INFO:teuthology.orchestra.run.smithi057.stdout:(81/196): postgresql-private-libs-13.11-1.el9.x 2.7 MB/s | 134 kB 00:00 2024-03-24T10:37:22.105 INFO:teuthology.orchestra.run.smithi005.stdout:(24/196): libcephfs-devel-19.0.0-1855.g21b3bd67 380 kB/s | 32 kB 00:00 2024-03-24T10:37:22.170 INFO:teuthology.orchestra.run.smithi057.stdout:(82/196): postgresql-server-13.11-1.el9.x86_64. 43 MB/s | 5.7 MB 00:00 2024-03-24T10:37:22.203 INFO:teuthology.orchestra.run.smithi057.stdout:(83/196): protobuf-3.14.0-13.el9.x86_64.rpm 7.2 MB/s | 1.0 MB 00:00 2024-03-24T10:37:22.228 INFO:teuthology.orchestra.run.smithi057.stdout:(84/196): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-24T10:37:22.322 INFO:teuthology.orchestra.run.smithi005.stdout:(25/196): libcephfs2-19.0.0-1855.g21b3bd67.el9. 3.4 MB/s | 744 kB 00:00 2024-03-24T10:37:22.346 INFO:teuthology.orchestra.run.smithi057.stdout:(85/196): python3-babel-2.9.1-2.el9.noarch.rpm 41 MB/s | 5.8 MB 00:00 2024-03-24T10:37:22.371 INFO:teuthology.orchestra.run.smithi057.stdout:(86/196): python3-jsonpatch-1.21-16.el9.noarch. 175 kB/s | 24 kB 00:00 2024-03-24T10:37:22.396 INFO:teuthology.orchestra.run.smithi057.stdout:(87/196): python3-jsonpointer-2.0-4.el9.noarch. 361 kB/s | 18 kB 00:00 2024-03-24T10:37:22.422 INFO:teuthology.orchestra.run.smithi057.stdout:(88/196): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-24T10:37:22.447 INFO:teuthology.orchestra.run.smithi057.stdout:(89/196): python3-markupsafe-1.1.1-12.el9.x86_6 631 kB/s | 32 kB 00:00 2024-03-24T10:37:22.547 INFO:teuthology.orchestra.run.smithi057.stdout:(90/196): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-24T10:37:22.572 INFO:teuthology.orchestra.run.smithi057.stdout:(91/196): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-03-24T10:37:22.598 INFO:teuthology.orchestra.run.smithi057.stdout:(92/196): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-24T10:37:22.623 INFO:teuthology.orchestra.run.smithi057.stdout:(93/196): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-24T10:37:22.640 INFO:teuthology.orchestra.run.smithi057.stdout:(94/196): python3-prettytable-0.7.2-27.el9.noar 981 kB/s | 41 kB 00:00 2024-03-24T10:37:22.665 INFO:teuthology.orchestra.run.smithi057.stdout:(95/196): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-24T10:37:22.691 INFO:teuthology.orchestra.run.smithi057.stdout:(96/196): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-24T10:37:22.716 INFO:teuthology.orchestra.run.smithi057.stdout:(97/196): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-24T10:37:22.750 INFO:teuthology.orchestra.run.smithi057.stdout:(98/196): python3-pytz-2021.1-5.el9.noarch.rpm 814 kB/s | 47 kB 00:00 2024-03-24T10:37:22.775 INFO:teuthology.orchestra.run.smithi057.stdout:(99/196): python3-requests-oauthlib-1.3.0-12.el 730 kB/s | 43 kB 00:00 2024-03-24T10:37:22.959 INFO:teuthology.orchestra.run.smithi057.stdout:(100/196): python3-toml-0.10.2-6.el9.noarch.rpm 213 kB/s | 39 kB 00:00 2024-03-24T10:37:23.101 INFO:teuthology.orchestra.run.smithi057.stdout:(101/196): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-24T10:37:23.126 INFO:teuthology.orchestra.run.smithi057.stdout:(102/196): python3-wcwidth-0.2.5-8.el9.noarch.r 247 kB/s | 41 kB 00:00 2024-03-24T10:37:23.151 INFO:teuthology.orchestra.run.smithi057.stdout:(103/196): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-24T10:37:23.179 INFO:teuthology.orchestra.run.smithi057.stdout:(104/196): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-24T10:37:23.202 INFO:teuthology.orchestra.run.smithi057.stdout:(105/196): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-24T10:37:23.227 INFO:teuthology.orchestra.run.smithi057.stdout:(106/196): zlib-devel-1.2.11-40.el9.x86_64.rpm 877 kB/s | 44 kB 00:00 2024-03-24T10:37:23.238 INFO:teuthology.orchestra.run.smithi005.stdout:(26/196): libcephfs2-debuginfo-19.0.0-1855.g21b 5.9 MB/s | 5.4 MB 00:00 2024-03-24T10:37:23.244 INFO:teuthology.orchestra.run.smithi057.stdout:(107/196): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-24T10:37:23.269 INFO:teuthology.orchestra.run.smithi057.stdout:(108/196): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-24T10:37:23.303 INFO:teuthology.orchestra.run.smithi057.stdout:(109/196): python3-cryptography-36.0.1-4.el9.x8 20 MB/s | 1.1 MB 00:00 2024-03-24T10:37:23.320 INFO:teuthology.orchestra.run.smithi057.stdout:(110/196): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 103 kB 00:00 2024-03-24T10:37:23.321 INFO:teuthology.orchestra.run.smithi005.stdout:(27/196): libcephsqlite-19.0.0-1855.g21b3bd67.e 1.9 MB/s | 163 kB 00:00 2024-03-24T10:37:23.345 INFO:teuthology.orchestra.run.smithi057.stdout:(111/196): python3-pycparser-2.20-6.el9.noarch. 2.9 MB/s | 124 kB 00:00 2024-03-24T10:37:23.371 INFO:teuthology.orchestra.run.smithi057.stdout:(112/196): python3-urllib3-1.26.5-5.el9.noarch. 3.7 MB/s | 187 kB 00:00 2024-03-24T10:37:23.414 INFO:teuthology.orchestra.run.smithi005.stdout:(28/196): librados-devel-19.0.0-1855.g21b3bd67. 1.3 MB/s | 127 kB 00:00 2024-03-24T10:37:23.506 INFO:teuthology.orchestra.run.smithi005.stdout:(29/196): librados-devel-debuginfo-19.0.0-1855. 1.2 MB/s | 109 kB 00:00 2024-03-24T10:37:23.538 INFO:teuthology.orchestra.run.smithi057.stdout:(113/196): jq-1.6-16.el9.x86_64.rpm 976 kB/s | 187 kB 00:00 2024-03-24T10:37:23.605 INFO:teuthology.orchestra.run.smithi057.stdout:(114/196): libgfortran-11.4.1-3.el9.x86_64.rpm 3.4 MB/s | 803 kB 00:00 2024-03-24T10:37:23.755 INFO:teuthology.orchestra.run.smithi057.stdout:(115/196): libquadmath-11.4.1-3.el9.x86_64.rpm 900 kB/s | 195 kB 00:00 2024-03-24T10:37:23.797 INFO:teuthology.orchestra.run.smithi057.stdout:(116/196): python3-requests-2.25.1-8.el9.noarch 650 kB/s | 125 kB 00:00 2024-03-24T10:37:23.889 INFO:teuthology.orchestra.run.smithi057.stdout:(117/196): elfutils-libelf-devel-0.190-2.el9.x8 161 kB/s | 21 kB 00:00 2024-03-24T10:37:23.922 INFO:teuthology.orchestra.run.smithi057.stdout:(118/196): libuuid-devel-2.37.4-18.el9.x86_64.r 203 kB/s | 25 kB 00:00 2024-03-24T10:37:24.081 INFO:teuthology.orchestra.run.smithi057.stdout:(119/196): python3-devel-3.9.17-2.el9.x86_64.rp 1.5 MB/s | 245 kB 00:00 2024-03-24T10:37:24.140 INFO:teuthology.orchestra.run.smithi057.stdout:(120/196): python3-jinja2-2.11.3-5.el9.noarch.r 4.2 MB/s | 248 kB 00:00 2024-03-24T10:37:24.198 INFO:teuthology.orchestra.run.smithi057.stdout:(121/196): qatlib-23.11.0-1.el9.x86_64.rpm 3.6 MB/s | 216 kB 00:00 2024-03-24T10:37:24.266 INFO:teuthology.orchestra.run.smithi057.stdout:(122/196): openssl-devel-3.0.7-27.el9.x86_64.rp 11 MB/s | 4.1 MB 00:00 2024-03-24T10:37:24.291 INFO:teuthology.orchestra.run.smithi057.stdout:(123/196): qatlib-service-23.11.0-1.el9.x86_64. 378 kB/s | 35 kB 00:00 2024-03-24T10:37:24.324 INFO:teuthology.orchestra.run.smithi057.stdout:(124/196): qatzip-libs-1.1.2-2.el9.x86_64.rpm 763 kB/s | 44 kB 00:00 2024-03-24T10:37:24.341 INFO:teuthology.orchestra.run.smithi057.stdout:(125/196): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-24T10:37:24.400 INFO:teuthology.orchestra.run.smithi057.stdout:(126/196): lua-devel-5.4.4-4.el9.x86_64.rpm 295 kB/s | 22 kB 00:00 2024-03-24T10:37:24.509 INFO:teuthology.orchestra.run.smithi057.stdout:(127/196): protobuf-compiler-3.14.0-13.el9.x86_ 5.0 MB/s | 863 kB 00:00 2024-03-24T10:37:24.659 INFO:teuthology.orchestra.run.smithi057.stdout:(128/196): abseil-cpp-20211102.0-3.el9.x86_64.r 2.1 MB/s | 548 kB 00:00 2024-03-24T10:37:24.701 INFO:teuthology.orchestra.run.smithi057.stdout:(129/196): grpc-data-1.46.7-10.el9.noarch.rpm 466 kB/s | 19 kB 00:00 2024-03-24T10:37:24.743 INFO:teuthology.orchestra.run.smithi057.stdout:(130/196): gperftools-libs-2.9.1-2.el9.x86_64.r 1.3 MB/s | 309 kB 00:00 2024-03-24T10:37:24.785 INFO:teuthology.orchestra.run.smithi057.stdout:(131/196): libarrow-doc-9.0.0-10.el9.noarch.rpm 623 kB/s | 26 kB 00:00 2024-03-24T10:37:24.836 INFO:teuthology.orchestra.run.smithi057.stdout:(132/196): liboath-2.6.7-2.el9.x86_64.rpm 965 kB/s | 49 kB 00:00 2024-03-24T10:37:24.911 INFO:teuthology.orchestra.run.smithi057.stdout:(133/196): libarrow-9.0.0-10.el9.x86_64.rpm 21 MB/s | 4.4 MB 00:00 2024-03-24T10:37:24.936 INFO:teuthology.orchestra.run.smithi057.stdout:(134/196): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2024-03-24T10:37:24.970 INFO:teuthology.orchestra.run.smithi057.stdout:(135/196): luarocks-3.9.2-1.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-03-24T10:37:25.028 INFO:teuthology.orchestra.run.smithi057.stdout:(136/196): python3-asyncssh-2.13.2-5.el9.noarch 9.2 MB/s | 548 kB 00:00 2024-03-24T10:37:25.070 INFO:teuthology.orchestra.run.smithi057.stdout:(137/196): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 838 kB 00:00 2024-03-24T10:37:25.095 INFO:teuthology.orchestra.run.smithi057.stdout:(138/196): python3-bcrypt-3.2.2-1.el9.x86_64.rp 648 kB/s | 43 kB 00:00 2024-03-24T10:37:25.121 INFO:teuthology.orchestra.run.smithi057.stdout:(139/196): python3-cachetools-4.2.4-1.el9.noarc 638 kB/s | 32 kB 00:00 2024-03-24T10:37:25.146 INFO:teuthology.orchestra.run.smithi057.stdout:(140/196): python3-certifi-2021.10.8-2.el9.noar 297 kB/s | 15 kB 00:00 2024-03-24T10:37:25.180 INFO:teuthology.orchestra.run.smithi057.stdout:(141/196): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-03-24T10:37:25.205 INFO:teuthology.orchestra.run.smithi057.stdout:(142/196): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-03-24T10:37:25.239 INFO:teuthology.orchestra.run.smithi057.stdout:(143/196): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-03-24T10:37:25.264 INFO:teuthology.orchestra.run.smithi057.stdout:(144/196): python3-colorama-0.4.4-12.el9.noarch 603 kB/s | 35 kB 00:00 2024-03-24T10:37:25.298 INFO:teuthology.orchestra.run.smithi057.stdout:(145/196): python3-google-auth-2.28.2-1.el9.noa 3.4 MB/s | 203 kB 00:00 2024-03-24T10:37:25.356 INFO:teuthology.orchestra.run.smithi057.stdout:(146/196): python3-grpcio-1.46.7-10.el9.x86_64. 22 MB/s | 2.0 MB 00:00 2024-03-24T10:37:25.382 INFO:teuthology.orchestra.run.smithi057.stdout:(147/196): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-03-24T10:37:25.407 INFO:teuthology.orchestra.run.smithi057.stdout:(148/196): python3-jaraco-8.2.1-3.el9.noarch.rp 212 kB/s | 11 kB 00:00 2024-03-24T10:37:25.432 INFO:teuthology.orchestra.run.smithi057.stdout:(149/196): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-03-24T10:37:25.458 INFO:teuthology.orchestra.run.smithi057.stdout:(150/196): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-03-24T10:37:25.483 INFO:teuthology.orchestra.run.smithi057.stdout:(151/196): python3-jaraco-functools-3.5.0-2.el9 385 kB/s | 19 kB 00:00 2024-03-24T10:37:25.508 INFO:teuthology.orchestra.run.smithi057.stdout:(152/196): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-03-24T10:37:25.575 INFO:teuthology.orchestra.run.smithi057.stdout:(153/196): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-03-24T10:37:25.601 INFO:teuthology.orchestra.run.smithi057.stdout:(154/196): python3-logutils-0.3.5-21.el9.noarch 501 kB/s | 46 kB 00:00 2024-03-24T10:37:25.618 INFO:teuthology.orchestra.run.smithi057.stdout:(155/196): python3-more-itertools-8.12.0-2.el9. 1.8 MB/s | 79 kB 00:00 2024-03-24T10:37:25.651 INFO:teuthology.orchestra.run.smithi057.stdout:(156/196): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-03-24T10:37:25.677 INFO:teuthology.orchestra.run.smithi057.stdout:(157/196): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-03-24T10:37:25.702 INFO:teuthology.orchestra.run.smithi057.stdout:(158/196): python3-portend-3.1.0-2.el9.noarch.r 325 kB/s | 16 kB 00:00 2024-03-24T10:37:25.727 INFO:teuthology.orchestra.run.smithi057.stdout:(159/196): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-03-24T10:37:25.752 INFO:teuthology.orchestra.run.smithi057.stdout:(160/196): python3-pyperclip-1.8.2-2.el9.noarch 563 kB/s | 28 kB 00:00 2024-03-24T10:37:25.778 INFO:teuthology.orchestra.run.smithi057.stdout:(161/196): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-03-24T10:37:25.811 INFO:teuthology.orchestra.run.smithi057.stdout:(162/196): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-03-24T10:37:25.837 INFO:teuthology.orchestra.run.smithi057.stdout:(163/196): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-24T10:37:25.862 INFO:teuthology.orchestra.run.smithi057.stdout:(164/196): python3-tempora-5.0.0-2.el9.noarch.r 710 kB/s | 36 kB 00:00 2024-03-24T10:37:25.887 INFO:teuthology.orchestra.run.smithi057.stdout:(165/196): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-03-24T10:37:25.921 INFO:teuthology.orchestra.run.smithi057.stdout:(166/196): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-03-24T10:37:25.946 INFO:teuthology.orchestra.run.smithi057.stdout:(167/196): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-03-24T10:37:25.978 INFO:teuthology.orchestra.run.smithi057.stdout:(168/196): python3-werkzeug-2.0.3-3.el9.1.noarc 7.4 MB/s | 427 kB 00:00 2024-03-24T10:37:26.003 INFO:teuthology.orchestra.run.smithi057.stdout:(169/196): python3-zc-lockfile-2.0-10.el9.noarc 356 kB/s | 20 kB 00:00 2024-03-24T10:37:26.037 INFO:teuthology.orchestra.run.smithi057.stdout:(170/196): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-24T10:37:26.095 INFO:teuthology.orchestra.run.smithi057.stdout:(171/196): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-03-24T10:37:27.321 INFO:teuthology.orchestra.run.smithi057.stdout:(172/196): librados2-19.0.0-1855.g21b3bd67.el9. 2.6 MB/s | 3.4 MB 00:01 2024-03-24T10:37:27.363 INFO:teuthology.orchestra.run.smithi057.stdout:(173/196): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-24T10:37:27.838 INFO:teuthology.orchestra.run.smithi057.stdout:(174/196): NetworkManager-1.47.2-1.el9.x86_64.r 4.9 MB/s | 2.3 MB 00:00 2024-03-24T10:37:28.138 INFO:teuthology.orchestra.run.smithi057.stdout:(175/196): NetworkManager-libnm-1.47.2-1.el9.x8 6.1 MB/s | 1.8 MB 00:00 2024-03-24T10:37:28.180 INFO:teuthology.orchestra.run.smithi057.stdout:(176/196): NetworkManager-team-1.47.2-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-03-24T10:37:28.264 INFO:teuthology.orchestra.run.smithi057.stdout:(177/196): NetworkManager-tui-1.47.2-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-03-24T10:37:28.314 INFO:teuthology.orchestra.run.smithi057.stdout:(178/196): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-03-24T10:37:28.364 INFO:teuthology.orchestra.run.smithi057.stdout:(179/196): elfutils-debuginfod-client-0.190-2.e 730 kB/s | 36 kB 00:00 2024-03-24T10:37:28.423 INFO:teuthology.orchestra.run.smithi057.stdout:(180/196): elfutils-libelf-0.190-2.el9.x86_64.r 3.2 MB/s | 192 kB 00:00 2024-03-24T10:37:28.482 INFO:teuthology.orchestra.run.smithi057.stdout:(181/196): elfutils-libs-0.190-2.el9.x86_64.rpm 4.3 MB/s | 256 kB 00:00 2024-03-24T10:37:28.532 INFO:teuthology.orchestra.run.smithi057.stdout:(182/196): libacl-2.3.1-4.el9.x86_64.rpm 463 kB/s | 23 kB 00:00 2024-03-24T10:37:28.582 INFO:teuthology.orchestra.run.smithi057.stdout:(183/196): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-24T10:37:28.633 INFO:teuthology.orchestra.run.smithi057.stdout:(184/196): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 155 kB 00:00 2024-03-24T10:37:28.675 INFO:teuthology.orchestra.run.smithi057.stdout:(185/196): libmount-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 136 kB 00:00 2024-03-24T10:37:28.725 INFO:teuthology.orchestra.run.smithi057.stdout:(186/196): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-24T10:37:28.775 INFO:teuthology.orchestra.run.smithi057.stdout:(187/196): libuuid-2.37.4-18.el9.x86_64.rpm 575 kB/s | 29 kB 00:00 2024-03-24T10:37:28.959 INFO:teuthology.orchestra.run.smithi057.stdout:(188/196): openssl-3.0.7-27.el9.x86_64.rpm 6.6 MB/s | 1.2 MB 00:00 2024-03-24T10:37:29.209 INFO:teuthology.orchestra.run.smithi057.stdout:(189/196): openssl-libs-3.0.7-27.el9.x86_64.rpm 8.6 MB/s | 2.2 MB 00:00 2024-03-24T10:37:29.460 INFO:teuthology.orchestra.run.smithi057.stdout:(190/196): util-linux-2.37.4-18.el9.x86_64.rpm 9.1 MB/s | 2.3 MB 00:00 2024-03-24T10:37:29.552 INFO:teuthology.orchestra.run.smithi057.stdout:(191/196): util-linux-core-2.37.4-18.el9.x86_64 5.0 MB/s | 465 kB 00:00 2024-03-24T10:37:29.602 INFO:teuthology.orchestra.run.smithi057.stdout:(192/196): util-linux-user-2.37.4-18.el9.x86_64 623 kB/s | 31 kB 00:00 2024-03-24T10:37:29.727 INFO:teuthology.orchestra.run.smithi057.stdout:(193/196): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.6 MB/s | 1.1 MB 00:00 2024-03-24T10:37:29.844 INFO:teuthology.orchestra.run.smithi057.stdout:(194/196): libacl-devel-2.3.1-4.el9.x86_64.rpm 636 kB/s | 74 kB 00:00 2024-03-24T10:37:33.171 INFO:teuthology.orchestra.run.smithi005.stdout:(30/196): librados2-debuginfo-19.0.0-1855.g21b3 4.6 MB/s | 44 MB 00:09 2024-03-24T10:37:33.355 INFO:teuthology.orchestra.run.smithi005.stdout:(31/196): libradosstriper1-19.0.0-1855.g21b3bd6 2.7 MB/s | 499 kB 00:00 2024-03-24T10:37:33.411 INFO:teuthology.orchestra.run.smithi057.stdout:(195/196): librbd1-19.0.0-1855.g21b3bd67.el9.x8 439 kB/s | 3.1 MB 00:07 2024-03-24T10:37:39.013 INFO:teuthology.orchestra.run.smithi005.stdout:(32/196): ceph-common-debuginfo-19.0.0-1855.g21 1.8 MB/s | 437 MB 04:00 2024-03-24T10:37:40.180 INFO:teuthology.orchestra.run.smithi005.stdout:(33/196): librgw2-19.0.0-1855.g21b3bd67.el9.x86 4.4 MB/s | 5.1 MB 00:01 2024-03-24T10:37:40.272 INFO:teuthology.orchestra.run.smithi005.stdout:(34/196): python3-ceph-argparse-19.0.0-1855.g21 494 kB/s | 45 kB 00:00 2024-03-24T10:37:40.372 INFO:teuthology.orchestra.run.smithi005.stdout:(35/196): python3-ceph-common-19.0.0-1855.g21b3 1.3 MB/s | 131 kB 00:00 2024-03-24T10:37:40.472 INFO:teuthology.orchestra.run.smithi005.stdout:(36/196): python3-cephfs-19.0.0-1855.g21b3bd67. 1.6 MB/s | 161 kB 00:00 2024-03-24T10:37:40.648 INFO:teuthology.orchestra.run.smithi005.stdout:(37/196): python3-cephfs-debuginfo-19.0.0-1855. 3.4 MB/s | 615 kB 00:00 2024-03-24T10:37:40.748 INFO:teuthology.orchestra.run.smithi005.stdout:(38/196): python3-rados-19.0.0-1855.g21b3bd67.e 3.1 MB/s | 320 kB 00:00 2024-03-24T10:37:40.948 INFO:teuthology.orchestra.run.smithi005.stdout:(39/196): python3-rados-debuginfo-19.0.0-1855.g 3.8 MB/s | 770 kB 00:00 2024-03-24T10:37:41.057 INFO:teuthology.orchestra.run.smithi005.stdout:(40/196): python3-rbd-19.0.0-1855.g21b3bd67.el9 2.7 MB/s | 298 kB 00:00 2024-03-24T10:37:41.324 INFO:teuthology.orchestra.run.smithi005.stdout:(41/196): python3-rbd-debuginfo-19.0.0-1855.g21 4.2 MB/s | 1.1 MB 00:00 2024-03-24T10:37:41.416 INFO:teuthology.orchestra.run.smithi005.stdout:(42/196): python3-rgw-19.0.0-1855.g21b3bd67.el9 1.1 MB/s | 99 kB 00:00 2024-03-24T10:37:41.532 INFO:teuthology.orchestra.run.smithi005.stdout:(43/196): python3-rgw-debuginfo-19.0.0-1855.g21 2.8 MB/s | 337 kB 00:00 2024-03-24T10:37:41.633 INFO:teuthology.orchestra.run.smithi005.stdout:(44/196): rbd-fuse-19.0.0-1855.g21b3bd67.el9.x8 853 kB/s | 85 kB 00:00 2024-03-24T10:37:41.750 INFO:teuthology.orchestra.run.smithi005.stdout:(45/196): rbd-fuse-debuginfo-19.0.0-1855.g21b3b 4.3 MB/s | 509 kB 00:00 2024-03-24T10:37:42.333 INFO:teuthology.orchestra.run.smithi005.stdout:(46/196): rbd-mirror-19.0.0-1855.g21b3bd67.el9. 5.3 MB/s | 3.1 MB 00:00 2024-03-24T10:37:44.058 INFO:teuthology.orchestra.run.smithi005.stdout:(47/196): librbd1-debuginfo-19.0.0-1855.g21b3bd 4.8 MB/s | 52 MB 00:10 2024-03-24T10:37:44.142 INFO:teuthology.orchestra.run.smithi005.stdout:(48/196): rbd-nbd-19.0.0-1855.g21b3bd67.el9.x86 2.0 MB/s | 170 kB 00:00 2024-03-24T10:37:44.400 INFO:teuthology.orchestra.run.smithi005.stdout:(49/196): rbd-nbd-debuginfo-19.0.0-1855.g21b3bd 4.9 MB/s | 1.3 MB 00:00 2024-03-24T10:37:44.476 INFO:teuthology.orchestra.run.smithi005.stdout:(50/196): ceph-grafana-dashboards-19.0.0-1855.g 419 kB/s | 31 kB 00:00 2024-03-24T10:37:44.568 INFO:teuthology.orchestra.run.smithi005.stdout:(51/196): ceph-mgr-cephadm-19.0.0-1855.g21b3bd6 1.5 MB/s | 138 kB 00:00 2024-03-24T10:37:44.910 INFO:teuthology.orchestra.run.smithi005.stdout:(52/196): ceph-mgr-dashboard-19.0.0-1855.g21b3b 5.2 MB/s | 1.8 MB 00:00 2024-03-24T10:37:46.135 INFO:teuthology.orchestra.run.smithi005.stdout:(53/196): ceph-mgr-diskprediction-local-19.0.0- 6.0 MB/s | 7.4 MB 00:01 2024-03-24T10:37:46.235 INFO:teuthology.orchestra.run.smithi005.stdout:(54/196): ceph-mgr-modules-core-19.0.0-1855.g21 2.4 MB/s | 248 kB 00:00 2024-03-24T10:37:46.319 INFO:teuthology.orchestra.run.smithi005.stdout:(55/196): ceph-mgr-rook-19.0.0-1855.g21b3bd67.e 592 kB/s | 49 kB 00:00 2024-03-24T10:37:46.394 INFO:teuthology.orchestra.run.smithi005.stdout:(56/196): ceph-prometheus-alerts-19.0.0-1855.g2 204 kB/s | 15 kB 00:00 2024-03-24T10:37:46.494 INFO:teuthology.orchestra.run.smithi005.stdout:(57/196): ceph-volume-19.0.0-1855.g21b3bd67.el9 2.6 MB/s | 264 kB 00:00 2024-03-24T10:37:46.669 INFO:teuthology.orchestra.run.smithi005.stdout:(58/196): cephadm-19.0.0-1855.g21b3bd67.el9.noa 4.2 MB/s | 748 kB 00:00 2024-03-24T10:37:46.761 INFO:teuthology.orchestra.run.smithi005.stdout:(59/196): cephfs-top-19.0.0-1855.g21b3bd67.el9. 199 kB/s | 18 kB 00:00 2024-03-24T10:37:46.929 INFO:teuthology.orchestra.run.smithi005.stdout:(60/196): automake-1.16.2-8.el9.noarch.rpm 3.9 MB/s | 662 kB 00:00 2024-03-24T10:37:46.979 INFO:teuthology.orchestra.run.smithi005.stdout:(61/196): bison-3.7.4-5.el9.x86_64.rpm 18 MB/s | 920 kB 00:00 2024-03-24T10:37:47.013 INFO:teuthology.orchestra.run.smithi005.stdout:(62/196): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-24T10:37:47.055 INFO:teuthology.orchestra.run.smithi005.stdout:(63/196): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-24T10:37:47.088 INFO:teuthology.orchestra.run.smithi005.stdout:(64/196): flexiblas-3.0.4-8.el9.x86_64.rpm 952 kB/s | 32 kB 00:00 2024-03-24T10:37:47.189 INFO:teuthology.orchestra.run.smithi005.stdout:(65/196): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-24T10:37:47.222 INFO:teuthology.orchestra.run.smithi005.stdout:(66/196): flexiblas-openblas-openmp-3.0.4-8.el9 517 kB/s | 17 kB 00:00 2024-03-24T10:37:47.256 INFO:teuthology.orchestra.run.smithi005.stdout:(67/196): libaio-devel-0.3.111-13.el9.x86_64.rp 303 kB/s | 10 kB 00:00 2024-03-24T10:37:47.289 INFO:teuthology.orchestra.run.smithi005.stdout:(68/196): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.6 MB/s | 159 kB 00:00 2024-03-24T10:37:47.323 INFO:teuthology.orchestra.run.smithi005.stdout:(69/196): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-24T10:37:47.382 INFO:teuthology.orchestra.run.smithi005.stdout:(70/196): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-24T10:37:47.424 INFO:teuthology.orchestra.run.smithi005.stdout:(71/196): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 578 kB 00:00 2024-03-24T10:37:47.450 INFO:teuthology.orchestra.run.smithi005.stdout:(72/196): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.8 MB/s | 46 kB 00:00 2024-03-24T10:37:47.482 INFO:teuthology.orchestra.run.smithi005.stdout:(73/196): lttng-ust-2.12.0-6.el9.x86_64.rpm 8.3 MB/s | 282 kB 00:00 2024-03-24T10:37:47.516 INFO:teuthology.orchestra.run.smithi005.stdout:(74/196): lua-5.4.4-4.el9.x86_64.rpm 5.5 MB/s | 187 kB 00:00 2024-03-24T10:37:47.550 INFO:teuthology.orchestra.run.smithi005.stdout:(75/196): oniguruma-6.9.6-1.el9.5.x86_64.rpm 6.3 MB/s | 217 kB 00:00 2024-03-24T10:37:47.575 INFO:teuthology.orchestra.run.smithi005.stdout:(76/196): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-24T10:37:47.693 INFO:teuthology.orchestra.run.smithi005.stdout:(77/196): openblas-openmp-0.3.21-2.el9.x86_64.r 41 MB/s | 4.7 MB 00:00 2024-03-24T10:37:47.751 INFO:teuthology.orchestra.run.smithi005.stdout:(78/196): postgresql-13.11-1.el9.x86_64.rpm 26 MB/s | 1.5 MB 00:00 2024-03-24T10:37:47.793 INFO:teuthology.orchestra.run.smithi005.stdout:(79/196): postgresql-contrib-13.11-1.el9.x86_64 19 MB/s | 811 kB 00:00 2024-03-24T10:37:47.826 INFO:teuthology.orchestra.run.smithi005.stdout:(80/196): postgresql-private-libs-13.11-1.el9.x 3.9 MB/s | 134 kB 00:00 2024-03-24T10:37:47.968 INFO:teuthology.orchestra.run.smithi005.stdout:(81/196): postgresql-server-13.11-1.el9.x86_64. 40 MB/s | 5.7 MB 00:00 2024-03-24T10:37:48.018 INFO:teuthology.orchestra.run.smithi005.stdout:(82/196): protobuf-3.14.0-13.el9.x86_64.rpm 20 MB/s | 1.0 MB 00:00 2024-03-24T10:37:48.052 INFO:teuthology.orchestra.run.smithi005.stdout:(83/196): python3-attrs-20.3.0-7.el9.noarch.rpm 2.3 MB/s | 78 kB 00:00 2024-03-24T10:37:48.202 INFO:teuthology.orchestra.run.smithi005.stdout:(84/196): python3-babel-2.9.1-2.el9.noarch.rpm 39 MB/s | 5.8 MB 00:00 2024-03-24T10:37:48.228 INFO:teuthology.orchestra.run.smithi005.stdout:(85/196): python3-jsonpatch-1.21-16.el9.noarch. 977 kB/s | 24 kB 00:00 2024-03-24T10:37:48.253 INFO:teuthology.orchestra.run.smithi005.stdout:(86/196): python3-jsonpointer-2.0-4.el9.noarch. 731 kB/s | 18 kB 00:00 2024-03-24T10:37:48.287 INFO:teuthology.orchestra.run.smithi005.stdout:(87/196): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-24T10:37:48.321 INFO:teuthology.orchestra.run.smithi005.stdout:(88/196): python3-markupsafe-1.1.1-12.el9.x86_6 945 kB/s | 32 kB 00:00 2024-03-24T10:37:48.454 INFO:teuthology.orchestra.run.smithi005.stdout:(89/196): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-24T10:37:48.488 INFO:teuthology.orchestra.run.smithi005.stdout:(90/196): python3-numpy-f2py-1.20.1-5.el9.x86_6 6.4 MB/s | 217 kB 00:00 2024-03-24T10:37:48.522 INFO:teuthology.orchestra.run.smithi005.stdout:(91/196): python3-oauthlib-3.1.1-5.el9.noarch.r 5.1 MB/s | 174 kB 00:00 2024-03-24T10:37:48.555 INFO:teuthology.orchestra.run.smithi005.stdout:(92/196): python3-packaging-20.9-5.el9.noarch.r 2.0 MB/s | 69 kB 00:00 2024-03-24T10:37:48.589 INFO:teuthology.orchestra.run.smithi005.stdout:(93/196): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-24T10:37:48.622 INFO:teuthology.orchestra.run.smithi005.stdout:(94/196): python3-protobuf-3.14.0-13.el9.noarch 7.0 MB/s | 238 kB 00:00 2024-03-24T10:37:48.656 INFO:teuthology.orchestra.run.smithi005.stdout:(95/196): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.9 MB/s | 132 kB 00:00 2024-03-24T10:37:48.690 INFO:teuthology.orchestra.run.smithi005.stdout:(96/196): python3-pyasn1-modules-0.4.8-6.el9.no 6.2 MB/s | 211 kB 00:00 2024-03-24T10:37:48.732 INFO:teuthology.orchestra.run.smithi005.stdout:(97/196): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-24T10:37:48.765 INFO:teuthology.orchestra.run.smithi005.stdout:(98/196): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-24T10:37:49.116 INFO:teuthology.orchestra.run.smithi005.stdout:(99/196): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-24T10:37:49.149 INFO:teuthology.orchestra.run.smithi005.stdout:(100/196): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-24T10:37:49.183 INFO:teuthology.orchestra.run.smithi005.stdout:(101/196): python3-wcwidth-0.2.5-8.el9.noarch.r 1.2 MB/s | 41 kB 00:00 2024-03-24T10:37:49.225 INFO:teuthology.orchestra.run.smithi005.stdout:(102/196): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-24T10:37:49.258 INFO:teuthology.orchestra.run.smithi005.stdout:(103/196): uuid-1.6.2-55.el9.x86_64.rpm 1.7 MB/s | 56 kB 00:00 2024-03-24T10:37:49.292 INFO:teuthology.orchestra.run.smithi005.stdout:(104/196): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-24T10:37:49.326 INFO:teuthology.orchestra.run.smithi005.stdout:(105/196): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-24T10:37:49.359 INFO:teuthology.orchestra.run.smithi005.stdout:(106/196): fuse-2.9.9-15.el9.x86_64.rpm 2.3 MB/s | 79 kB 00:00 2024-03-24T10:37:49.393 INFO:teuthology.orchestra.run.smithi005.stdout:(107/196): python3-cffi-1.14.5-5.el9.x86_64.rpm 7.1 MB/s | 241 kB 00:00 2024-03-24T10:37:49.452 INFO:teuthology.orchestra.run.smithi005.stdout:(108/196): python3-cryptography-36.0.1-4.el9.x8 20 MB/s | 1.1 MB 00:00 2024-03-24T10:37:49.485 INFO:teuthology.orchestra.run.smithi005.stdout:(109/196): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-24T10:37:49.519 INFO:teuthology.orchestra.run.smithi005.stdout:(110/196): python3-pycparser-2.20-6.el9.noarch. 3.7 MB/s | 124 kB 00:00 2024-03-24T10:37:50.261 INFO:teuthology.orchestra.run.smithi005.stdout:(111/196): rbd-mirror-debuginfo-19.0.0-1855.g21 6.5 MB/s | 51 MB 00:07 2024-03-24T10:37:50.536 INFO:teuthology.orchestra.run.smithi005.stdout:(112/196): python3-urllib3-1.26.5-5.el9.noarch. 184 kB/s | 187 kB 00:01 2024-03-24T10:37:50.878 INFO:teuthology.orchestra.run.smithi005.stdout:(113/196): jq-1.6-16.el9.x86_64.rpm 304 kB/s | 187 kB 00:00 2024-03-24T10:37:51.187 INFO:teuthology.orchestra.run.smithi005.stdout:(114/196): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-03-24T10:37:51.454 INFO:teuthology.orchestra.run.smithi005.stdout:(115/196): libquadmath-11.4.1-3.el9.x86_64.rpm 340 kB/s | 195 kB 00:00 2024-03-24T10:37:51.596 INFO:teuthology.orchestra.run.smithi005.stdout:(116/196): python3-requests-2.25.1-8.el9.noarch 306 kB/s | 125 kB 00:00 2024-03-24T10:37:51.713 INFO:teuthology.orchestra.run.smithi005.stdout:(117/196): elfutils-libelf-devel-0.190-2.el9.x8 83 kB/s | 21 kB 00:00 2024-03-24T10:37:51.730 INFO:teuthology.orchestra.run.smithi005.stdout:(118/196): libuuid-devel-2.37.4-18.el9.x86_64.r 191 kB/s | 25 kB 00:00 2024-03-24T10:37:51.888 INFO:teuthology.orchestra.run.smithi005.stdout:(119/196): python3-devel-3.9.17-2.el9.x86_64.rp 1.5 MB/s | 245 kB 00:00 2024-03-24T10:37:51.947 INFO:teuthology.orchestra.run.smithi005.stdout:(120/196): python3-jinja2-2.11.3-5.el9.noarch.r 4.2 MB/s | 248 kB 00:00 2024-03-24T10:37:52.005 INFO:teuthology.orchestra.run.smithi005.stdout:(121/196): qatlib-23.11.0-1.el9.x86_64.rpm 3.6 MB/s | 216 kB 00:00 2024-03-24T10:37:52.064 INFO:teuthology.orchestra.run.smithi005.stdout:(122/196): qatlib-service-23.11.0-1.el9.x86_64. 598 kB/s | 35 kB 00:00 2024-03-24T10:37:52.164 INFO:teuthology.orchestra.run.smithi005.stdout:(123/196): openssl-devel-3.0.7-27.el9.x86_64.rp 9.5 MB/s | 4.1 MB 00:00 2024-03-24T10:37:52.189 INFO:teuthology.orchestra.run.smithi005.stdout:(124/196): qatzip-libs-1.1.2-2.el9.x86_64.rpm 355 kB/s | 44 kB 00:00 2024-03-24T10:37:52.248 INFO:teuthology.orchestra.run.smithi005.stdout:(125/196): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.1 MB/s | 94 kB 00:00 2024-03-24T10:37:52.340 INFO:teuthology.orchestra.run.smithi005.stdout:(126/196): lua-devel-5.4.4-4.el9.x86_64.rpm 148 kB/s | 22 kB 00:00 2024-03-24T10:37:52.474 INFO:teuthology.orchestra.run.smithi005.stdout:(127/196): protobuf-compiler-3.14.0-13.el9.x86_ 3.7 MB/s | 863 kB 00:00 2024-03-24T10:37:52.774 INFO:teuthology.orchestra.run.smithi005.stdout:(128/196): abseil-cpp-20211102.0-3.el9.x86_64.r 1.2 MB/s | 548 kB 00:00 2024-03-24T10:37:52.841 INFO:teuthology.orchestra.run.smithi005.stdout:(129/196): grpc-data-1.46.7-10.el9.noarch.rpm 293 kB/s | 19 kB 00:00 2024-03-24T10:37:52.866 INFO:teuthology.orchestra.run.smithi005.stdout:(130/196): gperftools-libs-2.9.1-2.el9.x86_64.r 790 kB/s | 309 kB 00:00 2024-03-24T10:37:52.933 INFO:teuthology.orchestra.run.smithi005.stdout:(131/196): libarrow-doc-9.0.0-10.el9.noarch.rpm 390 kB/s | 26 kB 00:00 2024-03-24T10:37:53.000 INFO:teuthology.orchestra.run.smithi005.stdout:(132/196): liboath-2.6.7-2.el9.x86_64.rpm 728 kB/s | 49 kB 00:00 2024-03-24T10:37:53.109 INFO:teuthology.orchestra.run.smithi005.stdout:(133/196): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-03-24T10:37:53.134 INFO:teuthology.orchestra.run.smithi005.stdout:(134/196): libunwind-1.6.2-1.el9.x86_64.rpm 504 kB/s | 67 kB 00:00 2024-03-24T10:37:53.176 INFO:teuthology.orchestra.run.smithi005.stdout:(135/196): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-03-24T10:37:53.259 INFO:teuthology.orchestra.run.smithi005.stdout:(136/196): python3-asyncssh-2.13.2-5.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-03-24T10:37:53.293 INFO:teuthology.orchestra.run.smithi005.stdout:(137/196): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 838 kB 00:00 2024-03-24T10:37:53.327 INFO:teuthology.orchestra.run.smithi005.stdout:(138/196): python3-bcrypt-3.2.2-1.el9.x86_64.rp 649 kB/s | 43 kB 00:00 2024-03-24T10:37:53.361 INFO:teuthology.orchestra.run.smithi005.stdout:(139/196): python3-cachetools-4.2.4-1.el9.noarc 482 kB/s | 32 kB 00:00 2024-03-24T10:37:53.394 INFO:teuthology.orchestra.run.smithi005.stdout:(140/196): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-03-24T10:37:53.436 INFO:teuthology.orchestra.run.smithi005.stdout:(141/196): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-24T10:37:53.461 INFO:teuthology.orchestra.run.smithi005.stdout:(142/196): python3-cherrypy-18.6.1-2.el9.noarch 5.2 MB/s | 358 kB 00:00 2024-03-24T10:37:53.511 INFO:teuthology.orchestra.run.smithi005.stdout:(143/196): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-03-24T10:37:53.545 INFO:teuthology.orchestra.run.smithi005.stdout:(144/196): python3-colorama-0.4.4-12.el9.noarch 420 kB/s | 35 kB 00:00 2024-03-24T10:37:53.587 INFO:teuthology.orchestra.run.smithi005.stdout:(145/196): python3-google-auth-2.28.2-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-03-24T10:37:53.662 INFO:teuthology.orchestra.run.smithi005.stdout:(146/196): python3-grpcio-1.46.7-10.el9.x86_64. 17 MB/s | 2.0 MB 00:00 2024-03-24T10:37:53.688 INFO:teuthology.orchestra.run.smithi005.stdout:(147/196): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-24T10:37:53.721 INFO:teuthology.orchestra.run.smithi005.stdout:(148/196): python3-jaraco-8.2.1-3.el9.noarch.rp 182 kB/s | 11 kB 00:00 2024-03-24T10:37:53.755 INFO:teuthology.orchestra.run.smithi005.stdout:(149/196): python3-jaraco-classes-3.2.1-5.el9.n 264 kB/s | 18 kB 00:00 2024-03-24T10:37:53.780 INFO:teuthology.orchestra.run.smithi005.stdout:(150/196): python3-jaraco-collections-3.0.0-8.e 396 kB/s | 23 kB 00:00 2024-03-24T10:37:53.814 INFO:teuthology.orchestra.run.smithi005.stdout:(151/196): python3-jaraco-functools-3.5.0-2.el9 331 kB/s | 19 kB 00:00 2024-03-24T10:37:53.839 INFO:teuthology.orchestra.run.smithi005.stdout:(152/196): python3-jaraco-text-3.2.0-6.el9.noar 335 kB/s | 20 kB 00:00 2024-03-24T10:37:53.931 INFO:teuthology.orchestra.run.smithi005.stdout:(153/196): python3-logutils-0.3.5-21.el9.noarch 505 kB/s | 46 kB 00:00 2024-03-24T10:37:53.973 INFO:teuthology.orchestra.run.smithi005.stdout:(154/196): python3-kubernetes-26.1.0-2.el9.noar 6.4 MB/s | 1.0 MB 00:00 2024-03-24T10:37:53.998 INFO:teuthology.orchestra.run.smithi005.stdout:(155/196): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-03-24T10:37:54.040 INFO:teuthology.orchestra.run.smithi005.stdout:(156/196): python3-natsort-7.1.1-5.el9.noarch.r 864 kB/s | 58 kB 00:00 2024-03-24T10:37:54.066 INFO:teuthology.orchestra.run.smithi005.stdout:(157/196): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-24T10:37:54.108 INFO:teuthology.orchestra.run.smithi005.stdout:(158/196): python3-portend-3.1.0-2.el9.noarch.r 245 kB/s | 16 kB 00:00 2024-03-24T10:37:54.133 INFO:teuthology.orchestra.run.smithi005.stdout:(159/196): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-03-24T10:37:54.183 INFO:teuthology.orchestra.run.smithi005.stdout:(160/196): python3-pyperclip-1.8.2-2.el9.noarch 375 kB/s | 28 kB 00:00 2024-03-24T10:37:54.254 INFO:teuthology.orchestra.run.smithi005.stdout:(161/196): python3-repoze-lru-0.7-16.el9.noarch 254 kB/s | 31 kB 00:00 2024-03-24T10:37:54.305 INFO:teuthology.orchestra.run.smithi005.stdout:(162/196): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-03-24T10:37:54.414 INFO:teuthology.orchestra.run.smithi005.stdout:(163/196): python3-rsa-4.9-2.el9.noarch.rpm 372 kB/s | 59 kB 00:00 2024-03-24T10:37:54.439 INFO:teuthology.orchestra.run.smithi005.stdout:(164/196): python3-tempora-5.0.0-2.el9.noarch.r 268 kB/s | 36 kB 00:00 2024-03-24T10:37:54.481 INFO:teuthology.orchestra.run.smithi005.stdout:(165/196): python3-typing-extensions-4.4.0-2.el 762 kB/s | 51 kB 00:00 2024-03-24T10:37:54.515 INFO:teuthology.orchestra.run.smithi005.stdout:(166/196): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-24T10:37:54.540 INFO:teuthology.orchestra.run.smithi005.stdout:(167/196): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-03-24T10:37:54.599 INFO:teuthology.orchestra.run.smithi005.stdout:(168/196): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-24T10:37:54.616 INFO:teuthology.orchestra.run.smithi005.stdout:(169/196): python3-zc-lockfile-2.0-10.el9.noarc 266 kB/s | 20 kB 00:00 2024-03-24T10:37:54.683 INFO:teuthology.orchestra.run.smithi005.stdout:(170/196): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-24T10:37:54.733 INFO:teuthology.orchestra.run.smithi005.stdout:(171/196): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-03-24T10:37:55.725 INFO:teuthology.orchestra.run.smithi005.stdout:(172/196): librbd1-19.0.0-1855.g21b3bd67.el9.x8 3.2 MB/s | 3.1 MB 00:00 2024-03-24T10:37:55.792 INFO:teuthology.orchestra.run.smithi005.stdout:(173/196): xfsdump-3.1.12-4.el9.x86_64.rpm 4.7 MB/s | 321 kB 00:00 2024-03-24T10:37:55.984 INFO:teuthology.orchestra.run.smithi005.stdout:(174/196): NetworkManager-1.47.2-1.el9.x86_64.r 12 MB/s | 2.3 MB 00:00 2024-03-24T10:37:56.059 INFO:teuthology.orchestra.run.smithi005.stdout:(175/196): NetworkManager-libnm-1.47.2-1.el9.x8 24 MB/s | 1.8 MB 00:00 2024-03-24T10:37:56.102 INFO:teuthology.orchestra.run.smithi005.stdout:(176/196): NetworkManager-team-1.47.2-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-03-24T10:37:56.143 INFO:teuthology.orchestra.run.smithi005.stdout:(177/196): NetworkManager-tui-1.47.2-1.el9.x86_ 5.8 MB/s | 248 kB 00:00 2024-03-24T10:37:56.185 INFO:teuthology.orchestra.run.smithi005.stdout:(178/196): acl-2.3.1-4.el9.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2024-03-24T10:37:56.227 INFO:teuthology.orchestra.run.smithi005.stdout:(179/196): elfutils-debuginfod-client-0.190-2.e 876 kB/s | 36 kB 00:00 2024-03-24T10:37:56.269 INFO:teuthology.orchestra.run.smithi005.stdout:(180/196): elfutils-libelf-0.190-2.el9.x86_64.r 4.5 MB/s | 192 kB 00:00 2024-03-24T10:37:56.311 INFO:teuthology.orchestra.run.smithi005.stdout:(181/196): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 256 kB 00:00 2024-03-24T10:37:56.361 INFO:teuthology.orchestra.run.smithi005.stdout:(182/196): librados2-19.0.0-1855.g21b3bd67.el9. 2.0 MB/s | 3.4 MB 00:01 2024-03-24T10:37:56.386 INFO:teuthology.orchestra.run.smithi005.stdout:(183/196): libacl-2.3.1-4.el9.x86_64.rpm 306 kB/s | 23 kB 00:00 2024-03-24T10:37:56.428 INFO:teuthology.orchestra.run.smithi005.stdout:(184/196): libfdisk-2.37.4-18.el9.x86_64.rpm 3.7 MB/s | 155 kB 00:00 2024-03-24T10:37:56.453 INFO:teuthology.orchestra.run.smithi005.stdout:(185/196): libblkid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 108 kB 00:00 2024-03-24T10:37:56.479 INFO:teuthology.orchestra.run.smithi005.stdout:(186/196): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-03-24T10:37:56.504 INFO:teuthology.orchestra.run.smithi005.stdout:(187/196): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-24T10:37:56.529 INFO:teuthology.orchestra.run.smithi005.stdout:(188/196): libuuid-2.37.4-18.el9.x86_64.rpm 572 kB/s | 29 kB 00:00 2024-03-24T10:37:56.613 INFO:teuthology.orchestra.run.smithi005.stdout:(189/196): openssl-libs-3.0.7-27.el9.x86_64.rpm 26 MB/s | 2.2 MB 00:00 2024-03-24T10:37:56.655 INFO:teuthology.orchestra.run.smithi005.stdout:(190/196): openssl-3.0.7-27.el9.x86_64.rpm 8.0 MB/s | 1.2 MB 00:00 2024-03-24T10:37:56.714 INFO:teuthology.orchestra.run.smithi005.stdout:(191/196): util-linux-2.37.4-18.el9.x86_64.rpm 23 MB/s | 2.3 MB 00:00 2024-03-24T10:37:56.747 INFO:teuthology.orchestra.run.smithi005.stdout:(192/196): util-linux-core-2.37.4-18.el9.x86_64 4.9 MB/s | 465 kB 00:00 2024-03-24T10:37:56.773 INFO:teuthology.orchestra.run.smithi005.stdout:(193/196): util-linux-user-2.37.4-18.el9.x86_64 533 kB/s | 31 kB 00:00 2024-03-24T10:37:56.806 INFO:teuthology.orchestra.run.smithi005.stdout:(194/196): xfsprogs-6.3.0-1.el9.x86_64.rpm 19 MB/s | 1.1 MB 00:00 2024-03-24T10:37:56.890 INFO:teuthology.orchestra.run.smithi005.stdout:(195/196): libacl-devel-2.3.1-4.el9.x86_64.rpm 637 kB/s | 74 kB 00:00 2024-03-24T10:38:24.971 INFO:teuthology.orchestra.run.smithi005.stdout:(196/196): ceph-test-debuginfo-19.0.0-1855.g21b 10 MB/s | 674 MB 01:06 2024-03-24T10:38:24.975 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:38:24.975 INFO:teuthology.orchestra.run.smithi005.stdout:Total 7.6 MB/s | 2.2 GB 04:51 2024-03-24T10:38:25.424 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-24T10:38:25.730 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-24T10:38:25.730 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-24T10:38:27.254 INFO:teuthology.orchestra.run.smithi057.stdout:(196/196): ceph-test-debuginfo-19.0.0-1855.g21b 7.7 MB/s | 674 MB 01:27 2024-03-24T10:38:27.259 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:38:27.260 INFO:teuthology.orchestra.run.smithi057.stdout:Total 7.6 MB/s | 2.2 GB 04:52 2024-03-24T10:38:27.712 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2024-03-24T10:38:28.018 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2024-03-24T10:38:28.018 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2024-03-24T10:38:40.097 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-24T10:38:40.097 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-24T10:38:42.351 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2024-03-24T10:38:42.352 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2024-03-24T10:39:00.025 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-24T10:39:02.372 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2024-03-24T10:39:02.685 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/220 2024-03-24T10:39:02.729 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_6 2/220 2024-03-24T10:39:02.768 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 3/220 2024-03-24T10:39:02.827 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 4/220 2024-03-24T10:39:02.836 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 4/220 2024-03-24T10:39:02.879 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/220 2024-03-24T10:39:02.943 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/220 2024-03-24T10:39:02.983 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/220 2024-03-24T10:39:03.008 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/220 2024-03-24T10:39:03.071 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/220 2024-03-24T10:39:03.413 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/220 2024-03-24T10:39:03.467 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/220 2024-03-24T10:39:03.574 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/220 2024-03-24T10:39:03.601 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/220 2024-03-24T10:39:03.685 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/220 2024-03-24T10:39:03.723 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/220 2024-03-24T10:39:03.773 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/220 2024-03-24T10:39:03.964 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 15/220 2024-03-24T10:39:04.012 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : postgresql-13.11-1.el9.x86_64 16/220 2024-03-24T10:39:04.039 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/220 2024-03-24T10:39:04.128 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/220 2024-03-24T10:39:04.157 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 18/220 2024-03-24T10:39:04.223 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 18/220 2024-03-24T10:39:04.270 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 19/220 2024-03-24T10:39:04.292 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 20/220 2024-03-24T10:39:04.308 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 20/220 2024-03-24T10:39:04.442 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 21/220 2024-03-24T10:39:04.474 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 21/220 2024-03-24T10:39:04.505 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 22/220 2024-03-24T10:39:04.676 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 23/220 2024-03-24T10:39:04.807 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 24/220 2024-03-24T10:39:04.868 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 24/220 2024-03-24T10:39:04.967 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 25/220 2024-03-24T10:39:05.027 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 25/220 2024-03-24T10:39:05.045 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/220 2024-03-24T10:39:05.080 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_6 2/220 2024-03-24T10:39:05.115 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 3/220 2024-03-24T10:39:05.153 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 4/220 2024-03-24T10:39:05.161 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 4/220 2024-03-24T10:39:05.204 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/220 2024-03-24T10:39:05.254 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/220 2024-03-24T10:39:05.261 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 25/220 2024-03-24T10:39:05.295 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/220 2024-03-24T10:39:05.320 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/220 2024-03-24T10:39:05.385 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : re2-1:20211101-3.el9.x86_64 26/220 2024-03-24T10:39:05.397 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/220 2024-03-24T10:39:05.489 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 27/220 2024-03-24T10:39:05.532 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 28/220 2024-03-24T10:39:05.559 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : liboath-2.6.7-2.el9.x86_64 29/220 2024-03-24T10:39:05.594 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 30/220 2024-03-24T10:39:05.671 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 31/220 2024-03-24T10:39:05.723 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 32/220 2024-03-24T10:39:05.740 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/220 2024-03-24T10:39:05.761 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 33/220 2024-03-24T10:39:05.797 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/220 2024-03-24T10:39:05.837 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 34/220 2024-03-24T10:39:05.877 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 35/220 2024-03-24T10:39:05.920 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/220 2024-03-24T10:39:05.928 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 36/220 2024-03-24T10:39:05.956 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/220 2024-03-24T10:39:05.962 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 37/220 2024-03-24T10:39:05.989 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : lua-5.4.4-4.el9.x86_64 38/220 2024-03-24T10:39:06.048 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/220 2024-03-24T10:39:06.060 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 39/220 2024-03-24T10:39:06.084 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/220 2024-03-24T10:39:06.127 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/220 2024-03-24T10:39:06.312 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 15/220 2024-03-24T10:39:06.359 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : postgresql-13.11-1.el9.x86_64 16/220 2024-03-24T10:39:06.386 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/220 2024-03-24T10:39:06.393 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 40/220 2024-03-24T10:39:06.491 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/220 2024-03-24T10:39:06.496 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 41/220 2024-03-24T10:39:06.521 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 18/220 2024-03-24T10:39:06.525 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 42/220 2024-03-24T10:39:06.586 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 18/220 2024-03-24T10:39:06.599 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 43/220 2024-03-24T10:39:06.625 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 19/220 2024-03-24T10:39:06.640 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : luarocks-3.9.2-1.el9.noarch 44/220 2024-03-24T10:39:06.648 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 20/220 2024-03-24T10:39:06.662 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 20/220 2024-03-24T10:39:06.708 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 45/220 2024-03-24T10:39:06.856 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 46/220 2024-03-24T10:39:06.858 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 21/220 2024-03-24T10:39:06.890 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 21/220 2024-03-24T10:39:06.904 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 47/220 2024-03-24T10:39:06.929 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 22/220 2024-03-24T10:39:07.094 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 23/220 2024-03-24T10:39:07.239 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 24/220 2024-03-24T10:39:07.283 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 24/220 2024-03-24T10:39:07.394 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 25/220 2024-03-24T10:39:07.442 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 25/220 2024-03-24T10:39:07.458 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rsa-4.9-2.el9.noarch 48/220 2024-03-24T10:39:07.542 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 49/220 2024-03-24T10:39:07.638 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 50/220 2024-03-24T10:39:07.658 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 25/220 2024-03-24T10:39:07.692 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 51/220 2024-03-24T10:39:07.700 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 51/220 2024-03-24T10:39:07.740 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 52/220 2024-03-24T10:39:07.769 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : re2-1:20211101-3.el9.x86_64 26/220 2024-03-24T10:39:07.782 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 53/220 2024-03-24T10:39:07.820 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 54/220 2024-03-24T10:39:07.850 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 55/220 2024-03-24T10:39:07.866 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 27/220 2024-03-24T10:39:07.898 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 28/220 2024-03-24T10:39:07.916 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : liboath-2.6.7-2.el9.x86_64 29/220 2024-03-24T10:39:07.960 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 30/220 2024-03-24T10:39:08.037 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 31/220 2024-03-24T10:39:08.105 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 32/220 2024-03-24T10:39:08.128 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/220 2024-03-24T10:39:08.143 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 33/220 2024-03-24T10:39:08.218 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 34/220 2024-03-24T10:39:08.258 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 35/220 2024-03-24T10:39:08.310 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 36/220 2024-03-24T10:39:08.343 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 37/220 2024-03-24T10:39:08.348 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/220 2024-03-24T10:39:08.370 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : lua-5.4.4-4.el9.x86_64 38/220 2024-03-24T10:39:08.442 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 39/220 2024-03-24T10:39:08.444 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/220 2024-03-24T10:39:08.476 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 40/220 2024-03-24T10:39:08.488 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/220 2024-03-24T10:39:08.519 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 41/220 2024-03-24T10:39:08.547 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 42/220 2024-03-24T10:39:08.621 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 43/220 2024-03-24T10:39:08.638 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/220 2024-03-24T10:39:08.670 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : luarocks-3.9.2-1.el9.noarch 44/220 2024-03-24T10:39:08.683 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/220 2024-03-24T10:39:08.739 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/220 2024-03-24T10:39:08.747 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 45/220 2024-03-24T10:39:08.814 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/220 2024-03-24T10:39:08.850 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/220 2024-03-24T10:39:08.893 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 46/220 2024-03-24T10:39:08.895 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/220 2024-03-24T10:39:08.944 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/220 2024-03-24T10:39:08.949 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 47/220 2024-03-24T10:39:08.978 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/220 2024-03-24T10:39:09.021 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/220 2024-03-24T10:39:09.064 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/220 2024-03-24T10:39:09.206 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/220 2024-03-24T10:39:09.247 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/220 2024-03-24T10:39:09.274 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/220 2024-03-24T10:39:09.316 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/220 2024-03-24T10:39:09.415 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/220 2024-03-24T10:39:09.449 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 75/220 2024-03-24T10:39:09.498 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/220 2024-03-24T10:39:09.523 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/220 2024-03-24T10:39:09.546 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rsa-4.9-2.el9.noarch 48/220 2024-03-24T10:39:09.547 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/220 2024-03-24T10:39:09.629 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 49/220 2024-03-24T10:39:09.635 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/220 2024-03-24T10:39:09.709 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 50/220 2024-03-24T10:39:09.763 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/220 2024-03-24T10:39:09.770 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 51/220 2024-03-24T10:39:09.779 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 51/220 2024-03-24T10:39:09.812 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 52/220 2024-03-24T10:39:09.852 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 53/220 2024-03-24T10:39:09.870 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/220 2024-03-24T10:39:09.889 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 54/220 2024-03-24T10:39:09.907 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/220 2024-03-24T10:39:09.919 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 55/220 2024-03-24T10:39:09.965 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/220 2024-03-24T10:39:10.020 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/220 2024-03-24T10:39:10.182 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/220 2024-03-24T10:39:10.213 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/220 2024-03-24T10:39:10.261 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/220 2024-03-24T10:39:10.405 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/220 2024-03-24T10:39:10.457 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 88/220 2024-03-24T10:39:10.494 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/220 2024-03-24T10:39:10.517 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/220 2024-03-24T10:39:10.526 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : fuse-2.9.9-15.el9.x86_64 90/220 2024-03-24T10:39:10.545 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 91/220 2024-03-24T10:39:10.571 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 92/220 2024-03-24T10:39:10.611 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : uuid-1.6.2-55.el9.x86_64 93/220 2024-03-24T10:39:10.651 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 94/220 2024-03-24T10:39:10.681 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 95/220 2024-03-24T10:39:10.722 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/220 2024-03-24T10:39:10.761 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 96/220 2024-03-24T10:39:10.766 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/220 2024-03-24T10:39:10.791 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 97/220 2024-03-24T10:39:10.824 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 98/220 2024-03-24T10:39:10.883 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 99/220 2024-03-24T10:39:10.906 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/220 2024-03-24T10:39:10.917 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 100/220 2024-03-24T10:39:10.992 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 61/220 2024-03-24T10:39:11.012 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 101/220 2024-03-24T10:39:11.049 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/220 2024-03-24T10:39:11.124 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/220 2024-03-24T10:39:11.159 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/220 2024-03-24T10:39:11.175 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 102/220 2024-03-24T10:39:11.196 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/220 2024-03-24T10:39:11.224 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 103/220 2024-03-24T10:39:11.244 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/220 2024-03-24T10:39:11.255 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 104/220 2024-03-24T10:39:11.279 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/220 2024-03-24T10:39:11.303 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 105/220 2024-03-24T10:39:11.323 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/220 2024-03-24T10:39:11.364 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/220 2024-03-24T10:39:11.413 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-packaging-20.9-5.el9.noarch 106/220 2024-03-24T10:39:11.466 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 107/220 2024-03-24T10:39:11.507 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/220 2024-03-24T10:39:11.553 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/220 2024-03-24T10:39:11.583 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/220 2024-03-24T10:39:11.625 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/220 2024-03-24T10:39:11.716 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/220 2024-03-24T10:39:11.749 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 75/220 2024-03-24T10:39:11.791 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/220 2024-03-24T10:39:11.823 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/220 2024-03-24T10:39:11.847 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/220 2024-03-24T10:39:11.944 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/220 2024-03-24T10:39:12.071 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/220 2024-03-24T10:39:12.189 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/220 2024-03-24T10:39:12.237 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 108/220 2024-03-24T10:39:12.256 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/220 2024-03-24T10:39:12.287 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 109/220 2024-03-24T10:39:12.314 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/220 2024-03-24T10:39:12.321 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 110/220 2024-03-24T10:39:12.370 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 111/220 2024-03-24T10:39:12.394 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/220 2024-03-24T10:39:12.461 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 112/220 2024-03-24T10:39:12.555 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 113/220 2024-03-24T10:39:12.598 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/220 2024-03-24T10:39:12.667 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/220 2024-03-24T10:39:12.739 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : openblas-0.3.21-2.el9.x86_64 114/220 2024-03-24T10:39:12.769 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 115/220 2024-03-24T10:39:12.815 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/220 2024-03-24T10:39:12.849 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 116/220 2024-03-24T10:39:12.858 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 88/220 2024-03-24T10:39:12.889 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/220 2024-03-24T10:39:12.912 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : fuse-2.9.9-15.el9.x86_64 90/220 2024-03-24T10:39:12.940 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 91/220 2024-03-24T10:39:12.974 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 92/220 2024-03-24T10:39:13.012 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : uuid-1.6.2-55.el9.x86_64 93/220 2024-03-24T10:39:13.051 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 94/220 2024-03-24T10:39:13.081 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 95/220 2024-03-24T10:39:13.162 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 96/220 2024-03-24T10:39:13.200 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 97/220 2024-03-24T10:39:13.233 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 98/220 2024-03-24T10:39:13.275 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 99/220 2024-03-24T10:39:13.308 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 100/220 2024-03-24T10:39:13.412 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 101/220 2024-03-24T10:39:13.500 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 102/220 2024-03-24T10:39:13.540 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 117/220 2024-03-24T10:39:13.549 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 103/220 2024-03-24T10:39:13.580 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 104/220 2024-03-24T10:39:13.627 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 105/220 2024-03-24T10:39:13.655 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 118/220 2024-03-24T10:39:13.738 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-packaging-20.9-5.el9.noarch 106/220 2024-03-24T10:39:13.799 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 107/220 2024-03-24T10:39:14.584 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 108/220 2024-03-24T10:39:14.627 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 109/220 2024-03-24T10:39:14.661 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 110/220 2024-03-24T10:39:14.711 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 111/220 2024-03-24T10:39:14.803 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 112/220 2024-03-24T10:39:14.837 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 113/220 2024-03-24T10:39:14.945 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 119/220 2024-03-24T10:39:15.021 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 120/220 2024-03-24T10:39:15.022 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openblas-0.3.21-2.el9.x86_64 114/220 2024-03-24T10:39:15.042 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 115/220 2024-03-24T10:39:15.051 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 121/220 2024-03-24T10:39:15.083 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : jq-1.6-16.el9.x86_64 122/220 2024-03-24T10:39:15.111 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/220 2024-03-24T10:39:15.123 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 116/220 2024-03-24T10:39:15.433 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/220 2024-03-24T10:39:15.474 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/220 2024-03-24T10:39:15.565 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/220 2024-03-24T10:39:15.595 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 126/220 2024-03-24T10:39:15.632 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/220 2024-03-24T10:39:15.816 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 117/220 2024-03-24T10:39:15.936 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 118/220 2024-03-24T10:39:16.117 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/220 2024-03-24T10:39:16.143 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/220 2024-03-24T10:39:16.215 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/220 2024-03-24T10:39:16.237 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/220 2024-03-24T10:39:17.253 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 119/220 2024-03-24T10:39:17.350 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 120/220 2024-03-24T10:39:17.382 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 121/220 2024-03-24T10:39:17.412 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : jq-1.6-16.el9.x86_64 122/220 2024-03-24T10:39:17.441 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/220 2024-03-24T10:39:17.767 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/220 2024-03-24T10:39:17.795 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/220 2024-03-24T10:39:17.885 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/220 2024-03-24T10:39:17.916 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 126/220 2024-03-24T10:39:17.962 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/220 2024-03-24T10:39:18.438 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/220 2024-03-24T10:39:18.464 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/220 2024-03-24T10:39:18.544 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/220 2024-03-24T10:39:18.574 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/220 2024-03-24T10:39:18.590 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/220 2024-03-24T10:39:18.625 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/220 2024-03-24T10:39:19.192 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/220 2024-03-24T10:39:19.228 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/220 2024-03-24T10:39:19.330 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/220 2024-03-24T10:39:19.330 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T10:39:19.330 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T10:39:19.330 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:39:19.372 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 133/220 2024-03-24T10:39:20.895 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/220 2024-03-24T10:39:20.928 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/220 2024-03-24T10:39:21.502 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/220 2024-03-24T10:39:21.540 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/220 2024-03-24T10:39:21.618 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/220 2024-03-24T10:39:21.618 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T10:39:21.618 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T10:39:21.619 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:39:21.657 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 133/220 2024-03-24T10:39:32.152 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 133/220 2024-03-24T10:39:32.152 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /sys 2024-03-24T10:39:32.152 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /proc 2024-03-24T10:39:32.152 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /mnt 2024-03-24T10:39:32.152 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /var/tmp 2024-03-24T10:39:32.152 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /home 2024-03-24T10:39:32.152 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /root 2024-03-24T10:39:32.152 INFO:teuthology.orchestra.run.smithi005.stdout:skipping the directory /tmp 2024-03-24T10:39:32.152 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:39:32.319 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 134/220 2024-03-24T10:39:32.333 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 134/220 2024-03-24T10:39:32.333 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:39:32.333 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T10:39:32.333 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:39:33.882 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/220 2024-03-24T10:39:33.907 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/220 2024-03-24T10:39:33.907 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:39:33.907 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T10:39:33.907 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T10:39:33.907 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T10:39:33.907 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:39:34.141 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/220 2024-03-24T10:39:34.162 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/220 2024-03-24T10:39:34.163 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:39:34.163 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T10:39:34.163 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T10:39:34.163 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T10:39:34.163 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:39:34.612 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 133/220 2024-03-24T10:39:34.612 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /sys 2024-03-24T10:39:34.612 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /proc 2024-03-24T10:39:34.612 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /mnt 2024-03-24T10:39:34.613 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /var/tmp 2024-03-24T10:39:34.613 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /home 2024-03-24T10:39:34.613 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /root 2024-03-24T10:39:34.613 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /tmp 2024-03-24T10:39:34.613 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:39:34.625 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 137/220 2024-03-24T10:39:34.647 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 137/220 2024-03-24T10:39:34.647 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:39:34.647 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T10:39:34.647 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T10:39:34.647 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T10:39:34.647 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:39:34.743 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : automake-1.16.2-8.el9.noarch 138/220 2024-03-24T10:39:34.777 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 134/220 2024-03-24T10:39:34.792 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 134/220 2024-03-24T10:39:34.792 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:39:34.792 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T10:39:34.792 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:39:35.103 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 139/220 2024-03-24T10:39:35.134 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 139/220 2024-03-24T10:39:35.157 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 140/220 2024-03-24T10:39:35.211 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 141/220 2024-03-24T10:39:35.240 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 142/220 2024-03-24T10:39:35.483 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 142/220 2024-03-24T10:39:35.516 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 143/220 2024-03-24T10:39:36.330 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/220 2024-03-24T10:39:36.355 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/220 2024-03-24T10:39:36.355 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:39:36.355 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T10:39:36.355 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T10:39:36.356 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T10:39:36.356 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:39:36.533 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 143/220 2024-03-24T10:39:36.558 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 144/220 2024-03-24T10:39:36.589 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/220 2024-03-24T10:39:36.611 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/220 2024-03-24T10:39:36.611 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:39:36.611 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T10:39:36.611 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T10:39:36.611 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T10:39:36.611 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:39:36.663 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 144/220 2024-03-24T10:39:36.828 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 145/220 2024-03-24T10:39:36.852 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 146/220 2024-03-24T10:39:36.875 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 146/220 2024-03-24T10:39:36.875 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:39:36.875 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T10:39:36.875 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T10:39:36.875 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T10:39:36.875 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:39:36.919 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 147/220 2024-03-24T10:39:36.930 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 147/220 2024-03-24T10:39:36.985 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 148/220 2024-03-24T10:39:37.073 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 137/220 2024-03-24T10:39:37.095 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 137/220 2024-03-24T10:39:37.095 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:39:37.095 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T10:39:37.095 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T10:39:37.095 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T10:39:37.095 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:39:37.191 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : automake-1.16.2-8.el9.noarch 138/220 2024-03-24T10:39:37.626 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 139/220 2024-03-24T10:39:37.656 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 139/220 2024-03-24T10:39:37.688 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 140/220 2024-03-24T10:39:37.751 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 141/220 2024-03-24T10:39:37.780 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 142/220 2024-03-24T10:39:37.888 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/220 2024-03-24T10:39:37.922 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 150/220 2024-03-24T10:39:37.944 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 150/220 2024-03-24T10:39:37.944 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:39:37.944 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T10:39:37.944 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T10:39:37.944 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T10:39:37.944 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:39:37.989 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 151/220 2024-03-24T10:39:38.010 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 151/220 2024-03-24T10:39:38.010 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:39:38.010 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T10:39:38.010 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:39:38.024 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 142/220 2024-03-24T10:39:38.064 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 143/220 2024-03-24T10:39:38.065 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/220 2024-03-24T10:39:38.087 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/220 2024-03-24T10:39:38.087 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:39:38.087 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T10:39:38.087 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T10:39:38.087 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T10:39:38.087 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:39:38.407 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/220 2024-03-24T10:39:38.429 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/220 2024-03-24T10:39:38.429 INFO:teuthology.orchestra.run.smithi005.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:39:38.429 INFO:teuthology.orchestra.run.smithi005.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T10:39:38.429 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T10:39:38.429 INFO:teuthology.orchestra.run.smithi005.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T10:39:38.429 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:39:39.082 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 143/220 2024-03-24T10:39:39.114 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 144/220 2024-03-24T10:39:39.220 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 144/220 2024-03-24T10:39:39.384 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 145/220 2024-03-24T10:39:39.416 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 146/220 2024-03-24T10:39:39.438 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 146/220 2024-03-24T10:39:39.438 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:39:39.438 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T10:39:39.438 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T10:39:39.438 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T10:39:39.438 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:39:39.483 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 147/220 2024-03-24T10:39:39.494 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 147/220 2024-03-24T10:39:39.549 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 148/220 2024-03-24T10:39:40.449 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/220 2024-03-24T10:39:40.477 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 150/220 2024-03-24T10:39:40.499 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 150/220 2024-03-24T10:39:40.499 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:39:40.499 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T10:39:40.499 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T10:39:40.499 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T10:39:40.499 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:39:40.544 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 151/220 2024-03-24T10:39:40.565 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 151/220 2024-03-24T10:39:40.565 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:39:40.565 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T10:39:40.566 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:39:40.620 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/220 2024-03-24T10:39:40.642 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/220 2024-03-24T10:39:40.642 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:39:40.642 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T10:39:40.642 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T10:39:40.642 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T10:39:40.642 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:39:40.954 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/220 2024-03-24T10:39:40.975 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/220 2024-03-24T10:39:40.976 INFO:teuthology.orchestra.run.smithi057.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T10:39:40.976 INFO:teuthology.orchestra.run.smithi057.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T10:39:40.976 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T10:39:40.976 INFO:teuthology.orchestra.run.smithi057.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T10:39:40.976 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:39:43.129 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 154/220 2024-03-24T10:39:43.171 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 155/220 2024-03-24T10:39:43.209 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 156/220 2024-03-24T10:39:43.236 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/220 2024-03-24T10:39:43.431 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 158/220 2024-03-24T10:39:43.582 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/220 2024-03-24T10:39:43.627 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 160/220 2024-03-24T10:39:43.684 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/220 2024-03-24T10:39:43.743 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/220 2024-03-24T10:39:43.778 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/220 2024-03-24T10:39:43.811 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 164/220 2024-03-24T10:39:43.843 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/220 2024-03-24T10:39:43.873 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 166/220 2024-03-24T10:39:43.949 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 167/220 2024-03-24T10:39:44.018 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 168/220 2024-03-24T10:39:44.063 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/220 2024-03-24T10:39:44.707 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-24T10:39:44.753 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-24T10:39:44.772 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-24T10:39:45.658 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 154/220 2024-03-24T10:39:45.700 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 155/220 2024-03-24T10:39:45.730 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 156/220 2024-03-24T10:39:45.757 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/220 2024-03-24T10:39:45.967 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 158/220 2024-03-24T10:39:46.085 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/220 2024-03-24T10:39:46.130 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 160/220 2024-03-24T10:39:46.175 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/220 2024-03-24T10:39:46.230 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/220 2024-03-24T10:39:46.273 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/220 2024-03-24T10:39:46.305 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 164/220 2024-03-24T10:39:46.346 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/220 2024-03-24T10:39:46.376 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 166/220 2024-03-24T10:39:46.441 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 167/220 2024-03-24T10:39:46.500 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 168/220 2024-03-24T10:39:46.532 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/220 2024-03-24T10:39:47.173 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-24T10:39:47.215 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-24T10:39:47.234 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 170/220 2024-03-24T10:39:52.279 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 171/220 2024-03-24T10:39:54.715 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 171/220 2024-03-24T10:40:36.930 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 172/220 2024-03-24T10:40:37.645 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1855.g21b3bd67.el 173/220 2024-03-24T10:40:37.863 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 174/220 2024-03-24T10:40:38.774 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 172/220 2024-03-24T10:40:39.428 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1855.g21b3bd67.el 173/220 2024-03-24T10:40:39.603 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 174/220 2024-03-24T10:40:42.974 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1 175/220 2024-03-24T10:40:44.711 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1 175/220 2024-03-24T10:40:45.455 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 176/220 2024-03-24T10:40:46.865 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 176/220 2024-03-24T10:40:55.671 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 177/220 2024-03-24T10:40:57.327 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 177/220 2024-03-24T10:41:25.042 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 178/220 2024-03-24T10:41:26.633 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 178/220 2024-03-24T10:41:43.039 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 179/220 2024-03-24T10:41:44.623 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 179/220 2024-03-24T10:42:54.305 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1855.g21b3bd67.e 180/220 2024-03-24T10:42:54.896 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 181/220 2024-03-24T10:42:54.934 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1855.g21b3bd67.el9 182/220 2024-03-24T10:42:55.732 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1855.g21b3bd67.e 180/220 2024-03-24T10:42:56.324 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 181/220 2024-03-24T10:42:56.363 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1855.g21b3bd67.el9 182/220 2024-03-24T10:42:59.939 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1855.g21b3bd67 183/220 2024-03-24T10:43:01.361 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1855.g21b3bd67 183/220 2024-03-24T10:43:05.996 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librados2-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 184/220 2024-03-24T10:43:06.141 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : librbd1-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x8 185/220 2024-03-24T10:43:06.248 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1855.g21b3bd67 186/220 2024-03-24T10:43:06.395 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1855.g21b3bd67. 187/220 2024-03-24T10:43:06.456 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1855.g21b3bd67.el 188/220 2024-03-24T10:43:06.537 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1855.g21b3bd67.el 189/220 2024-03-24T10:43:07.299 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librados2-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 184/220 2024-03-24T10:43:07.387 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librbd1-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x8 185/220 2024-03-24T10:43:07.486 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1855.g21b3bd67 186/220 2024-03-24T10:43:07.624 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1855.g21b3bd67. 187/220 2024-03-24T10:43:07.685 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1855.g21b3bd67.el 188/220 2024-03-24T10:43:07.774 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1855.g21b3bd67.el 189/220 2024-03-24T10:43:12.401 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 190/220 2024-03-24T10:43:12.800 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1855.g21b3bd67.el9 191/220 2024-03-24T10:43:13.632 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 190/220 2024-03-24T10:43:13.794 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1855.g21b3bd67.el9 191/220 2024-03-24T10:43:13.941 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x8 192/220 2024-03-24T10:43:14.029 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 193/220 2024-03-24T10:43:14.075 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 194/220 2024-03-24T10:43:14.221 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : flex-2.6.4-9.el9.x86_64 195/220 2024-03-24T10:43:14.258 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : bison-3.7.4-5.el9.x86_64 196/220 2024-03-24T10:43:14.283 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 197/220 2024-03-24T10:43:14.308 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 198/220 2024-03-24T10:43:14.335 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 199/220 2024-03-24T10:43:14.335 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 200/220 2024-03-24T10:43:14.423 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 200/220 2024-03-24T10:43:14.461 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 201/220 2024-03-24T10:43:14.461 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 202/220 2024-03-24T10:43:14.537 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 202/220 2024-03-24T10:43:14.570 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 203/220 2024-03-24T10:43:14.605 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 204/220 2024-03-24T10:43:14.631 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 205/220 2024-03-24T10:43:14.656 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 206/220 2024-03-24T10:43:14.681 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x8 192/220 2024-03-24T10:43:14.690 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 207/220 2024-03-24T10:43:14.737 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 208/220 2024-03-24T10:43:14.749 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 193/220 2024-03-24T10:43:14.769 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 209/220 2024-03-24T10:43:14.794 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 210/220 2024-03-24T10:43:14.796 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 194/220 2024-03-24T10:43:14.820 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 211/220 2024-03-24T10:43:14.845 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 212/220 2024-03-24T10:43:14.870 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 213/220 2024-03-24T10:43:14.876 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-24T10:43:14.877 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-24T10:43:14.901 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : flex-2.6.4-9.el9.x86_64 195/220 2024-03-24T10:43:14.936 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : bison-3.7.4-5.el9.x86_64 196/220 2024-03-24T10:43:14.961 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 197/220 2024-03-24T10:43:14.987 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 198/220 2024-03-24T10:43:15.013 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 199/220 2024-03-24T10:43:15.013 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 200/220 2024-03-24T10:43:15.065 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-24T10:43:15.065 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 215/220 2024-03-24T10:43:15.118 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 200/220 2024-03-24T10:43:15.148 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 215/220 2024-03-24T10:43:15.155 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 201/220 2024-03-24T10:43:15.155 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 202/220 2024-03-24T10:43:15.173 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 216/220 2024-03-24T10:43:15.205 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 217/220 2024-03-24T10:43:15.232 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 218/220 2024-03-24T10:43:15.240 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 202/220 2024-03-24T10:43:15.257 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 219/220 2024-03-24T10:43:15.283 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 203/220 2024-03-24T10:43:15.316 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 204/220 2024-03-24T10:43:15.326 INFO:teuthology.orchestra.run.smithi005.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 220/220 2024-03-24T10:43:15.342 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 205/220 2024-03-24T10:43:15.367 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 206/220 2024-03-24T10:43:15.401 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 207/220 2024-03-24T10:43:15.448 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 208/220 2024-03-24T10:43:15.488 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 209/220 2024-03-24T10:43:15.513 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 210/220 2024-03-24T10:43:15.538 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 211/220 2024-03-24T10:43:15.564 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 212/220 2024-03-24T10:43:15.589 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 213/220 2024-03-24T10:43:15.596 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-24T10:43:15.596 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-24T10:43:15.775 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 214/220 2024-03-24T10:43:15.775 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 215/220 2024-03-24T10:43:15.858 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 215/220 2024-03-24T10:43:15.883 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 216/220 2024-03-24T10:43:15.915 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 217/220 2024-03-24T10:43:15.942 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 218/220 2024-03-24T10:43:15.967 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 219/220 2024-03-24T10:43:16.036 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 220/220 2024-03-24T10:43:24.704 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 220/220 2024-03-24T10:43:24.704 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 1/220 2024-03-24T10:43:24.704 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2/220 2024-03-24T10:43:24.704 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 3/220 2024-03-24T10:43:24.704 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 4/220 2024-03-24T10:43:24.704 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1855.g21b3bd67.el 5/220 2024-03-24T10:43:24.705 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_6 6/220 2024-03-24T10:43:24.705 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 7/220 2024-03-24T10:43:24.705 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 8/220 2024-03-24T10:43:24.705 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 9/220 2024-03-24T10:43:24.705 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1 10/220 2024-03-24T10:43:24.705 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/220 2024-03-24T10:43:24.705 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 12/220 2024-03-24T10:43:24.705 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/220 2024-03-24T10:43:24.705 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 14/220 2024-03-24T10:43:24.705 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 15/220 2024-03-24T10:43:24.705 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 16/220 2024-03-24T10:43:24.705 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/220 2024-03-24T10:43:24.705 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 18/220 2024-03-24T10:43:24.706 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 19/220 2024-03-24T10:43:24.706 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1855.g21b3bd67.e 20/220 2024-03-24T10:43:24.706 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 21/220 2024-03-24T10:43:24.706 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 22/220 2024-03-24T10:43:24.706 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 23/220 2024-03-24T10:43:24.706 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 24/220 2024-03-24T10:43:24.706 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 25/220 2024-03-24T10:43:24.706 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 26/220 2024-03-24T10:43:24.706 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 27/220 2024-03-24T10:43:24.706 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1855.g21b3bd67.el9 28/220 2024-03-24T10:43:24.706 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 29/220 2024-03-24T10:43:24.725 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 30/220 2024-03-24T10:43:24.725 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1855.g21b3bd67 31/220 2024-03-24T10:43:24.725 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados2-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 32/220 2024-03-24T10:43:24.725 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 33/220 2024-03-24T10:43:24.725 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x8 34/220 2024-03-24T10:43:24.725 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 35/220 2024-03-24T10:43:24.725 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 36/220 2024-03-24T10:43:24.725 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 37/220 2024-03-24T10:43:24.725 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 38/220 2024-03-24T10:43:24.725 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1855.g21b3bd67 39/220 2024-03-24T10:43:24.725 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 40/220 2024-03-24T10:43:24.726 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1855.g21b3bd67. 41/220 2024-03-24T10:43:24.726 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 42/220 2024-03-24T10:43:24.726 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1855.g21b3bd67.el 43/220 2024-03-24T10:43:24.726 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 44/220 2024-03-24T10:43:24.726 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1855.g21b3bd67.el 45/220 2024-03-24T10:43:24.726 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 46/220 2024-03-24T10:43:24.726 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 47/220 2024-03-24T10:43:24.726 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 48/220 2024-03-24T10:43:24.726 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1855.g21b3bd67.el9 49/220 2024-03-24T10:43:24.726 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 50/220 2024-03-24T10:43:24.726 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x8 51/220 2024-03-24T10:43:24.727 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 52/220 2024-03-24T10:43:24.727 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 53/220 2024-03-24T10:43:24.727 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 54/220 2024-03-24T10:43:24.727 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 55/220 2024-03-24T10:43:24.727 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 56/220 2024-03-24T10:43:24.727 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 57/220 2024-03-24T10:43:24.727 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 58/220 2024-03-24T10:43:24.727 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 59/220 2024-03-24T10:43:24.727 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 60/220 2024-03-24T10:43:24.727 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 61/220 2024-03-24T10:43:24.727 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : automake-1.16.2-8.el9.noarch 62/220 2024-03-24T10:43:24.727 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : bison-3.7.4-5.el9.x86_64 63/220 2024-03-24T10:43:24.727 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 64/220 2024-03-24T10:43:24.728 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : flex-2.6.4-9.el9.x86_64 65/220 2024-03-24T10:43:24.728 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 66/220 2024-03-24T10:43:24.728 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 67/220 2024-03-24T10:43:24.728 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 68/220 2024-03-24T10:43:24.728 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 69/220 2024-03-24T10:43:24.728 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 70/220 2024-03-24T10:43:24.728 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 71/220 2024-03-24T10:43:24.728 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 72/220 2024-03-24T10:43:24.728 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 73/220 2024-03-24T10:43:24.728 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 74/220 2024-03-24T10:43:24.728 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 75/220 2024-03-24T10:43:24.728 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : lua-5.4.4-4.el9.x86_64 76/220 2024-03-24T10:43:24.729 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 77/220 2024-03-24T10:43:24.729 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 78/220 2024-03-24T10:43:24.729 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 79/220 2024-03-24T10:43:24.729 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : postgresql-13.11-1.el9.x86_64 80/220 2024-03-24T10:43:24.729 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 81/220 2024-03-24T10:43:24.729 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 82/220 2024-03-24T10:43:24.729 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 83/220 2024-03-24T10:43:24.729 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 84/220 2024-03-24T10:43:24.729 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 85/220 2024-03-24T10:43:24.729 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 86/220 2024-03-24T10:43:24.729 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 87/220 2024-03-24T10:43:24.729 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 88/220 2024-03-24T10:43:24.729 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 89/220 2024-03-24T10:43:24.729 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 90/220 2024-03-24T10:43:24.730 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 91/220 2024-03-24T10:43:24.730 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/220 2024-03-24T10:43:24.730 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 93/220 2024-03-24T10:43:24.730 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 94/220 2024-03-24T10:43:24.730 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 95/220 2024-03-24T10:43:24.730 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 96/220 2024-03-24T10:43:24.730 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 97/220 2024-03-24T10:43:24.730 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 98/220 2024-03-24T10:43:24.730 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 99/220 2024-03-24T10:43:24.730 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 100/220 2024-03-24T10:43:24.730 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 101/220 2024-03-24T10:43:24.730 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 102/220 2024-03-24T10:43:24.730 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 103/220 2024-03-24T10:43:24.730 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 104/220 2024-03-24T10:43:24.731 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 105/220 2024-03-24T10:43:24.731 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 106/220 2024-03-24T10:43:24.731 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 107/220 2024-03-24T10:43:24.731 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 108/220 2024-03-24T10:43:24.731 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 109/220 2024-03-24T10:43:24.731 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 110/220 2024-03-24T10:43:24.731 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-ply-3.11-14.el9.noarch 111/220 2024-03-24T10:43:24.731 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 112/220 2024-03-24T10:43:24.731 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 113/220 2024-03-24T10:43:24.731 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : jq-1.6-16.el9.x86_64 114/220 2024-03-24T10:43:24.731 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 115/220 2024-03-24T10:43:24.731 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 116/220 2024-03-24T10:43:24.731 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 117/220 2024-03-24T10:43:24.732 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 118/220 2024-03-24T10:43:24.732 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 119/220 2024-03-24T10:43:24.732 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 120/220 2024-03-24T10:43:24.732 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 121/220 2024-03-24T10:43:24.732 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 122/220 2024-03-24T10:43:24.732 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 123/220 2024-03-24T10:43:24.732 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 124/220 2024-03-24T10:43:24.732 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 125/220 2024-03-24T10:43:24.732 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 126/220 2024-03-24T10:43:24.732 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 127/220 2024-03-24T10:43:24.732 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 128/220 2024-03-24T10:43:24.732 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 129/220 2024-03-24T10:43:24.732 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 130/220 2024-03-24T10:43:24.732 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 131/220 2024-03-24T10:43:24.732 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 132/220 2024-03-24T10:43:24.733 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 133/220 2024-03-24T10:43:24.733 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 134/220 2024-03-24T10:43:24.733 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 135/220 2024-03-24T10:43:24.733 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 136/220 2024-03-24T10:43:24.733 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 137/220 2024-03-24T10:43:24.733 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 138/220 2024-03-24T10:43:24.733 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 139/220 2024-03-24T10:43:24.733 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 140/220 2024-03-24T10:43:24.733 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 141/220 2024-03-24T10:43:24.733 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 142/220 2024-03-24T10:43:24.733 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 143/220 2024-03-24T10:43:24.733 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 144/220 2024-03-24T10:43:24.733 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 145/220 2024-03-24T10:43:24.733 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 146/220 2024-03-24T10:43:24.733 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 147/220 2024-03-24T10:43:24.733 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 148/220 2024-03-24T10:43:24.734 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 149/220 2024-03-24T10:43:24.734 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 150/220 2024-03-24T10:43:24.734 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 151/220 2024-03-24T10:43:24.734 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 152/220 2024-03-24T10:43:24.734 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 153/220 2024-03-24T10:43:24.734 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 154/220 2024-03-24T10:43:24.734 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 155/220 2024-03-24T10:43:24.734 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 156/220 2024-03-24T10:43:24.734 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 157/220 2024-03-24T10:43:24.734 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 158/220 2024-03-24T10:43:24.734 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 159/220 2024-03-24T10:43:24.734 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 160/220 2024-03-24T10:43:24.734 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 161/220 2024-03-24T10:43:24.735 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 162/220 2024-03-24T10:43:24.735 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 163/220 2024-03-24T10:43:24.735 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 164/220 2024-03-24T10:43:24.735 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 165/220 2024-03-24T10:43:24.735 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 166/220 2024-03-24T10:43:24.735 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 167/220 2024-03-24T10:43:24.735 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 168/220 2024-03-24T10:43:24.735 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 169/220 2024-03-24T10:43:24.735 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 170/220 2024-03-24T10:43:24.735 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : re2-1:20211101-3.el9.x86_64 171/220 2024-03-24T10:43:24.735 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 172/220 2024-03-24T10:43:24.735 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 173/220 2024-03-24T10:43:24.735 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 174/220 2024-03-24T10:43:24.735 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 175/220 2024-03-24T10:43:24.736 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 176/220 2024-03-24T10:43:24.736 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 177/220 2024-03-24T10:43:24.736 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 178/220 2024-03-24T10:43:24.736 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 179/220 2024-03-24T10:43:24.736 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 180/220 2024-03-24T10:43:24.736 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 181/220 2024-03-24T10:43:24.736 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 182/220 2024-03-24T10:43:24.736 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 183/220 2024-03-24T10:43:24.736 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 184/220 2024-03-24T10:43:24.736 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 185/220 2024-03-24T10:43:24.736 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 186/220 2024-03-24T10:43:24.736 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : acl-2.3.1-4.el9.x86_64 187/220 2024-03-24T10:43:24.736 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : acl-2.3.1-3.el9.x86_64 188/220 2024-03-24T10:43:24.736 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 189/220 2024-03-24T10:43:24.736 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 190/220 2024-03-24T10:43:24.736 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 191/220 2024-03-24T10:43:24.737 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 192/220 2024-03-24T10:43:24.737 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 193/220 2024-03-24T10:43:24.737 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 194/220 2024-03-24T10:43:24.737 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 195/220 2024-03-24T10:43:24.737 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 196/220 2024-03-24T10:43:24.737 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 197/220 2024-03-24T10:43:24.737 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 198/220 2024-03-24T10:43:24.737 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 199/220 2024-03-24T10:43:24.737 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 200/220 2024-03-24T10:43:24.737 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 201/220 2024-03-24T10:43:24.737 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 202/220 2024-03-24T10:43:24.737 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 203/220 2024-03-24T10:43:24.737 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 204/220 2024-03-24T10:43:24.737 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 205/220 2024-03-24T10:43:24.738 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 206/220 2024-03-24T10:43:24.738 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 207/220 2024-03-24T10:43:24.738 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 208/220 2024-03-24T10:43:24.738 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 209/220 2024-03-24T10:43:24.738 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 210/220 2024-03-24T10:43:24.738 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 211/220 2024-03-24T10:43:24.738 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 212/220 2024-03-24T10:43:24.738 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 213/220 2024-03-24T10:43:24.738 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 214/220 2024-03-24T10:43:24.738 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 215/220 2024-03-24T10:43:24.738 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 216/220 2024-03-24T10:43:24.738 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 217/220 2024-03-24T10:43:24.738 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 218/220 2024-03-24T10:43:24.738 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 219/220 2024-03-24T10:43:25.465 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 220/220 2024-03-24T10:43:25.465 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 1/220 2024-03-24T10:43:25.465 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2/220 2024-03-24T10:43:25.465 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 3/220 2024-03-24T10:43:25.465 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 4/220 2024-03-24T10:43:25.465 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1855.g21b3bd67.el 5/220 2024-03-24T10:43:25.465 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_6 6/220 2024-03-24T10:43:25.465 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 7/220 2024-03-24T10:43:25.465 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 8/220 2024-03-24T10:43:25.465 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 9/220 2024-03-24T10:43:25.465 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1 10/220 2024-03-24T10:43:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/220 2024-03-24T10:43:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 12/220 2024-03-24T10:43:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/220 2024-03-24T10:43:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 14/220 2024-03-24T10:43:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 15/220 2024-03-24T10:43:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 16/220 2024-03-24T10:43:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/220 2024-03-24T10:43:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 18/220 2024-03-24T10:43:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 19/220 2024-03-24T10:43:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1855.g21b3bd67.e 20/220 2024-03-24T10:43:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 21/220 2024-03-24T10:43:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 22/220 2024-03-24T10:43:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 23/220 2024-03-24T10:43:25.466 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 24/220 2024-03-24T10:43:25.467 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 25/220 2024-03-24T10:43:25.467 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 26/220 2024-03-24T10:43:25.467 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 27/220 2024-03-24T10:43:25.467 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1855.g21b3bd67.el9 28/220 2024-03-24T10:43:25.467 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 29/220 2024-03-24T10:43:25.467 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 30/220 2024-03-24T10:43:25.485 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1855.g21b3bd67 31/220 2024-03-24T10:43:25.485 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados2-debuginfo-2:19.0.0-1855.g21b3bd67.el9. 32/220 2024-03-24T10:43:25.485 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 33/220 2024-03-24T10:43:25.485 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x8 34/220 2024-03-24T10:43:25.485 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 35/220 2024-03-24T10:43:25.485 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 36/220 2024-03-24T10:43:25.485 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 37/220 2024-03-24T10:43:25.485 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 38/220 2024-03-24T10:43:25.485 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1855.g21b3bd67 39/220 2024-03-24T10:43:25.485 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 40/220 2024-03-24T10:43:25.485 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1855.g21b3bd67. 41/220 2024-03-24T10:43:25.485 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 42/220 2024-03-24T10:43:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1855.g21b3bd67.el 43/220 2024-03-24T10:43:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 44/220 2024-03-24T10:43:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1855.g21b3bd67.el 45/220 2024-03-24T10:43:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 46/220 2024-03-24T10:43:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x 47/220 2024-03-24T10:43:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 48/220 2024-03-24T10:43:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1855.g21b3bd67.el9 49/220 2024-03-24T10:43:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 50/220 2024-03-24T10:43:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x8 51/220 2024-03-24T10:43:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 52/220 2024-03-24T10:43:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 53/220 2024-03-24T10:43:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 54/220 2024-03-24T10:43:25.486 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 55/220 2024-03-24T10:43:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 56/220 2024-03-24T10:43:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 57/220 2024-03-24T10:43:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 58/220 2024-03-24T10:43:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 59/220 2024-03-24T10:43:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 60/220 2024-03-24T10:43:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 61/220 2024-03-24T10:43:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : automake-1.16.2-8.el9.noarch 62/220 2024-03-24T10:43:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : bison-3.7.4-5.el9.x86_64 63/220 2024-03-24T10:43:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 64/220 2024-03-24T10:43:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : flex-2.6.4-9.el9.x86_64 65/220 2024-03-24T10:43:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 66/220 2024-03-24T10:43:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 67/220 2024-03-24T10:43:25.487 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 68/220 2024-03-24T10:43:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 69/220 2024-03-24T10:43:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 70/220 2024-03-24T10:43:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 71/220 2024-03-24T10:43:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 72/220 2024-03-24T10:43:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 73/220 2024-03-24T10:43:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 74/220 2024-03-24T10:43:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 75/220 2024-03-24T10:43:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : lua-5.4.4-4.el9.x86_64 76/220 2024-03-24T10:43:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 77/220 2024-03-24T10:43:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 78/220 2024-03-24T10:43:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 79/220 2024-03-24T10:43:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : postgresql-13.11-1.el9.x86_64 80/220 2024-03-24T10:43:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 81/220 2024-03-24T10:43:25.488 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 82/220 2024-03-24T10:43:25.489 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 83/220 2024-03-24T10:43:25.489 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 84/220 2024-03-24T10:43:25.489 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 85/220 2024-03-24T10:43:25.489 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 86/220 2024-03-24T10:43:25.489 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 87/220 2024-03-24T10:43:25.489 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 88/220 2024-03-24T10:43:25.489 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 89/220 2024-03-24T10:43:25.489 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 90/220 2024-03-24T10:43:25.489 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 91/220 2024-03-24T10:43:25.489 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 92/220 2024-03-24T10:43:25.489 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 93/220 2024-03-24T10:43:25.489 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 94/220 2024-03-24T10:43:25.489 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 95/220 2024-03-24T10:43:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 96/220 2024-03-24T10:43:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 97/220 2024-03-24T10:43:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 98/220 2024-03-24T10:43:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 99/220 2024-03-24T10:43:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 100/220 2024-03-24T10:43:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 101/220 2024-03-24T10:43:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 102/220 2024-03-24T10:43:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 103/220 2024-03-24T10:43:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 104/220 2024-03-24T10:43:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 105/220 2024-03-24T10:43:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 106/220 2024-03-24T10:43:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 107/220 2024-03-24T10:43:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 108/220 2024-03-24T10:43:25.490 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 109/220 2024-03-24T10:43:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 110/220 2024-03-24T10:43:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ply-3.11-14.el9.noarch 111/220 2024-03-24T10:43:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 112/220 2024-03-24T10:43:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 113/220 2024-03-24T10:43:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : jq-1.6-16.el9.x86_64 114/220 2024-03-24T10:43:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 115/220 2024-03-24T10:43:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 116/220 2024-03-24T10:43:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 117/220 2024-03-24T10:43:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 118/220 2024-03-24T10:43:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 119/220 2024-03-24T10:43:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 120/220 2024-03-24T10:43:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 121/220 2024-03-24T10:43:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 122/220 2024-03-24T10:43:25.491 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 123/220 2024-03-24T10:43:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 124/220 2024-03-24T10:43:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 125/220 2024-03-24T10:43:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 126/220 2024-03-24T10:43:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 127/220 2024-03-24T10:43:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 128/220 2024-03-24T10:43:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 129/220 2024-03-24T10:43:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 130/220 2024-03-24T10:43:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 131/220 2024-03-24T10:43:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 132/220 2024-03-24T10:43:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 133/220 2024-03-24T10:43:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 134/220 2024-03-24T10:43:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 135/220 2024-03-24T10:43:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 136/220 2024-03-24T10:43:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 137/220 2024-03-24T10:43:25.492 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 138/220 2024-03-24T10:43:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 139/220 2024-03-24T10:43:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 140/220 2024-03-24T10:43:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 141/220 2024-03-24T10:43:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 142/220 2024-03-24T10:43:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 143/220 2024-03-24T10:43:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 144/220 2024-03-24T10:43:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 145/220 2024-03-24T10:43:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 146/220 2024-03-24T10:43:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 147/220 2024-03-24T10:43:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 148/220 2024-03-24T10:43:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 149/220 2024-03-24T10:43:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 150/220 2024-03-24T10:43:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 151/220 2024-03-24T10:43:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 152/220 2024-03-24T10:43:25.493 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 153/220 2024-03-24T10:43:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 154/220 2024-03-24T10:43:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 155/220 2024-03-24T10:43:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 156/220 2024-03-24T10:43:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 157/220 2024-03-24T10:43:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 158/220 2024-03-24T10:43:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 159/220 2024-03-24T10:43:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 160/220 2024-03-24T10:43:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 161/220 2024-03-24T10:43:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 162/220 2024-03-24T10:43:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 163/220 2024-03-24T10:43:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 164/220 2024-03-24T10:43:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 165/220 2024-03-24T10:43:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 166/220 2024-03-24T10:43:25.494 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 167/220 2024-03-24T10:43:25.495 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 168/220 2024-03-24T10:43:25.495 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 169/220 2024-03-24T10:43:25.495 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 170/220 2024-03-24T10:43:25.495 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : re2-1:20211101-3.el9.x86_64 171/220 2024-03-24T10:43:25.495 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 172/220 2024-03-24T10:43:25.495 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 173/220 2024-03-24T10:43:25.495 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 174/220 2024-03-24T10:43:25.495 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 175/220 2024-03-24T10:43:25.495 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 176/220 2024-03-24T10:43:25.495 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 177/220 2024-03-24T10:43:25.495 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 178/220 2024-03-24T10:43:25.495 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 179/220 2024-03-24T10:43:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 180/220 2024-03-24T10:43:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 181/220 2024-03-24T10:43:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 182/220 2024-03-24T10:43:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 183/220 2024-03-24T10:43:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 184/220 2024-03-24T10:43:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 185/220 2024-03-24T10:43:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 186/220 2024-03-24T10:43:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : acl-2.3.1-4.el9.x86_64 187/220 2024-03-24T10:43:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : acl-2.3.1-3.el9.x86_64 188/220 2024-03-24T10:43:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 189/220 2024-03-24T10:43:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 190/220 2024-03-24T10:43:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 191/220 2024-03-24T10:43:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 192/220 2024-03-24T10:43:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 193/220 2024-03-24T10:43:25.496 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 194/220 2024-03-24T10:43:25.497 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 195/220 2024-03-24T10:43:25.497 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 196/220 2024-03-24T10:43:25.497 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 197/220 2024-03-24T10:43:25.497 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 198/220 2024-03-24T10:43:25.497 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 199/220 2024-03-24T10:43:25.497 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 200/220 2024-03-24T10:43:25.497 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 201/220 2024-03-24T10:43:25.497 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 202/220 2024-03-24T10:43:25.497 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 203/220 2024-03-24T10:43:25.497 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 204/220 2024-03-24T10:43:25.497 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 205/220 2024-03-24T10:43:25.497 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 206/220 2024-03-24T10:43:25.498 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 207/220 2024-03-24T10:43:25.498 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 208/220 2024-03-24T10:43:25.498 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 209/220 2024-03-24T10:43:25.498 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 210/220 2024-03-24T10:43:25.498 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 211/220 2024-03-24T10:43:25.498 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 212/220 2024-03-24T10:43:25.498 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 213/220 2024-03-24T10:43:25.498 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 214/220 2024-03-24T10:43:25.498 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 215/220 2024-03-24T10:43:25.498 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 216/220 2024-03-24T10:43:25.498 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 217/220 2024-03-24T10:43:25.498 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 218/220 2024-03-24T10:43:25.499 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 219/220 2024-03-24T10:43:27.611 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 220/220 2024-03-24T10:43:27.612 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:27.612 INFO:teuthology.orchestra.run.smithi005.stdout:Upgraded: 2024-03-24T10:43:27.612 INFO:teuthology.orchestra.run.smithi005.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T10:43:27.612 INFO:teuthology.orchestra.run.smithi005.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T10:43:27.612 INFO:teuthology.orchestra.run.smithi005.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T10:43:27.612 INFO:teuthology.orchestra.run.smithi005.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T10:43:27.612 INFO:teuthology.orchestra.run.smithi005.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T10:43:27.612 INFO:teuthology.orchestra.run.smithi005.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T10:43:27.612 INFO:teuthology.orchestra.run.smithi005.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T10:43:27.612 INFO:teuthology.orchestra.run.smithi005.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T10:43:27.612 INFO:teuthology.orchestra.run.smithi005.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T10:43:27.613 INFO:teuthology.orchestra.run.smithi005.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T10:43:27.613 INFO:teuthology.orchestra.run.smithi005.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T10:43:27.613 INFO:teuthology.orchestra.run.smithi005.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T10:43:27.613 INFO:teuthology.orchestra.run.smithi005.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T10:43:27.613 INFO:teuthology.orchestra.run.smithi005.stdout: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.613 INFO:teuthology.orchestra.run.smithi005.stdout: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.613 INFO:teuthology.orchestra.run.smithi005.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T10:43:27.613 INFO:teuthology.orchestra.run.smithi005.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T10:43:27.613 INFO:teuthology.orchestra.run.smithi005.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T10:43:27.613 INFO:teuthology.orchestra.run.smithi005.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T10:43:27.613 INFO:teuthology.orchestra.run.smithi005.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T10:43:27.614 INFO:teuthology.orchestra.run.smithi005.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T10:43:27.614 INFO:teuthology.orchestra.run.smithi005.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T10:43:27.614 INFO:teuthology.orchestra.run.smithi005.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T10:43:27.614 INFO:teuthology.orchestra.run.smithi005.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T10:43:27.614 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-24T10:43:27.614 INFO:teuthology.orchestra.run.smithi005.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T10:43:27.614 INFO:teuthology.orchestra.run.smithi005.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T10:43:27.614 INFO:teuthology.orchestra.run.smithi005.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T10:43:27.614 INFO:teuthology.orchestra.run.smithi005.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T10:43:27.614 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.614 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.615 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-base-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.615 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.615 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-common-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.615 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.615 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.615 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.615 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:43:27.615 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-immutable-object-cache-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.615 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.615 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.615 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mds-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.616 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.616 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:43:27.616 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:43:27.616 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.616 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:43:27.616 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:43:27.616 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:43:27.616 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.616 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-mon-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.616 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.616 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-osd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.617 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:43:27.617 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.617 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-radosgw-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.617 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.617 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.617 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-test-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.617 INFO:teuthology.orchestra.run.smithi005.stdout: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:43:27.617 INFO:teuthology.orchestra.run.smithi005.stdout: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:43:27.617 INFO:teuthology.orchestra.run.smithi005.stdout: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.617 INFO:teuthology.orchestra.run.smithi005.stdout: cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.617 INFO:teuthology.orchestra.run.smithi005.stdout: cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:43:27.618 INFO:teuthology.orchestra.run.smithi005.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T10:43:27.618 INFO:teuthology.orchestra.run.smithi005.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T10:43:27.618 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T10:43:27.618 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T10:43:27.618 INFO:teuthology.orchestra.run.smithi005.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T10:43:27.618 INFO:teuthology.orchestra.run.smithi005.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T10:43:27.618 INFO:teuthology.orchestra.run.smithi005.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T10:43:27.618 INFO:teuthology.orchestra.run.smithi005.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T10:43:27.618 INFO:teuthology.orchestra.run.smithi005.stdout: jq-1.6-16.el9.x86_64 2024-03-24T10:43:27.618 INFO:teuthology.orchestra.run.smithi005.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T10:43:27.618 INFO:teuthology.orchestra.run.smithi005.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T10:43:27.619 INFO:teuthology.orchestra.run.smithi005.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T10:43:27.619 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.619 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.619 INFO:teuthology.orchestra.run.smithi005.stdout: libcephfs2-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.619 INFO:teuthology.orchestra.run.smithi005.stdout: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.619 INFO:teuthology.orchestra.run.smithi005.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T10:43:27.619 INFO:teuthology.orchestra.run.smithi005.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T10:43:27.619 INFO:teuthology.orchestra.run.smithi005.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T10:43:27.619 INFO:teuthology.orchestra.run.smithi005.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T10:43:27.619 INFO:teuthology.orchestra.run.smithi005.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T10:43:27.619 INFO:teuthology.orchestra.run.smithi005.stdout: librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.619 INFO:teuthology.orchestra.run.smithi005.stdout: librados-devel-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.620 INFO:teuthology.orchestra.run.smithi005.stdout: librados2-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.620 INFO:teuthology.orchestra.run.smithi005.stdout: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.620 INFO:teuthology.orchestra.run.smithi005.stdout: librbd1-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.620 INFO:teuthology.orchestra.run.smithi005.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T10:43:27.620 INFO:teuthology.orchestra.run.smithi005.stdout: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.620 INFO:teuthology.orchestra.run.smithi005.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T10:43:27.620 INFO:teuthology.orchestra.run.smithi005.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T10:43:27.620 INFO:teuthology.orchestra.run.smithi005.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T10:43:27.620 INFO:teuthology.orchestra.run.smithi005.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T10:43:27.620 INFO:teuthology.orchestra.run.smithi005.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T10:43:27.620 INFO:teuthology.orchestra.run.smithi005.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T10:43:27.620 INFO:teuthology.orchestra.run.smithi005.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T10:43:27.621 INFO:teuthology.orchestra.run.smithi005.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T10:43:27.621 INFO:teuthology.orchestra.run.smithi005.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T10:43:27.621 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T10:43:27.621 INFO:teuthology.orchestra.run.smithi005.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T10:43:27.621 INFO:teuthology.orchestra.run.smithi005.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T10:43:27.621 INFO:teuthology.orchestra.run.smithi005.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T10:43:27.621 INFO:teuthology.orchestra.run.smithi005.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T10:43:27.621 INFO:teuthology.orchestra.run.smithi005.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T10:43:27.621 INFO:teuthology.orchestra.run.smithi005.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T10:43:27.621 INFO:teuthology.orchestra.run.smithi005.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T10:43:27.621 INFO:teuthology.orchestra.run.smithi005.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T10:43:27.622 INFO:teuthology.orchestra.run.smithi005.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T10:43:27.622 INFO:teuthology.orchestra.run.smithi005.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T10:43:27.622 INFO:teuthology.orchestra.run.smithi005.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T10:43:27.622 INFO:teuthology.orchestra.run.smithi005.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T10:43:27.622 INFO:teuthology.orchestra.run.smithi005.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T10:43:27.622 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T10:43:27.622 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.622 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.622 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.622 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cephfs-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.622 INFO:teuthology.orchestra.run.smithi005.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T10:43:27.622 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T10:43:27.623 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T10:43:27.623 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T10:43:27.623 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T10:43:27.623 INFO:teuthology.orchestra.run.smithi005.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T10:43:27.623 INFO:teuthology.orchestra.run.smithi005.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T10:43:27.623 INFO:teuthology.orchestra.run.smithi005.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T10:43:27.623 INFO:teuthology.orchestra.run.smithi005.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T10:43:27.623 INFO:teuthology.orchestra.run.smithi005.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T10:43:27.623 INFO:teuthology.orchestra.run.smithi005.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T10:43:27.623 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T10:43:27.623 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T10:43:27.623 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T10:43:27.624 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T10:43:27.624 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T10:43:27.624 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T10:43:27.624 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T10:43:27.624 INFO:teuthology.orchestra.run.smithi005.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T10:43:27.624 INFO:teuthology.orchestra.run.smithi005.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T10:43:27.624 INFO:teuthology.orchestra.run.smithi005.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T10:43:27.624 INFO:teuthology.orchestra.run.smithi005.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T10:43:27.624 INFO:teuthology.orchestra.run.smithi005.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T10:43:27.624 INFO:teuthology.orchestra.run.smithi005.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T10:43:27.624 INFO:teuthology.orchestra.run.smithi005.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T10:43:27.624 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T10:43:27.625 INFO:teuthology.orchestra.run.smithi005.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T10:43:27.625 INFO:teuthology.orchestra.run.smithi005.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T10:43:27.625 INFO:teuthology.orchestra.run.smithi005.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T10:43:27.625 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T10:43:27.625 INFO:teuthology.orchestra.run.smithi005.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T10:43:27.625 INFO:teuthology.orchestra.run.smithi005.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T10:43:27.625 INFO:teuthology.orchestra.run.smithi005.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T10:43:27.625 INFO:teuthology.orchestra.run.smithi005.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T10:43:27.625 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T10:43:27.625 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T10:43:27.625 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T10:43:27.626 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T10:43:27.626 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T10:43:27.626 INFO:teuthology.orchestra.run.smithi005.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T10:43:27.626 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.626 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rados-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.626 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.626 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rbd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.626 INFO:teuthology.orchestra.run.smithi005.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T10:43:27.626 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T10:43:27.626 INFO:teuthology.orchestra.run.smithi005.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T10:43:27.627 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.627 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rgw-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.627 INFO:teuthology.orchestra.run.smithi005.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T10:43:27.627 INFO:teuthology.orchestra.run.smithi005.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T10:43:27.627 INFO:teuthology.orchestra.run.smithi005.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T10:43:27.627 INFO:teuthology.orchestra.run.smithi005.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T10:43:27.627 INFO:teuthology.orchestra.run.smithi005.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T10:43:27.627 INFO:teuthology.orchestra.run.smithi005.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T10:43:27.627 INFO:teuthology.orchestra.run.smithi005.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T10:43:27.627 INFO:teuthology.orchestra.run.smithi005.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T10:43:27.627 INFO:teuthology.orchestra.run.smithi005.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T10:43:27.628 INFO:teuthology.orchestra.run.smithi005.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T10:43:27.628 INFO:teuthology.orchestra.run.smithi005.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T10:43:27.628 INFO:teuthology.orchestra.run.smithi005.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T10:43:27.628 INFO:teuthology.orchestra.run.smithi005.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T10:43:27.628 INFO:teuthology.orchestra.run.smithi005.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T10:43:27.628 INFO:teuthology.orchestra.run.smithi005.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T10:43:27.628 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.628 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.628 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.628 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-mirror-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.628 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.628 INFO:teuthology.orchestra.run.smithi005.stdout: rbd-nbd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:27.629 INFO:teuthology.orchestra.run.smithi005.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T10:43:27.629 INFO:teuthology.orchestra.run.smithi005.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T10:43:27.629 INFO:teuthology.orchestra.run.smithi005.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T10:43:27.629 INFO:teuthology.orchestra.run.smithi005.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T10:43:27.629 INFO:teuthology.orchestra.run.smithi005.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T10:43:27.629 INFO:teuthology.orchestra.run.smithi005.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T10:43:27.629 INFO:teuthology.orchestra.run.smithi005.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T10:43:27.629 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:27.630 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-24T10:43:28.359 DEBUG:teuthology.parallel:result is None 2024-03-24T10:43:28.663 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 220/220 2024-03-24T10:43:28.663 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:28.663 INFO:teuthology.orchestra.run.smithi057.stdout:Upgraded: 2024-03-24T10:43:28.663 INFO:teuthology.orchestra.run.smithi057.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T10:43:28.663 INFO:teuthology.orchestra.run.smithi057.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T10:43:28.663 INFO:teuthology.orchestra.run.smithi057.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T10:43:28.663 INFO:teuthology.orchestra.run.smithi057.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T10:43:28.663 INFO:teuthology.orchestra.run.smithi057.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T10:43:28.664 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T10:43:28.664 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T10:43:28.664 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T10:43:28.664 INFO:teuthology.orchestra.run.smithi057.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T10:43:28.664 INFO:teuthology.orchestra.run.smithi057.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T10:43:28.664 INFO:teuthology.orchestra.run.smithi057.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T10:43:28.664 INFO:teuthology.orchestra.run.smithi057.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T10:43:28.664 INFO:teuthology.orchestra.run.smithi057.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T10:43:28.664 INFO:teuthology.orchestra.run.smithi057.stdout: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.664 INFO:teuthology.orchestra.run.smithi057.stdout: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.664 INFO:teuthology.orchestra.run.smithi057.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T10:43:28.664 INFO:teuthology.orchestra.run.smithi057.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T10:43:28.664 INFO:teuthology.orchestra.run.smithi057.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T10:43:28.664 INFO:teuthology.orchestra.run.smithi057.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T10:43:28.665 INFO:teuthology.orchestra.run.smithi057.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T10:43:28.665 INFO:teuthology.orchestra.run.smithi057.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T10:43:28.665 INFO:teuthology.orchestra.run.smithi057.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T10:43:28.665 INFO:teuthology.orchestra.run.smithi057.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T10:43:28.665 INFO:teuthology.orchestra.run.smithi057.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T10:43:28.665 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2024-03-24T10:43:28.665 INFO:teuthology.orchestra.run.smithi057.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T10:43:28.665 INFO:teuthology.orchestra.run.smithi057.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T10:43:28.665 INFO:teuthology.orchestra.run.smithi057.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T10:43:28.665 INFO:teuthology.orchestra.run.smithi057.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T10:43:28.665 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.665 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.665 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-base-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.665 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.665 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-common-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.666 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.666 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.666 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.666 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:43:28.666 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-immutable-object-cache-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.666 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.666 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.666 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mds-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.666 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.666 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:43:28.666 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:43:28.666 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.666 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:43:28.666 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:43:28.666 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:43:28.667 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.667 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mon-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.667 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.667 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-osd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.667 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:43:28.667 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.667 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-radosgw-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.667 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.667 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.667 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-test-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.667 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:43:28.667 INFO:teuthology.orchestra.run.smithi057.stdout: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:43:28.667 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.668 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.668 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-24T10:43:28.668 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T10:43:28.668 INFO:teuthology.orchestra.run.smithi057.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T10:43:28.668 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T10:43:28.668 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T10:43:28.668 INFO:teuthology.orchestra.run.smithi057.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T10:43:28.668 INFO:teuthology.orchestra.run.smithi057.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T10:43:28.668 INFO:teuthology.orchestra.run.smithi057.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T10:43:28.668 INFO:teuthology.orchestra.run.smithi057.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T10:43:28.668 INFO:teuthology.orchestra.run.smithi057.stdout: jq-1.6-16.el9.x86_64 2024-03-24T10:43:28.669 INFO:teuthology.orchestra.run.smithi057.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T10:43:28.669 INFO:teuthology.orchestra.run.smithi057.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T10:43:28.669 INFO:teuthology.orchestra.run.smithi057.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T10:43:28.669 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.669 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.669 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs2-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.669 INFO:teuthology.orchestra.run.smithi057.stdout: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.669 INFO:teuthology.orchestra.run.smithi057.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T10:43:28.669 INFO:teuthology.orchestra.run.smithi057.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T10:43:28.669 INFO:teuthology.orchestra.run.smithi057.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T10:43:28.669 INFO:teuthology.orchestra.run.smithi057.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T10:43:28.669 INFO:teuthology.orchestra.run.smithi057.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T10:43:28.669 INFO:teuthology.orchestra.run.smithi057.stdout: librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.670 INFO:teuthology.orchestra.run.smithi057.stdout: librados-devel-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.670 INFO:teuthology.orchestra.run.smithi057.stdout: librados2-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.670 INFO:teuthology.orchestra.run.smithi057.stdout: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.670 INFO:teuthology.orchestra.run.smithi057.stdout: librbd1-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.670 INFO:teuthology.orchestra.run.smithi057.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T10:43:28.670 INFO:teuthology.orchestra.run.smithi057.stdout: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.670 INFO:teuthology.orchestra.run.smithi057.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T10:43:28.670 INFO:teuthology.orchestra.run.smithi057.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T10:43:28.670 INFO:teuthology.orchestra.run.smithi057.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T10:43:28.670 INFO:teuthology.orchestra.run.smithi057.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T10:43:28.670 INFO:teuthology.orchestra.run.smithi057.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T10:43:28.670 INFO:teuthology.orchestra.run.smithi057.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T10:43:28.670 INFO:teuthology.orchestra.run.smithi057.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T10:43:28.671 INFO:teuthology.orchestra.run.smithi057.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T10:43:28.671 INFO:teuthology.orchestra.run.smithi057.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T10:43:28.671 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T10:43:28.671 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T10:43:28.671 INFO:teuthology.orchestra.run.smithi057.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T10:43:28.671 INFO:teuthology.orchestra.run.smithi057.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T10:43:28.671 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T10:43:28.671 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T10:43:28.671 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T10:43:28.671 INFO:teuthology.orchestra.run.smithi057.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T10:43:28.671 INFO:teuthology.orchestra.run.smithi057.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T10:43:28.671 INFO:teuthology.orchestra.run.smithi057.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T10:43:28.671 INFO:teuthology.orchestra.run.smithi057.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T10:43:28.671 INFO:teuthology.orchestra.run.smithi057.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T10:43:28.671 INFO:teuthology.orchestra.run.smithi057.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T10:43:28.672 INFO:teuthology.orchestra.run.smithi057.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T10:43:28.672 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T10:43:28.672 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.672 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.672 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.672 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cephfs-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.672 INFO:teuthology.orchestra.run.smithi057.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T10:43:28.672 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T10:43:28.672 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T10:43:28.672 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T10:43:28.672 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T10:43:28.672 INFO:teuthology.orchestra.run.smithi057.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T10:43:28.672 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T10:43:28.672 INFO:teuthology.orchestra.run.smithi057.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T10:43:28.673 INFO:teuthology.orchestra.run.smithi057.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T10:43:28.673 INFO:teuthology.orchestra.run.smithi057.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T10:43:28.673 INFO:teuthology.orchestra.run.smithi057.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T10:43:28.673 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T10:43:28.673 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T10:43:28.673 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T10:43:28.673 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T10:43:28.673 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T10:43:28.673 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T10:43:28.673 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T10:43:28.673 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T10:43:28.673 INFO:teuthology.orchestra.run.smithi057.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T10:43:28.673 INFO:teuthology.orchestra.run.smithi057.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T10:43:28.673 INFO:teuthology.orchestra.run.smithi057.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T10:43:28.674 INFO:teuthology.orchestra.run.smithi057.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T10:43:28.674 INFO:teuthology.orchestra.run.smithi057.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T10:43:28.674 INFO:teuthology.orchestra.run.smithi057.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T10:43:28.674 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T10:43:28.674 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T10:43:28.674 INFO:teuthology.orchestra.run.smithi057.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T10:43:28.674 INFO:teuthology.orchestra.run.smithi057.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T10:43:28.674 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T10:43:28.674 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T10:43:28.674 INFO:teuthology.orchestra.run.smithi057.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T10:43:28.674 INFO:teuthology.orchestra.run.smithi057.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T10:43:28.674 INFO:teuthology.orchestra.run.smithi057.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T10:43:28.674 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T10:43:28.674 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T10:43:28.674 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T10:43:28.675 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T10:43:28.675 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T10:43:28.675 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T10:43:28.675 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.675 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rados-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.675 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.675 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rbd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.675 INFO:teuthology.orchestra.run.smithi057.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T10:43:28.675 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T10:43:28.675 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T10:43:28.675 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.675 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rgw-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.675 INFO:teuthology.orchestra.run.smithi057.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T10:43:28.675 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T10:43:28.675 INFO:teuthology.orchestra.run.smithi057.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T10:43:28.675 INFO:teuthology.orchestra.run.smithi057.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T10:43:28.676 INFO:teuthology.orchestra.run.smithi057.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T10:43:28.676 INFO:teuthology.orchestra.run.smithi057.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T10:43:28.676 INFO:teuthology.orchestra.run.smithi057.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T10:43:28.676 INFO:teuthology.orchestra.run.smithi057.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T10:43:28.676 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T10:43:28.676 INFO:teuthology.orchestra.run.smithi057.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T10:43:28.676 INFO:teuthology.orchestra.run.smithi057.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T10:43:28.676 INFO:teuthology.orchestra.run.smithi057.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T10:43:28.676 INFO:teuthology.orchestra.run.smithi057.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T10:43:28.676 INFO:teuthology.orchestra.run.smithi057.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T10:43:28.676 INFO:teuthology.orchestra.run.smithi057.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T10:43:28.676 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.676 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-fuse-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.676 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.676 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-mirror-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.677 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.677 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-nbd-debuginfo-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-24T10:43:28.677 INFO:teuthology.orchestra.run.smithi057.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T10:43:28.677 INFO:teuthology.orchestra.run.smithi057.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T10:43:28.677 INFO:teuthology.orchestra.run.smithi057.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T10:43:28.677 INFO:teuthology.orchestra.run.smithi057.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T10:43:28.677 INFO:teuthology.orchestra.run.smithi057.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T10:43:28.677 INFO:teuthology.orchestra.run.smithi057.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T10:43:28.677 INFO:teuthology.orchestra.run.smithi057.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T10:43:28.678 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:28.678 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2024-03-24T10:43:29.351 DEBUG:teuthology.parallel:result is None 2024-03-24T10:43:29.352 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:43:29.493 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T10:43:29.518 INFO:teuthology.orchestra.run.smithi005.stdout:19.0.0-1855.g21b3bd67.el9 2024-03-24T10:43:29.518 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1855.g21b3bd67.el9 2024-03-24T10:43:29.518 INFO:teuthology.task.install:The correct ceph version 19.0.0-1855.g21b3bd67 is installed. 2024-03-24T10:43:29.519 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-24T10:43:29.662 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T10:43:29.684 INFO:teuthology.orchestra.run.smithi057.stdout:19.0.0-1855.g21b3bd67.el9 2024-03-24T10:43:29.684 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1855.g21b3bd67.el9 2024-03-24T10:43:29.684 INFO:teuthology.task.install:The correct ceph version 19.0.0-1855.g21b3bd67 is installed. 2024-03-24T10:43:29.685 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-24T10:43:29.685 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:43:29.686 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T10:43:29.720 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:43:29.720 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T10:43:29.754 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-24T10:43:29.754 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:43:29.754 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T10:43:29.789 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T10:43:29.859 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:43:29.859 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T10:43:29.887 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T10:43:29.955 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-24T10:43:29.955 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:43:29.955 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T10:43:29.985 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T10:43:30.055 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:43:30.055 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T10:43:30.081 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T10:43:30.147 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-24T10:43:30.148 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:43:30.148 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T10:43:30.176 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T10:43:30.245 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:43:30.246 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T10:43:30.271 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T10:43:30.338 INFO:teuthology.run_tasks:Running task ceph... 2024-03-24T10:43:30.421 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-24T10:43:30.421 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 777 /var/log/ceph 2024-03-24T10:43:30.423 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 777 /var/log/ceph 2024-03-24T10:43:30.451 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-24T10:43:30.451 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-24T10:43:30.492 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-24T10:43:30.519 INFO:tasks.ceph:Creating extra log directories... 2024-03-24T10:43:30.519 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-24T10:43:30.563 DEBUG:teuthology.orchestra.run.smithi057:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-24T10:43:30.596 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-24T10:43:30.596 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-24T10:43:30.596 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-24T10:43:30.596 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:43:30.596 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-24T10:43:30.645 DEBUG:teuthology.orchestra.run.smithi005:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-24T10:43:30.764 DEBUG:teuthology.orchestra.run.smithi005:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-24T10:43:30.792 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:43:30.792 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-24T10:43:30.806 DEBUG:teuthology.orchestra.run.smithi057:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-24T10:43:30.923 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-24T10:43:30.955 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-24T10:43:30.955 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-24T10:43:30.955 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619063', 'branch': 'wip-batrick-testing-20240323.003144-squid', 'description': 'fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/fsstress validater/valgrind}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7619063', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-batrick-testing-20240323.003144-squid'}, 'ceph': {'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}, 'watchdog': {'daemon_restart': 'normal'}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'ceph-fuse': {'client.0': {'valgrind': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-batrick-testing-20240323.003144-squid', 'sha1': 'd39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 4284, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'sleep_before_teardown': 0, 'subset': '405/512', 'suite': 'fs', 'suite_branch': 'wip-batrick-testing-20240323.003144-squid', 'suite_path': '/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/batrick/ceph', 'suite_sha1': 'd39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi005.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFn+kiqlbs5zbTuBBHsHzEb/WYC+FI9IOgEygpTTmkkomwTk+KNf0/B2ea6hscWCvKN0mtgrm8ZyKJLkRdZ3nW8=', 'smithi057.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMokMhSL3Mys8idkxCh6J+sEHRTJAMEjyaIKAU1fo3o+pBIjGzyT4BjzlGmoyU6eIQ8lbjbW+2u15K3TYKI+qsI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'debuginfo': True, 'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}, 'watchdog': {'daemon_restart': 'normal'}, 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'workunit': {'clients': {'all': ['suites/fsstress.sh']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-24_04:56:01', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-24T10:43:30.956 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-24T10:43:30.958 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-24T10:43:31.004 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m0777 -- /var/run/ceph 2024-03-24T10:43:31.013 DEBUG:teuthology.orchestra.run.smithi057:> sudo install -d -m0777 -- /var/run/ceph 2024-03-24T10:43:31.071 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:43:31.071 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T10:43:31.094 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T10:43:31.094 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_1 2024-03-24T10:43:31.149 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T10:43:31.149 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:43:31.149 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-24T10:43:31.149 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:43:31.149 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:43:31.149 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-03-24 10:43:15.680028661 +0000 2024-03-24T10:43:31.149 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-03-24 10:32:37.805912021 +0000 2024-03-24T10:43:31.149 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-03-24 10:32:37.805912021 +0000 2024-03-24T10:43:31.150 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-03-24 10:32:37.805912021 +0000 2024-03-24T10:43:31.150 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T10:43:31.215 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-03-24T10:43:31.215 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-03-24T10:43:31.215 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 8.5985e-05 s, 6.0 MB/s 2024-03-24T10:43:31.216 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T10:43:31.272 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_2 2024-03-24T10:43:31.327 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T10:43:31.327 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:43:31.327 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-24T10:43:31.327 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:43:31.327 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:43:31.327 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-03-24 10:43:15.681028641 +0000 2024-03-24T10:43:31.328 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-03-24 10:32:38.208903852 +0000 2024-03-24T10:43:31.328 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-03-24 10:32:38.208903852 +0000 2024-03-24T10:43:31.328 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-03-24 10:32:38.208903852 +0000 2024-03-24T10:43:31.328 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T10:43:31.393 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-03-24T10:43:31.393 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-03-24T10:43:31.393 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 8.1666e-05 s, 6.3 MB/s 2024-03-24T10:43:31.394 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T10:43:31.450 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_3 2024-03-24T10:43:31.505 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T10:43:31.505 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:43:31.505 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-24T10:43:31.505 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:43:31.505 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:43:31.505 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-03-24 10:43:15.681028641 +0000 2024-03-24T10:43:31.505 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-03-24 10:32:38.593896048 +0000 2024-03-24T10:43:31.505 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-03-24 10:32:38.593896048 +0000 2024-03-24T10:43:31.505 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-03-24 10:32:38.593896048 +0000 2024-03-24T10:43:31.506 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T10:43:31.571 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-03-24T10:43:31.571 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-03-24T10:43:31.571 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 9.0455e-05 s, 5.7 MB/s 2024-03-24T10:43:31.572 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T10:43:31.627 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_4 2024-03-24T10:43:31.683 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T10:43:31.684 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:43:31.684 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-24T10:43:31.684 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:43:31.684 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:43:31.684 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-03-24 10:43:15.681028641 +0000 2024-03-24T10:43:31.684 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-03-24 10:32:38.946888894 +0000 2024-03-24T10:43:31.684 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-03-24 10:32:38.946888894 +0000 2024-03-24T10:43:31.684 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-03-24 10:32:38.946888894 +0000 2024-03-24T10:43:31.684 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T10:43:31.748 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-03-24T10:43:31.748 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-03-24T10:43:31.748 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 9.7292e-05 s, 5.3 MB/s 2024-03-24T10:43:31.749 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T10:43:31.805 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T10:43:31.806 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:43:31.806 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T10:43:31.823 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T10:43:31.823 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_1 2024-03-24T10:43:31.882 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T10:43:31.882 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:43:31.882 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-24T10:43:31.883 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:43:31.883 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:43:31.883 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2024-03-24 10:43:16.359507571 +0000 2024-03-24T10:43:31.883 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2024-03-24 10:32:32.752060802 +0000 2024-03-24T10:43:31.883 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2024-03-24 10:32:32.752060802 +0000 2024-03-24T10:43:31.883 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2024-03-24 10:32:32.752060802 +0000 2024-03-24T10:43:31.883 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T10:43:31.952 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2024-03-24T10:43:31.952 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2024-03-24T10:43:31.952 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000283538 s, 1.8 MB/s 2024-03-24T10:43:31.954 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T10:43:32.012 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_2 2024-03-24T10:43:32.069 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T10:43:32.069 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:43:32.069 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-24T10:43:32.070 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:43:32.070 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:43:32.070 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2024-03-24 10:43:16.359507571 +0000 2024-03-24T10:43:32.070 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2024-03-24 10:32:33.437051550 +0000 2024-03-24T10:43:32.070 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2024-03-24 10:32:33.437051550 +0000 2024-03-24T10:43:32.070 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2024-03-24 10:32:33.437051550 +0000 2024-03-24T10:43:32.070 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T10:43:32.138 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2024-03-24T10:43:32.138 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2024-03-24T10:43:32.138 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000289243 s, 1.8 MB/s 2024-03-24T10:43:32.139 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T10:43:32.197 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_3 2024-03-24T10:43:32.253 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T10:43:32.254 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:43:32.254 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-24T10:43:32.254 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:43:32.254 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:43:32.254 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2024-03-24 10:43:16.360507557 +0000 2024-03-24T10:43:32.254 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2024-03-24 10:32:33.850045972 +0000 2024-03-24T10:43:32.254 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2024-03-24 10:32:33.850045972 +0000 2024-03-24T10:43:32.254 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2024-03-24 10:32:33.850045972 +0000 2024-03-24T10:43:32.254 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T10:43:32.322 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2024-03-24T10:43:32.323 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2024-03-24T10:43:32.323 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000423558 s, 1.2 MB/s 2024-03-24T10:43:32.324 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T10:43:32.380 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_4 2024-03-24T10:43:32.436 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T10:43:32.436 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T10:43:32.436 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-24T10:43:32.436 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T10:43:32.436 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T10:43:32.436 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2024-03-24 10:43:16.360507557 +0000 2024-03-24T10:43:32.437 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2024-03-24 10:32:34.233040799 +0000 2024-03-24T10:43:32.437 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2024-03-24 10:32:34.233040799 +0000 2024-03-24T10:43:32.437 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: 2024-03-24 10:32:34.233040799 +0000 2024-03-24T10:43:32.437 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T10:43:32.503 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2024-03-24T10:43:32.503 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2024-03-24T10:43:32.503 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000499257 s, 1.0 MB/s 2024-03-24T10:43:32.504 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T10:43:32.560 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T10:43:32.560 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi057.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-24T10:43:32.560 INFO:tasks.ceph:Generating config... 2024-03-24T10:43:32.561 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-24T10:43:32.561 INFO:tasks.ceph:[client] debug client = 20 2024-03-24T10:43:32.561 INFO:tasks.ceph:[client] debug ms = 1 2024-03-24T10:43:32.562 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-24T10:43:32.562 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-24T10:43:32.562 INFO:tasks.ceph:[global] mds beacon grace = 60 2024-03-24T10:43:32.562 INFO:tasks.ceph:[global] mds heartbeat grace = 60 2024-03-24T10:43:32.562 INFO:tasks.ceph:[global] osd heartbeat grace = 60 2024-03-24T10:43:32.562 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-24T10:43:32.562 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-24T10:43:32.562 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-24T10:43:32.562 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-24T10:43:32.562 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-24T10:43:32.562 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-24T10:43:32.562 INFO:tasks.ceph:[mds] mds valgrind exit = True 2024-03-24T10:43:32.562 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-24T10:43:32.562 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-24T10:43:32.562 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-24T10:43:32.563 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-24T10:43:32.563 INFO:tasks.ceph:[mgr] debug client = 20 2024-03-24T10:43:32.563 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-24T10:43:32.563 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-24T10:43:32.563 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-24T10:43:32.563 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-24T10:43:32.563 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-24T10:43:32.563 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-03-24T10:43:32.563 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-24T10:43:32.563 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-24T10:43:32.563 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-24T10:43:32.563 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-24T10:43:32.563 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-24T10:43:32.563 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-24T10:43:32.563 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-24T10:43:32.563 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-24T10:43:32.564 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-24T10:43:32.564 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-24T10:43:32.564 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-24T10:43:32.564 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-24T10:43:32.564 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-24T10:43:32.564 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-24T10:43:32.564 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-24T10:43:32.564 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-24T10:43:32.564 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-24T10:43:32.564 INFO:tasks.ceph:Setting up mon.a... 2024-03-24T10:43:32.564 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-24T10:43:32.620 INFO:teuthology.orchestra.run.smithi005.stdout:creating /etc/ceph/ceph.keyring 2024-03-24T10:43:32.623 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-24T10:43:32.669 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-24T10:43:32.696 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.5'), ('mon.b', '172.21.15.57'), ('mon.c', '[v2:172.21.15.57:3301,v1:172.21.15.57:6790]')] 2024-03-24T10:43:32.697 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.5,172.21.15.57,[v2:172.21.15.57:3301,v1:172.21.15.57:6790]', 'mds beacon grace': 60, 'mds heartbeat grace': 60, 'osd heartbeat grace': 60}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-24T10:43:32.698 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:43:32.698 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-24T10:43:32.752 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.5 --add b 172.21.15.57 --addv c '[v2:172.21.15.57:3301,v1:172.21.15.57:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:43:32.837 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:43:32.837 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: generated fsid 7d9848db-2fe7-4d95-a6e2-b04f4f639ebc 2024-03-24T10:43:32.837 INFO:teuthology.orchestra.run.smithi005.stdout:setting min_mon_release = quincy 2024-03-24T10:43:32.837 INFO:teuthology.orchestra.run.smithi005.stdout:epoch 0 2024-03-24T10:43:32.837 INFO:teuthology.orchestra.run.smithi005.stdout:fsid 7d9848db-2fe7-4d95-a6e2-b04f4f639ebc 2024-03-24T10:43:32.837 INFO:teuthology.orchestra.run.smithi005.stdout:last_changed 2024-03-24T10:43:32.838058+0000 2024-03-24T10:43:32.837 INFO:teuthology.orchestra.run.smithi005.stdout:created 2024-03-24T10:43:32.838058+0000 2024-03-24T10:43:32.837 INFO:teuthology.orchestra.run.smithi005.stdout:min_mon_release 17 (quincy) 2024-03-24T10:43:32.837 INFO:teuthology.orchestra.run.smithi005.stdout:election_strategy: 1 2024-03-24T10:43:32.838 INFO:teuthology.orchestra.run.smithi005.stdout:0: [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon.a 2024-03-24T10:43:32.838 INFO:teuthology.orchestra.run.smithi005.stdout:1: [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] mon.b 2024-03-24T10:43:32.838 INFO:teuthology.orchestra.run.smithi005.stdout:2: [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] mon.c 2024-03-24T10:43:32.838 INFO:teuthology.orchestra.run.smithi005.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-24T10:43:32.839 DEBUG:teuthology.orchestra.run.smithi005:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-24T10:43:32.853 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7d9848db-2fe7-4d95-a6e2-b04f4f639ebc... 2024-03-24T10:43:32.854 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-24T10:43:32.895 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-24T10:43:32.943 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2024-03-24T10:43:32.943 INFO:teuthology.orchestra.run.smithi005.stdout: chdir = "" 2024-03-24T10:43:32.943 INFO:teuthology.orchestra.run.smithi005.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-24T10:43:32.944 INFO:teuthology.orchestra.run.smithi005.stdout: auth supported = cephx 2024-03-24T10:43:32.944 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.944 INFO:teuthology.orchestra.run.smithi005.stdout: filestore xattr use omap = true 2024-03-24T10:43:32.944 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.944 INFO:teuthology.orchestra.run.smithi005.stdout: mon clock drift allowed = 1.000 2024-03-24T10:43:32.944 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.944 INFO:teuthology.orchestra.run.smithi005.stdout: osd crush chooseleaf type = 0 2024-03-24T10:43:32.944 INFO:teuthology.orchestra.run.smithi005.stdout: auth debug = true 2024-03-24T10:43:32.944 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.944 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on old message = true 2024-03-24T10:43:32.944 INFO:teuthology.orchestra.run.smithi005.stdout: ms die on bug = true 2024-03-24T10:43:32.944 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.944 INFO:teuthology.orchestra.run.smithi005.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-24T10:43:32.944 INFO:teuthology.orchestra.run.smithi005.stdout: mon pg warn max object skew = 0 2024-03-24T10:43:32.944 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.945 INFO:teuthology.orchestra.run.smithi005.stdout: # disable pg_autoscaler by default for new pools 2024-03-24T10:43:32.945 INFO:teuthology.orchestra.run.smithi005.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-24T10:43:32.945 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.945 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default size = 2 2024-03-24T10:43:32.945 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.945 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow primary affinity = true 2024-03-24T10:43:32.945 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd allow pg remap = true 2024-03-24T10:43:32.945 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on legacy crush tunables = false 2024-03-24T10:43:32.945 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on crush straw calc version zero = false 2024-03-24T10:43:32.945 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on no sortbitwise = false 2024-03-24T10:43:32.945 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on osd down out interval zero = false 2024-03-24T10:43:32.945 INFO:teuthology.orchestra.run.smithi005.stdout: mon warn on too few osds = false 2024-03-24T10:43:32.945 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-24T10:43:32.945 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-24T10:43:32.945 INFO:teuthology.orchestra.run.smithi005.stdout: mon_allow_pool_size_one = true 2024-03-24T10:43:32.945 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.946 INFO:teuthology.orchestra.run.smithi005.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-24T10:43:32.946 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.946 INFO:teuthology.orchestra.run.smithi005.stdout: osd default data pool replay window = 5 2024-03-24T10:43:32.946 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.946 INFO:teuthology.orchestra.run.smithi005.stdout: mon allow pool delete = true 2024-03-24T10:43:32.946 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.946 INFO:teuthology.orchestra.run.smithi005.stdout: mon cluster log file level = debug 2024-03-24T10:43:32.946 INFO:teuthology.orchestra.run.smithi005.stdout: debug asserts on shutdown = true 2024-03-24T10:43:32.946 INFO:teuthology.orchestra.run.smithi005.stdout: mon health detail to clog = false 2024-03-24T10:43:32.946 INFO:teuthology.orchestra.run.smithi005.stdout: mon host = "172.21.15.5,172.21.15.57,[v2:172.21.15.57:3301,v1:172.21.15.57:6790]" 2024-03-24T10:43:32.946 INFO:teuthology.orchestra.run.smithi005.stdout: mds beacon grace = 60 2024-03-24T10:43:32.946 INFO:teuthology.orchestra.run.smithi005.stdout: mds heartbeat grace = 60 2024-03-24T10:43:32.946 INFO:teuthology.orchestra.run.smithi005.stdout: osd heartbeat grace = 60 2024-03-24T10:43:32.946 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = 7d9848db-2fe7-4d95-a6e2-b04f4f639ebc 2024-03-24T10:43:32.947 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.947 INFO:teuthology.orchestra.run.smithi005.stdout:[osd] 2024-03-24T10:43:32.947 INFO:teuthology.orchestra.run.smithi005.stdout: osd journal size = 100 2024-03-24T10:43:32.947 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.947 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub load threshold = 5.0 2024-03-24T10:43:32.947 INFO:teuthology.orchestra.run.smithi005.stdout: osd scrub max interval = 600 2024-03-24T10:43:32.947 INFO:teuthology.orchestra.run.smithi005.stdout: osd mclock profile = high_recovery_ops 2024-03-24T10:43:32.947 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.947 INFO:teuthology.orchestra.run.smithi005.stdout: osd recover clone overlap = true 2024-03-24T10:43:32.947 INFO:teuthology.orchestra.run.smithi005.stdout: osd recovery max chunk = 1048576 2024-03-24T10:43:32.947 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.947 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug shutdown = true 2024-03-24T10:43:32.947 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug op order = true 2024-03-24T10:43:32.947 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug verify stray on activate = true 2024-03-24T10:43:32.947 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug trim objects = true 2024-03-24T10:43:32.947 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.948 INFO:teuthology.orchestra.run.smithi005.stdout: osd open classes on start = true 2024-03-24T10:43:32.948 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug pg log writeout = true 2024-03-24T10:43:32.948 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.948 INFO:teuthology.orchestra.run.smithi005.stdout: osd deep scrub update digest min age = 30 2024-03-24T10:43:32.948 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.948 INFO:teuthology.orchestra.run.smithi005.stdout: osd map max advance = 10 2024-03-24T10:43:32.948 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.948 INFO:teuthology.orchestra.run.smithi005.stdout: journal zero on create = true 2024-03-24T10:43:32.948 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.948 INFO:teuthology.orchestra.run.smithi005.stdout: filestore ondisk finisher threads = 3 2024-03-24T10:43:32.948 INFO:teuthology.orchestra.run.smithi005.stdout: filestore apply finisher threads = 3 2024-03-24T10:43:32.948 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.948 INFO:teuthology.orchestra.run.smithi005.stdout: bdev debug aio = true 2024-03-24T10:43:32.948 INFO:teuthology.orchestra.run.smithi005.stdout: osd debug misdirected ops = true 2024-03-24T10:43:32.948 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore block size = 96636764160 2024-03-24T10:43:32.948 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore compression mode = aggressive 2024-03-24T10:43:32.948 INFO:teuthology.orchestra.run.smithi005.stdout: bluestore fsck on mount = True 2024-03-24T10:43:32.949 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluefs = 20 2024-03-24T10:43:32.949 INFO:teuthology.orchestra.run.smithi005.stdout: debug bluestore = 20 2024-03-24T10:43:32.949 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-03-24T10:43:32.949 INFO:teuthology.orchestra.run.smithi005.stdout: debug osd = 20 2024-03-24T10:43:32.949 INFO:teuthology.orchestra.run.smithi005.stdout: debug rocksdb = 10 2024-03-24T10:43:32.949 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd backfillfull_ratio = 0.85 2024-03-24T10:43:32.949 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd full ratio = 0.9 2024-03-24T10:43:32.949 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd nearfull ratio = 0.8 2024-03-24T10:43:32.949 INFO:teuthology.orchestra.run.smithi005.stdout: osd failsafe full ratio = 0.95 2024-03-24T10:43:32.949 INFO:teuthology.orchestra.run.smithi005.stdout: osd fast shutdown = False 2024-03-24T10:43:32.949 INFO:teuthology.orchestra.run.smithi005.stdout: osd objectstore = bluestore 2024-03-24T10:43:32.949 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2024-03-24T10:43:32.949 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.949 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr] 2024-03-24T10:43:32.949 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-03-24T10:43:32.950 INFO:teuthology.orchestra.run.smithi005.stdout: debug mgr = 20 2024-03-24T10:43:32.950 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2024-03-24T10:43:32.950 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2024-03-24T10:43:32.950 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2024-03-24T10:43:32.950 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2024-03-24T10:43:32.950 INFO:teuthology.orchestra.run.smithi005.stdout: mgr/telemetry/nag = false 2024-03-24T10:43:32.950 INFO:teuthology.orchestra.run.smithi005.stdout: debug client = 20 2024-03-24T10:43:32.950 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.950 INFO:teuthology.orchestra.run.smithi005.stdout:[mon] 2024-03-24T10:43:32.950 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-03-24T10:43:32.950 INFO:teuthology.orchestra.run.smithi005.stdout: debug mon = 20 2024-03-24T10:43:32.950 INFO:teuthology.orchestra.run.smithi005.stdout: debug paxos = 20 2024-03-24T10:43:32.950 INFO:teuthology.orchestra.run.smithi005.stdout: debug auth = 20 2024-03-24T10:43:32.950 INFO:teuthology.orchestra.run.smithi005.stdout: mon data avail warn = 5 2024-03-24T10:43:32.950 INFO:teuthology.orchestra.run.smithi005.stdout: mon mgr mkfs grace = 240 2024-03-24T10:43:32.951 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min pgs per osd = 4 2024-03-24T10:43:32.951 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd reporter subtree level = osd 2024-03-24T10:43:32.951 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd prime pg temp = true 2024-03-24T10:43:32.951 INFO:teuthology.orchestra.run.smithi005.stdout: mon reweight min bytes per osd = 10 2024-03-24T10:43:32.951 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.951 INFO:teuthology.orchestra.run.smithi005.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-24T10:43:32.951 INFO:teuthology.orchestra.run.smithi005.stdout: auth mon ticket ttl = 660 # 11m 2024-03-24T10:43:32.951 INFO:teuthology.orchestra.run.smithi005.stdout: auth service ticket ttl = 240 # 4m 2024-03-24T10:43:32.951 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.951 INFO:teuthology.orchestra.run.smithi005.stdout: # don't complain about insecure global_id in the test suite 2024-03-24T10:43:32.951 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-24T10:43:32.951 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-24T10:43:32.951 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.951 INFO:teuthology.orchestra.run.smithi005.stdout: # 1m isn't quite enough 2024-03-24T10:43:32.951 INFO:teuthology.orchestra.run.smithi005.stdout: mon_down_mkfs_grace = 2m 2024-03-24T10:43:32.951 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.952 INFO:teuthology.orchestra.run.smithi005.stdout: mon_warn_on_filestore_osds = false 2024-03-24T10:43:32.952 INFO:teuthology.orchestra.run.smithi005.stdout: mon down mkfs grace = 300 2024-03-24T10:43:32.952 INFO:teuthology.orchestra.run.smithi005.stdout: mon op complaint time = 120 2024-03-24T10:43:32.952 INFO:teuthology.orchestra.run.smithi005.stdout: mon osd crush smoke test = False 2024-03-24T10:43:32.952 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:43:32.952 INFO:teuthology.orchestra.run.smithi005.stdout:[client] 2024-03-24T10:43:32.952 INFO:teuthology.orchestra.run.smithi005.stdout: rgw cache enabled = true 2024-03-24T10:43:32.952 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable ops log = true 2024-03-24T10:43:32.952 INFO:teuthology.orchestra.run.smithi005.stdout: rgw enable usage log = true 2024-03-24T10:43:32.952 INFO:teuthology.orchestra.run.smithi005.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-24T10:43:32.952 INFO:teuthology.orchestra.run.smithi005.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-24T10:43:32.952 INFO:teuthology.orchestra.run.smithi005.stdout: client mount timeout = 600 2024-03-24T10:43:32.952 INFO:teuthology.orchestra.run.smithi005.stdout: debug client = 20 2024-03-24T10:43:32.952 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-03-24T10:43:32.952 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 900 2024-03-24T10:43:32.952 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 900 2024-03-24T10:43:32.953 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.a] 2024-03-24T10:43:32.953 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.b] 2024-03-24T10:43:32.953 INFO:teuthology.orchestra.run.smithi005.stdout:[mon.c] 2024-03-24T10:43:32.953 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.a] 2024-03-24T10:43:32.953 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.c] 2024-03-24T10:43:32.953 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.e] 2024-03-24T10:43:32.953 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2024-03-24T10:43:32.953 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.d] 2024-03-24T10:43:32.953 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.f] 2024-03-24T10:43:32.953 INFO:teuthology.orchestra.run.smithi005.stdout:[mds] 2024-03-24T10:43:32.953 INFO:teuthology.orchestra.run.smithi005.stdout: debug mds = 20 2024-03-24T10:43:32.953 INFO:teuthology.orchestra.run.smithi005.stdout: debug mds balancer = 20 2024-03-24T10:43:32.953 INFO:teuthology.orchestra.run.smithi005.stdout: debug ms = 1 2024-03-24T10:43:32.953 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug frag = True 2024-03-24T10:43:32.953 INFO:teuthology.orchestra.run.smithi005.stdout: mds debug scatterstat = True 2024-03-24T10:43:32.953 INFO:teuthology.orchestra.run.smithi005.stdout: mds op complaint time = 180 2024-03-24T10:43:32.953 INFO:teuthology.orchestra.run.smithi005.stdout: mds valgrind exit = True 2024-03-24T10:43:32.954 INFO:teuthology.orchestra.run.smithi005.stdout: mds verify scatter = True 2024-03-24T10:43:32.954 INFO:teuthology.orchestra.run.smithi005.stdout: osd op complaint time = 180 2024-03-24T10:43:32.954 INFO:teuthology.orchestra.run.smithi005.stdout: rados mon op timeout = 900 2024-03-24T10:43:32.954 INFO:teuthology.orchestra.run.smithi005.stdout: rados osd op timeout = 900 2024-03-24T10:43:32.954 INFO:teuthology.orchestra.run.smithi057.stdout:[global] 2024-03-24T10:43:32.954 INFO:teuthology.orchestra.run.smithi057.stdout: chdir = "" 2024-03-24T10:43:32.954 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-24T10:43:32.954 INFO:teuthology.orchestra.run.smithi057.stdout: auth supported = cephx 2024-03-24T10:43:32.954 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.954 INFO:teuthology.orchestra.run.smithi057.stdout: filestore xattr use omap = true 2024-03-24T10:43:32.955 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.955 INFO:teuthology.orchestra.run.smithi057.stdout: mon clock drift allowed = 1.000 2024-03-24T10:43:32.955 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.955 INFO:teuthology.orchestra.run.smithi057.stdout: osd crush chooseleaf type = 0 2024-03-24T10:43:32.955 INFO:teuthology.orchestra.run.smithi057.stdout: auth debug = true 2024-03-24T10:43:32.955 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.955 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on old message = true 2024-03-24T10:43:32.955 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on bug = true 2024-03-24T10:43:32.955 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.955 INFO:teuthology.orchestra.run.smithi057.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-24T10:43:32.955 INFO:teuthology.orchestra.run.smithi057.stdout: mon pg warn max object skew = 0 2024-03-24T10:43:32.955 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.955 INFO:teuthology.orchestra.run.smithi057.stdout: # disable pg_autoscaler by default for new pools 2024-03-24T10:43:32.955 INFO:teuthology.orchestra.run.smithi057.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-24T10:43:32.955 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.956 INFO:teuthology.orchestra.run.smithi057.stdout: osd pool default size = 2 2024-03-24T10:43:32.956 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.956 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd allow primary affinity = true 2024-03-24T10:43:32.956 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd allow pg remap = true 2024-03-24T10:43:32.956 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on legacy crush tunables = false 2024-03-24T10:43:32.956 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on crush straw calc version zero = false 2024-03-24T10:43:32.956 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on no sortbitwise = false 2024-03-24T10:43:32.956 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on osd down out interval zero = false 2024-03-24T10:43:32.956 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on too few osds = false 2024-03-24T10:43:32.956 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-24T10:43:32.956 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-24T10:43:32.956 INFO:teuthology.orchestra.run.smithi057.stdout: mon_allow_pool_size_one = true 2024-03-24T10:43:32.956 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.956 INFO:teuthology.orchestra.run.smithi057.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-24T10:43:32.956 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.956 INFO:teuthology.orchestra.run.smithi057.stdout: osd default data pool replay window = 5 2024-03-24T10:43:32.957 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.957 INFO:teuthology.orchestra.run.smithi057.stdout: mon allow pool delete = true 2024-03-24T10:43:32.957 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.957 INFO:teuthology.orchestra.run.smithi057.stdout: mon cluster log file level = debug 2024-03-24T10:43:32.957 INFO:teuthology.orchestra.run.smithi057.stdout: debug asserts on shutdown = true 2024-03-24T10:43:32.957 INFO:teuthology.orchestra.run.smithi057.stdout: mon health detail to clog = false 2024-03-24T10:43:32.957 INFO:teuthology.orchestra.run.smithi057.stdout: mon host = "172.21.15.5,172.21.15.57,[v2:172.21.15.57:3301,v1:172.21.15.57:6790]" 2024-03-24T10:43:32.957 INFO:teuthology.orchestra.run.smithi057.stdout: mds beacon grace = 60 2024-03-24T10:43:32.957 INFO:teuthology.orchestra.run.smithi057.stdout: mds heartbeat grace = 60 2024-03-24T10:43:32.957 INFO:teuthology.orchestra.run.smithi057.stdout: osd heartbeat grace = 60 2024-03-24T10:43:32.957 INFO:teuthology.orchestra.run.smithi057.stdout: fsid = 7d9848db-2fe7-4d95-a6e2-b04f4f639ebc 2024-03-24T10:43:32.957 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.957 INFO:teuthology.orchestra.run.smithi057.stdout:[osd] 2024-03-24T10:43:32.957 INFO:teuthology.orchestra.run.smithi057.stdout: osd journal size = 100 2024-03-24T10:43:32.957 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.957 INFO:teuthology.orchestra.run.smithi057.stdout: osd scrub load threshold = 5.0 2024-03-24T10:43:32.957 INFO:teuthology.orchestra.run.smithi057.stdout: osd scrub max interval = 600 2024-03-24T10:43:32.958 INFO:teuthology.orchestra.run.smithi057.stdout: osd mclock profile = high_recovery_ops 2024-03-24T10:43:32.958 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.958 INFO:teuthology.orchestra.run.smithi057.stdout: osd recover clone overlap = true 2024-03-24T10:43:32.958 INFO:teuthology.orchestra.run.smithi057.stdout: osd recovery max chunk = 1048576 2024-03-24T10:43:32.958 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.958 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug shutdown = true 2024-03-24T10:43:32.958 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug op order = true 2024-03-24T10:43:32.958 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug verify stray on activate = true 2024-03-24T10:43:32.958 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug trim objects = true 2024-03-24T10:43:32.958 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.958 INFO:teuthology.orchestra.run.smithi057.stdout: osd open classes on start = true 2024-03-24T10:43:32.958 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug pg log writeout = true 2024-03-24T10:43:32.958 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.958 INFO:teuthology.orchestra.run.smithi057.stdout: osd deep scrub update digest min age = 30 2024-03-24T10:43:32.958 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.958 INFO:teuthology.orchestra.run.smithi057.stdout: osd map max advance = 10 2024-03-24T10:43:32.959 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.959 INFO:teuthology.orchestra.run.smithi057.stdout: journal zero on create = true 2024-03-24T10:43:32.959 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.959 INFO:teuthology.orchestra.run.smithi057.stdout: filestore ondisk finisher threads = 3 2024-03-24T10:43:32.959 INFO:teuthology.orchestra.run.smithi057.stdout: filestore apply finisher threads = 3 2024-03-24T10:43:32.959 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.959 INFO:teuthology.orchestra.run.smithi057.stdout: bdev debug aio = true 2024-03-24T10:43:32.959 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug misdirected ops = true 2024-03-24T10:43:32.959 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore block size = 96636764160 2024-03-24T10:43:32.959 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore compression mode = aggressive 2024-03-24T10:43:32.959 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore fsck on mount = True 2024-03-24T10:43:32.959 INFO:teuthology.orchestra.run.smithi057.stdout: debug bluefs = 20 2024-03-24T10:43:32.959 INFO:teuthology.orchestra.run.smithi057.stdout: debug bluestore = 20 2024-03-24T10:43:32.959 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2024-03-24T10:43:32.959 INFO:teuthology.orchestra.run.smithi057.stdout: debug osd = 20 2024-03-24T10:43:32.959 INFO:teuthology.orchestra.run.smithi057.stdout: debug rocksdb = 10 2024-03-24T10:43:32.959 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd backfillfull_ratio = 0.85 2024-03-24T10:43:32.960 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd full ratio = 0.9 2024-03-24T10:43:32.960 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd nearfull ratio = 0.8 2024-03-24T10:43:32.960 INFO:teuthology.orchestra.run.smithi057.stdout: osd failsafe full ratio = 0.95 2024-03-24T10:43:32.960 INFO:teuthology.orchestra.run.smithi057.stdout: osd fast shutdown = False 2024-03-24T10:43:32.960 INFO:teuthology.orchestra.run.smithi057.stdout: osd objectstore = bluestore 2024-03-24T10:43:32.960 INFO:teuthology.orchestra.run.smithi057.stdout: osd op complaint time = 180 2024-03-24T10:43:32.960 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.960 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr] 2024-03-24T10:43:32.960 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2024-03-24T10:43:32.960 INFO:teuthology.orchestra.run.smithi057.stdout: debug mgr = 20 2024-03-24T10:43:32.960 INFO:teuthology.orchestra.run.smithi057.stdout: debug mon = 20 2024-03-24T10:43:32.960 INFO:teuthology.orchestra.run.smithi057.stdout: debug auth = 20 2024-03-24T10:43:32.960 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min pgs per osd = 4 2024-03-24T10:43:32.960 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min bytes per osd = 10 2024-03-24T10:43:32.960 INFO:teuthology.orchestra.run.smithi057.stdout: mgr/telemetry/nag = false 2024-03-24T10:43:32.960 INFO:teuthology.orchestra.run.smithi057.stdout: debug client = 20 2024-03-24T10:43:32.961 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.961 INFO:teuthology.orchestra.run.smithi057.stdout:[mon] 2024-03-24T10:43:32.961 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2024-03-24T10:43:32.961 INFO:teuthology.orchestra.run.smithi057.stdout: debug mon = 20 2024-03-24T10:43:32.961 INFO:teuthology.orchestra.run.smithi057.stdout: debug paxos = 20 2024-03-24T10:43:32.961 INFO:teuthology.orchestra.run.smithi057.stdout: debug auth = 20 2024-03-24T10:43:32.961 INFO:teuthology.orchestra.run.smithi057.stdout: mon data avail warn = 5 2024-03-24T10:43:32.961 INFO:teuthology.orchestra.run.smithi057.stdout: mon mgr mkfs grace = 240 2024-03-24T10:43:32.961 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min pgs per osd = 4 2024-03-24T10:43:32.961 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd reporter subtree level = osd 2024-03-24T10:43:32.961 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd prime pg temp = true 2024-03-24T10:43:32.961 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min bytes per osd = 10 2024-03-24T10:43:32.961 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.961 INFO:teuthology.orchestra.run.smithi057.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-24T10:43:32.961 INFO:teuthology.orchestra.run.smithi057.stdout: auth mon ticket ttl = 660 # 11m 2024-03-24T10:43:32.961 INFO:teuthology.orchestra.run.smithi057.stdout: auth service ticket ttl = 240 # 4m 2024-03-24T10:43:32.961 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.962 INFO:teuthology.orchestra.run.smithi057.stdout: # don't complain about insecure global_id in the test suite 2024-03-24T10:43:32.962 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-24T10:43:32.962 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-24T10:43:32.962 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.962 INFO:teuthology.orchestra.run.smithi057.stdout: # 1m isn't quite enough 2024-03-24T10:43:32.962 INFO:teuthology.orchestra.run.smithi057.stdout: mon_down_mkfs_grace = 2m 2024-03-24T10:43:32.962 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.962 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_filestore_osds = false 2024-03-24T10:43:32.962 INFO:teuthology.orchestra.run.smithi057.stdout: mon down mkfs grace = 300 2024-03-24T10:43:32.962 INFO:teuthology.orchestra.run.smithi057.stdout: mon op complaint time = 120 2024-03-24T10:43:32.962 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd crush smoke test = False 2024-03-24T10:43:32.962 INFO:teuthology.orchestra.run.smithi057.stdout: 2024-03-24T10:43:32.962 INFO:teuthology.orchestra.run.smithi057.stdout:[client] 2024-03-24T10:43:32.962 INFO:teuthology.orchestra.run.smithi057.stdout: rgw cache enabled = true 2024-03-24T10:43:32.962 INFO:teuthology.orchestra.run.smithi057.stdout: rgw enable ops log = true 2024-03-24T10:43:32.962 INFO:teuthology.orchestra.run.smithi057.stdout: rgw enable usage log = true 2024-03-24T10:43:32.963 INFO:teuthology.orchestra.run.smithi057.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-24T10:43:32.963 INFO:teuthology.orchestra.run.smithi057.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-24T10:43:32.963 INFO:teuthology.orchestra.run.smithi057.stdout: client mount timeout = 600 2024-03-24T10:43:32.963 INFO:teuthology.orchestra.run.smithi057.stdout: debug client = 20 2024-03-24T10:43:32.963 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2024-03-24T10:43:32.963 INFO:teuthology.orchestra.run.smithi057.stdout: rados mon op timeout = 900 2024-03-24T10:43:32.963 INFO:teuthology.orchestra.run.smithi057.stdout: rados osd op timeout = 900 2024-03-24T10:43:32.963 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.a] 2024-03-24T10:43:32.963 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.b] 2024-03-24T10:43:32.963 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.c] 2024-03-24T10:43:32.963 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.a] 2024-03-24T10:43:32.963 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.c] 2024-03-24T10:43:32.963 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.e] 2024-03-24T10:43:32.963 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.b] 2024-03-24T10:43:32.963 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.d] 2024-03-24T10:43:32.963 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.f] 2024-03-24T10:43:32.964 INFO:teuthology.orchestra.run.smithi057.stdout:[mds] 2024-03-24T10:43:32.964 INFO:teuthology.orchestra.run.smithi057.stdout: debug mds = 20 2024-03-24T10:43:32.964 INFO:teuthology.orchestra.run.smithi057.stdout: debug mds balancer = 20 2024-03-24T10:43:32.964 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2024-03-24T10:43:32.964 INFO:teuthology.orchestra.run.smithi057.stdout: mds debug frag = True 2024-03-24T10:43:32.964 INFO:teuthology.orchestra.run.smithi057.stdout: mds debug scatterstat = True 2024-03-24T10:43:32.964 INFO:teuthology.orchestra.run.smithi057.stdout: mds op complaint time = 180 2024-03-24T10:43:32.964 INFO:teuthology.orchestra.run.smithi057.stdout: mds valgrind exit = True 2024-03-24T10:43:32.964 INFO:teuthology.orchestra.run.smithi057.stdout: mds verify scatter = True 2024-03-24T10:43:32.964 INFO:teuthology.orchestra.run.smithi057.stdout: osd op complaint time = 180 2024-03-24T10:43:32.964 INFO:teuthology.orchestra.run.smithi057.stdout: rados mon op timeout = 900 2024-03-24T10:43:32.964 INFO:teuthology.orchestra.run.smithi057.stdout: rados osd op timeout = 900 2024-03-24T10:43:32.966 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-24T10:43:32.966 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-24T10:43:33.059 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-24T10:43:33.060 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:43:33.060 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-24T10:43:33.074 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:43:33.074 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-24T10:43:33.128 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi005.front.sepia.ceph.com 2024-03-24T10:43:33.129 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:43:33.129 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-24T10:43:33.129 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-24T10:43:33.207 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:43:33.207 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:43:33.262 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi057.front.sepia.ceph.com 2024-03-24T10:43:33.262 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:43:33.262 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-24T10:43:33.262 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-24T10:43:33.306 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:43:33.306 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:43:33.362 INFO:tasks.ceph:Setting up mon nodes... 2024-03-24T10:43:33.362 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-24T10:43:33.362 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-24T10:43:33.416 INFO:teuthology.orchestra.run.smithi005.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-24T10:43:33.418 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-24T10:43:33.497 INFO:teuthology.orchestra.run.smithi057.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-24T10:43:33.499 INFO:tasks.ceph:Setting up mds nodes... 2024-03-24T10:43:33.499 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-24T10:43:33.553 INFO:teuthology.orchestra.run.smithi005.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-24T10:43:33.556 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-24T10:43:33.583 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-24T10:43:33.676 INFO:teuthology.orchestra.run.smithi005.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-24T10:43:33.679 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-24T10:43:33.704 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-03-24T10:43:33.798 INFO:teuthology.orchestra.run.smithi005.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-03-24T10:43:33.801 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-03-24T10:43:33.827 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-24T10:43:33.887 INFO:teuthology.orchestra.run.smithi057.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-24T10:43:33.890 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-24T10:43:33.917 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-24T10:43:34.013 INFO:teuthology.orchestra.run.smithi057.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-24T10:43:34.015 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-24T10:43:34.042 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-03-24T10:43:34.137 INFO:teuthology.orchestra.run.smithi057.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-03-24T10:43:34.150 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-03-24T10:43:34.177 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-24T10:43:34.178 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T10:43:34.221 INFO:teuthology.orchestra.run.smithi005.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-24T10:43:34.237 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-24T10:43:34.237 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi005.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi057.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-24T10:43:34.237 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-24T10:43:34.303 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T10:43:34.303 INFO:tasks.ceph:role: osd.0 2024-03-24T10:43:34.303 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi005.front.sepia.ceph.com 2024-03-24T10:43:34.303 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-24T10:43:34.370 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T10:43:34.370 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T10:43:34.370 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T10:43:34.370 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T10:43:34.370 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T10:43:34.370 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-03-24T10:43:34.371 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T10:43:34.371 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T10:43:34.371 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T10:43:34.371 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T10:43:34.406 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2024-03-24T10:43:34.407 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-03-24T10:43:34.408 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-24T10:43:34.479 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-24T10:43:34.549 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-24T10:43:34.616 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T10:43:34.616 INFO:tasks.ceph:role: osd.1 2024-03-24T10:43:34.616 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi005.front.sepia.ceph.com 2024-03-24T10:43:34.616 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-24T10:43:34.682 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T10:43:34.682 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T10:43:34.682 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T10:43:34.682 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T10:43:34.682 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T10:43:34.682 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-03-24T10:43:34.682 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T10:43:34.683 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T10:43:34.683 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T10:43:34.683 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T10:43:34.717 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2024-03-24T10:43:34.719 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-03-24T10:43:34.719 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-24T10:43:34.800 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-24T10:43:34.871 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-24T10:43:34.937 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T10:43:34.937 INFO:tasks.ceph:role: osd.2 2024-03-24T10:43:34.938 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi005.front.sepia.ceph.com 2024-03-24T10:43:34.938 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-24T10:43:35.003 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T10:43:35.003 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T10:43:35.004 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T10:43:35.004 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T10:43:35.004 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T10:43:35.004 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-03-24T10:43:35.004 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T10:43:35.004 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T10:43:35.004 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T10:43:35.004 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T10:43:35.038 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2024-03-24T10:43:35.040 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-03-24T10:43:35.041 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-24T10:43:35.111 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-24T10:43:35.182 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-24T10:43:35.247 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T10:43:35.248 INFO:tasks.ceph:role: osd.3 2024-03-24T10:43:35.248 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi005.front.sepia.ceph.com 2024-03-24T10:43:35.248 DEBUG:teuthology.orchestra.run.smithi005:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-24T10:43:35.314 INFO:teuthology.orchestra.run.smithi005.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T10:43:35.314 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T10:43:35.314 INFO:teuthology.orchestra.run.smithi005.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T10:43:35.314 INFO:teuthology.orchestra.run.smithi005.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T10:43:35.314 INFO:teuthology.orchestra.run.smithi005.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T10:43:35.314 INFO:teuthology.orchestra.run.smithi005.stdout: = sunit=0 swidth=0 blks 2024-03-24T10:43:35.314 INFO:teuthology.orchestra.run.smithi005.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T10:43:35.314 INFO:teuthology.orchestra.run.smithi005.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T10:43:35.314 INFO:teuthology.orchestra.run.smithi005.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T10:43:35.314 INFO:teuthology.orchestra.run.smithi005.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T10:43:35.349 INFO:teuthology.orchestra.run.smithi005.stdout:Discarding blocks...Done. 2024-03-24T10:43:35.351 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi005.front.sepia.ceph.com -o noatime 2024-03-24T10:43:35.351 DEBUG:teuthology.orchestra.run.smithi005:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-24T10:43:35.421 DEBUG:teuthology.orchestra.run.smithi005:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-24T10:43:35.491 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:43:35.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:35.589+0000 7f421d41a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-24T10:43:35.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:35.589+0000 7f421d41a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-24T10:43:35.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:35.589+0000 7f421d41a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-24T10:43:35.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:35.589+0000 7f421d41a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-24T10:43:35.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:35.589+0000 7f421d41a640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-24T10:43:35.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:35.590+0000 7f421d41a640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-24T10:43:38.435 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T10:43:38.466 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:43:38.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:38.558+0000 7fd3945e1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-24T10:43:38.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:38.558+0000 7fd3945e1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-24T10:43:38.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:38.558+0000 7fd3945e1640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-24T10:43:38.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:38.558+0000 7fd3945e1640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-24T10:43:38.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:38.558+0000 7fd3945e1640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-24T10:43:38.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:38.558+0000 7fd3945e1640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-24T10:43:41.411 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-24T10:43:41.441 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:43:41.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:41.532+0000 7f836af36640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-24T10:43:41.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:41.532+0000 7f836af36640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-24T10:43:41.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:41.532+0000 7f836af36640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-24T10:43:41.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:41.532+0000 7f836af36640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-24T10:43:41.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:41.532+0000 7f836af36640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-24T10:43:41.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:41.533+0000 7f836af36640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-24T10:43:44.379 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-24T10:43:44.409 DEBUG:teuthology.orchestra.run.smithi005:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:43:44.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:44.503+0000 7fd60248b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-24T10:43:44.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:44.503+0000 7fd60248b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-24T10:43:44.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:44.503+0000 7fd60248b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-24T10:43:44.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:44.503+0000 7fd60248b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-24T10:43:44.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:44.503+0000 7fd60248b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-24T10:43:44.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:43:44.503+0000 7fd60248b640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-24T10:43:47.350 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-24T10:43:47.381 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-24T10:43:47.412 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T10:43:47.413 INFO:tasks.ceph:role: osd.4 2024-03-24T10:43:47.413 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi057.front.sepia.ceph.com 2024-03-24T10:43:47.413 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-24T10:43:47.512 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T10:43:47.513 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T10:43:47.513 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T10:43:47.513 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T10:43:47.513 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T10:43:47.513 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2024-03-24T10:43:47.513 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T10:43:47.513 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T10:43:47.513 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T10:43:47.513 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T10:43:48.871 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2024-03-24T10:43:48.873 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2024-03-24T10:43:48.873 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-24T10:43:48.907 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-24T10:43:48.980 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-24T10:43:49.047 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T10:43:49.047 INFO:tasks.ceph:role: osd.5 2024-03-24T10:43:49.047 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi057.front.sepia.ceph.com 2024-03-24T10:43:49.047 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-24T10:43:49.141 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T10:43:49.141 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T10:43:49.141 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T10:43:49.141 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T10:43:49.141 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T10:43:49.141 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2024-03-24T10:43:49.141 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T10:43:49.142 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T10:43:49.142 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T10:43:49.142 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T10:43:50.462 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2024-03-24T10:43:50.465 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2024-03-24T10:43:50.465 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-24T10:43:50.501 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-24T10:43:50.575 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-24T10:43:50.646 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T10:43:50.646 INFO:tasks.ceph:role: osd.6 2024-03-24T10:43:50.646 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi057.front.sepia.ceph.com 2024-03-24T10:43:50.646 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-24T10:43:50.743 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T10:43:50.743 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T10:43:50.743 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T10:43:50.743 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T10:43:50.743 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T10:43:50.743 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2024-03-24T10:43:50.744 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T10:43:50.744 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T10:43:50.744 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T10:43:50.744 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T10:43:52.108 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2024-03-24T10:43:52.111 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2024-03-24T10:43:52.111 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-24T10:43:52.152 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-24T10:43:52.226 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-24T10:43:52.295 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T10:43:52.295 INFO:tasks.ceph:role: osd.7 2024-03-24T10:43:52.295 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi057.front.sepia.ceph.com 2024-03-24T10:43:52.296 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-24T10:43:52.391 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T10:43:52.391 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T10:43:52.391 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T10:43:52.391 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-24T10:43:52.391 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T10:43:52.391 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2024-03-24T10:43:52.392 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T10:43:52.392 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-24T10:43:52.392 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T10:43:52.392 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T10:43:53.704 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2024-03-24T10:43:53.706 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2024-03-24T10:43:53.706 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-24T10:43:53.743 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-24T10:43:53.816 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:43:53.912 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:43:53.912+0000 7f8484810640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-24T10:43:53.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:43:53.913+0000 7f8484810640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-24T10:43:53.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:43:53.913+0000 7f8484810640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-24T10:43:53.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:43:53.913+0000 7f8484810640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-24T10:43:53.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:43:53.913+0000 7f8484810640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-24T10:43:53.914 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:43:53.913+0000 7f8484810640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-24T10:43:56.774 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-24T10:43:56.804 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:43:56.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:43:56.896+0000 7f20a1413640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-24T10:43:56.897 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:43:56.897+0000 7f20a1413640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-24T10:43:56.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:43:56.897+0000 7f20a1413640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-24T10:43:56.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:43:56.897+0000 7f20a1413640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-24T10:43:56.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:43:56.897+0000 7f20a1413640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-24T10:43:56.898 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:43:56.897+0000 7f20a1413640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-24T10:43:59.758 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T10:43:59.790 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:43:59.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:43:59.879+0000 7fb857c0e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-24T10:43:59.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:43:59.879+0000 7fb857c0e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-24T10:43:59.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:43:59.879+0000 7fb857c0e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-24T10:43:59.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:43:59.879+0000 7fb857c0e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-24T10:43:59.880 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:43:59.879+0000 7fb857c0e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-24T10:43:59.881 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:43:59.880+0000 7fb857c0e640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-24T10:44:00.957 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:44:00.960 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:44:02.748 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-24T10:44:02.778 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:44:02.868 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:02.868+0000 7f1507e11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-24T10:44:02.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:02.868+0000 7f1507e11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-24T10:44:02.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:02.868+0000 7f1507e11640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-24T10:44:02.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:02.868+0000 7f1507e11640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-24T10:44:02.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:02.868+0000 7f1507e11640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-24T10:44:02.869 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:02.869+0000 7f1507e11640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-24T10:44:05.738 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-24T10:44:05.770 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-24T10:44:05.770 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:44:05.770 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-24T10:44:05.802 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:44:05.802 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-24T10:44:05.871 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:44:05.872 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-24T10:44:05.941 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:44:05.941 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-03-24T10:44:06.010 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:44:06.011 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-24T10:44:06.079 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:44:06.079 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-24T10:44:06.146 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:44:06.146 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-24T10:44:06.214 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:44:06.214 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-24T10:44:06.280 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:44:06.280 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-24T10:44:06.312 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:44:06.312 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-24T10:44:06.382 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:44:06.382 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-24T10:44:06.452 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:44:06.452 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-03-24T10:44:06.521 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:44:06.521 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-24T10:44:06.589 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:44:06.589 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-24T10:44:06.657 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:44:06.657 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-24T10:44:06.724 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:44:06.724 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-24T10:44:06.791 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:44:06.791 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-24T10:44:06.807 INFO:tasks.ceph:Adding keys to all mons... 2024-03-24T10:44:06.807 DEBUG:teuthology.orchestra.run.smithi005:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-24T10:44:06.849 DEBUG:teuthology.orchestra.run.smithi057:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-24T10:44:06.874 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr.y] 2024-03-24T10:44:06.874 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDVAwBmHgLeGBAA20CetDSp5sqy8c4SDm1ujw== 2024-03-24T10:44:06.874 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.a] 2024-03-24T10:44:06.874 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDVAwBmlToTIRAAvzFy9VpwQ/HsxDJO4YE5ow== 2024-03-24T10:44:06.874 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.c] 2024-03-24T10:44:06.874 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDVAwBmo+ZiKBAAOv4mqiCtzCB5UayvKyT0Fw== 2024-03-24T10:44:06.874 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.e] 2024-03-24T10:44:06.875 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDVAwBmmEisLxAAXZSJbBuR7JnZI3NQerPiEQ== 2024-03-24T10:44:06.875 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.0] 2024-03-24T10:44:06.875 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDXAwBmXWQxIxAA7Sm2E+JFqNY1ZlGUVxkyBA== 2024-03-24T10:44:06.875 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.1] 2024-03-24T10:44:06.875 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDaAwBm609TIRAA3gQWs8SNSdF6A0eBVa4WlA== 2024-03-24T10:44:06.875 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.2] 2024-03-24T10:44:06.875 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDdAwBmZGjLHxAAfPCVuam0nE7I++va3n5u8Q== 2024-03-24T10:44:06.875 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.3] 2024-03-24T10:44:06.875 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDgAwBm4x8LHhAAgYN7mX46gBf3haaUhNK2oA== 2024-03-24T10:44:06.875 INFO:teuthology.orchestra.run.smithi005.stdout:[mgr.x] 2024-03-24T10:44:06.875 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDVAwBmpYOsHRAAQ+GSXgBrbUiUUYHXm3d/pw== 2024-03-24T10:44:06.875 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.b] 2024-03-24T10:44:06.875 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDVAwBmGeH5NBAAiwLjjv2rbfanTNwCJy/dXQ== 2024-03-24T10:44:06.875 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.d] 2024-03-24T10:44:06.876 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDWAwBmcNvaABAA8mwfXwsDWxdMKXzQ9TUzAA== 2024-03-24T10:44:06.876 INFO:teuthology.orchestra.run.smithi005.stdout:[mds.f] 2024-03-24T10:44:06.876 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDWAwBmDeE/CBAAPtz99hRUh0VRHBz67+SOlA== 2024-03-24T10:44:06.876 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.4] 2024-03-24T10:44:06.876 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDpAwBmsS52NhAAIoFGZc0ped/K6qjF+NNPOw== 2024-03-24T10:44:06.876 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.5] 2024-03-24T10:44:06.876 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDsAwBmjkeJNRAAD5lg81xx1DPWAvWClueFkQ== 2024-03-24T10:44:06.876 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.6] 2024-03-24T10:44:06.876 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDvAwBm4xCANBAAjwke9Pk03RLvNCVmYSSzzQ== 2024-03-24T10:44:06.876 INFO:teuthology.orchestra.run.smithi005.stdout:[osd.7] 2024-03-24T10:44:06.876 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDyAwBmrTLVMxAAr73A32/XZWunnM+XUPujWg== 2024-03-24T10:44:06.876 INFO:teuthology.orchestra.run.smithi005.stdout:[client.0] 2024-03-24T10:44:06.876 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDWAwBm7z9FDRAA1RBVZU/P7IwsR64SR9stmg== 2024-03-24T10:44:06.878 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr.y] 2024-03-24T10:44:06.878 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDVAwBmHgLeGBAA20CetDSp5sqy8c4SDm1ujw== 2024-03-24T10:44:06.878 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.a] 2024-03-24T10:44:06.878 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDVAwBmlToTIRAAvzFy9VpwQ/HsxDJO4YE5ow== 2024-03-24T10:44:06.878 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.c] 2024-03-24T10:44:06.878 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDVAwBmo+ZiKBAAOv4mqiCtzCB5UayvKyT0Fw== 2024-03-24T10:44:06.878 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.e] 2024-03-24T10:44:06.878 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDVAwBmmEisLxAAXZSJbBuR7JnZI3NQerPiEQ== 2024-03-24T10:44:06.878 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.0] 2024-03-24T10:44:06.878 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDXAwBmXWQxIxAA7Sm2E+JFqNY1ZlGUVxkyBA== 2024-03-24T10:44:06.878 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.1] 2024-03-24T10:44:06.879 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDaAwBm609TIRAA3gQWs8SNSdF6A0eBVa4WlA== 2024-03-24T10:44:06.879 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.2] 2024-03-24T10:44:06.879 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDdAwBmZGjLHxAAfPCVuam0nE7I++va3n5u8Q== 2024-03-24T10:44:06.879 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.3] 2024-03-24T10:44:06.879 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDgAwBm4x8LHhAAgYN7mX46gBf3haaUhNK2oA== 2024-03-24T10:44:06.886 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr.x] 2024-03-24T10:44:06.887 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDVAwBmpYOsHRAAQ+GSXgBrbUiUUYHXm3d/pw== 2024-03-24T10:44:06.887 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.b] 2024-03-24T10:44:06.887 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDVAwBmGeH5NBAAiwLjjv2rbfanTNwCJy/dXQ== 2024-03-24T10:44:06.887 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.d] 2024-03-24T10:44:06.887 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDWAwBmcNvaABAA8mwfXwsDWxdMKXzQ9TUzAA== 2024-03-24T10:44:06.887 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.f] 2024-03-24T10:44:06.887 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDWAwBmDeE/CBAAPtz99hRUh0VRHBz67+SOlA== 2024-03-24T10:44:06.887 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.4] 2024-03-24T10:44:06.887 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDpAwBmsS52NhAAIoFGZc0ped/K6qjF+NNPOw== 2024-03-24T10:44:06.887 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.5] 2024-03-24T10:44:06.887 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDsAwBmjkeJNRAAD5lg81xx1DPWAvWClueFkQ== 2024-03-24T10:44:06.888 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.6] 2024-03-24T10:44:06.888 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDvAwBm4xCANBAAjwke9Pk03RLvNCVmYSSzzQ== 2024-03-24T10:44:06.888 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.7] 2024-03-24T10:44:06.888 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDyAwBmrTLVMxAAr73A32/XZWunnM+XUPujWg== 2024-03-24T10:44:06.888 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2024-03-24T10:44:06.888 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQDWAwBm7z9FDRAA1RBVZU/P7IwsR64SR9stmg== 2024-03-24T10:44:06.888 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-24T10:44:06.921 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-24T10:44:06.973 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T10:44:06.976 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T10:44:07.030 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T10:44:07.032 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T10:44:07.086 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T10:44:07.088 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T10:44:07.141 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:44:07.143 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:44:07.194 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:44:07.196 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:44:07.247 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:44:07.250 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:44:07.300 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:44:07.303 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:44:07.354 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-24T10:44:07.356 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-24T10:44:07.408 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T10:44:07.410 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T10:44:07.464 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T10:44:07.466 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T10:44:07.520 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T10:44:07.522 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T10:44:07.576 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:44:07.578 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:44:07.629 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:44:07.631 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:44:07.682 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:44:07.685 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:44:07.736 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:44:07.738 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T10:44:07.789 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T10:44:07.791 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T10:44:07.845 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-24T10:44:07.845 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-24T10:44:07.871 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-24T10:44:07.967 DEBUG:teuthology.orchestra.run.smithi005:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-24T10:44:07.993 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-24T10:44:08.021 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-24T10:44:08.117 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-24T10:44:08.143 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-24T10:44:08.209 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-24T10:44:08.309 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-24T10:44:08.336 DEBUG:teuthology.orchestra.run.smithi005:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:44:08.338 DEBUG:teuthology.orchestra.run.smithi057:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-24T10:44:08.390 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-24T10:44:08.391 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-24T10:44:08.391 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-24T10:44:08.391 DEBUG:teuthology.orchestra.run.smithi005:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-24T10:44:08.393 INFO:tasks.ceph.mon.a:Started 2024-03-24T10:44:08.393 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-24T10:44:08.394 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-24T10:44:08.394 DEBUG:teuthology.orchestra.run.smithi057:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-24T10:44:08.433 INFO:tasks.ceph.mon.b:Started 2024-03-24T10:44:08.433 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-24T10:44:08.433 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-24T10:44:08.433 DEBUG:teuthology.orchestra.run.smithi057:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-24T10:44:08.435 INFO:tasks.ceph.mon.c:Started 2024-03-24T10:44:08.435 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-24T10:44:08.435 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-24T10:44:08.435 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-24T10:44:08.437 INFO:tasks.ceph.mgr.y:Started 2024-03-24T10:44:08.437 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-24T10:44:08.438 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-24T10:44:08.439 INFO:tasks.ceph.mgr.x:Started 2024-03-24T10:44:08.439 INFO:tasks.ceph:Setting crush tunables to default 2024-03-24T10:44:08.440 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph osd crush tunables default 2024-03-24T10:44:08.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:08.540+0000 7f7f0f50c640 1 Processor -- start 2024-03-24T10:44:08.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:08.540+0000 7f7f0f50c640 1 -- start start 2024-03-24T10:44:08.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:08.541+0000 7f7f0f50c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f080c86e0 0x7f7f080c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:08.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:08.541+0000 7f7f0f50c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7f080c9420 0x7f7f080c9870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:08.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:08.541+0000 7f7f0f50c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f080ca3a0 con 0x7f7f08154760 2024-03-24T10:44:08.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:08.541+0000 7f7f0f50c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f7f0815d840 con 0x7f7f080c9420 2024-03-24T10:44:08.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:08.541+0000 7f7f0f50c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7f080c4810 con 0x7f7f080c86e0 2024-03-24T10:44:08.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:08.541+0000 7f7f0d2a8640 1 -- >> v1:172.21.15.57:6789/0 conn(0x7f7f08154760 legacy=0x7f7f0815a9a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.57:6789/0 2024-03-24T10:44:08.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:08.541+0000 7f7f0daa9640 1 -- >> v2:172.21.15.57:3300/0 conn(0x7f7f080c9420 msgr2=0x7f7f080c9870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.57:3300/0 2024-03-24T10:44:08.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:08.541+0000 7f7f0caa7640 1 -- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f080c86e0 msgr2=0x7f7f080c42d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.57:3301/0 2024-03-24T10:44:08.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:08.541+0000 7f7f0caa7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f080c86e0 0x7f7f080c42d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:44:08.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:08.541+0000 7f7f0daa9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7f080c9420 0x7f7f080c9870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:44:08.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:08.742+0000 7f7f0d2a8640 1 -- >> v1:172.21.15.57:6789/0 conn(0x7f7f08154760 legacy=0x7f7f0815a9a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.57:6789/0 2024-03-24T10:44:08.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:08.742+0000 7f7f0caa7640 1 -- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f080c86e0 msgr2=0x7f7f080c42d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.57:3301/0 2024-03-24T10:44:08.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:08.742+0000 7f7f0daa9640 1 -- >> v2:172.21.15.57:3300/0 conn(0x7f7f080c9420 msgr2=0x7f7f080c9870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.57:3300/0 2024-03-24T10:44:08.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:08.742+0000 7f7f0caa7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f080c86e0 0x7f7f080c42d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:44:08.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:08.742+0000 7f7f0daa9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7f080c9420 0x7f7f080c9870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:44:09.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:09.143+0000 7f7f0d2a8640 1 -- >> v1:172.21.15.57:6789/0 conn(0x7f7f08154760 legacy=0x7f7f0815a9a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.57:6789/0 2024-03-24T10:44:09.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:09.143+0000 7f7f0caa7640 1 -- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f080c86e0 msgr2=0x7f7f080c42d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.57:3301/0 2024-03-24T10:44:09.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:09.143+0000 7f7f0daa9640 1 -- >> v2:172.21.15.57:3300/0 conn(0x7f7f080c9420 msgr2=0x7f7f080c9870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.57:3300/0 2024-03-24T10:44:09.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:09.143+0000 7f7f0caa7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f080c86e0 0x7f7f080c42d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T10:44:09.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:09.143+0000 7f7f0daa9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7f080c9420 0x7f7f080c9870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T10:44:09.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:09.944+0000 7f7f0d2a8640 1 -- >> v1:172.21.15.57:6789/0 conn(0x7f7f08154760 legacy=0x7f7f0815a9a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.57:6789/0 2024-03-24T10:44:09.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:09.944+0000 7f7f0daa9640 1 -- >> v2:172.21.15.57:3300/0 conn(0x7f7f080c9420 msgr2=0x7f7f080c9870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.57:3300/0 2024-03-24T10:44:09.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:09.944+0000 7f7f0caa7640 1 -- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f080c86e0 msgr2=0x7f7f080c42d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.57:3301/0 2024-03-24T10:44:09.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:09.944+0000 7f7f0daa9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7f080c9420 0x7f7f080c9870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T10:44:09.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:09.944+0000 7f7f0caa7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f080c86e0 0x7f7f080c42d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T10:44:11.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:11.541+0000 7f7efeffd640 1 -- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f080c86e0 msgr2=0x7f7f080c42d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T10:44:11.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:11.541+0000 7f7efeffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f080c86e0 0x7f7f080c42d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:11.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:11.541+0000 7f7efeffd640 1 -- >> v2:172.21.15.57:3300/0 conn(0x7f7f080c9420 msgr2=0x7f7f080c9870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T10:44:11.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:11.541+0000 7f7efeffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7f080c9420 0x7f7f080c9870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:11.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:11.541+0000 7f7efeffd640 1 -- >> v1:172.21.15.57:6789/0 conn(0x7f7f08154760 legacy=0x7f7f0815a9a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T10:44:11.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:11.541+0000 7f7efeffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7eec007400 0x7f7eec0098c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:11.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:11.541+0000 7f7efeffd640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f080ca680 con 0x7f7eec0036c0 2024-03-24T10:44:11.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:11.541+0000 7f7efeffd640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f7f080c4810 con 0x7f7eec007400 2024-03-24T10:44:11.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:11.541+0000 7f7f0d2a8640 1 -- >> v1:172.21.15.5:6789/0 conn(0x7f7eec0036c0 legacy=0x7f7eec005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.5:6789/0 2024-03-24T10:44:11.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:11.541+0000 7f7f0caa7640 1 -- >> v2:172.21.15.5:3300/0 conn(0x7f7eec007400 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3300/0 2024-03-24T10:44:11.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:11.541+0000 7f7f0caa7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7eec007400 0x7f7eec0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:44:11.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:11.742+0000 7f7f0d2a8640 1 -- >> v1:172.21.15.5:6789/0 conn(0x7f7eec0036c0 legacy=0x7f7eec005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.5:6789/0 2024-03-24T10:44:11.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:11.742+0000 7f7f0caa7640 1 -- >> v2:172.21.15.5:3300/0 conn(0x7f7eec007400 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3300/0 2024-03-24T10:44:11.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:11.742+0000 7f7f0caa7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7eec007400 0x7f7eec0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:44:12.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:12.143+0000 7f7f0d2a8640 1 -- >> v1:172.21.15.5:6789/0 conn(0x7f7eec0036c0 legacy=0x7f7eec005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.5:6789/0 2024-03-24T10:44:12.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:12.143+0000 7f7f0caa7640 1 -- >> v2:172.21.15.5:3300/0 conn(0x7f7eec007400 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3300/0 2024-03-24T10:44:12.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:12.143+0000 7f7f0caa7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7eec007400 0x7f7eec0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T10:44:12.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:12.944+0000 7f7f0d2a8640 1 -- >> v1:172.21.15.5:6789/0 conn(0x7f7eec0036c0 legacy=0x7f7eec005b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.5:6789/0 2024-03-24T10:44:12.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:12.944+0000 7f7f0caa7640 1 -- >> v2:172.21.15.5:3300/0 conn(0x7f7eec007400 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3300/0 2024-03-24T10:44:12.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:12.944+0000 7f7f0caa7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7eec007400 0x7f7eec0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T10:44:14.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:14.542+0000 7f7efeffd640 1 -- >> v2:172.21.15.5:3300/0 conn(0x7f7eec007400 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T10:44:14.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:14.542+0000 7f7efeffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7eec007400 0x7f7eec0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:14.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:14.542+0000 7f7efeffd640 1 -- >> v1:172.21.15.5:6789/0 conn(0x7f7eec0036c0 legacy=0x7f7eec005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T10:44:14.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:14.542+0000 7f7f0daa9640 1 -- reap_dead start 2024-03-24T10:44:14.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:14.542+0000 7f7efeffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:14.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:14.542+0000 7f7efeffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 0x7f7eec00d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:14.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:14.542+0000 7f7efeffd640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f080ca3a0 con 0x7f7eec009f70 2024-03-24T10:44:14.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:14.542+0000 7f7efeffd640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f7f080c4810 con 0x7f7eec00aa60 2024-03-24T10:44:14.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:14.542+0000 7f7efeffd640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7eec00e740 con 0x7f7eec005b40 2024-03-24T10:44:14.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:14.542+0000 7f7f0d2a8640 1 -- >> v1:172.21.15.5:6789/0 conn(0x7f7eec009f70 legacy=0x7f7eec00a350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.5:6789/0 2024-03-24T10:44:14.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:14.542+0000 7f7f0caa7640 1 -- >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3300/0 2024-03-24T10:44:14.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:14.542+0000 7f7f0caa7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:44:14.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:14.542+0000 7f7f0daa9640 1 -- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 msgr2=0x7f7eec00d1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.57:3301/0 2024-03-24T10:44:14.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:14.542+0000 7f7f0daa9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 0x7f7eec00d1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:44:14.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:14.743+0000 7f7f0d2a8640 1 -- >> v1:172.21.15.5:6789/0 conn(0x7f7eec009f70 legacy=0x7f7eec00a350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.5:6789/0 2024-03-24T10:44:14.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:14.743+0000 7f7f0caa7640 1 -- >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3300/0 2024-03-24T10:44:14.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:14.743+0000 7f7f0caa7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:44:14.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:14.743+0000 7f7f0daa9640 1 -- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 msgr2=0x7f7eec00d1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.57:3301/0 2024-03-24T10:44:14.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:14.743+0000 7f7f0daa9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 0x7f7eec00d1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:44:15.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:15.144+0000 7f7f0d2a8640 1 -- >> v1:172.21.15.5:6789/0 conn(0x7f7eec009f70 legacy=0x7f7eec00a350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.5:6789/0 2024-03-24T10:44:15.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:15.144+0000 7f7f0caa7640 1 -- >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3300/0 2024-03-24T10:44:15.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:15.144+0000 7f7f0caa7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T10:44:15.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:15.144+0000 7f7f0daa9640 1 -- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 msgr2=0x7f7eec00d1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.57:3301/0 2024-03-24T10:44:15.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:15.144+0000 7f7f0daa9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 0x7f7eec00d1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T10:44:15.757 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:00:07.258 36946== Warning: unimplemented fcntl command: 1036 2024-03-24T10:44:15.843 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:00:07.307 36958== Warning: unimplemented fcntl command: 1036 2024-03-24T10:44:15.846 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:00:07.310 36957== Warning: unimplemented fcntl command: 1036 2024-03-24T10:44:15.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:15.945+0000 7f7f0d2a8640 1 -- >> v1:172.21.15.5:6789/0 conn(0x7f7eec009f70 legacy=0x7f7eec00a350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.5:6789/0 2024-03-24T10:44:15.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:15.945+0000 7f7f0caa7640 1 -- >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.5:3300/0 2024-03-24T10:44:15.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:15.945+0000 7f7f0caa7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T10:44:15.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:15.945+0000 7f7f0daa9640 1 -- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 msgr2=0x7f7eec00d1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.57:3301/0 2024-03-24T10:44:15.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:15.945+0000 7f7f0daa9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 0x7f7eec00d1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T10:44:16.084 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:00:07.585 36946== Warning: unimplemented fcntl command: 1036 2024-03-24T10:44:16.171 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:00:07.635 36958== Warning: unimplemented fcntl command: 1036 2024-03-24T10:44:16.173 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:00:07.637 36957== Warning: unimplemented fcntl command: 1036 2024-03-24T10:44:17.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:17.542+0000 7f7efeffd640 1 -- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 msgr2=0x7f7eec00d1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T10:44:17.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:17.542+0000 7f7efeffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 0x7f7eec00d1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:17.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:17.542+0000 7f7efeffd640 1 -- >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T10:44:17.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:17.542+0000 7f7efeffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:17.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:17.542+0000 7f7efeffd640 1 -- >> v1:172.21.15.5:6789/0 conn(0x7f7eec009f70 legacy=0x7f7eec00a350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T10:44:17.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:17.542+0000 7f7efeffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 0x7f7eec00db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:17.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:17.542+0000 7f7efeffd640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f080ca680 con 0x7f7eec00e240 2024-03-24T10:44:17.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:17.542+0000 7f7efeffd640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f7f080c4810 con 0x7f7eec00d760 2024-03-24T10:44:17.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:17.712+0000 7f7f0d2a8640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 0x7f7eec00db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:17.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:17.720+0000 7f7f0caa7640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f7eec00e240 0x7f7eec0129a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:48810/0 (socket says 172.21.15.5:48810) 2024-03-24T10:44:17.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:17.720+0000 7f7f0caa7640 1 -- 172.21.15.5:0/519364642 learned_addr learned my addr 172.21.15.5:0/519364642 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:44:17.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:17.944+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 msgr2=0x7f7eec00db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:17.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:17.944+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 msgr2=0x7f7eec00db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:17.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:17.944+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 0x7f7eec00db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:17.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:17.944+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 0x7f7eec00db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:44:18.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:18.048+0000 7f7f0caa7640 1 -- 172.21.15.5:0/519364642 >> v1:172.21.15.57:6789/0 conn(0x7f7eec00e240 legacy=0x7f7eec0129a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-24T10:44:18.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:18.048+0000 7f7f0caa7640 1 -- 172.21.15.5:0/519364642 >> v1:172.21.15.57:6789/0 conn(0x7f7eec00e240 legacy=0x7f7eec0129a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T10:44:18.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:18.048+0000 7f7f0caa7640 1 --1- 172.21.15.5:0/519364642 >> v1:172.21.15.57:6789/0 conn(0x7f7eec00e240 0x7f7eec0129a0 :-1 s=OPENED pgs=1 cs=1 l=1).handle_message read tag failed 2024-03-24T10:44:18.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:18.048+0000 7f7f0caa7640 1 --1- 172.21.15.5:0/519364642 >> v1:172.21.15.57:6789/0 conn(0x7f7eec00e240 0x7f7eec0129a0 :-1 s=OPENED pgs=1 cs=1 l=1).fault on lossy channel, failing 2024-03-24T10:44:18.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:18.191+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 0x7f7eec00db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:18.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:18.495+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 msgr2=0x7f7eec00db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:18.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:18.495+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 msgr2=0x7f7eec00db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:18.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:18.495+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 0x7f7eec00db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:18.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:18.495+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 0x7f7eec00db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:44:18.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:18.963+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 0x7f7eec00db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:18.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:18.994+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 msgr2=0x7f7eec00db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:18.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:18.994+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 msgr2=0x7f7eec00db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:18.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:18.994+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 0x7f7eec00db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:18.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:18.994+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 0x7f7eec00db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T10:44:19.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:19.796+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 0x7f7eec00db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:19.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:19.798+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 msgr2=0x7f7eec00db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:19.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:19.798+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 msgr2=0x7f7eec00db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:19.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:19.798+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 0x7f7eec00db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:19.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:19.798+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 0x7f7eec00db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T10:44:20.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.542+0000 7f7efeffd640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 msgr2=0x7f7eec00db70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T10:44:20.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.542+0000 7f7efeffd640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00d760 0x7f7eec00db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:20.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.542+0000 7f7efeffd640 1 -- 172.21.15.5:0/519364642 >> v1:172.21.15.57:6789/0 conn(0x7f7eec00e240 legacy=0x7f7eec0129a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-24T10:44:20.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.542+0000 7f7f0daa9640 1 -- 172.21.15.5:0/519364642 reap_dead start 2024-03-24T10:44:20.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.542+0000 7f7efeffd640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:20.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.542+0000 7f7efeffd640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:20.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.543+0000 7f7efeffd640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 0x7f7eec00d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:20.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.543+0000 7f7efeffd640 1 -- 172.21.15.5:0/519364642 --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f7f080c4810 con 0x7f7eec00aa60 2024-03-24T10:44:20.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.543+0000 7f7efeffd640 1 -- 172.21.15.5:0/519364642 --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f7eec00e740 con 0x7f7eec00a010 2024-03-24T10:44:20.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.543+0000 7f7efeffd640 1 -- 172.21.15.5:0/519364642 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7eec00a560 con 0x7f7eec005b40 2024-03-24T10:44:20.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.543+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:20.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.543+0000 7f7f0daa9640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 0x7f7eec00d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:20.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.543+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:20.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.544+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:20.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.544+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:20.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.544+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:20.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.544+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:44:20.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.545+0000 7f7f0daa9640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 msgr2=0x7f7eec00d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-24T10:44:20.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.545+0000 7f7f0daa9640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 msgr2=0x7f7eec00d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:20.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.545+0000 7f7f0daa9640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 0x7f7eec00d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:20.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.545+0000 7f7f0daa9640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 0x7f7eec00d1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:44:20.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.554+0000 7f7f0caa7640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 msgr2=0x7f7eec00db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-24T10:44:20.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.554+0000 7f7f0caa7640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 msgr2=0x7f7eec00db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:20.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.554+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:20.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.554+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:44:20.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.745+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:20.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.746+0000 7f7f0daa9640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 0x7f7eec00d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:20.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.747+0000 7f7f0daa9640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 msgr2=0x7f7eec00d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-24T10:44:20.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.747+0000 7f7f0daa9640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 msgr2=0x7f7eec00d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:20.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.748+0000 7f7f0daa9640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 0x7f7eec00d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:20.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.748+0000 7f7f0daa9640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 0x7f7eec00d1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:44:20.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.748+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:20.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.748+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:20.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.748+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:20.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.748+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:44:20.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.756+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:20.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.757+0000 7f7f0caa7640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 msgr2=0x7f7eec00db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:20.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.757+0000 7f7f0caa7640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 msgr2=0x7f7eec00db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:20.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.757+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:20.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:20.757+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:44:21.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.149+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:21.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.149+0000 7f7f0daa9640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 0x7f7eec00d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:21.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.150+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:21.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.150+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:21.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.150+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:21.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.150+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T10:44:21.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.151+0000 7f7f0daa9640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 msgr2=0x7f7eec00d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-24T10:44:21.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.151+0000 7f7f0daa9640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 msgr2=0x7f7eec00d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:21.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.151+0000 7f7f0daa9640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 0x7f7eec00d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:21.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.151+0000 7f7f0daa9640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 0x7f7eec00d1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T10:44:21.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.158+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:21.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.159+0000 7f7f0caa7640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 msgr2=0x7f7eec00db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:21.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.159+0000 7f7f0caa7640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 msgr2=0x7f7eec00db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:21.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.159+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:21.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.159+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T10:44:21.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.952+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:21.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.952+0000 7f7f0daa9640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 0x7f7eec00d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:21.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.953+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:21.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.953+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:21.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.953+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:21.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.953+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T10:44:21.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.954+0000 7f7f0daa9640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 msgr2=0x7f7eec00d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-24T10:44:21.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.954+0000 7f7f0daa9640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 msgr2=0x7f7eec00d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:21.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.954+0000 7f7f0daa9640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 0x7f7eec00d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:21.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.954+0000 7f7f0daa9640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 0x7f7eec00d1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T10:44:21.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.961+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:21.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.962+0000 7f7f0caa7640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 msgr2=0x7f7eec00db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:21.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.962+0000 7f7f0caa7640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 msgr2=0x7f7eec00db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:21.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.962+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:21.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:21.962+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T10:44:23.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:23.543+0000 7f7efeffd640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 msgr2=0x7f7eec00db40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T10:44:23.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:23.543+0000 7f7efeffd640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:23.543+0000 7f7efeffd640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 msgr2=0x7f7eec00d1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T10:44:23.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:23.543+0000 7f7efeffd640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec00a010 0x7f7eec00d1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:23.543+0000 7f7efeffd640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T10:44:23.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:23.543+0000 7f7efeffd640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:23.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:23.543+0000 7f7efeffd640 1 -- 172.21.15.5:0/519364642 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f080ca3a0 con 0x7f7eec006d60 2024-03-24T10:44:23.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:23.543+0000 7f7efeffd640 1 -- 172.21.15.5:0/519364642 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eec00a8b0 con 0x7f7eec006270 2024-03-24T10:44:26.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.543+0000 7f7efeffd640 1 -- 172.21.15.5:0/519364642 >> v1:172.21.15.57:6789/0 conn(0x7f7eec006270 legacy=0x7f7eec006650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:26.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.543+0000 7f7efeffd640 1 -- 172.21.15.5:0/519364642 >> v1:172.21.15.5:6789/0 conn(0x7f7eec006d60 legacy=0x7f7eec0139b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:26.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.543+0000 7f7efeffd640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:26.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.543+0000 7f7f0daa9640 1 -- 172.21.15.5:0/519364642 reap_dead start 2024-03-24T10:44:26.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.543+0000 7f7efeffd640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:26.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.543+0000 7f7efeffd640 1 -- 172.21.15.5:0/519364642 --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eec0171d0 con 0x7f7eec00a010 2024-03-24T10:44:26.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.543+0000 7f7efeffd640 1 -- 172.21.15.5:0/519364642 --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f7f080c4810 con 0x7f7eec00aa60 2024-03-24T10:44:26.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.543+0000 7f7efeffd640 1 -- 172.21.15.5:0/519364642 --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f7eec00e740 con 0x7f7eec005b40 2024-03-24T10:44:26.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.545+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:26.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.545+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:26.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.546+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:26.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.546+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:26.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.546+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:26.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.546+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:44:26.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.548+0000 7f7f0caa7640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 msgr2=0x7f7eec00db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-24T10:44:26.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.548+0000 7f7f0caa7640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 msgr2=0x7f7eec00db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:26.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.548+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:26.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.548+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:44:26.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.747+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:26.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.749+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:26.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.749+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:26.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.749+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:26.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.749+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:44:26.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.749+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:26.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.751+0000 7f7f0caa7640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 msgr2=0x7f7eec00db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-24T10:44:26.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.751+0000 7f7f0caa7640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 msgr2=0x7f7eec00db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:26.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.751+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:26.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:26.751+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:44:27.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:27.150+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:27.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:27.151+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:27.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:27.151+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:27.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:27.151+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:27.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:27.151+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T10:44:27.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:27.152+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:27.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:27.154+0000 7f7f0caa7640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 msgr2=0x7f7eec00db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:27.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:27.154+0000 7f7f0caa7640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 msgr2=0x7f7eec00db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:27.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:27.154+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:27.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:27.154+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T10:44:27.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:27.953+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:27.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:27.954+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:27.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:27.954+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:27.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:27.954+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:27.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:27.954+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T10:44:27.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:27.955+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:27.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:27.957+0000 7f7f0caa7640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 msgr2=0x7f7eec00db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:27.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:27.957+0000 7f7f0caa7640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 msgr2=0x7f7eec00db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:27.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:27.957+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:27.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:27.957+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T10:44:29.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:29.544+0000 7f7efeffd640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 msgr2=0x7f7eec00db40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T10:44:29.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:29.544+0000 7f7efeffd640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.57:3300/0 conn(0x7f7eec005b40 0x7f7eec00db40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:29.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:29.544+0000 7f7efeffd640 1 -- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 msgr2=0x7f7eec0098c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T10:44:29.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:29.544+0000 7f7efeffd640 1 --2- 172.21.15.5:0/519364642 >> v2:172.21.15.5:3300/0 conn(0x7f7eec00aa60 0x7f7eec0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:29.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:29.544+0000 7f7efeffd640 1 -- 172.21.15.5:0/519364642 >> v1:172.21.15.57:6789/0 conn(0x7f7eec00a010 legacy=0x7f7eec006270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:29.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:29.544+0000 7f7efeffd640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 0x7f7eec00c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:29.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:29.544+0000 7f7efeffd640 1 -- 172.21.15.5:0/519364642 --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eec006980 con 0x7f7eec00cb10 2024-03-24T10:44:29.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:29.544+0000 7f7efeffd640 1 -- 172.21.15.5:0/519364642 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7f080c4810 con 0x7f7eec00c1f0 2024-03-24T10:44:29.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:29.544+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 0x7f7eec00c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:29.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:29.546+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 msgr2=0x7f7eec00c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:29.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:29.546+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 msgr2=0x7f7eec00c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:29.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:29.546+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 0x7f7eec00c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:29.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:29.546+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 0x7f7eec00c5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:44:29.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:29.747+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 0x7f7eec00c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:29.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:29.748+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 msgr2=0x7f7eec00c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:29.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:29.748+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 msgr2=0x7f7eec00c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:29.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:29.748+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 0x7f7eec00c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:29.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:29.749+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 0x7f7eec00c5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:44:30.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:30.150+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 0x7f7eec00c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:30.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:30.152+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 msgr2=0x7f7eec00c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:30.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:30.152+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 msgr2=0x7f7eec00c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:30.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:30.152+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 0x7f7eec00c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:30.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:30.152+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 0x7f7eec00c5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T10:44:30.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:30.955+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 0x7f7eec00c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:30.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:30.991+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 msgr2=0x7f7eec00c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T10:44:30.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:30.991+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 msgr2=0x7f7eec00c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T10:44:30.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:30.991+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 0x7f7eec00c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T10:44:30.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:30.991+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 0x7f7eec00c5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T10:44:31.047 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:44:31.049 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:44:31.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.308+0000 7f7efffff640 1 -- 172.21.15.5:0/519364642 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4258225907 0 0) 0x7f7eec006980 con 0x7f7eec00cb10 2024-03-24T10:44:31.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.308+0000 7f7efffff640 1 -- 172.21.15.5:0/519364642 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ef4003520 con 0x7f7eec00cb10 2024-03-24T10:44:31.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.354+0000 7f7efffff640 1 -- 172.21.15.5:0/519364642 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3039649112 0 0) 0x7f7ef4003520 con 0x7f7eec00cb10 2024-03-24T10:44:31.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.354+0000 7f7efffff640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 msgr2=0x7f7eec00c5d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T10:44:31.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.354+0000 7f7efffff640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7eec00c1f0 0x7f7eec00c5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.354+0000 7f7efffff640 1 -- 172.21.15.5:0/519364642 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eec014a50 con 0x7f7eec00cb10 2024-03-24T10:44:31.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.362+0000 7f7efffff640 1 -- 172.21.15.5:0/519364642 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 100681612 0 0) 0x7f7ef8014550 con 0x7f7eec00cb10 2024-03-24T10:44:31.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.362+0000 7f7efffff640 1 -- 172.21.15.5:0/519364642 >> v1:172.21.15.5:6789/0 conn(0x7f7eec00cb10 legacy=0x7f7eec00cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.362+0000 7f7f0daa9640 1 -- 172.21.15.5:0/519364642 reap_dead start 2024-03-24T10:44:31.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.362+0000 7f7efffff640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ef4003970 0x7f7ef4003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.362+0000 7f7efffff640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7ef40047c0 0x7f7ef4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.362+0000 7f7efffff640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ef40051d0 0x7f7ef400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.362+0000 7f7efffff640 1 -- 172.21.15.5:0/519364642 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f7f080c4810 con 0x7f7ef40051d0 2024-03-24T10:44:31.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.362+0000 7f7efffff640 1 -- 172.21.15.5:0/519364642 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef400c100 con 0x7f7ef4003970 2024-03-24T10:44:31.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.362+0000 7f7efffff640 1 -- 172.21.15.5:0/519364642 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7ef400c2c0 con 0x7f7ef40047c0 2024-03-24T10:44:31.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.363+0000 7f7f0caa7640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7ef40047c0 0x7f7ef4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.363+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ef4003970 0x7f7ef4003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.366+0000 7f7f0daa9640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ef40051d0 0x7f7ef400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.384+0000 7f7f0daa9640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7ef40047c0 msgr2=0x7f7ef4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:31.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.384+0000 7f7f0daa9640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7ef40047c0 0x7f7ef4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.384+0000 7f7f0daa9640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ef4003970 msgr2=0x7f7ef4003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:31.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.384+0000 7f7f0daa9640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ef4003970 0x7f7ef4003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.384+0000 7f7f0daa9640 1 -- 172.21.15.5:0/519364642 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef4003520 con 0x7f7ef40051d0 2024-03-24T10:44:31.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.392+0000 7f7f0daa9640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ef40051d0 0x7f7ef400b980 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7f080711a0 tx=0x7f7ef40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728d226b556ab352 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:31.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.424+0000 7f7efffff640 1 -- 172.21.15.5:0/519364642 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0000f2d0 con 0x7f7ef40051d0 2024-03-24T10:44:31.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.424+0000 7f7efffff640 1 -- 172.21.15.5:0/519364642 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f0002f050 con 0x7f7ef40051d0 2024-03-24T10:44:31.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.424+0000 7f7efffff640 1 -- 172.21.15.5:0/519364642 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0000d650 con 0x7f7ef40051d0 2024-03-24T10:44:31.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.425+0000 7f7f0f50c640 1 -- 172.21.15.5:0/519364642 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ef40051d0 msgr2=0x7f7ef400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.425+0000 7f7f0f50c640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ef40051d0 0x7f7ef400b980 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7f080711a0 tx=0x7f7ef40057f0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.425+0000 7f7f0f50c640 1 -- 172.21.15.5:0/519364642 shutdown_connections 2024-03-24T10:44:31.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.425+0000 7f7f0f50c640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ef40051d0 0x7f7ef400b980 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.425+0000 7f7f0f50c640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7ef40047c0 0x7f7ef4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.425+0000 7f7f0f50c640 1 --2- 172.21.15.5:0/519364642 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ef4003970 0x7f7ef4003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.425+0000 7f7f0f50c640 1 -- 172.21.15.5:0/519364642 >> 172.21.15.5:0/519364642 conn(0x7f7f080711f0 msgr2=0x7f7f080715d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:31.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.425+0000 7f7f0f50c640 1 -- 172.21.15.5:0/519364642 shutdown_connections 2024-03-24T10:44:31.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.425+0000 7f7f0f50c640 1 -- 172.21.15.5:0/519364642 wait complete. 2024-03-24T10:44:31.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.427+0000 7f7f0f50c640 1 Processor -- start 2024-03-24T10:44:31.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.427+0000 7f7f0f50c640 1 -- start start 2024-03-24T10:44:31.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.429+0000 7f7f0f50c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ef4003970 0x7f7f080ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.429+0000 7f7f0f50c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ef40047c0 0x7f7f0815cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.429+0000 7f7f0f50c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7ef40051d0 0x7f7f080afab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:31.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.429+0000 7f7f0f50c640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f7f080b37d0 con 0x7f7ef4003970 2024-03-24T10:44:31.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.429+0000 7f7f0f50c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7f080adf40 con 0x7f7ef40047c0 2024-03-24T10:44:31.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.429+0000 7f7f0f50c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7f080ae220 con 0x7f7ef40051d0 2024-03-24T10:44:31.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.429+0000 7f7f0daa9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7ef40051d0 0x7f7f080afab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.429+0000 7f7f0caa7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ef40047c0 0x7f7f0815cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.429+0000 7f7f0daa9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7ef40051d0 0x7f7f080afab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:39720/0 (socket says 172.21.15.5:39720) 2024-03-24T10:44:31.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.429+0000 7f7f0daa9640 1 -- 172.21.15.5:0/164590834 learned_addr learned my addr 172.21.15.5:0/164590834 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:31.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.430+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/164590834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ef4003970 0x7f7f080ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:31.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.441+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/164590834 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7ef40051d0 msgr2=0x7f7f080afab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.441+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/164590834 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7ef40051d0 0x7f7f080afab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.441+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/164590834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ef40047c0 msgr2=0x7f7f0815cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.441+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/164590834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ef40047c0 0x7f7f0815cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.442+0000 7f7f0d2a8640 1 -- 172.21.15.5:0/164590834 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f080ae500 con 0x7f7ef4003970 2024-03-24T10:44:31.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.444+0000 7f7f0d2a8640 1 --2- 172.21.15.5:0/164590834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ef4003970 0x7f7f080ad8b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7f04003930 tx=0x7f7f04014780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:31.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.446+0000 7f7efdffb640 1 -- 172.21.15.5:0/164590834 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0400b040 con 0x7f7ef4003970 2024-03-24T10:44:31.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.446+0000 7f7f0f50c640 1 -- 172.21.15.5:0/164590834 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f080ae7e0 con 0x7f7ef4003970 2024-03-24T10:44:31.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.447+0000 7f7f0f50c640 1 -- 172.21.15.5:0/164590834 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f080af590 con 0x7f7ef4003970 2024-03-24T10:44:31.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.448+0000 7f7f0f50c640 1 -- 172.21.15.5:0/164590834 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f080c9ee0 con 0x7f7ef4003970 2024-03-24T10:44:31.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.637+0000 7f7efdffb640 1 -- 172.21.15.5:0/164590834 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f0401c040 con 0x7f7ef4003970 2024-03-24T10:44:31.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.637+0000 7f7efdffb640 1 -- 172.21.15.5:0/164590834 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f04007ea0 con 0x7f7ef4003970 2024-03-24T10:44:31.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.637+0000 7f7efdffb640 1 -- 172.21.15.5:0/164590834 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7f0402a660 con 0x7f7ef4003970 2024-03-24T10:44:31.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.637+0000 7f7efdffb640 1 -- 172.21.15.5:0/164590834 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7f04020db0 con 0x7f7ef4003970 2024-03-24T10:44:31.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.637+0000 7f7efdffb640 1 -- 172.21.15.5:0/164590834 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7f04025070 con 0x7f7ef4003970 2024-03-24T10:44:31.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.704+0000 7f7f0f50c640 1 -- 172.21.15.5:0/164590834 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f7f080688e0 con 0x7f7ef4003970 2024-03-24T10:44:31.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.820+0000 7f7efdffb640 1 -- 172.21.15.5:0/164590834 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f7f040203f0 con 0x7f7ef4003970 2024-03-24T10:44:31.821 INFO:teuthology.orchestra.run.smithi005.stderr:adjusted tunables profile to default 2024-03-24T10:44:31.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.821+0000 7f7f0f50c640 1 -- 172.21.15.5:0/164590834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ef4003970 msgr2=0x7f7f080ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:31.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.821+0000 7f7f0f50c640 1 --2- 172.21.15.5:0/164590834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ef4003970 0x7f7f080ad8b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7f04003930 tx=0x7f7f04014780 comp rx=0 tx=0).stop 2024-03-24T10:44:31.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.821+0000 7f7f0f50c640 1 -- 172.21.15.5:0/164590834 shutdown_connections 2024-03-24T10:44:31.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.821+0000 7f7f0f50c640 1 --2- 172.21.15.5:0/164590834 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7ef40051d0 0x7f7f080afab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.821+0000 7f7f0f50c640 1 --2- 172.21.15.5:0/164590834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7ef40047c0 0x7f7f0815cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.821+0000 7f7f0f50c640 1 --2- 172.21.15.5:0/164590834 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7ef4003970 0x7f7f080ad8b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:31.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.821+0000 7f7f0f50c640 1 -- 172.21.15.5:0/164590834 >> 172.21.15.5:0/164590834 conn(0x7f7f080711f0 msgr2=0x7f7f080cba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:31.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.821+0000 7f7f0f50c640 1 -- 172.21.15.5:0/164590834 shutdown_connections 2024-03-24T10:44:31.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:31.821+0000 7f7f0f50c640 1 -- 172.21.15.5:0/164590834 wait complete. 2024-03-24T10:44:31.832 INFO:tasks.ceph:check_enable_crimson: False 2024-03-24T10:44:31.833 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-24T10:44:31.833 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:44:31.833 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-24T10:44:31.861 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:44:31.861 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-24T10:44:31.927 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:44:31.927 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-24T10:44:31.994 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-24T10:44:31.994 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-24T10:44:32.061 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:44:32.061 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-24T10:44:32.089 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:44:32.089 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-24T10:44:32.156 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:44:32.157 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-24T10:44:32.224 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2024-03-24T10:44:32.224 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-24T10:44:32.291 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph osd new 50f590b3-f5ec-4e1b-ac46-25cab307a27a 0 2024-03-24T10:44:32.431 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.431+0000 7fa36ca2a640 1 Processor -- start 2024-03-24T10:44:32.431 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.431+0000 7fa36ca2a640 1 -- start start 2024-03-24T10:44:32.432 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.431+0000 7fa36ca2a640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa3680cc000 0x7fa36815e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.432 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.431+0000 7fa36ca2a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa36806bf50 0x7fa36806c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.432 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.431+0000 7fa36ca2a640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3680cc3e0 con 0x7fa3680cf340 2024-03-24T10:44:32.432 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.431+0000 7fa36ca2a640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fa36815a400 con 0x7fa36806bf50 2024-03-24T10:44:32.432 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.431+0000 7fa36ca2a640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fa368154bd0 con 0x7fa3680cc000 2024-03-24T10:44:32.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.432+0000 7fa36659c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa3680cc000 0x7fa36815e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.432+0000 7fa36659c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa3680cc000 0x7fa36815e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:33978/0 (socket says 172.21.15.57:33978) 2024-03-24T10:44:32.433 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.432+0000 7fa36659c640 1 -- 172.21.15.57:0/741296794 learned_addr learned my addr 172.21.15.57:0/741296794 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-03-24T10:44:32.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.434+0000 7fa36659c640 1 -- 172.21.15.57:0/741296794 >> v2:172.21.15.5:3300/0 conn(0x7fa36806bf50 msgr2=0x7fa36806c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:32.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.434+0000 7fa36659c640 1 --2- 172.21.15.57:0/741296794 >> v2:172.21.15.5:3300/0 conn(0x7fa36806bf50 0x7fa36806c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.434+0000 7fa36659c640 1 -- 172.21.15.57:0/741296794 >> v1:172.21.15.5:6789/0 conn(0x7fa3680cf340 legacy=0x7fa3680cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:32.435 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.434+0000 7fa36659c640 1 -- 172.21.15.57:0/741296794 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa36806c8e0 con 0x7fa3680cc000 2024-03-24T10:44:32.437 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.437+0000 7fa36659c640 1 --2- 172.21.15.57:0/741296794 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3680cc000 0x7fa36815e3b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa3680b27b0 tx=0x7fa35802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10b5cd68dcaa6e25 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:32.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.473+0000 7fa365d9b640 1 -- 172.21.15.57:0/741296794 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa358031870 con 0x7fa3680cc000 2024-03-24T10:44:32.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.473+0000 7fa365d9b640 1 -- 172.21.15.57:0/741296794 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3580319d0 con 0x7fa3680cc000 2024-03-24T10:44:32.473 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.473+0000 7fa365d9b640 1 -- 172.21.15.57:0/741296794 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa358031d60 con 0x7fa3680cc000 2024-03-24T10:44:32.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.473+0000 7fa36ca2a640 1 -- 172.21.15.57:0/741296794 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3680cc000 msgr2=0x7fa36815e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.473+0000 7fa36ca2a640 1 --2- 172.21.15.57:0/741296794 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3680cc000 0x7fa36815e3b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa3680b27b0 tx=0x7fa35802b910 comp rx=0 tx=0).stop 2024-03-24T10:44:32.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.473+0000 7fa36ca2a640 1 -- 172.21.15.57:0/741296794 shutdown_connections 2024-03-24T10:44:32.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.473+0000 7fa36ca2a640 1 --2- 172.21.15.57:0/741296794 >> v2:172.21.15.5:3300/0 conn(0x7fa36806bf50 0x7fa36806c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.474 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.473+0000 7fa36ca2a640 1 --2- 172.21.15.57:0/741296794 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3680cc000 0x7fa36815e3b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.473+0000 7fa36ca2a640 1 -- 172.21.15.57:0/741296794 >> 172.21.15.57:0/741296794 conn(0x7fa3680cfc30 msgr2=0x7fa3680d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:32.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.473+0000 7fa36ca2a640 1 -- 172.21.15.57:0/741296794 shutdown_connections 2024-03-24T10:44:32.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.473+0000 7fa36ca2a640 1 -- 172.21.15.57:0/741296794 wait complete. 2024-03-24T10:44:32.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.475+0000 7fa36ca2a640 1 Processor -- start 2024-03-24T10:44:32.475 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.475+0000 7fa36ca2a640 1 -- start start 2024-03-24T10:44:32.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.475+0000 7fa36ca2a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa36806bf50 0x7fa368159a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.475+0000 7fa36ca2a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3680cc000 0x7fa3680c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.475+0000 7fa36ca2a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3680cf340 0x7fa3680c2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:32.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.475+0000 7fa36ca2a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa3681e6290 con 0x7fa36806bf50 2024-03-24T10:44:32.476 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.475+0000 7fa36ca2a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa3681f1030 con 0x7fa3680cf340 2024-03-24T10:44:32.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.475+0000 7fa36ca2a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fa3681f04b0 con 0x7fa3680cc000 2024-03-24T10:44:32.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.476+0000 7fa366d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3680cf340 0x7fa3680c2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.476+0000 7fa35ffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3680cc000 0x7fa3680c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.476+0000 7fa36659c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa36806bf50 0x7fa368159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:32.477 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.476+0000 7fa35ffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3680cc000 0x7fa3680c22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.57:52006/0 (socket says 172.21.15.57:52006) 2024-03-24T10:44:32.478 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.476+0000 7fa35ffff640 1 -- 172.21.15.57:0/727746224 learned_addr learned my addr 172.21.15.57:0/727746224 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-03-24T10:44:32.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.480+0000 7fa35ffff640 1 -- 172.21.15.57:0/727746224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3680cf340 msgr2=0x7fa3680c2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.480+0000 7fa35ffff640 1 --2- 172.21.15.57:0/727746224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3680cf340 0x7fa3680c2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.480 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.480+0000 7fa35ffff640 1 -- 172.21.15.57:0/727746224 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa36806bf50 msgr2=0x7fa368159a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:32.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.480+0000 7fa35ffff640 1 --2- 172.21.15.57:0/727746224 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa36806bf50 0x7fa368159a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:32.481 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.480+0000 7fa35ffff640 1 -- 172.21.15.57:0/727746224 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3580094d0 con 0x7fa3680cc000 2024-03-24T10:44:32.483 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.483+0000 7fa35ffff640 1 --2- 172.21.15.57:0/727746224 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3680cc000 0x7fa3680c22f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa35000c410 tx=0x7fa35000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:32.574 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.571+0000 7fa35effd640 1 -- 172.21.15.57:0/727746224 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3500035f0 con 0x7fa3680cc000 2024-03-24T10:44:32.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.571+0000 7fa35effd640 1 -- 172.21.15.57:0/727746224 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa350003750 con 0x7fa3680cc000 2024-03-24T10:44:32.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.571+0000 7fa36ca2a640 1 -- 172.21.15.57:0/727746224 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3681f0790 con 0x7fa3680cc000 2024-03-24T10:44:32.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.571+0000 7fa35effd640 1 -- 172.21.15.57:0/727746224 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa350003b40 con 0x7fa3680cc000 2024-03-24T10:44:32.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.571+0000 7fa36ca2a640 1 -- 172.21.15.57:0/727746224 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3681f0d50 con 0x7fa3680cc000 2024-03-24T10:44:32.575 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.572+0000 7fa36ca2a640 1 -- 172.21.15.57:0/727746224 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3680732f0 con 0x7fa3680cc000 2024-03-24T10:44:32.811 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.810+0000 7fa35effd640 1 -- 172.21.15.57:0/727746224 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa350003ca0 con 0x7fa3680cc000 2024-03-24T10:44:32.811 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.810+0000 7fa35effd640 1 -- 172.21.15.57:0/727746224 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fa3500127d0 con 0x7fa3680cc000 2024-03-24T10:44:32.811 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.810+0000 7fa35effd640 1 -- 172.21.15.57:0/727746224 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa350017070 con 0x7fa3680cc000 2024-03-24T10:44:32.884 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:32.884+0000 7fa36ca2a640 1 -- 172.21.15.57:0/727746224 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "osd new", "uuid": "50f590b3-f5ec-4e1b-ac46-25cab307a27a", "id": 0} v 0) -- 0x7fa36806d440 con 0x7fa3680cc000 2024-03-24T10:44:33.076 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.075+0000 7fa35effd640 1 -- 172.21.15.57:0/727746224 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "50f590b3-f5ec-4e1b-ac46-25cab307a27a", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fa35001c440 con 0x7fa3680cc000 2024-03-24T10:44:33.076 INFO:teuthology.orchestra.run.smithi057.stdout:0 2024-03-24T10:44:33.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.076+0000 7fa36ca2a640 1 -- 172.21.15.57:0/727746224 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3680cc000 msgr2=0x7fa3680c22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.076+0000 7fa36ca2a640 1 --2- 172.21.15.57:0/727746224 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3680cc000 0x7fa3680c22f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa35000c410 tx=0x7fa35000c8d0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.076+0000 7fa36ca2a640 1 -- 172.21.15.57:0/727746224 shutdown_connections 2024-03-24T10:44:33.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.076+0000 7fa36ca2a640 1 --2- 172.21.15.57:0/727746224 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3680cf340 0x7fa3680c2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.076+0000 7fa36ca2a640 1 --2- 172.21.15.57:0/727746224 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3680cc000 0x7fa3680c22f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.077 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.076+0000 7fa36ca2a640 1 --2- 172.21.15.57:0/727746224 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa36806bf50 0x7fa368159a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.076+0000 7fa36ca2a640 1 -- 172.21.15.57:0/727746224 >> 172.21.15.57:0/727746224 conn(0x7fa3680cfc30 msgr2=0x7fa368154100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:33.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.076+0000 7fa36ca2a640 1 -- 172.21.15.57:0/727746224 shutdown_connections 2024-03-24T10:44:33.078 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.076+0000 7fa36ca2a640 1 -- 172.21.15.57:0/727746224 wait complete. 2024-03-24T10:44:33.087 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph osd new ccba9e93-c704-472b-a832-8d4eeb456c47 1 2024-03-24T10:44:33.154 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:33.154+0000 7f764ec40040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T10:44:33.164 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:33.164+0000 7ff1af22f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T10:44:33.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.187+0000 7f8fe4b84640 1 Processor -- start 2024-03-24T10:44:33.188 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.187+0000 7f8fe4b84640 1 -- start start 2024-03-24T10:44:33.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.188+0000 7f8fe4b84640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8fe0154760 0x7f8fe00c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.188+0000 7f8fe4b84640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fe00ca3a0 con 0x7f8fe00c86e0 2024-03-24T10:44:33.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.188+0000 7f8fe4b84640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fe00ca680 con 0x7f8fe00c9420 2024-03-24T10:44:33.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.188+0000 7f8fe4b84640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f8fe015d840 con 0x7f8fe0154760 2024-03-24T10:44:33.189 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.189+0000 7f8fde59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8fe0154760 0x7f8fe00c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.189+0000 7f8fded9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f8fe00c9420 0x7f8fe0164bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46182/0 (socket says 172.21.15.57:46182) 2024-03-24T10:44:33.190 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.189+0000 7f8fded9d640 1 -- 172.21.15.57:0/1572293202 learned_addr learned my addr 172.21.15.57:0/1572293202 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-24T10:44:33.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.191+0000 7f8fde59c640 1 -- 172.21.15.57:0/1572293202 >> v1:172.21.15.57:6789/0 conn(0x7f8fe00c9420 legacy=0x7f8fe0164bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.191+0000 7f8fde59c640 1 -- 172.21.15.57:0/1572293202 >> v1:172.21.15.5:6789/0 conn(0x7f8fe00c86e0 legacy=0x7f8fe015a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.192 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.191+0000 7f8fde59c640 1 -- 172.21.15.57:0/1572293202 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe00ca960 con 0x7f8fe0154760 2024-03-24T10:44:33.194 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.193+0000 7f8fde59c640 1 --2- 172.21.15.57:0/1572293202 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8fe0154760 0x7f8fe00c42d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8fd00097d0 tx=0x7f8fd002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96329838ce879116 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:33.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.197+0000 7f8fddd9b640 1 -- 172.21.15.57:0/1572293202 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0030880 con 0x7f8fe0154760 2024-03-24T10:44:33.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.197+0000 7f8fddd9b640 1 -- 172.21.15.57:0/1572293202 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fd00309e0 con 0x7f8fe0154760 2024-03-24T10:44:33.198 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.197+0000 7f8fddd9b640 1 -- 172.21.15.57:0/1572293202 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0030d70 con 0x7f8fe0154760 2024-03-24T10:44:33.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.198+0000 7f8fe4b84640 1 -- 172.21.15.57:0/1572293202 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8fe0154760 msgr2=0x7f8fe00c42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.198+0000 7f8fe4b84640 1 --2- 172.21.15.57:0/1572293202 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8fe0154760 0x7f8fe00c42d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8fd00097d0 tx=0x7f8fd002b9c0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.198+0000 7f8fe4b84640 1 -- 172.21.15.57:0/1572293202 shutdown_connections 2024-03-24T10:44:33.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.198+0000 7f8fe4b84640 1 --2- 172.21.15.57:0/1572293202 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8fe0154760 0x7f8fe00c42d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.198+0000 7f8fe4b84640 1 -- 172.21.15.57:0/1572293202 >> 172.21.15.57:0/1572293202 conn(0x7f8fe00711f0 msgr2=0x7f8fe00715d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:33.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.198+0000 7f8fe4b84640 1 -- 172.21.15.57:0/1572293202 shutdown_connections 2024-03-24T10:44:33.199 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.198+0000 7f8fe4b84640 1 -- 172.21.15.57:0/1572293202 wait complete. 2024-03-24T10:44:33.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.200+0000 7f8fe4b84640 1 Processor -- start 2024-03-24T10:44:33.200 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.200+0000 7f8fe4b84640 1 -- start start 2024-03-24T10:44:33.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.201+0000 7f8fe4b84640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8fe00c86e0 0x7f8fe01f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.201+0000 7f8fe4b84640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8fe00c9420 0x7f8fe014bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.201+0000 7f8fe4b84640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fe0154760 0x7f8fe006cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.201+0000 7f8fe4b84640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe00add10 con 0x7f8fe00c86e0 2024-03-24T10:44:33.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.201+0000 7f8fe4b84640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe01e36a0 con 0x7f8fe0154760 2024-03-24T10:44:33.201 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.201+0000 7f8fe4b84640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f8fe01f4290 con 0x7f8fe00c9420 2024-03-24T10:44:33.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.201+0000 7f8fd7fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8fe00c9420 0x7f8fe014bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.201+0000 7f8fded9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fe0154760 0x7f8fe006cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.201+0000 7f8fd7fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8fe00c9420 0x7f8fe014bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.57:52040/0 (socket says 172.21.15.57:52040) 2024-03-24T10:44:33.202 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.201+0000 7f8fd7fff640 1 -- 172.21.15.57:0/933876892 learned_addr learned my addr 172.21.15.57:0/933876892 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-03-24T10:44:33.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.205+0000 7f8fd7fff640 1 -- 172.21.15.57:0/933876892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fe0154760 msgr2=0x7f8fe006cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.205+0000 7f8fd7fff640 1 --2- 172.21.15.57:0/933876892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fe0154760 0x7f8fe006cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.205+0000 7f8fd7fff640 1 -- 172.21.15.57:0/933876892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8fe00c86e0 msgr2=0x7f8fe01f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.205+0000 7f8fd7fff640 1 --2- 172.21.15.57:0/933876892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8fe00c86e0 0x7f8fe01f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.205 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.205+0000 7f8fd7fff640 1 -- 172.21.15.57:0/933876892 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd0009400 con 0x7f8fe00c9420 2024-03-24T10:44:33.208 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.208+0000 7f8fd7fff640 1 --2- 172.21.15.57:0/933876892 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8fe00c9420 0x7f8fe014bfe0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800c400 tx=0x7f8fc800c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:33.217 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:33.216+0000 7f764ec40040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T10:44:33.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.217+0000 7f8fd6ffd640 1 -- 172.21.15.57:0/933876892 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc80052d0 con 0x7f8fe00c9420 2024-03-24T10:44:33.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.217+0000 7f8fd6ffd640 1 -- 172.21.15.57:0/933876892 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fc8005450 con 0x7f8fe00c9420 2024-03-24T10:44:33.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.217+0000 7f8fe4b84640 1 -- 172.21.15.57:0/933876892 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fe01e44e0 con 0x7f8fe00c9420 2024-03-24T10:44:33.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.217+0000 7f8fd6ffd640 1 -- 172.21.15.57:0/933876892 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8007260 con 0x7f8fe00c9420 2024-03-24T10:44:33.218 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.217+0000 7f8fe4b84640 1 -- 172.21.15.57:0/933876892 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fe00ae2e0 con 0x7f8fe00c9420 2024-03-24T10:44:33.219 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.219+0000 7f8fe4b84640 1 -- 172.21.15.57:0/933876892 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fe00688e0 con 0x7f8fe00c9420 2024-03-24T10:44:33.223 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:33.223+0000 7ff1af22f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T10:44:33.327 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.327+0000 7f8fd6ffd640 1 -- 172.21.15.57:0/933876892 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8fc80073c0 con 0x7f8fe00c9420 2024-03-24T10:44:33.327 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.327+0000 7f8fd6ffd640 1 -- 172.21.15.57:0/933876892 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f8fc8003910 con 0x7f8fe00c9420 2024-03-24T10:44:33.327 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.327+0000 7f8fd6ffd640 1 -- 172.21.15.57:0/933876892 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f8fc8003b40 con 0x7f8fe00c9420 2024-03-24T10:44:33.403 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.402+0000 7f8fe4b84640 1 -- 172.21.15.57:0/933876892 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ccba9e93-c704-472b-a832-8d4eeb456c47", "id": 1} v 0) -- 0x7f8fe01594b0 con 0x7f8fe00c9420 2024-03-24T10:44:33.488 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.488+0000 7f8fd6ffd640 1 -- 172.21.15.57:0/933876892 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ccba9e93-c704-472b-a832-8d4eeb456c47", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f8fc8003ce0 con 0x7f8fe00c9420 2024-03-24T10:44:33.489 INFO:teuthology.orchestra.run.smithi057.stdout:1 2024-03-24T10:44:33.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.489+0000 7f8fe4b84640 1 -- 172.21.15.57:0/933876892 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8fe00c9420 msgr2=0x7f8fe014bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.489+0000 7f8fe4b84640 1 --2- 172.21.15.57:0/933876892 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8fe00c9420 0x7f8fe014bfe0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800c400 tx=0x7f8fc800c8c0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.489+0000 7f8fe4b84640 1 -- 172.21.15.57:0/933876892 shutdown_connections 2024-03-24T10:44:33.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.489+0000 7f8fe4b84640 1 --2- 172.21.15.57:0/933876892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8fe0154760 0x7f8fe006cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.489+0000 7f8fe4b84640 1 --2- 172.21.15.57:0/933876892 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8fe00c9420 0x7f8fe014bfe0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.489+0000 7f8fe4b84640 1 --2- 172.21.15.57:0/933876892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8fe00c86e0 0x7f8fe01f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.490 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.489+0000 7f8fe4b84640 1 -- 172.21.15.57:0/933876892 >> 172.21.15.57:0/933876892 conn(0x7f8fe00711f0 msgr2=0x7f8fe0158d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:33.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.489+0000 7f8fe4b84640 1 -- 172.21.15.57:0/933876892 shutdown_connections 2024-03-24T10:44:33.491 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.489+0000 7f8fe4b84640 1 -- 172.21.15.57:0/933876892 wait complete. 2024-03-24T10:44:33.500 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph osd new e614c5b1-6504-452e-82e0-95c7ca271737 2 2024-03-24T10:44:33.602 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.601+0000 7fd11286f640 1 Processor -- start 2024-03-24T10:44:33.602 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.601+0000 7fd11286f640 1 -- start start 2024-03-24T10:44:33.602 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.602+0000 7fd11286f640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd10c150da0 0x7fd10c15acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.602 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.602+0000 7fd11286f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd10c0cae50 0x7fd10c071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.602 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.602+0000 7fd11286f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd10c0c85c0 0x7fd10c0717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.602+0000 7fd11286f640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fd10c151180 con 0x7fd10c150da0 2024-03-24T10:44:33.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.602+0000 7fd11286f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fd10c15b210 con 0x7fd10c0c85c0 2024-03-24T10:44:33.603 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.602+0000 7fd11286f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd10c15d910 con 0x7fd10c0cae50 2024-03-24T10:44:33.604 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.602+0000 7fd10b7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd10c0cae50 0x7fd10c071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.604 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.602+0000 7fd110e0c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd10c0c85c0 0x7fd10c0717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.604 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.603+0000 7fd10bfff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd10c150da0 0x7fd10c15acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.604 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.603+0000 7fd10b7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd10c0cae50 0x7fd10c071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.57:52056/0 (socket says 172.21.15.57:52056) 2024-03-24T10:44:33.604 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.603+0000 7fd10b7fe640 1 -- 172.21.15.57:0/370505987 learned_addr learned my addr 172.21.15.57:0/370505987 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-03-24T10:44:33.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.605+0000 7fd10b7fe640 1 -- 172.21.15.57:0/370505987 >> v2:172.21.15.57:3300/0 conn(0x7fd10c0c85c0 msgr2=0x7fd10c0717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:33.605 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.605+0000 7fd10b7fe640 1 --2- 172.21.15.57:0/370505987 >> v2:172.21.15.57:3300/0 conn(0x7fd10c0c85c0 0x7fd10c0717a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.606 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.605+0000 7fd10b7fe640 1 -- 172.21.15.57:0/370505987 >> v2:172.21.15.5:3300/0 conn(0x7fd10c150da0 msgr2=0x7fd10c15acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:33.606 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.605+0000 7fd10b7fe640 1 --2- 172.21.15.57:0/370505987 >> v2:172.21.15.5:3300/0 conn(0x7fd10c150da0 0x7fd10c15acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.606 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.605+0000 7fd10b7fe640 1 -- 172.21.15.57:0/370505987 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd10c15dbf0 con 0x7fd10c0cae50 2024-03-24T10:44:33.607 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.607+0000 7fd10b7fe640 1 --2- 172.21.15.57:0/370505987 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd10c0cae50 0x7fd10c071260 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd10c06f630 tx=0x7fd0f402b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=521c8ce3585a7f16 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:33.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.611+0000 7fd10affd640 1 -- 172.21.15.57:0/370505987 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4031870 con 0x7fd10c0cae50 2024-03-24T10:44:33.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.611+0000 7fd10affd640 1 -- 172.21.15.57:0/370505987 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0f40319d0 con 0x7fd10c0cae50 2024-03-24T10:44:33.611 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.611+0000 7fd10affd640 1 -- 172.21.15.57:0/370505987 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4031d60 con 0x7fd10c0cae50 2024-03-24T10:44:33.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.611+0000 7fd11286f640 1 -- 172.21.15.57:0/370505987 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd10c0cae50 msgr2=0x7fd10c071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.611+0000 7fd11286f640 1 --2- 172.21.15.57:0/370505987 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd10c0cae50 0x7fd10c071260 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd10c06f630 tx=0x7fd0f402b910 comp rx=0 tx=0).stop 2024-03-24T10:44:33.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.612+0000 7fd11286f640 1 -- 172.21.15.57:0/370505987 shutdown_connections 2024-03-24T10:44:33.612 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.612+0000 7fd11286f640 1 --2- 172.21.15.57:0/370505987 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd10c0cae50 0x7fd10c071260 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.612+0000 7fd11286f640 1 --2- 172.21.15.57:0/370505987 >> v2:172.21.15.57:3300/0 conn(0x7fd10c0c85c0 0x7fd10c0717a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.612+0000 7fd11286f640 1 --2- 172.21.15.57:0/370505987 >> v2:172.21.15.5:3300/0 conn(0x7fd10c150da0 0x7fd10c15acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.612+0000 7fd11286f640 1 -- 172.21.15.57:0/370505987 >> 172.21.15.57:0/370505987 conn(0x7fd10c0706e0 msgr2=0x7fd10c06f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:33.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.612+0000 7fd11286f640 1 -- 172.21.15.57:0/370505987 shutdown_connections 2024-03-24T10:44:33.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.612+0000 7fd11286f640 1 -- 172.21.15.57:0/370505987 wait complete. 2024-03-24T10:44:33.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.613+0000 7fd11286f640 1 Processor -- start 2024-03-24T10:44:33.613 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.613+0000 7fd11286f640 1 -- start start 2024-03-24T10:44:33.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.614+0000 7fd11286f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd10c0c85c0 0x7fd10c159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.614+0000 7fd11286f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd10c0cae50 0x7fd10c0cde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.614 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.614+0000 7fd11286f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd10c150da0 0x7fd10c1f3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.614+0000 7fd11286f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd10c153960 con 0x7fd10c0c85c0 2024-03-24T10:44:33.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.614+0000 7fd11286f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd10c15a3a0 con 0x7fd10c0cae50 2024-03-24T10:44:33.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.614+0000 7fd11286f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd10c1e74f0 con 0x7fd10c150da0 2024-03-24T10:44:33.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.614+0000 7fd110e0c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd10c150da0 0x7fd10c1f3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.614+0000 7fd10b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd10c0cae50 0x7fd10c0cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.614+0000 7fd10bfff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd10c0c85c0 0x7fd10c159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:33.615 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.615+0000 7fd110e0c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd10c150da0 0x7fd10c1f3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.57:52066/0 (socket says 172.21.15.57:52066) 2024-03-24T10:44:33.616 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.615+0000 7fd110e0c640 1 -- 172.21.15.57:0/4041297709 learned_addr learned my addr 172.21.15.57:0/4041297709 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-03-24T10:44:33.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.618+0000 7fd110e0c640 1 -- 172.21.15.57:0/4041297709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd10c0cae50 msgr2=0x7fd10c0cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.618+0000 7fd110e0c640 1 --2- 172.21.15.57:0/4041297709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd10c0cae50 0x7fd10c0cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.618+0000 7fd110e0c640 1 -- 172.21.15.57:0/4041297709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd10c0c85c0 msgr2=0x7fd10c159e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.618 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.618+0000 7fd110e0c640 1 --2- 172.21.15.57:0/4041297709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd10c0c85c0 0x7fd10c159e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.619 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.618+0000 7fd110e0c640 1 -- 172.21.15.57:0/4041297709 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f40094d0 con 0x7fd10c150da0 2024-03-24T10:44:33.620 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.620+0000 7fd110e0c640 1 --2- 172.21.15.57:0/4041297709 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd10c150da0 0x7fd10c1f3a20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd10400ef00 tx=0x7fd10400b480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:33.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.624+0000 7fd108ff9640 1 -- 172.21.15.57:0/4041297709 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd104010040 con 0x7fd10c150da0 2024-03-24T10:44:33.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.624+0000 7fd108ff9640 1 -- 172.21.15.57:0/4041297709 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd104014400 con 0x7fd10c150da0 2024-03-24T10:44:33.624 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.624+0000 7fd108ff9640 1 -- 172.21.15.57:0/4041297709 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd104009ea0 con 0x7fd10c150da0 2024-03-24T10:44:33.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.624+0000 7fd11286f640 1 -- 172.21.15.57:0/4041297709 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd10c15a610 con 0x7fd10c150da0 2024-03-24T10:44:33.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.624+0000 7fd11286f640 1 -- 172.21.15.57:0/4041297709 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd10c1530c0 con 0x7fd10c150da0 2024-03-24T10:44:33.625 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.625+0000 7fd11286f640 1 -- 172.21.15.57:0/4041297709 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd10c154950 con 0x7fd10c150da0 2024-03-24T10:44:33.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.730+0000 7fd108ff9640 1 -- 172.21.15.57:0/4041297709 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd1040147b0 con 0x7fd10c150da0 2024-03-24T10:44:33.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.730+0000 7fd108ff9640 1 -- 172.21.15.57:0/4041297709 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fd10401c3b0 con 0x7fd10c150da0 2024-03-24T10:44:33.730 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.730+0000 7fd108ff9640 1 -- 172.21.15.57:0/4041297709 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd10403bdd0 con 0x7fd10c150da0 2024-03-24T10:44:33.774 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:33.774+0000 7ff1af22f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T10:44:33.790 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:33.790+0000 7f764ec40040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T10:44:33.805 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.804+0000 7fd11286f640 1 -- 172.21.15.57:0/4041297709 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e614c5b1-6504-452e-82e0-95c7ca271737", "id": 2} v 0) -- 0x7fd10c06dfd0 con 0x7fd10c150da0 2024-03-24T10:44:33.830 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:33.830+0000 7ff1af22f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T10:44:33.849 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:33.848+0000 7f764ec40040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T10:44:33.879 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:33.879+0000 7ff1af22f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T10:44:33.886 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.886+0000 7fd108ff9640 1 -- 172.21.15.57:0/4041297709 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e614c5b1-6504-452e-82e0-95c7ca271737", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fd10402a070 con 0x7fd10c150da0 2024-03-24T10:44:33.886 INFO:teuthology.orchestra.run.smithi057.stdout:2 2024-03-24T10:44:33.887 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.887+0000 7fd11286f640 1 -- 172.21.15.57:0/4041297709 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd10c150da0 msgr2=0x7fd10c1f3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:33.888 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.887+0000 7fd11286f640 1 --2- 172.21.15.57:0/4041297709 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd10c150da0 0x7fd10c1f3a20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd10400ef00 tx=0x7fd10400b480 comp rx=0 tx=0).stop 2024-03-24T10:44:33.888 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.887+0000 7fd11286f640 1 -- 172.21.15.57:0/4041297709 shutdown_connections 2024-03-24T10:44:33.888 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.887+0000 7fd11286f640 1 --2- 172.21.15.57:0/4041297709 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd10c150da0 0x7fd10c1f3a20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.888 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.887+0000 7fd11286f640 1 --2- 172.21.15.57:0/4041297709 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd10c0cae50 0x7fd10c0cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.888 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.887+0000 7fd11286f640 1 --2- 172.21.15.57:0/4041297709 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd10c0c85c0 0x7fd10c159e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:33.888 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.887+0000 7fd11286f640 1 -- 172.21.15.57:0/4041297709 >> 172.21.15.57:0/4041297709 conn(0x7fd10c0706e0 msgr2=0x7fd10c15cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:33.888 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.887+0000 7fd11286f640 1 -- 172.21.15.57:0/4041297709 shutdown_connections 2024-03-24T10:44:33.888 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.887+0000 7fd11286f640 1 -- 172.21.15.57:0/4041297709 wait complete. 2024-03-24T10:44:33.898 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph osd new c60154d0-85e0-46a1-8273-4863bd8ad2dc 3 2024-03-24T10:44:33.898 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:33.898+0000 7f764ec40040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T10:44:33.935 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:33.935+0000 7ff1af22f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T10:44:33.956 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:33.955+0000 7f764ec40040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T10:44:33.979 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:33.979+0000 7ff1af22f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T10:44:33.998 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.998+0000 7f7c6dc37640 1 Processor -- start 2024-03-24T10:44:33.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.998+0000 7f7c6dc37640 1 -- start start 2024-03-24T10:44:33.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.999+0000 7f7c6dc37640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7c68139ac0 0x7f7c680701b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.999+0000 7f7c6dc37640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7c6813d080 0x7f7c680c6590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:33.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.999+0000 7f7c6dc37640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c6813c4e0 con 0x7f7c680706f0 2024-03-24T10:44:33.999 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.999+0000 7f7c6dc37640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f7c680ca110 con 0x7f7c6813d080 2024-03-24T10:44:34.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.999+0000 7f7c6dc37640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7c680c4460 con 0x7f7c68139ac0 2024-03-24T10:44:34.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.999+0000 7f7c66ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f7c680706f0 0x7f7c6813c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46190/0 (socket says 172.21.15.57:46190) 2024-03-24T10:44:34.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.999+0000 7f7c66ffd640 1 -- 172.21.15.57:0/1627627555 learned_addr learned my addr 172.21.15.57:0/1627627555 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-24T10:44:34.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.999+0000 7f7c677fe640 1 --2- 172.21.15.57:0/1627627555 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7c68139ac0 0x7f7c680701b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:34.000 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:33.999+0000 7f7c67fff640 1 --2- 172.21.15.57:0/1627627555 >> v2:172.21.15.5:3300/0 conn(0x7f7c6813d080 0x7f7c680c6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:34.001 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:34.000+0000 7f764ec40040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T10:44:34.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.002+0000 7f7c677fe640 1 -- 172.21.15.57:0/1627627555 >> v2:172.21.15.5:3300/0 conn(0x7f7c6813d080 msgr2=0x7f7c680c6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:34.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.002+0000 7f7c677fe640 1 --2- 172.21.15.57:0/1627627555 >> v2:172.21.15.5:3300/0 conn(0x7f7c6813d080 0x7f7c680c6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.002 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.002+0000 7f7c677fe640 1 -- 172.21.15.57:0/1627627555 >> v1:172.21.15.57:6789/0 conn(0x7f7c680706f0 legacy=0x7f7c6813c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:34.003 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.002+0000 7f7c677fe640 1 -- 172.21.15.57:0/1627627555 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c680c4740 con 0x7f7c68139ac0 2024-03-24T10:44:34.004 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.004+0000 7f7c677fe640 1 --2- 172.21.15.57:0/1627627555 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7c68139ac0 0x7f7c680701b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7c580097a0 tx=0x7f7c5802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26434e3d007e7f2e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:34.010 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.009+0000 7f7c667fc640 1 -- 172.21.15.57:0/1627627555 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c58031830 con 0x7f7c68139ac0 2024-03-24T10:44:34.010 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.009+0000 7f7c667fc640 1 -- 172.21.15.57:0/1627627555 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c58031990 con 0x7f7c68139ac0 2024-03-24T10:44:34.010 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.009+0000 7f7c667fc640 1 -- 172.21.15.57:0/1627627555 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c58031d20 con 0x7f7c68139ac0 2024-03-24T10:44:34.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.010+0000 7f7c6dc37640 1 -- 172.21.15.57:0/1627627555 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7c68139ac0 msgr2=0x7f7c680701b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:34.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.010+0000 7f7c6dc37640 1 --2- 172.21.15.57:0/1627627555 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7c68139ac0 0x7f7c680701b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7c580097a0 tx=0x7f7c5802b8d0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.010+0000 7f7c6dc37640 1 -- 172.21.15.57:0/1627627555 shutdown_connections 2024-03-24T10:44:34.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.010+0000 7f7c6dc37640 1 --2- 172.21.15.57:0/1627627555 >> v2:172.21.15.5:3300/0 conn(0x7f7c6813d080 0x7f7c680c6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.010+0000 7f7c6dc37640 1 --2- 172.21.15.57:0/1627627555 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7c68139ac0 0x7f7c680701b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.011 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.010+0000 7f7c6dc37640 1 -- 172.21.15.57:0/1627627555 >> 172.21.15.57:0/1627627555 conn(0x7f7c68061d30 msgr2=0x7f7c68072680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:34.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.010+0000 7f7c6dc37640 1 -- 172.21.15.57:0/1627627555 shutdown_connections 2024-03-24T10:44:34.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.010+0000 7f7c6dc37640 1 -- 172.21.15.57:0/1627627555 wait complete. 2024-03-24T10:44:34.012 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.012+0000 7f7c6dc37640 1 Processor -- start 2024-03-24T10:44:34.013 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.012+0000 7f7c6dc37640 1 -- start start 2024-03-24T10:44:34.015 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.014+0000 7f7c6dc37640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7c680706f0 0x7f7c68149b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:34.015 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.014+0000 7f7c6dc37640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7c68139ac0 0x7f7c680cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:34.015 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.014+0000 7f7c6dc37640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c6813d080 0x7f7c680cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:34.015 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.014+0000 7f7c6dc37640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6814a270 con 0x7f7c680706f0 2024-03-24T10:44:34.015 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.014+0000 7f7c6dc37640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7c680cb210 con 0x7f7c6813d080 2024-03-24T10:44:34.015 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.014+0000 7f7c6dc37640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7c680cd220 con 0x7f7c68139ac0 2024-03-24T10:44:34.015 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.014+0000 7f7c66ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7c68139ac0 0x7f7c680cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:34.015 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.014+0000 7f7c67fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c6813d080 0x7f7c680cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:34.015 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.014+0000 7f7c677fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7c680706f0 0x7f7c68149b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:34.016 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.014+0000 7f7c66ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7c68139ac0 0x7f7c680cd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.57:52082/0 (socket says 172.21.15.57:52082) 2024-03-24T10:44:34.016 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.014+0000 7f7c66ffd640 1 -- 172.21.15.57:0/1712898724 learned_addr learned my addr 172.21.15.57:0/1712898724 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-03-24T10:44:34.018 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.018+0000 7f7c66ffd640 1 -- 172.21.15.57:0/1712898724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c6813d080 msgr2=0x7f7c680cbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:34.018 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.018+0000 7f7c66ffd640 1 --2- 172.21.15.57:0/1712898724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c6813d080 0x7f7c680cbb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.019 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.018+0000 7f7c66ffd640 1 -- 172.21.15.57:0/1712898724 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7c680706f0 msgr2=0x7f7c68149b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:34.019 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.018+0000 7f7c66ffd640 1 --2- 172.21.15.57:0/1712898724 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7c680706f0 0x7f7c68149b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.019 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.018+0000 7f7c66ffd640 1 -- 172.21.15.57:0/1712898724 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c58009400 con 0x7f7c68139ac0 2024-03-24T10:44:34.021 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.020+0000 7f7c66ffd640 1 --2- 172.21.15.57:0/1712898724 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7c68139ac0 0x7f7c680cd6b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00e710 tx=0x7f7c5c00ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:34.025 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.024+0000 7f7c4ffff640 1 -- 172.21.15.57:0/1712898724 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c005ce0 con 0x7f7c68139ac0 2024-03-24T10:44:34.025 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.024+0000 7f7c4ffff640 1 -- 172.21.15.57:0/1712898724 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c5c005e40 con 0x7f7c68139ac0 2024-03-24T10:44:34.025 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.024+0000 7f7c4ffff640 1 -- 172.21.15.57:0/1712898724 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c001920 con 0x7f7c68139ac0 2024-03-24T10:44:34.025 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.024+0000 7f7c6dc37640 1 -- 172.21.15.57:0/1712898724 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c680caa00 con 0x7f7c68139ac0 2024-03-24T10:44:34.025 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.024+0000 7f7c6dc37640 1 -- 172.21.15.57:0/1712898724 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c680cac50 con 0x7f7c68139ac0 2024-03-24T10:44:34.025 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.025+0000 7f7c6dc37640 1 -- 172.21.15.57:0/1712898724 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c680cd3d0 con 0x7f7c68139ac0 2024-03-24T10:44:34.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.027+0000 7f7c4ffff640 1 -- 172.21.15.57:0/1712898724 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7c5c001a80 con 0x7f7c68139ac0 2024-03-24T10:44:34.028 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.027+0000 7f7c4ffff640 1 -- 172.21.15.57:0/1712898724 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f7c5c024a20 con 0x7f7c68139ac0 2024-03-24T10:44:34.105 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:34.105+0000 7ff1af22f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T10:44:34.129 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:34.128+0000 7f764ec40040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T10:44:34.133 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.132+0000 7f7c4ffff640 1 -- 172.21.15.57:0/1712898724 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7c680cd3d0 con 0x7f7c68139ac0 2024-03-24T10:44:34.178 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:34.178+0000 7ff1af22f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T10:44:34.202 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:34.202+0000 7f764ec40040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T10:44:34.209 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.207+0000 7f7c6dc37640 1 -- 172.21.15.57:0/1712898724 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c60154d0-85e0-46a1-8273-4863bd8ad2dc", "id": 3} v 0) -- 0x7f7c680688e0 con 0x7f7c68139ac0 2024-03-24T10:44:34.226 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:34.226+0000 7ff1af22f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T10:44:34.253 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:34.252+0000 7f764ec40040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T10:44:34.279 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.279+0000 7f7c4ffff640 1 -- 172.21.15.57:0/1712898724 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c60154d0-85e0-46a1-8273-4863bd8ad2dc", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f7c5c0290b0 con 0x7f7c68139ac0 2024-03-24T10:44:34.279 INFO:teuthology.orchestra.run.smithi057.stdout:3 2024-03-24T10:44:34.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.280+0000 7f7c6dc37640 1 -- 172.21.15.57:0/1712898724 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7c68139ac0 msgr2=0x7f7c680cd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:34.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.280+0000 7f7c6dc37640 1 --2- 172.21.15.57:0/1712898724 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7c68139ac0 0x7f7c680cd6b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00e710 tx=0x7f7c5c00ebd0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.280+0000 7f7c6dc37640 1 -- 172.21.15.57:0/1712898724 shutdown_connections 2024-03-24T10:44:34.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.280+0000 7f7c6dc37640 1 --2- 172.21.15.57:0/1712898724 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c6813d080 0x7f7c680cbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.280+0000 7f7c6dc37640 1 --2- 172.21.15.57:0/1712898724 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7c68139ac0 0x7f7c680cd6b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.280+0000 7f7c6dc37640 1 --2- 172.21.15.57:0/1712898724 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7c680706f0 0x7f7c68149b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.280+0000 7f7c6dc37640 1 -- 172.21.15.57:0/1712898724 >> 172.21.15.57:0/1712898724 conn(0x7f7c68061d30 msgr2=0x7f7c6813b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:34.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.280+0000 7f7c6dc37640 1 -- 172.21.15.57:0/1712898724 shutdown_connections 2024-03-24T10:44:34.281 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.280+0000 7f7c6dc37640 1 -- 172.21.15.57:0/1712898724 wait complete. 2024-03-24T10:44:34.284 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:34.284+0000 7ff1af22f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T10:44:34.291 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph osd new 21d74109-da3a-4fc0-b2d8-93e9ae3ed30c 4 2024-03-24T10:44:34.316 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:34.315+0000 7f764ec40040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T10:44:34.336 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:34.336+0000 7ff1af22f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T10:44:34.379 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:34.379+0000 7f764ec40040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T10:44:34.396 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.395+0000 7f48b2da7640 1 Processor -- start 2024-03-24T10:44:34.396 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.395+0000 7f48b2da7640 1 -- start start 2024-03-24T10:44:34.396 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.396+0000 7f48b2da7640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f48ac06b3a0 0x7f48ac15e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:34.397 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.396+0000 7f48b2da7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f48ac070cd0 0x7f48ac0c5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:34.397 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.396+0000 7f48b2da7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f48ac06b780 0x7f48ac1540e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:34.397 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.396+0000 7f48b2da7640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f48ac15a440 con 0x7f48ac070cd0 2024-03-24T10:44:34.397 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.396+0000 7f48b2da7640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f48ac073610 con 0x7f48ac06b3a0 2024-03-24T10:44:34.397 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.396+0000 7f48b2da7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f48ac15e980 con 0x7f48ac06b780 2024-03-24T10:44:34.398 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.397+0000 7f48b1344640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f48ac06b780 0x7f48ac1540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:34.398 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.397+0000 7f48b0b43640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f48ac06b3a0 0x7f48ac15e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:34.398 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.397+0000 7f48abfff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f48ac070cd0 0x7f48ac0c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:34.398 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.397+0000 7f48b1344640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f48ac06b780 0x7f48ac1540e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.57:52096/0 (socket says 172.21.15.57:52096) 2024-03-24T10:44:34.398 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.397+0000 7f48b1344640 1 -- 172.21.15.57:0/245379323 learned_addr learned my addr 172.21.15.57:0/245379323 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-03-24T10:44:34.399 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.399+0000 7f48b1344640 1 -- 172.21.15.57:0/245379323 >> v2:172.21.15.57:3300/0 conn(0x7f48ac06b3a0 msgr2=0x7f48ac15e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:34.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.399+0000 7f48b1344640 1 --2- 172.21.15.57:0/245379323 >> v2:172.21.15.57:3300/0 conn(0x7f48ac06b3a0 0x7f48ac15e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.399+0000 7f48b1344640 1 -- 172.21.15.57:0/245379323 >> v2:172.21.15.5:3300/0 conn(0x7f48ac070cd0 msgr2=0x7f48ac0c5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:34.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.399+0000 7f48b1344640 1 --2- 172.21.15.57:0/245379323 >> v2:172.21.15.5:3300/0 conn(0x7f48ac070cd0 0x7f48ac0c5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.400 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.399+0000 7f48b1344640 1 -- 172.21.15.57:0/245379323 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48ac15b410 con 0x7f48ac06b780 2024-03-24T10:44:34.402 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.401+0000 7f48b1344640 1 --2- 172.21.15.57:0/245379323 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f48ac06b780 0x7f48ac1540e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f48ac072cf0 tx=0x7f48a402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af74e8050a88619c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:34.405 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.405+0000 7f48ab7fe640 1 -- 172.21.15.57:0/245379323 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a402cb00 con 0x7f48ac06b780 2024-03-24T10:44:34.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.405+0000 7f48ab7fe640 1 -- 172.21.15.57:0/245379323 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48a402cc60 con 0x7f48ac06b780 2024-03-24T10:44:34.406 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.405+0000 7f48ab7fe640 1 -- 172.21.15.57:0/245379323 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a40343f0 con 0x7f48ac06b780 2024-03-24T10:44:34.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.406+0000 7f48b2da7640 1 -- 172.21.15.57:0/245379323 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f48ac06b780 msgr2=0x7f48ac1540e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:34.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.406+0000 7f48b2da7640 1 --2- 172.21.15.57:0/245379323 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f48ac06b780 0x7f48ac1540e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f48ac072cf0 tx=0x7f48a402bf60 comp rx=0 tx=0).stop 2024-03-24T10:44:34.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.406+0000 7f48b2da7640 1 -- 172.21.15.57:0/245379323 shutdown_connections 2024-03-24T10:44:34.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.406+0000 7f48b2da7640 1 --2- 172.21.15.57:0/245379323 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f48ac06b780 0x7f48ac1540e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.406+0000 7f48b2da7640 1 --2- 172.21.15.57:0/245379323 >> v2:172.21.15.5:3300/0 conn(0x7f48ac070cd0 0x7f48ac0c5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.406+0000 7f48b2da7640 1 --2- 172.21.15.57:0/245379323 >> v2:172.21.15.57:3300/0 conn(0x7f48ac06b3a0 0x7f48ac15e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.406+0000 7f48b2da7640 1 -- 172.21.15.57:0/245379323 >> 172.21.15.57:0/245379323 conn(0x7f48ac06c240 msgr2=0x7f48ac06c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:34.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.406+0000 7f48b2da7640 1 -- 172.21.15.57:0/245379323 shutdown_connections 2024-03-24T10:44:34.407 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.406+0000 7f48b2da7640 1 -- 172.21.15.57:0/245379323 wait complete. 2024-03-24T10:44:34.408 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.407+0000 7f48b2da7640 1 Processor -- start 2024-03-24T10:44:34.408 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.407+0000 7f48b2da7640 1 -- start start 2024-03-24T10:44:34.408 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.408+0000 7f48b2da7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48ac06b3a0 0x7f48ac0cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:34.408 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.408+0000 7f48b2da7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f48ac06b780 0x7f48ac156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:34.409 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.408+0000 7f48b2da7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48ac070cd0 0x7f48ac157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:34.409 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.408+0000 7f48b2da7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f48ac1555d0 con 0x7f48ac06b3a0 2024-03-24T10:44:34.409 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.408+0000 7f48b2da7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f48ac0c3820 con 0x7f48ac070cd0 2024-03-24T10:44:34.409 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.408+0000 7f48b2da7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f48ac15da90 con 0x7f48ac06b780 2024-03-24T10:44:34.409 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.409+0000 7f48abfff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f48ac06b780 0x7f48ac156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:34.409 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.409+0000 7f48b0b43640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48ac06b3a0 0x7f48ac0cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:34.410 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.409+0000 7f48b1344640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48ac070cd0 0x7f48ac157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:34.410 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.409+0000 7f48abfff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f48ac06b780 0x7f48ac156fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.57:52100/0 (socket says 172.21.15.57:52100) 2024-03-24T10:44:34.410 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.409+0000 7f48abfff640 1 -- 172.21.15.57:0/1320629926 learned_addr learned my addr 172.21.15.57:0/1320629926 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-03-24T10:44:34.413 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.412+0000 7f48abfff640 1 -- 172.21.15.57:0/1320629926 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48ac070cd0 msgr2=0x7f48ac157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:34.413 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.412+0000 7f48abfff640 1 --2- 172.21.15.57:0/1320629926 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48ac070cd0 0x7f48ac157510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.413 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.412+0000 7f48abfff640 1 -- 172.21.15.57:0/1320629926 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48ac06b3a0 msgr2=0x7f48ac0cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:34.413 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.412+0000 7f48abfff640 1 --2- 172.21.15.57:0/1320629926 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48ac06b3a0 0x7f48ac0cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.413 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.412+0000 7f48abfff640 1 -- 172.21.15.57:0/1320629926 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48a4009b30 con 0x7f48ac06b780 2024-03-24T10:44:34.415 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.414+0000 7f48abfff640 1 --2- 172.21.15.57:0/1320629926 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f48ac06b780 0x7f48ac156fd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f489800e8a0 tx=0x7f489800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:34.419 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.419+0000 7f48a97fa640 1 -- 172.21.15.57:0/1320629926 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489800ee90 con 0x7f48ac06b780 2024-03-24T10:44:34.420 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.419+0000 7f48a97fa640 1 -- 172.21.15.57:0/1320629926 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f489800c940 con 0x7f48ac06b780 2024-03-24T10:44:34.420 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.419+0000 7f48a97fa640 1 -- 172.21.15.57:0/1320629926 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489800cd10 con 0x7f48ac06b780 2024-03-24T10:44:34.420 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.419+0000 7f48b2da7640 1 -- 172.21.15.57:0/1320629926 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48ac1f01e0 con 0x7f48ac06b780 2024-03-24T10:44:34.420 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.419+0000 7f48b2da7640 1 -- 172.21.15.57:0/1320629926 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f48ac154f10 con 0x7f48ac06b780 2024-03-24T10:44:34.420 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.420+0000 7f48b2da7640 1 -- 172.21.15.57:0/1320629926 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48ac155840 con 0x7f48ac06b780 2024-03-24T10:44:34.423 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.422+0000 7f48a97fa640 1 -- 172.21.15.57:0/1320629926 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4898005990 con 0x7f48ac06b780 2024-03-24T10:44:34.423 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.422+0000 7f48a97fa640 1 -- 172.21.15.57:0/1320629926 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f48980067a0 con 0x7f48ac06b780 2024-03-24T10:44:34.526 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.526+0000 7f48a97fa640 1 -- 172.21.15.57:0/1320629926 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4898021060 con 0x7f48ac06b780 2024-03-24T10:44:34.600 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.599+0000 7f48b2da7640 1 -- 172.21.15.57:0/1320629926 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "osd new", "uuid": "21d74109-da3a-4fc0-b2d8-93e9ae3ed30c", "id": 4} v 0) -- 0x7f48ac0688e0 con 0x7f48ac06b780 2024-03-24T10:44:34.670 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.670+0000 7f48a97fa640 1 -- 172.21.15.57:0/1320629926 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "21d74109-da3a-4fc0-b2d8-93e9ae3ed30c", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f4898017070 con 0x7f48ac06b780 2024-03-24T10:44:34.671 INFO:teuthology.orchestra.run.smithi057.stdout:4 2024-03-24T10:44:34.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.671+0000 7f48b2da7640 1 -- 172.21.15.57:0/1320629926 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f48ac06b780 msgr2=0x7f48ac156fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:34.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.671+0000 7f48b2da7640 1 --2- 172.21.15.57:0/1320629926 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f48ac06b780 0x7f48ac156fd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f489800e8a0 tx=0x7f489800ed60 comp rx=0 tx=0).stop 2024-03-24T10:44:34.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.671+0000 7f48b2da7640 1 -- 172.21.15.57:0/1320629926 shutdown_connections 2024-03-24T10:44:34.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.671+0000 7f48b2da7640 1 --2- 172.21.15.57:0/1320629926 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f48ac070cd0 0x7f48ac157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.671+0000 7f48b2da7640 1 --2- 172.21.15.57:0/1320629926 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f48ac06b780 0x7f48ac156fd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.671+0000 7f48b2da7640 1 --2- 172.21.15.57:0/1320629926 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f48ac06b3a0 0x7f48ac0cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.671+0000 7f48b2da7640 1 -- 172.21.15.57:0/1320629926 >> 172.21.15.57:0/1320629926 conn(0x7f48ac06c240 msgr2=0x7f48ac1588c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:34.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.671+0000 7f48b2da7640 1 -- 172.21.15.57:0/1320629926 shutdown_connections 2024-03-24T10:44:34.672 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.671+0000 7f48b2da7640 1 -- 172.21.15.57:0/1320629926 wait complete. 2024-03-24T10:44:34.682 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph osd new 5cf19cc7-215f-4d69-a30a-80543c0293e7 5 2024-03-24T10:44:34.738 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:34.738+0000 7ff1af22f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T10:44:34.782 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:34.781+0000 7f764ec40040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T10:44:34.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.822+0000 7ff870aa4640 1 Processor -- start 2024-03-24T10:44:34.823 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.822+0000 7ff870aa4640 1 -- start start 2024-03-24T10:44:34.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.823+0000 7ff870aa4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff86c0cf340 0x7ff86c0cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:34.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.823+0000 7ff870aa4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff86c0cc3e0 con 0x7ff86c0ccb80 2024-03-24T10:44:34.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.823+0000 7ff870aa4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff86c15a5f0 con 0x7ff86c0cc000 2024-03-24T10:44:34.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.823+0000 7ff870aa4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7ff86c15a400 con 0x7ff86c0cf340 2024-03-24T10:44:34.824 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.824+0000 7ff863fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff86c0cf340 0x7ff86c0cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:34.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.824+0000 7ff86ad9d640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7ff86c0ccb80 0x7ff86c06bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.57:58778/0 (socket says 172.21.15.57:58778) 2024-03-24T10:44:34.825 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.824+0000 7ff86ad9d640 1 -- 172.21.15.57:0/1287611378 learned_addr learned my addr 172.21.15.57:0/1287611378 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-24T10:44:34.827 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.827+0000 7ff869d9b640 1 -- 172.21.15.57:0/1287611378 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 269017791 0 0) 0x7ff86c0cc3e0 con 0x7ff86c0ccb80 2024-03-24T10:44:34.827 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.827+0000 7ff869d9b640 1 -- 172.21.15.57:0/1287611378 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff858003520 con 0x7ff86c0ccb80 2024-03-24T10:44:34.828 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.827+0000 7ff869d9b640 1 -- 172.21.15.57:0/1287611378 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1306508911 0 0) 0x7ff86c15a5f0 con 0x7ff86c0cc000 2024-03-24T10:44:34.828 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.827+0000 7ff869d9b640 1 -- 172.21.15.57:0/1287611378 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff86c0cc3e0 con 0x7ff86c0cc000 2024-03-24T10:44:34.829 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.829+0000 7ff863fff640 1 -- 172.21.15.57:0/1287611378 >> v1:172.21.15.57:6789/0 conn(0x7ff86c0cc000 legacy=0x7ff86c15e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:34.829 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.829+0000 7ff863fff640 1 -- 172.21.15.57:0/1287611378 >> v1:172.21.15.5:6789/0 conn(0x7ff86c0ccb80 legacy=0x7ff86c06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:34.829 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.829+0000 7ff863fff640 1 -- 172.21.15.57:0/1287611378 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff86c15a8b0 con 0x7ff86c0cf340 2024-03-24T10:44:34.832 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.831+0000 7ff863fff640 1 --2- 172.21.15.57:0/1287611378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff86c0cf340 0x7ff86c0cc640 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff8540097a0 tx=0x7ff85402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c5c87cf2c3edfa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:34.837 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.836+0000 7ff869d9b640 1 -- 172.21.15.57:0/1287611378 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff854030610 con 0x7ff86c0cf340 2024-03-24T10:44:34.837 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.836+0000 7ff869d9b640 1 -- 172.21.15.57:0/1287611378 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff854030770 con 0x7ff86c0cf340 2024-03-24T10:44:34.837 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.836+0000 7ff869d9b640 1 -- 172.21.15.57:0/1287611378 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff854030b00 con 0x7ff86c0cf340 2024-03-24T10:44:34.838 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.837+0000 7ff870aa4640 1 -- 172.21.15.57:0/1287611378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff86c0cf340 msgr2=0x7ff86c0cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:34.838 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.837+0000 7ff870aa4640 1 --2- 172.21.15.57:0/1287611378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff86c0cf340 0x7ff86c0cc640 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff8540097a0 tx=0x7ff85402b960 comp rx=0 tx=0).stop 2024-03-24T10:44:34.838 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.838+0000 7ff870aa4640 1 -- 172.21.15.57:0/1287611378 shutdown_connections 2024-03-24T10:44:34.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.838+0000 7ff870aa4640 1 --2- 172.21.15.57:0/1287611378 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff86c0cf340 0x7ff86c0cc640 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.838+0000 7ff870aa4640 1 -- 172.21.15.57:0/1287611378 >> 172.21.15.57:0/1287611378 conn(0x7ff86c0cfc30 msgr2=0x7ff86c0d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:34.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.838+0000 7ff870aa4640 1 -- 172.21.15.57:0/1287611378 shutdown_connections 2024-03-24T10:44:34.839 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.838+0000 7ff870aa4640 1 -- 172.21.15.57:0/1287611378 wait complete. 2024-03-24T10:44:34.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.840+0000 7ff870aa4640 1 Processor -- start 2024-03-24T10:44:34.840 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.840+0000 7ff870aa4640 1 -- start start 2024-03-24T10:44:34.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.841+0000 7ff870aa4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff86c0cc000 0x7ff86c1508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:34.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.841+0000 7ff870aa4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff86c0ccb80 0x7ff86c0c6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:34.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.841+0000 7ff870aa4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff86c0cf340 0x7ff86c0cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:34.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.841+0000 7ff870aa4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff86c1efb70 con 0x7ff86c0cc000 2024-03-24T10:44:34.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.841+0000 7ff870aa4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff86c150e00 con 0x7ff86c0cf340 2024-03-24T10:44:34.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.841+0000 7ff870aa4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff86c1f3210 con 0x7ff86c0ccb80 2024-03-24T10:44:34.842 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.841+0000 7ff86ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff86c0cf340 0x7ff86c0cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:34.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.841+0000 7ff86a59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff86c0cc000 0x7ff86c1508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:34.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.841+0000 7ff863fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff86c0ccb80 0x7ff86c0c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:34.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.842+0000 7ff86ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff86c0cf340 0x7ff86c0cda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34050/0 (socket says 172.21.15.57:34050) 2024-03-24T10:44:34.843 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.842+0000 7ff86ad9d640 1 -- 172.21.15.57:0/1341039274 learned_addr learned my addr 172.21.15.57:0/1341039274 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-03-24T10:44:34.845 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.845+0000 7ff86ad9d640 1 -- 172.21.15.57:0/1341039274 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff86c0ccb80 msgr2=0x7ff86c0c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:34.845 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.845+0000 7ff86ad9d640 1 --2- 172.21.15.57:0/1341039274 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff86c0ccb80 0x7ff86c0c6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.845 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.845+0000 7ff86ad9d640 1 -- 172.21.15.57:0/1341039274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff86c0cc000 msgr2=0x7ff86c1508c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:34.846 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.845+0000 7ff86ad9d640 1 --2- 172.21.15.57:0/1341039274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff86c0cc000 0x7ff86c1508c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:34.846 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.845+0000 7ff86ad9d640 1 -- 172.21.15.57:0/1341039274 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff854009400 con 0x7ff86c0cf340 2024-03-24T10:44:34.847 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.847+0000 7ff86ad9d640 1 --2- 172.21.15.57:0/1341039274 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff86c0cf340 0x7ff86c0cda10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff86400cb40 tx=0x7ff86400a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:34.851 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.851+0000 7ff862ffd640 1 -- 172.21.15.57:0/1341039274 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86400aca0 con 0x7ff86c0cf340 2024-03-24T10:44:34.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.851+0000 7ff862ffd640 1 -- 172.21.15.57:0/1341039274 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff86400ae20 con 0x7ff86c0cf340 2024-03-24T10:44:34.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.851+0000 7ff870aa4640 1 -- 172.21.15.57:0/1341039274 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff86c1f34f0 con 0x7ff86c0cf340 2024-03-24T10:44:34.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.851+0000 7ff862ffd640 1 -- 172.21.15.57:0/1341039274 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff864012640 con 0x7ff86c0cf340 2024-03-24T10:44:34.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.851+0000 7ff870aa4640 1 -- 172.21.15.57:0/1341039274 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff86c1f3ab0 con 0x7ff86c0cf340 2024-03-24T10:44:34.852 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:34.852+0000 7ff870aa4640 1 -- 172.21.15.57:0/1341039274 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff86c1efce0 con 0x7ff86c0cf340 2024-03-24T10:44:34.884 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:34.885+0000 7ff1af22f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T10:44:34.929 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:34.928+0000 7f764ec40040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T10:44:35.005 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:35.005+0000 7ff1af22f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T10:44:35.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.036+0000 7ff862ffd640 1 -- 172.21.15.57:0/1341039274 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff8640127a0 con 0x7ff86c0cf340 2024-03-24T10:44:35.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.036+0000 7ff862ffd640 1 -- 172.21.15.57:0/1341039274 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7ff8640075f0 con 0x7ff86c0cf340 2024-03-24T10:44:35.036 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.036+0000 7ff862ffd640 1 -- 172.21.15.57:0/1341039274 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff86c1efce0 con 0x7ff86c0cf340 2024-03-24T10:44:35.050 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:35.049+0000 7f764ec40040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T10:44:35.053 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:35.053+0000 7ff1af22f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T10:44:35.097 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:35.097+0000 7f764ec40040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T10:44:35.110 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.110+0000 7ff870aa4640 1 -- 172.21.15.57:0/1341039274 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5cf19cc7-215f-4d69-a30a-80543c0293e7", "id": 5} v 0) -- 0x7ff86c0681d0 con 0x7ff86c0cf340 2024-03-24T10:44:35.171 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:35.171+0000 7ff1af22f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:44:35.216 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:35.216+0000 7f764ec40040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:44:35.224 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.224+0000 7ff862ffd640 1 -- 172.21.15.57:0/1341039274 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5cf19cc7-215f-4d69-a30a-80543c0293e7", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7ff864017070 con 0x7ff86c0cf340 2024-03-24T10:44:35.224 INFO:teuthology.orchestra.run.smithi057.stdout:5 2024-03-24T10:44:35.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.225+0000 7ff870aa4640 1 -- 172.21.15.57:0/1341039274 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff86c0cf340 msgr2=0x7ff86c0cda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:35.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.225+0000 7ff870aa4640 1 --2- 172.21.15.57:0/1341039274 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff86c0cf340 0x7ff86c0cda10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff86400cb40 tx=0x7ff86400a060 comp rx=0 tx=0).stop 2024-03-24T10:44:35.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.225+0000 7ff870aa4640 1 -- 172.21.15.57:0/1341039274 shutdown_connections 2024-03-24T10:44:35.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.225+0000 7ff870aa4640 1 --2- 172.21.15.57:0/1341039274 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff86c0cf340 0x7ff86c0cda10 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:35.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.225+0000 7ff870aa4640 1 --2- 172.21.15.57:0/1341039274 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff86c0ccb80 0x7ff86c0c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:35.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.225+0000 7ff870aa4640 1 --2- 172.21.15.57:0/1341039274 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff86c0cc000 0x7ff86c1508c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:35.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.225+0000 7ff870aa4640 1 -- 172.21.15.57:0/1341039274 >> 172.21.15.57:0/1341039274 conn(0x7ff86c0cfc30 msgr2=0x7ff86c159100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:35.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.225+0000 7ff870aa4640 1 -- 172.21.15.57:0/1341039274 shutdown_connections 2024-03-24T10:44:35.226 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.225+0000 7ff870aa4640 1 -- 172.21.15.57:0/1341039274 wait complete. 2024-03-24T10:44:35.237 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph osd new a5e76c69-c3fc-47b2-8f92-e560cf503e45 6 2024-03-24T10:44:35.274 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:35.274+0000 7ff1af22f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T10:44:35.321 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:35.320+0000 7f764ec40040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T10:44:35.341 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.340+0000 7f9b7f834640 1 Processor -- start 2024-03-24T10:44:35.341 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.340+0000 7f9b7f834640 1 -- start start 2024-03-24T10:44:35.342 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.341+0000 7f9b7f834640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9b781523e0 0x7f9b7815c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:35.342 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.341+0000 7f9b7f834640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9b780cae50 0x7f9b78071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:35.342 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.341+0000 7f9b7f834640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b780c2e10 con 0x7f9b78158090 2024-03-24T10:44:35.342 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.341+0000 7f9b7f834640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f9b781590c0 con 0x7f9b780cae50 2024-03-24T10:44:35.342 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.341+0000 7f9b7f834640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f9b781527c0 con 0x7f9b781523e0 2024-03-24T10:44:35.344 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.344+0000 7f9b7d5d0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9b781523e0 0x7f9b7815c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:35.345 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.344+0000 7f9b7d5d0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9b781523e0 0x7f9b7815c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34054/0 (socket says 172.21.15.57:34054) 2024-03-24T10:44:35.345 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.344+0000 7f9b7d5d0640 1 -- 172.21.15.57:0/35733388 learned_addr learned my addr 172.21.15.57:0/35733388 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-03-24T10:44:35.349 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.349+0000 7f9b7d5d0640 1 -- 172.21.15.57:0/35733388 >> v2:172.21.15.5:3300/0 conn(0x7f9b780cae50 msgr2=0x7f9b78071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:35.349 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.349+0000 7f9b7d5d0640 1 --2- 172.21.15.57:0/35733388 >> v2:172.21.15.5:3300/0 conn(0x7f9b780cae50 0x7f9b78071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:35.349 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.349+0000 7f9b7d5d0640 1 -- 172.21.15.57:0/35733388 >> v1:172.21.15.57:6789/0 conn(0x7f9b78158090 legacy=0x7f9b780c32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:35.349 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.349+0000 7f9b7d5d0640 1 -- 172.21.15.57:0/35733388 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b78158540 con 0x7f9b781523e0 2024-03-24T10:44:35.354 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.354+0000 7f9b7d5d0640 1 --2- 172.21.15.57:0/35733388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b781523e0 0x7f9b7815c040 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9b7806ffe0 tx=0x7f9b7402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f91364b0ec2c6c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:35.378 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.377+0000 7f9b6ffff640 1 -- 172.21.15.57:0/35733388 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b74030860 con 0x7f9b781523e0 2024-03-24T10:44:35.378 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.377+0000 7f9b6ffff640 1 -- 172.21.15.57:0/35733388 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b740309c0 con 0x7f9b781523e0 2024-03-24T10:44:35.378 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.377+0000 7f9b6ffff640 1 -- 172.21.15.57:0/35733388 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b74030d50 con 0x7f9b781523e0 2024-03-24T10:44:35.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.378+0000 7f9b7f834640 1 -- 172.21.15.57:0/35733388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b781523e0 msgr2=0x7f9b7815c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:35.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.378+0000 7f9b7f834640 1 --2- 172.21.15.57:0/35733388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b781523e0 0x7f9b7815c040 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9b7806ffe0 tx=0x7f9b7402b9a0 comp rx=0 tx=0).stop 2024-03-24T10:44:35.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.378+0000 7f9b7f834640 1 -- 172.21.15.57:0/35733388 shutdown_connections 2024-03-24T10:44:35.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.378+0000 7f9b7f834640 1 --2- 172.21.15.57:0/35733388 >> v2:172.21.15.5:3300/0 conn(0x7f9b780cae50 0x7f9b78071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:35.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.378+0000 7f9b7f834640 1 --2- 172.21.15.57:0/35733388 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b781523e0 0x7f9b7815c040 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:35.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.378+0000 7f9b7f834640 1 -- 172.21.15.57:0/35733388 >> 172.21.15.57:0/35733388 conn(0x7f9b780706e0 msgr2=0x7f9b7806f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:35.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.378+0000 7f9b7f834640 1 -- 172.21.15.57:0/35733388 shutdown_connections 2024-03-24T10:44:35.379 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.378+0000 7f9b7f834640 1 -- 172.21.15.57:0/35733388 wait complete. 2024-03-24T10:44:35.380 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.379+0000 7f9b7f834640 1 Processor -- start 2024-03-24T10:44:35.380 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.379+0000 7f9b7f834640 1 -- start start 2024-03-24T10:44:35.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.380+0000 7f9b7f834640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b780cae50 0x7f9b781487b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:35.381 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.380+0000 7f9b7f834640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9b781523e0 0x7f9b781ede50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:35.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.380+0000 7f9b7f834640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b78158090 0x7f9b780cca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:35.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.380+0000 7f9b7f834640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f9b780ca300 con 0x7f9b780cae50 2024-03-24T10:44:35.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.380+0000 7f9b7f834640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9b780c5490 con 0x7f9b78158090 2024-03-24T10:44:35.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.380+0000 7f9b7f834640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f9b7815adc0 con 0x7f9b781523e0 2024-03-24T10:44:35.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.381+0000 7f9b7cdcf640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9b781523e0 0x7f9b781ede50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:35.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.381+0000 7f9b7ddd1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b78158090 0x7f9b780cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:35.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.381+0000 7f9b7d5d0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b780cae50 0x7f9b781487b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:35.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.381+0000 7f9b7cdcf640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9b781523e0 0x7f9b781ede50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.57:52120/0 (socket says 172.21.15.57:52120) 2024-03-24T10:44:35.382 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.381+0000 7f9b7cdcf640 1 -- 172.21.15.57:0/804948868 learned_addr learned my addr 172.21.15.57:0/804948868 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-03-24T10:44:35.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.384+0000 7f9b7cdcf640 1 -- 172.21.15.57:0/804948868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b78158090 msgr2=0x7f9b780cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:35.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.384+0000 7f9b7cdcf640 1 --2- 172.21.15.57:0/804948868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b78158090 0x7f9b780cca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:35.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.384+0000 7f9b7cdcf640 1 -- 172.21.15.57:0/804948868 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b780cae50 msgr2=0x7f9b781487b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:35.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.384+0000 7f9b7cdcf640 1 --2- 172.21.15.57:0/804948868 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b780cae50 0x7f9b781487b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:35.385 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.384+0000 7f9b7cdcf640 1 -- 172.21.15.57:0/804948868 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b740094d0 con 0x7f9b781523e0 2024-03-24T10:44:35.387 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.387+0000 7f9b7cdcf640 1 --2- 172.21.15.57:0/804948868 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9b781523e0 0x7f9b781ede50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9b6800c440 tx=0x7f9b68001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:35.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.390+0000 7f9b6dffb640 1 -- 172.21.15.57:0/804948868 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b680052d0 con 0x7f9b781523e0 2024-03-24T10:44:35.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.390+0000 7f9b6dffb640 1 -- 172.21.15.57:0/804948868 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b68005430 con 0x7f9b781523e0 2024-03-24T10:44:35.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.390+0000 7f9b6dffb640 1 -- 172.21.15.57:0/804948868 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b68007260 con 0x7f9b781523e0 2024-03-24T10:44:35.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.390+0000 7f9b7f834640 1 -- 172.21.15.57:0/804948868 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b7815b830 con 0x7f9b781523e0 2024-03-24T10:44:35.391 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.391+0000 7f9b7f834640 1 -- 172.21.15.57:0/804948868 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b7815b030 con 0x7f9b781523e0 2024-03-24T10:44:35.392 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.391+0000 7f9b7f834640 1 -- 172.21.15.57:0/804948868 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b780688e0 con 0x7f9b781523e0 2024-03-24T10:44:35.443 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:35.443+0000 7ff1af22f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:44:35.486 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:35.486+0000 7ff1af22f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T10:44:35.487 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:35.487+0000 7f764ec40040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T10:44:35.496 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.495+0000 7f9b6dffb640 1 -- 172.21.15.57:0/804948868 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9b680073c0 con 0x7f9b781523e0 2024-03-24T10:44:35.549 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.496+0000 7f9b6dffb640 1 -- 172.21.15.57:0/804948868 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f9b68011590 con 0x7f9b781523e0 2024-03-24T10:44:35.549 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.496+0000 7f9b6dffb640 1 -- 172.21.15.57:0/804948868 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9b68020060 con 0x7f9b781523e0 2024-03-24T10:44:35.549 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:35.531+0000 7f764ec40040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T10:44:35.570 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.569+0000 7f9b7f834640 1 -- 172.21.15.57:0/804948868 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a5e76c69-c3fc-47b2-8f92-e560cf503e45", "id": 6} v 0) -- 0x7f9b7815c0e0 con 0x7f9b781523e0 2024-03-24T10:44:35.634 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.634+0000 7f9b6dffb640 1 -- 172.21.15.57:0/804948868 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a5e76c69-c3fc-47b2-8f92-e560cf503e45", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f9b68003910 con 0x7f9b781523e0 2024-03-24T10:44:35.634 INFO:teuthology.orchestra.run.smithi057.stdout:6 2024-03-24T10:44:35.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.635+0000 7f9b7f834640 1 -- 172.21.15.57:0/804948868 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9b781523e0 msgr2=0x7f9b781ede50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:35.635 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.635+0000 7f9b7f834640 1 --2- 172.21.15.57:0/804948868 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9b781523e0 0x7f9b781ede50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9b6800c440 tx=0x7f9b68001e70 comp rx=0 tx=0).stop 2024-03-24T10:44:35.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.635+0000 7f9b7f834640 1 -- 172.21.15.57:0/804948868 shutdown_connections 2024-03-24T10:44:35.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.635+0000 7f9b7f834640 1 --2- 172.21.15.57:0/804948868 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b78158090 0x7f9b780cca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:35.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.635+0000 7f9b7f834640 1 --2- 172.21.15.57:0/804948868 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9b781523e0 0x7f9b781ede50 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:35.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.635+0000 7f9b7f834640 1 --2- 172.21.15.57:0/804948868 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b780cae50 0x7f9b781487b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:35.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.635+0000 7f9b7f834640 1 -- 172.21.15.57:0/804948868 >> 172.21.15.57:0/804948868 conn(0x7f9b780706e0 msgr2=0x7f9b7814d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:35.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.635+0000 7f9b7f834640 1 -- 172.21.15.57:0/804948868 shutdown_connections 2024-03-24T10:44:35.636 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.635+0000 7f9b7f834640 1 -- 172.21.15.57:0/804948868 wait complete. 2024-03-24T10:44:35.646 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph osd new 47d0b873-e3ac-4f40-a372-681436e45d0c 7 2024-03-24T10:44:35.651 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:35.651+0000 7ff1af22f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T10:44:35.697 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:35.697+0000 7f764ec40040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T10:44:35.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.746+0000 7fa2981a7640 1 Processor -- start 2024-03-24T10:44:35.746 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.746+0000 7fa2981a7640 1 -- start start 2024-03-24T10:44:35.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.747+0000 7fa2981a7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa290061f80 0x7fa29006da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:35.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.747+0000 7fa2981a7640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa29006b130 con 0x7fa29006df50 2024-03-24T10:44:35.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.747+0000 7fa2981a7640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa29006bcb0 con 0x7fa29006d630 2024-03-24T10:44:35.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.747+0000 7fa2981a7640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fa2900c46d0 con 0x7fa290061f80 2024-03-24T10:44:35.748 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.748+0000 7fa295742640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa290061f80 0x7fa29006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:35.749 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.748+0000 7fa295f43640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa29006d630 0x7fa2900c7030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:46208/0 (socket says 172.21.15.57:46208) 2024-03-24T10:44:35.749 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.748+0000 7fa295f43640 1 -- 172.21.15.57:0/2754969362 learned_addr learned my addr 172.21.15.57:0/2754969362 (peer_addr_for_me v1:172.21.15.57:0/0) 2024-03-24T10:44:35.751 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.751+0000 7fa294f41640 1 -- 172.21.15.57:0/2754969362 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 504713054 0 0) 0x7fa29006bcb0 con 0x7fa29006d630 2024-03-24T10:44:35.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.751+0000 7fa294f41640 1 -- 172.21.15.57:0/2754969362 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa27c003520 con 0x7fa29006d630 2024-03-24T10:44:35.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.751+0000 7fa295742640 1 -- 172.21.15.57:0/2754969362 >> v1:172.21.15.57:6789/0 conn(0x7fa29006d630 legacy=0x7fa2900c7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:35.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.751+0000 7fa295742640 1 -- 172.21.15.57:0/2754969362 >> v1:172.21.15.5:6789/0 conn(0x7fa29006df50 legacy=0x7fa290164ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:35.752 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.751+0000 7fa295742640 1 -- 172.21.15.57:0/2754969362 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa29015a570 con 0x7fa290061f80 2024-03-24T10:44:35.755 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.755+0000 7fa295742640 1 --2- 172.21.15.57:0/2754969362 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa290061f80 0x7fa29006da10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa2800097a0 tx=0x7fa28002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8831ee5aec2c72 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:35.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.759+0000 7fa294f41640 1 -- 172.21.15.57:0/2754969362 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa280030610 con 0x7fa290061f80 2024-03-24T10:44:35.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.760+0000 7fa294f41640 1 -- 172.21.15.57:0/2754969362 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa280030770 con 0x7fa290061f80 2024-03-24T10:44:35.760 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.760+0000 7fa294f41640 1 -- 172.21.15.57:0/2754969362 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa280030b00 con 0x7fa290061f80 2024-03-24T10:44:35.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.760+0000 7fa2981a7640 1 -- 172.21.15.57:0/2754969362 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa290061f80 msgr2=0x7fa29006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:35.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.760+0000 7fa2981a7640 1 --2- 172.21.15.57:0/2754969362 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa290061f80 0x7fa29006da10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa2800097a0 tx=0x7fa28002b960 comp rx=0 tx=0).stop 2024-03-24T10:44:35.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.760+0000 7fa2981a7640 1 -- 172.21.15.57:0/2754969362 shutdown_connections 2024-03-24T10:44:35.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.760+0000 7fa2981a7640 1 --2- 172.21.15.57:0/2754969362 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa290061f80 0x7fa29006da10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:35.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.760+0000 7fa2981a7640 1 -- 172.21.15.57:0/2754969362 >> 172.21.15.57:0/2754969362 conn(0x7fa2900cc270 msgr2=0x7fa2900cc680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:35.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.761+0000 7fa2981a7640 1 -- 172.21.15.57:0/2754969362 shutdown_connections 2024-03-24T10:44:35.761 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.761+0000 7fa2981a7640 1 -- 172.21.15.57:0/2754969362 wait complete. 2024-03-24T10:44:35.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.762+0000 7fa2981a7640 1 Processor -- start 2024-03-24T10:44:35.762 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.762+0000 7fa2981a7640 1 -- start start 2024-03-24T10:44:35.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.762+0000 7fa2981a7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa290061f80 0x7fa2900c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:35.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.762+0000 7fa2981a7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa29006d630 0x7fa2900c5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:35.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.762+0000 7fa2981a7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa29006df50 0x7fa290146cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:35.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.762+0000 7fa2981a7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa290149640 con 0x7fa29006d630 2024-03-24T10:44:35.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.762+0000 7fa2981a7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa2900c9980 con 0x7fa29006df50 2024-03-24T10:44:35.763 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.762+0000 7fa2981a7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fa2901516d0 con 0x7fa290061f80 2024-03-24T10:44:35.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.763+0000 7fa295742640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa29006d630 0x7fa2900c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:35.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.763+0000 7fa295f43640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa290061f80 0x7fa2900c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:35.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.763+0000 7fa296744640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa29006df50 0x7fa290146cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:35.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.763+0000 7fa295f43640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa290061f80 0x7fa2900c9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.57:52132/0 (socket says 172.21.15.57:52132) 2024-03-24T10:44:35.764 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.763+0000 7fa295f43640 1 -- 172.21.15.57:0/3936295908 learned_addr learned my addr 172.21.15.57:0/3936295908 (peer_addr_for_me v2:172.21.15.57:0/0) 2024-03-24T10:44:35.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.766+0000 7fa295f43640 1 -- 172.21.15.57:0/3936295908 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa29006df50 msgr2=0x7fa290146cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:35.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.766+0000 7fa295f43640 1 --2- 172.21.15.57:0/3936295908 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa29006df50 0x7fa290146cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:35.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.766+0000 7fa295f43640 1 -- 172.21.15.57:0/3936295908 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa29006d630 msgr2=0x7fa2900c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:35.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.766+0000 7fa295f43640 1 --2- 172.21.15.57:0/3936295908 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa29006d630 0x7fa2900c5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:35.767 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.767+0000 7fa295f43640 1 -- 172.21.15.57:0/3936295908 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa280009400 con 0x7fa290061f80 2024-03-24T10:44:35.769 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.769+0000 7fa295f43640 1 --2- 172.21.15.57:0/3936295908 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa290061f80 0x7fa2900c9440 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa28c00c500 tx=0x7fa28c00c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:35.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.772+0000 7fa2867fc640 1 -- 172.21.15.57:0/3936295908 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa28c005ce0 con 0x7fa290061f80 2024-03-24T10:44:35.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.772+0000 7fa2867fc640 1 -- 172.21.15.57:0/3936295908 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa28c005e40 con 0x7fa290061f80 2024-03-24T10:44:35.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.772+0000 7fa2981a7640 1 -- 172.21.15.57:0/3936295908 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa290152be0 con 0x7fa290061f80 2024-03-24T10:44:35.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.772+0000 7fa2867fc640 1 -- 172.21.15.57:0/3936295908 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa28c007330 con 0x7fa290061f80 2024-03-24T10:44:35.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.772+0000 7fa2981a7640 1 -- 172.21.15.57:0/3936295908 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa290148f80 con 0x7fa290061f80 2024-03-24T10:44:35.773 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.773+0000 7fa2981a7640 1 -- 172.21.15.57:0/3936295908 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa290148dd0 con 0x7fa290061f80 2024-03-24T10:44:35.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.899+0000 7fa2867fc640 1 -- 172.21.15.57:0/3936295908 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa28c007550 con 0x7fa290061f80 2024-03-24T10:44:35.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.899+0000 7fa2867fc640 1 -- 172.21.15.57:0/3936295908 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7fa28c01ba10 con 0x7fa290061f80 2024-03-24T10:44:35.900 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.899+0000 7fa2867fc640 1 -- 172.21.15.57:0/3936295908 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa28c017070 con 0x7fa290061f80 2024-03-24T10:44:35.973 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:35.973+0000 7fa2981a7640 1 -- 172.21.15.57:0/3936295908 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "osd new", "uuid": "47d0b873-e3ac-4f40-a372-681436e45d0c", "id": 7} v 0) -- 0x7fa2900688e0 con 0x7fa290061f80 2024-03-24T10:44:36.074 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:36.073+0000 7fa2867fc640 1 -- 172.21.15.57:0/3936295908 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "47d0b873-e3ac-4f40-a372-681436e45d0c", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7fa28c01bdb0 con 0x7fa290061f80 2024-03-24T10:44:36.074 INFO:teuthology.orchestra.run.smithi057.stdout:7 2024-03-24T10:44:36.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:36.074+0000 7fa2981a7640 1 -- 172.21.15.57:0/3936295908 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa290061f80 msgr2=0x7fa2900c9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:36.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:36.074+0000 7fa2981a7640 1 --2- 172.21.15.57:0/3936295908 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa290061f80 0x7fa2900c9440 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa28c00c500 tx=0x7fa28c00c9c0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:36.074+0000 7fa2981a7640 1 -- 172.21.15.57:0/3936295908 shutdown_connections 2024-03-24T10:44:36.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:36.074+0000 7fa2981a7640 1 --2- 172.21.15.57:0/3936295908 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa29006d630 0x7fa2900c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:36.074+0000 7fa2981a7640 1 --2- 172.21.15.57:0/3936295908 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa29006df50 0x7fa290146cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:36.074+0000 7fa2981a7640 1 --2- 172.21.15.57:0/3936295908 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa290061f80 0x7fa2900c9440 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:36.074+0000 7fa2981a7640 1 -- 172.21.15.57:0/3936295908 >> 172.21.15.57:0/3936295908 conn(0x7fa2900cc270 msgr2=0x7fa29014fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:36.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:36.074+0000 7fa2981a7640 1 -- 172.21.15.57:0/3936295908 shutdown_connections 2024-03-24T10:44:36.075 INFO:teuthology.orchestra.run.smithi057.stderr:2024-03-24T10:44:36.074+0000 7fa2981a7640 1 -- 172.21.15.57:0/3936295908 wait complete. 2024-03-24T10:44:36.085 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-24T10:44:36.085 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 0 2024-03-24T10:44:36.087 INFO:tasks.ceph.osd.0:Started 2024-03-24T10:44:36.088 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-24T10:44:36.088 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 1 2024-03-24T10:44:36.090 INFO:tasks.ceph.osd.1:Started 2024-03-24T10:44:36.090 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-24T10:44:36.090 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 2 2024-03-24T10:44:36.092 INFO:tasks.ceph.osd.2:Started 2024-03-24T10:44:36.092 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-24T10:44:36.093 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 3 2024-03-24T10:44:36.095 INFO:tasks.ceph.osd.3:Started 2024-03-24T10:44:36.095 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-24T10:44:36.095 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 4 2024-03-24T10:44:36.097 INFO:tasks.ceph.osd.4:Started 2024-03-24T10:44:36.097 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-24T10:44:36.097 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 5 2024-03-24T10:44:36.099 INFO:tasks.ceph.osd.5:Started 2024-03-24T10:44:36.099 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-24T10:44:36.099 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 6 2024-03-24T10:44:36.101 INFO:tasks.ceph.osd.6:Started 2024-03-24T10:44:36.102 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-24T10:44:36.102 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 7 2024-03-24T10:44:36.104 INFO:tasks.ceph.osd.7:Started 2024-03-24T10:44:36.104 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:44:36.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.234+0000 7fef3f215640 1 Processor -- start 2024-03-24T10:44:36.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.234+0000 7fef3f215640 1 -- start start 2024-03-24T10:44:36.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.235+0000 7fef3f215640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fef381543c0 0x7fef380c4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.235+0000 7fef3f215640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fef380cbe10 0x7fef380ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.235+0000 7fef3f215640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fef380caee0 0x7fef3815a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.235+0000 7fef3f215640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fef3815af60 con 0x7fef380cbe10 2024-03-24T10:44:36.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.235+0000 7fef3f215640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fef3815b240 con 0x7fef381543c0 2024-03-24T10:44:36.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.235+0000 7fef3f215640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fef3815d970 con 0x7fef380caee0 2024-03-24T10:44:36.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.236+0000 7fef3cfb1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fef381543c0 0x7fef380c4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.236+0000 7fef3cfb1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fef381543c0 0x7fef380c4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:53894/0 (socket says 172.21.15.5:53894) 2024-03-24T10:44:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.236+0000 7fef3cfb1640 1 -- 172.21.15.5:0/4154903382 learned_addr learned my addr 172.21.15.5:0/4154903382 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:36.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.238+0000 7fef3cfb1640 1 -- 172.21.15.5:0/4154903382 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fef380caee0 msgr2=0x7fef3815a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:36.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.238+0000 7fef3cfb1640 1 --2- 172.21.15.5:0/4154903382 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fef380caee0 0x7fef3815a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.238+0000 7fef3cfb1640 1 -- 172.21.15.5:0/4154903382 >> v2:172.21.15.5:3300/0 conn(0x7fef380cbe10 msgr2=0x7fef380ca9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:36.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.238+0000 7fef3cfb1640 1 --2- 172.21.15.5:0/4154903382 >> v2:172.21.15.5:3300/0 conn(0x7fef380cbe10 0x7fef380ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.239+0000 7fef3cfb1640 1 -- 172.21.15.5:0/4154903382 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef3815dc50 con 0x7fef381543c0 2024-03-24T10:44:36.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.242+0000 7fef3cfb1640 1 --2- 172.21.15.5:0/4154903382 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef381543c0 0x7fef380c4a00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fef3806d5d0 tx=0x7fef2802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58899175fd34cf86 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:36.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.246+0000 7fef377fe640 1 -- 172.21.15.5:0/4154903382 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef28031870 con 0x7fef381543c0 2024-03-24T10:44:36.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.246+0000 7fef377fe640 1 -- 172.21.15.5:0/4154903382 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef280319d0 con 0x7fef381543c0 2024-03-24T10:44:36.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.246+0000 7fef377fe640 1 -- 172.21.15.5:0/4154903382 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef28031d60 con 0x7fef381543c0 2024-03-24T10:44:36.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.247+0000 7fef3f215640 1 -- 172.21.15.5:0/4154903382 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef381543c0 msgr2=0x7fef380c4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:36.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.247+0000 7fef3f215640 1 --2- 172.21.15.5:0/4154903382 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef381543c0 0x7fef380c4a00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fef3806d5d0 tx=0x7fef2802b910 comp rx=0 tx=0).stop 2024-03-24T10:44:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.247+0000 7fef3f215640 1 -- 172.21.15.5:0/4154903382 shutdown_connections 2024-03-24T10:44:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.247+0000 7fef3f215640 1 --2- 172.21.15.5:0/4154903382 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fef380caee0 0x7fef3815a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.247+0000 7fef3f215640 1 --2- 172.21.15.5:0/4154903382 >> v2:172.21.15.5:3300/0 conn(0x7fef380cbe10 0x7fef380ca9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.247+0000 7fef3f215640 1 --2- 172.21.15.5:0/4154903382 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef381543c0 0x7fef380c4a00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.247+0000 7fef3f215640 1 -- 172.21.15.5:0/4154903382 >> 172.21.15.5:0/4154903382 conn(0x7fef38061b50 msgr2=0x7fef38061f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.247+0000 7fef3f215640 1 -- 172.21.15.5:0/4154903382 shutdown_connections 2024-03-24T10:44:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.247+0000 7fef3f215640 1 -- 172.21.15.5:0/4154903382 wait complete. 2024-03-24T10:44:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.248+0000 7fef3f215640 1 Processor -- start 2024-03-24T10:44:36.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.249+0000 7fef3f215640 1 -- start start 2024-03-24T10:44:36.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.249+0000 7fef3f215640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fef380caee0 0x7fef381f4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.249+0000 7fef3f215640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef380cbe10 0x7fef3814a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.250+0000 7fef3f215640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef381543c0 0x7fef380b53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:36.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.250+0000 7fef3f215640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fef380c9840 con 0x7fef380cbe10 2024-03-24T10:44:36.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.250+0000 7fef3f215640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fef381f46d0 con 0x7fef381543c0 2024-03-24T10:44:36.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.250+0000 7fef3f215640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fef381518a0 con 0x7fef380caee0 2024-03-24T10:44:36.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.250+0000 7fef37fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef380cbe10 0x7fef3814a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.250+0000 7fef3d7b2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef381543c0 0x7fef380b53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.250+0000 7fef3d7b2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef381543c0 0x7fef380b53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:53902/0 (socket says 172.21.15.5:53902) 2024-03-24T10:44:36.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.250+0000 7fef3d7b2640 1 -- 172.21.15.5:0/668433038 learned_addr learned my addr 172.21.15.5:0/668433038 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:36.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.250+0000 7fef3cfb1640 1 --2- 172.21.15.5:0/668433038 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fef380caee0 0x7fef381f4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:36.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.254+0000 7fef37fff640 1 -- 172.21.15.5:0/668433038 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fef380caee0 msgr2=0x7fef381f4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:36.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.254+0000 7fef37fff640 1 --2- 172.21.15.5:0/668433038 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fef380caee0 0x7fef381f4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.254+0000 7fef37fff640 1 -- 172.21.15.5:0/668433038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef381543c0 msgr2=0x7fef380b53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:36.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.254+0000 7fef37fff640 1 --2- 172.21.15.5:0/668433038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef381543c0 0x7fef380b53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.254+0000 7fef37fff640 1 -- 172.21.15.5:0/668433038 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef280094d0 con 0x7fef380cbe10 2024-03-24T10:44:36.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.254+0000 7fef3d7b2640 1 --2- 172.21.15.5:0/668433038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef381543c0 0x7fef380b53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:36.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.256+0000 7fef37fff640 1 --2- 172.21.15.5:0/668433038 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef380cbe10 0x7fef3814a7b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fef2000b3c0 tx=0x7fef2000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:36.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.260+0000 7fef357fa640 1 -- 172.21.15.5:0/668433038 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef20011830 con 0x7fef380cbe10 2024-03-24T10:44:36.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.260+0000 7fef357fa640 1 -- 172.21.15.5:0/668433038 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef200119b0 con 0x7fef380cbe10 2024-03-24T10:44:36.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.260+0000 7fef357fa640 1 -- 172.21.15.5:0/668433038 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef20011d60 con 0x7fef380cbe10 2024-03-24T10:44:36.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.260+0000 7fef3f215640 1 -- 172.21.15.5:0/668433038 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef380c8fa0 con 0x7fef380cbe10 2024-03-24T10:44:36.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.260+0000 7fef3f215640 1 -- 172.21.15.5:0/668433038 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef380c9560 con 0x7fef380cbe10 2024-03-24T10:44:36.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.261+0000 7fef3f215640 1 -- 172.21.15.5:0/668433038 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef38150a40 con 0x7fef380cbe10 2024-03-24T10:44:36.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.372+0000 7fef357fa640 1 -- 172.21.15.5:0/668433038 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fef2001b460 con 0x7fef380cbe10 2024-03-24T10:44:36.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.372+0000 7fef357fa640 1 -- 172.21.15.5:0/668433038 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fef200243e0 con 0x7fef380cbe10 2024-03-24T10:44:36.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.372+0000 7fef357fa640 1 -- 172.21.15.5:0/668433038 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fef20024610 con 0x7fef380cbe10 2024-03-24T10:44:36.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.435+0000 7fef3f215640 1 -- 172.21.15.5:0/668433038 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fef380688e0 con 0x7fef380cbe10 2024-03-24T10:44:36.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.484+0000 7fef357fa640 1 -- 172.21.15.5:0/668433038 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fef2001b8a0 con 0x7fef380cbe10 2024-03-24T10:44:36.485 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:44:36.485 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":10,"fsid":"7d9848db-2fe7-4d95-a6e2-b04f4f639ebc","created":"2024-03-24T10:44:30.435176+0000","modified":"2024-03-24T10:44:35.986391+0000","last_up_change":"0.000000","last_in_change":"2024-03-24T10:44:35.986391+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"50f590b3-f5ec-4e1b-ac46-25cab307a27a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccba9e93-c704-472b-a832-8d4eeb456c47","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e614c5b1-6504-452e-82e0-95c7ca271737","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c60154d0-85e0-46a1-8273-4863bd8ad2dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"21d74109-da3a-4fc0-b2d8-93e9ae3ed30c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5cf19cc7-215f-4d69-a30a-80543c0293e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a5e76c69-c3fc-47b2-8f92-e560cf503e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47d0b873-e3ac-4f40-a372-681436e45d0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:44:36.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.485+0000 7fef3f215640 1 -- 172.21.15.5:0/668433038 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef380cbe10 msgr2=0x7fef3814a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:36.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.485+0000 7fef3f215640 1 --2- 172.21.15.5:0/668433038 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef380cbe10 0x7fef3814a7b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fef2000b3c0 tx=0x7fef2000b880 comp rx=0 tx=0).stop 2024-03-24T10:44:36.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.485+0000 7fef3f215640 1 -- 172.21.15.5:0/668433038 shutdown_connections 2024-03-24T10:44:36.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.485+0000 7fef3f215640 1 --2- 172.21.15.5:0/668433038 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef380cbe10 0x7fef3814a7b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.485+0000 7fef3f215640 1 --2- 172.21.15.5:0/668433038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef381543c0 0x7fef380b53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.485+0000 7fef3f215640 1 --2- 172.21.15.5:0/668433038 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fef380caee0 0x7fef381f4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:36.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.485+0000 7fef3f215640 1 -- 172.21.15.5:0/668433038 >> 172.21.15.5:0/668433038 conn(0x7fef38061b50 msgr2=0x7fef380cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:36.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.486+0000 7fef3f215640 1 -- 172.21.15.5:0/668433038 shutdown_connections 2024-03-24T10:44:36.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:36.486+0000 7fef3f215640 1 -- 172.21.15.5:0/668433038 wait complete. 2024-03-24T10:44:36.496 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-24T10:44:36.497 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-24T10:44:36.520 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:36.521+0000 7ff1af22f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T10:44:36.607 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:36.607+0000 7f764ec40040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T10:44:36.718 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-03-24T10:44:36.718+0000 7f0183211640 -1 Falling back to public interface 2024-03-24T10:44:36.719 INFO:tasks.ceph.osd.3.smithi005.stderr:2024-03-24T10:44:36.719+0000 7f00b4819640 -1 Falling back to public interface 2024-03-24T10:44:36.723 INFO:tasks.ceph.osd.2.smithi005.stderr:2024-03-24T10:44:36.723+0000 7fb5e2dbd640 -1 Falling back to public interface 2024-03-24T10:44:36.736 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-03-24T10:44:36.736+0000 7fc3e356d640 -1 Falling back to public interface 2024-03-24T10:44:36.743 INFO:tasks.ceph.osd.6.smithi057.stderr:2024-03-24T10:44:36.742+0000 7f9b77c9b640 -1 Falling back to public interface 2024-03-24T10:44:36.752 INFO:tasks.ceph.osd.5.smithi057.stderr:2024-03-24T10:44:36.752+0000 7f0b6f637640 -1 Falling back to public interface 2024-03-24T10:44:36.759 INFO:tasks.ceph.osd.4.smithi057.stderr:2024-03-24T10:44:36.759+0000 7f7d03024640 -1 Falling back to public interface 2024-03-24T10:44:36.764 INFO:tasks.ceph.osd.7.smithi057.stderr:2024-03-24T10:44:36.763+0000 7fb691ec4640 -1 Falling back to public interface 2024-03-24T10:44:36.810 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:36.810+0000 7ff1af22f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T10:44:36.863 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:36.863+0000 7ff1af22f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T10:44:36.897 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:36.897+0000 7f764ec40040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T10:44:36.950 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:36.950+0000 7f764ec40040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T10:44:36.982 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:36.982+0000 7ff1af22f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T10:44:37.072 INFO:tasks.ceph.mgr.x.smithi057.stderr:2024-03-24T10:44:37.072+0000 7f764ec40040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T10:44:37.303 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-24T10:44:37.396 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.396+0000 7fdcc1b3e640 1 Processor -- start 2024-03-24T10:44:37.396 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.396+0000 7fdcc1b3e640 1 -- start start 2024-03-24T10:44:37.397 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.396+0000 7fdcc1b3e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdcbc0cacd0 0x7fdcbc15a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:37.397 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.396+0000 7fdcc1b3e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fdcbc0ce080 0x7fdcbc0cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:37.397 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.396+0000 7fdcc1b3e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcbc06e240 con 0x7fdcbc0cd430 2024-03-24T10:44:37.397 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.396+0000 7fdcc1b3e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fdcbc0cdc30 con 0x7fdcbc0ce080 2024-03-24T10:44:37.397 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.396+0000 7fdcc1b3e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fdcbc0ce8a0 con 0x7fdcbc0cacd0 2024-03-24T10:44:37.398 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.397+0000 7fdcbaffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fdcbc0ce080 0x7fdcbc0cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:37.398 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.397+0000 7fdcbaffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fdcbc0ce080 0x7fdcbc0cc4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:53910/0 (socket says 172.21.15.5:53910) 2024-03-24T10:44:37.398 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.397+0000 7fdcbaffd640 1 -- 172.21.15.5:0/4096796671 learned_addr learned my addr 172.21.15.5:0/4096796671 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:37.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.400+0000 7fdcbaffd640 1 -- 172.21.15.5:0/4096796671 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdcbc0cacd0 msgr2=0x7fdcbc15a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:37.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.400+0000 7fdcbaffd640 1 --2- 172.21.15.5:0/4096796671 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdcbc0cacd0 0x7fdcbc15a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:37.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.400+0000 7fdcbaffd640 1 -- 172.21.15.5:0/4096796671 >> v1:172.21.15.57:6789/0 conn(0x7fdcbc0cd430 legacy=0x7fdcbc0cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:37.400 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.400+0000 7fdcbaffd640 1 -- 172.21.15.5:0/4096796671 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcbc154cc0 con 0x7fdcbc0ce080 2024-03-24T10:44:37.403 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.403+0000 7fdcbaffd640 1 --2- 172.21.15.5:0/4096796671 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcbc0ce080 0x7fdcbc0cc4f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0009870 tx=0x7fdcb002b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f42e802261067e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:37.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.407+0000 7fdcba7fc640 1 -- 172.21.15.5:0/4096796671 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb00317c0 con 0x7fdcbc0ce080 2024-03-24T10:44:37.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.407+0000 7fdcba7fc640 1 -- 172.21.15.5:0/4096796671 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcb0031920 con 0x7fdcbc0ce080 2024-03-24T10:44:37.407 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.407+0000 7fdcba7fc640 1 -- 172.21.15.5:0/4096796671 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb0031cb0 con 0x7fdcbc0ce080 2024-03-24T10:44:37.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.407+0000 7fdcc1b3e640 1 -- 172.21.15.5:0/4096796671 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcbc0ce080 msgr2=0x7fdcbc0cc4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:37.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.407+0000 7fdcc1b3e640 1 --2- 172.21.15.5:0/4096796671 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcbc0ce080 0x7fdcbc0cc4f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0009870 tx=0x7fdcb002b860 comp rx=0 tx=0).stop 2024-03-24T10:44:37.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.407+0000 7fdcc1b3e640 1 -- 172.21.15.5:0/4096796671 shutdown_connections 2024-03-24T10:44:37.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.407+0000 7fdcc1b3e640 1 --2- 172.21.15.5:0/4096796671 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcbc0ce080 0x7fdcbc0cc4f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:37.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.407+0000 7fdcc1b3e640 1 --2- 172.21.15.5:0/4096796671 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdcbc0cacd0 0x7fdcbc15a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:37.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.407+0000 7fdcc1b3e640 1 -- 172.21.15.5:0/4096796671 >> 172.21.15.5:0/4096796671 conn(0x7fdcbc0b1830 msgr2=0x7fdcbc0b1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:37.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.407+0000 7fdcc1b3e640 1 -- 172.21.15.5:0/4096796671 shutdown_connections 2024-03-24T10:44:37.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.407+0000 7fdcc1b3e640 1 -- 172.21.15.5:0/4096796671 wait complete. 2024-03-24T10:44:37.408 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.408+0000 7fdcc1b3e640 1 Processor -- start 2024-03-24T10:44:37.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.409+0000 7fdcc1b3e640 1 -- start start 2024-03-24T10:44:37.409 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.409+0000 7fdcc1b3e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdcbc0cacd0 0x7fdcbc15b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:37.410 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.409+0000 7fdcc1b3e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcbc0cd430 0x7fdcbc1f07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:37.410 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.409+0000 7fdcc1b3e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdcbc156a70 0x7fdcbc0c5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:37.410 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.409+0000 7fdcc1b3e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc1f3a00 con 0x7fdcbc156a70 2024-03-24T10:44:37.410 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.409+0000 7fdcc1b3e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc1f3ce0 con 0x7fdcbc0cd430 2024-03-24T10:44:37.410 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.409+0000 7fdcc1b3e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fdcbc1f3fc0 con 0x7fdcbc0cacd0 2024-03-24T10:44:37.410 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.410+0000 7fdcbaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcbc0cd430 0x7fdcbc1f07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:37.410 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.410+0000 7fdcbbfff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdcbc156a70 0x7fdcbc0c5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:37.410 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.410+0000 7fdcbaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcbc0cd430 0x7fdcbc1f07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:53926/0 (socket says 172.21.15.5:53926) 2024-03-24T10:44:37.411 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.410+0000 7fdcbaffd640 1 -- 172.21.15.5:0/2773858624 learned_addr learned my addr 172.21.15.5:0/2773858624 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:37.413 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.413+0000 7fdcbaffd640 1 -- 172.21.15.5:0/2773858624 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdcbc0cacd0 msgr2=0x7fdcbc15b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:37.413 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.413+0000 7fdcbaffd640 1 --2- 172.21.15.5:0/2773858624 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdcbc0cacd0 0x7fdcbc15b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:37.413 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.413+0000 7fdcbaffd640 1 -- 172.21.15.5:0/2773858624 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdcbc156a70 msgr2=0x7fdcbc0c5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:37.413 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.413+0000 7fdcbaffd640 1 --2- 172.21.15.5:0/2773858624 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdcbc156a70 0x7fdcbc0c5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:37.413 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.413+0000 7fdcbaffd640 1 -- 172.21.15.5:0/2773858624 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb00094d0 con 0x7fdcbc0cd430 2024-03-24T10:44:37.415 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.415+0000 7fdcbaffd640 1 --2- 172.21.15.5:0/2773858624 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcbc0cd430 0x7fdcbc1f07c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0000c00 tx=0x7fdcb00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:37.419 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.419+0000 7fdca3fff640 1 -- 172.21.15.5:0/2773858624 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb0032960 con 0x7fdcbc0cd430 2024-03-24T10:44:37.419 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.419+0000 7fdca3fff640 1 -- 172.21.15.5:0/2773858624 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcb0032ae0 con 0x7fdcbc0cd430 2024-03-24T10:44:37.419 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.419+0000 7fdcc1b3e640 1 -- 172.21.15.5:0/2773858624 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcbc1f42a0 con 0x7fdcbc0cd430 2024-03-24T10:44:37.419 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.419+0000 7fdca3fff640 1 -- 172.21.15.5:0/2773858624 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb0032e90 con 0x7fdcbc0cd430 2024-03-24T10:44:37.419 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.419+0000 7fdcc1b3e640 1 -- 172.21.15.5:0/2773858624 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcbc1eff50 con 0x7fdcbc0cd430 2024-03-24T10:44:37.420 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.420+0000 7fdcc1b3e640 1 -- 172.21.15.5:0/2773858624 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcbc0cf4e0 con 0x7fdcbc0cd430 2024-03-24T10:44:37.430 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.430+0000 7fdca3fff640 1 -- 172.21.15.5:0/2773858624 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 2) ==== 44915+0+0 (secure 0 0 0) 0x7fdcb004d3f0 con 0x7fdcbc0cd430 2024-03-24T10:44:37.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.536+0000 7fdca3fff640 1 -- 172.21.15.5:0/2773858624 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fdcb00394d0 con 0x7fdcbc0cd430 2024-03-24T10:44:37.536 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.536+0000 7fdca3fff640 1 -- 172.21.15.5:0/2773858624 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fdcb00443a0 con 0x7fdcbc0cd430 2024-03-24T10:44:37.606 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.606+0000 7fdcc1b3e640 1 -- 172.21.15.5:0/2773858624 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdcbc1f04e0 con 0x7fdcbc0cd430 2024-03-24T10:44:37.696 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.696+0000 7fdca3fff640 1 -- 172.21.15.5:0/2773858624 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fdcbc1f04e0 con 0x7fdcbc0cd430 2024-03-24T10:44:37.696 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-24T10:44:37.696 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":10,"fsid":"7d9848db-2fe7-4d95-a6e2-b04f4f639ebc","created":"2024-03-24T10:44:30.435176+0000","modified":"2024-03-24T10:44:35.986391+0000","last_up_change":"0.000000","last_in_change":"2024-03-24T10:44:35.986391+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"50f590b3-f5ec-4e1b-ac46-25cab307a27a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ccba9e93-c704-472b-a832-8d4eeb456c47","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e614c5b1-6504-452e-82e0-95c7ca271737","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c60154d0-85e0-46a1-8273-4863bd8ad2dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"21d74109-da3a-4fc0-b2d8-93e9ae3ed30c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5cf19cc7-215f-4d69-a30a-80543c0293e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a5e76c69-c3fc-47b2-8f92-e560cf503e45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"47d0b873-e3ac-4f40-a372-681436e45d0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:44:37.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.697+0000 7fdcc1b3e640 1 -- 172.21.15.5:0/2773858624 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcbc0cd430 msgr2=0x7fdcbc1f07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:37.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.697+0000 7fdcc1b3e640 1 --2- 172.21.15.5:0/2773858624 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcbc0cd430 0x7fdcbc1f07c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0000c00 tx=0x7fdcb00046c0 comp rx=0 tx=0).stop 2024-03-24T10:44:37.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.697+0000 7fdcc1b3e640 1 -- 172.21.15.5:0/2773858624 shutdown_connections 2024-03-24T10:44:37.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.697+0000 7fdcc1b3e640 1 --2- 172.21.15.5:0/2773858624 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdcbc156a70 0x7fdcbc0c5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:37.697 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.697+0000 7fdcc1b3e640 1 --2- 172.21.15.5:0/2773858624 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdcbc0cd430 0x7fdcbc1f07c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:37.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.697+0000 7fdcc1b3e640 1 --2- 172.21.15.5:0/2773858624 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdcbc0cacd0 0x7fdcbc15b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:37.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.697+0000 7fdcc1b3e640 1 -- 172.21.15.5:0/2773858624 >> 172.21.15.5:0/2773858624 conn(0x7fdcbc0b1830 msgr2=0x7fdcbc0c8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:37.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.697+0000 7fdcc1b3e640 1 -- 172.21.15.5:0/2773858624 shutdown_connections 2024-03-24T10:44:37.698 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:37.697+0000 7fdcc1b3e640 1 -- 172.21.15.5:0/2773858624 wait complete. 2024-03-24T10:44:37.707 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-24T10:44:38.765 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:38.765+0000 7ff14f730640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-24T10:44:38.772 INFO:tasks.ceph.mgr.y.smithi005.stderr:2024-03-24T10:44:38.772+0000 7ff14f730640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-24T10:44:40.178 INFO:tasks.ceph.osd.3.smithi005.stderr:2024-03-24T10:44:40.178+0000 7f00b4819640 -1 osd.3 0 log_to_monitors true 2024-03-24T10:44:40.200 INFO:tasks.ceph.osd.6.smithi057.stderr:2024-03-24T10:44:40.201+0000 7f9b77c9b640 -1 osd.6 0 log_to_monitors true 2024-03-24T10:44:40.223 INFO:tasks.ceph.osd.5.smithi057.stderr:2024-03-24T10:44:40.223+0000 7f0b6f637640 -1 osd.5 0 log_to_monitors true 2024-03-24T10:44:40.234 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-03-24T10:44:40.234+0000 7f0183211640 -1 osd.1 0 log_to_monitors true 2024-03-24T10:44:40.246 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-03-24T10:44:40.246+0000 7fc3e356d640 -1 osd.0 0 log_to_monitors true 2024-03-24T10:44:40.247 INFO:tasks.ceph.osd.2.smithi005.stderr:2024-03-24T10:44:40.247+0000 7fb5e2dbd640 -1 osd.2 0 log_to_monitors true 2024-03-24T10:44:40.259 INFO:tasks.ceph.osd.4.smithi057.stderr:2024-03-24T10:44:40.260+0000 7f7d03024640 -1 osd.4 0 log_to_monitors true 2024-03-24T10:44:40.271 INFO:tasks.ceph.osd.7.smithi057.stderr:2024-03-24T10:44:40.272+0000 7fb691ec4640 -1 osd.7 0 log_to_monitors true 2024-03-24T10:44:43.051 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-03-24T10:44:43.051+0000 7fc3df503640 -1 osd.0 0 waiting for initial osdmap 2024-03-24T10:44:43.109 INFO:tasks.ceph.osd.7.smithi057.stderr:2024-03-24T10:44:43.110+0000 7fb68de5a640 -1 osd.7 0 waiting for initial osdmap 2024-03-24T10:44:43.112 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-03-24T10:44:43.112+0000 7fc3d7323640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:44:43.120 INFO:tasks.ceph.osd.4.smithi057.stderr:2024-03-24T10:44:43.120+0000 7f7cfefba640 -1 osd.4 0 waiting for initial osdmap 2024-03-24T10:44:43.130 INFO:tasks.ceph.osd.6.smithi057.stderr:2024-03-24T10:44:43.131+0000 7f9b73c31640 -1 osd.6 0 waiting for initial osdmap 2024-03-24T10:44:43.162 INFO:tasks.ceph.osd.2.smithi005.stderr:2024-03-24T10:44:43.162+0000 7fb5ded53640 -1 osd.2 0 waiting for initial osdmap 2024-03-24T10:44:43.168 INFO:tasks.ceph.osd.4.smithi057.stderr:2024-03-24T10:44:43.169+0000 7f7cf6dda640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:44:43.172 INFO:tasks.ceph.osd.6.smithi057.stderr:2024-03-24T10:44:43.173+0000 7f9b6ba51640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:44:43.203 INFO:tasks.ceph.osd.2.smithi005.stderr:2024-03-24T10:44:43.203+0000 7fb5d6b73640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:44:43.208 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-03-24T10:44:43.208+0000 7f017ebd3640 -1 osd.1 0 waiting for initial osdmap 2024-03-24T10:44:43.224 INFO:tasks.ceph.osd.5.smithi057.stderr:2024-03-24T10:44:43.224+0000 7f0b6b5cd640 -1 osd.5 0 waiting for initial osdmap 2024-03-24T10:44:43.258 INFO:tasks.ceph.osd.5.smithi057.stderr:2024-03-24T10:44:43.259+0000 7f0b633ed640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:44:43.259 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-03-24T10:44:43.259+0000 7f01769f3640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:44:43.345 INFO:tasks.ceph.osd.7.smithi057.stderr:2024-03-24T10:44:43.346+0000 7fb685c7a640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:44:43.353 INFO:tasks.ceph.osd.3.smithi005.stderr:2024-03-24T10:44:43.353+0000 7f00b0a91640 -1 osd.3 0 waiting for initial osdmap 2024-03-24T10:44:43.456 INFO:tasks.ceph.osd.3.smithi005.stderr:2024-03-24T10:44:43.456+0000 7f00a809d640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T10:44:44.513 DEBUG:teuthology.orchestra.run.smithi005:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-24T10:44:44.603 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.603+0000 7ff923db6640 1 Processor -- start 2024-03-24T10:44:44.603 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.603+0000 7ff923db6640 1 -- start start 2024-03-24T10:44:44.604 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.604+0000 7ff923db6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff91c0ce080 0x7ff91c0cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:44.604 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.604+0000 7ff923db6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff91c0cd430 0x7ff91c0cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:44.604 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.604+0000 7ff923db6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff91c06e240 con 0x7ff91c0cacd0 2024-03-24T10:44:44.604 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.604+0000 7ff923db6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7ff91c0cdc30 con 0x7ff91c0cd430 2024-03-24T10:44:44.604 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.604+0000 7ff923db6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff91c0ce8a0 con 0x7ff91c0ce080 2024-03-24T10:44:44.605 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.604+0000 7ff921351640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff91c0ce080 0x7ff91c0cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:44.605 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.604+0000 7ff922353640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff91c0cd430 0x7ff91c0cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:44.605 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.605+0000 7ff921351640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff91c0ce080 0x7ff91c0cc4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:45764/0 (socket says 172.21.15.5:45764) 2024-03-24T10:44:44.605 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.605+0000 7ff921351640 1 -- 172.21.15.5:0/1003641216 learned_addr learned my addr 172.21.15.5:0/1003641216 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:44.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.607+0000 7ff921351640 1 -- 172.21.15.5:0/1003641216 >> v2:172.21.15.57:3300/0 conn(0x7ff91c0cd430 msgr2=0x7ff91c0cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:44.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.607+0000 7ff921351640 1 --2- 172.21.15.5:0/1003641216 >> v2:172.21.15.57:3300/0 conn(0x7ff91c0cd430 0x7ff91c0cca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:44.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.607+0000 7ff921351640 1 -- 172.21.15.5:0/1003641216 >> v1:172.21.15.57:6789/0 conn(0x7ff91c0cacd0 legacy=0x7ff91c15a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:44.607 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.607+0000 7ff921351640 1 -- 172.21.15.5:0/1003641216 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff91c154cc0 con 0x7ff91c0ce080 2024-03-24T10:44:44.609 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.609+0000 7ff921351640 1 --2- 172.21.15.5:0/1003641216 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff91c0ce080 0x7ff91c0cc4f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff90c009870 tx=0x7ff90c02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d69be2354dd43097 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:44.613 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.613+0000 7ff920b50640 1 -- 172.21.15.5:0/1003641216 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff90c0317c0 con 0x7ff91c0ce080 2024-03-24T10:44:44.613 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.613+0000 7ff920b50640 1 -- 172.21.15.5:0/1003641216 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff90c031920 con 0x7ff91c0ce080 2024-03-24T10:44:44.613 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.613+0000 7ff920b50640 1 -- 172.21.15.5:0/1003641216 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff90c031cb0 con 0x7ff91c0ce080 2024-03-24T10:44:44.614 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.614+0000 7ff923db6640 1 -- 172.21.15.5:0/1003641216 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff91c0ce080 msgr2=0x7ff91c0cc4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:44.614 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.614+0000 7ff923db6640 1 --2- 172.21.15.5:0/1003641216 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff91c0ce080 0x7ff91c0cc4f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff90c009870 tx=0x7ff90c02b860 comp rx=0 tx=0).stop 2024-03-24T10:44:44.614 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.614+0000 7ff923db6640 1 -- 172.21.15.5:0/1003641216 shutdown_connections 2024-03-24T10:44:44.615 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.614+0000 7ff923db6640 1 --2- 172.21.15.5:0/1003641216 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff91c0ce080 0x7ff91c0cc4f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:44.615 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.614+0000 7ff923db6640 1 --2- 172.21.15.5:0/1003641216 >> v2:172.21.15.57:3300/0 conn(0x7ff91c0cd430 0x7ff91c0cca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:44.615 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.614+0000 7ff923db6640 1 -- 172.21.15.5:0/1003641216 >> 172.21.15.5:0/1003641216 conn(0x7ff91c0b1830 msgr2=0x7ff91c0b1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:44.615 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.614+0000 7ff923db6640 1 -- 172.21.15.5:0/1003641216 shutdown_connections 2024-03-24T10:44:44.615 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.614+0000 7ff923db6640 1 -- 172.21.15.5:0/1003641216 wait complete. 2024-03-24T10:44:44.615 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.615+0000 7ff923db6640 1 Processor -- start 2024-03-24T10:44:44.615 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.615+0000 7ff923db6640 1 -- start start 2024-03-24T10:44:44.616 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.616+0000 7ff923db6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff91c0cacd0 0x7ff91c14f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:44.617 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.616+0000 7ff923db6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff91c0cd430 0x7ff91c15ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:44.617 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.616+0000 7ff923db6640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff91c14a060 0x7ff91c15bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:44.617 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.616+0000 7ff923db6640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff91c15e590 con 0x7ff91c14a060 2024-03-24T10:44:44.617 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.616+0000 7ff923db6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff91c15dfb0 con 0x7ff91c0cd430 2024-03-24T10:44:44.617 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.616+0000 7ff923db6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff91c0ceb30 con 0x7ff91c0cacd0 2024-03-24T10:44:44.617 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.616+0000 7ff921351640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff91c0cd430 0x7ff91c15ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:44.617 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.616+0000 7ff921b52640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff91c0cacd0 0x7ff91c14f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:44.617 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.617+0000 7ff921351640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff91c0cd430 0x7ff91c15ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:38038/0 (socket says 172.21.15.5:38038) 2024-03-24T10:44:44.617 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.617+0000 7ff921351640 1 -- 172.21.15.5:0/1728094669 learned_addr learned my addr 172.21.15.5:0/1728094669 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:44.620 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.620+0000 7ff921b52640 1 -- 172.21.15.5:0/1728094669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff91c0cd430 msgr2=0x7ff91c15ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:44.620 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.620+0000 7ff921b52640 1 --2- 172.21.15.5:0/1728094669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff91c0cd430 0x7ff91c15ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:44.620 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.620+0000 7ff921b52640 1 -- 172.21.15.5:0/1728094669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff91c14a060 msgr2=0x7ff91c15bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:44.620 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.620+0000 7ff921b52640 1 --2- 172.21.15.5:0/1728094669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff91c14a060 0x7ff91c15bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:44.620 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.620+0000 7ff921b52640 1 -- 172.21.15.5:0/1728094669 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff90c0094d0 con 0x7ff91c0cacd0 2024-03-24T10:44:44.622 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.622+0000 7ff921b52640 1 --2- 172.21.15.5:0/1728094669 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff91c0cacd0 0x7ff91c14f690 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff91800c3f0 tx=0x7ff91800c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:44.626 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.625+0000 7ff9127fc640 1 -- 172.21.15.5:0/1728094669 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9180052d0 con 0x7ff91c0cacd0 2024-03-24T10:44:44.626 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.625+0000 7ff9127fc640 1 -- 172.21.15.5:0/1728094669 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff918005450 con 0x7ff91c0cacd0 2024-03-24T10:44:44.626 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.625+0000 7ff9127fc640 1 -- 172.21.15.5:0/1728094669 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff918007260 con 0x7ff91c0cacd0 2024-03-24T10:44:44.626 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.625+0000 7ff923db6640 1 -- 172.21.15.5:0/1728094669 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff91c1f07f0 con 0x7ff91c0cacd0 2024-03-24T10:44:44.626 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.625+0000 7ff923db6640 1 -- 172.21.15.5:0/1728094669 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff91c15e220 con 0x7ff91c0cacd0 2024-03-24T10:44:44.626 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.626+0000 7ff923db6640 1 -- 172.21.15.5:0/1728094669 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff91c0c7da0 con 0x7ff91c0cacd0 2024-03-24T10:44:44.645 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.645+0000 7ff9127fc640 1 -- 172.21.15.5:0/1728094669 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 4) ==== 89724+0+0 (secure 0 0 0) 0x7ff9180073c0 con 0x7ff91c0cacd0 2024-03-24T10:44:44.645 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.645+0000 7ff9127fc640 1 --2- 172.21.15.5:0/1728094669 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff90006b7d0 0x7ff90006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:44.646 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.645+0000 7ff921351640 1 --2- 172.21.15.5:0/1728094669 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff90006b7d0 0x7ff90006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:44.646 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.646+0000 7ff921351640 1 --2- 172.21.15.5:0/1728094669 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff90006b7d0 0x7ff90006dc90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff90c009870 tx=0x7ff90c03d040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:44.648 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.648+0000 7ff9127fc640 1 -- 172.21.15.5:0/1728094669 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(13..13 src has 1..13) ==== 3071+0+0 (secure 0 0 0) 0x7ff918058440 con 0x7ff91c0cacd0 2024-03-24T10:44:44.981 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:44.981+0000 7ff9127fc640 1 -- 172.21.15.5:0/1728094669 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff9180546e0 con 0x7ff91c0cacd0 2024-03-24T10:44:45.172 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:45.172+0000 7ff923db6640 1 -- 172.21.15.5:0/1728094669 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff91c04f470 con 0x7ff91c0cacd0 2024-03-24T10:44:45.298 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:45.298+0000 7ff9127fc640 1 -- 172.21.15.5:0/1728094669 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+10858 (secure 0 0 0) 0x7ff918057da0 con 0x7ff91c0cacd0 2024-03-24T10:44:45.298 INFO:teuthology.misc.health.smithi005.stdout: 2024-03-24T10:44:45.298 INFO:teuthology.misc.health.smithi005.stdout:{"epoch":14,"fsid":"7d9848db-2fe7-4d95-a6e2-b04f4f639ebc","created":"2024-03-24T10:44:30.435176+0000","modified":"2024-03-24T10:44:44.634047+0000","last_up_change":"2024-03-24T10:44:44.634047+0000","last_in_change":"2024-03-24T10:44:35.986391+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"50f590b3-f5ec-4e1b-ac46-25cab307a27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6825","nonce":1991336051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6827","nonce":1991336051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6831","nonce":1991336051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6829","nonce":1991336051}]},"public_addr":"172.21.15.5:6825/1991336051","cluster_addr":"172.21.15.5:6827/1991336051","heartbeat_back_addr":"172.21.15.5:6831/1991336051","heartbeat_front_addr":"172.21.15.5:6829/1991336051","state":["exists","up"]},{"osd":1,"uuid":"ccba9e93-c704-472b-a832-8d4eeb456c47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6801","nonce":2140667501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6803","nonce":2140667501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6807","nonce":2140667501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6805","nonce":2140667501}]},"public_addr":"172.21.15.5:6801/2140667501","cluster_addr":"172.21.15.5:6803/2140667501","heartbeat_back_addr":"172.21.15.5:6807/2140667501","heartbeat_front_addr":"172.21.15.5:6805/2140667501","state":["exists","up"]},{"osd":2,"uuid":"e614c5b1-6504-452e-82e0-95c7ca271737","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6817","nonce":90414679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6819","nonce":90414679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6823","nonce":90414679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6821","nonce":90414679}]},"public_addr":"172.21.15.5:6817/90414679","cluster_addr":"172.21.15.5:6819/90414679","heartbeat_back_addr":"172.21.15.5:6823/90414679","heartbeat_front_addr":"172.21.15.5:6821/90414679","state":["exists","up"]},{"osd":3,"uuid":"c60154d0-85e0-46a1-8273-4863bd8ad2dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6809","nonce":4212473119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6811","nonce":4212473119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6815","nonce":4212473119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6813","nonce":4212473119}]},"public_addr":"172.21.15.5:6809/4212473119","cluster_addr":"172.21.15.5:6811/4212473119","heartbeat_back_addr":"172.21.15.5:6815/4212473119","heartbeat_front_addr":"172.21.15.5:6813/4212473119","state":["exists","up"]},{"osd":4,"uuid":"21d74109-da3a-4fc0-b2d8-93e9ae3ed30c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6817","nonce":2193986505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6819","nonce":2193986505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6823","nonce":2193986505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6821","nonce":2193986505}]},"public_addr":"172.21.15.57:6817/2193986505","cluster_addr":"172.21.15.57:6819/2193986505","heartbeat_back_addr":"172.21.15.57:6823/2193986505","heartbeat_front_addr":"172.21.15.57:6821/2193986505","state":["exists","up"]},{"osd":5,"uuid":"5cf19cc7-215f-4d69-a30a-80543c0293e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6809","nonce":3044987466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6811","nonce":3044987466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6815","nonce":3044987466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6813","nonce":3044987466}]},"public_addr":"172.21.15.57:6809/3044987466","cluster_addr":"172.21.15.57:6811/3044987466","heartbeat_back_addr":"172.21.15.57:6815/3044987466","heartbeat_front_addr":"172.21.15.57:6813/3044987466","state":["exists","up"]},{"osd":6,"uuid":"a5e76c69-c3fc-47b2-8f92-e560cf503e45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6801","nonce":4290897445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6803","nonce":4290897445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6807","nonce":4290897445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6805","nonce":4290897445}]},"public_addr":"172.21.15.57:6801/4290897445","cluster_addr":"172.21.15.57:6803/4290897445","heartbeat_back_addr":"172.21.15.57:6807/4290897445","heartbeat_front_addr":"172.21.15.57:6805/4290897445","state":["exists","up"]},{"osd":7,"uuid":"47d0b873-e3ac-4f40-a372-681436e45d0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6825","nonce":212605698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6827","nonce":212605698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6831","nonce":212605698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6829","nonce":212605698}]},"public_addr":"172.21.15.57:6825/212605698","cluster_addr":"172.21.15.57:6827/212605698","heartbeat_back_addr":"172.21.15.57:6831/212605698","heartbeat_front_addr":"172.21.15.57:6829/212605698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.211237+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.277082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.190915+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:44:45.300 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:45.300+0000 7ff923db6640 1 -- 172.21.15.5:0/1728094669 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff90006b7d0 msgr2=0x7ff90006dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:45.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:45.300+0000 7ff923db6640 1 --2- 172.21.15.5:0/1728094669 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff90006b7d0 0x7ff90006dc90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff90c009870 tx=0x7ff90c03d040 comp rx=0 tx=0).stop 2024-03-24T10:44:45.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:45.300+0000 7ff923db6640 1 -- 172.21.15.5:0/1728094669 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff91c0cacd0 msgr2=0x7ff91c14f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:45.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:45.300+0000 7ff923db6640 1 --2- 172.21.15.5:0/1728094669 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff91c0cacd0 0x7ff91c14f690 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff91800c3f0 tx=0x7ff91800c8b0 comp rx=0 tx=0).stop 2024-03-24T10:44:45.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:45.300+0000 7ff923db6640 1 -- 172.21.15.5:0/1728094669 shutdown_connections 2024-03-24T10:44:45.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:45.300+0000 7ff923db6640 1 --2- 172.21.15.5:0/1728094669 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff90006b7d0 0x7ff90006dc90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:45.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:45.300+0000 7ff923db6640 1 --2- 172.21.15.5:0/1728094669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff91c14a060 0x7ff91c15bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:45.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:45.300+0000 7ff923db6640 1 --2- 172.21.15.5:0/1728094669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff91c0cd430 0x7ff91c15ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:45.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:45.300+0000 7ff923db6640 1 --2- 172.21.15.5:0/1728094669 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff91c0cacd0 0x7ff91c14f690 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:45.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:45.300+0000 7ff923db6640 1 -- 172.21.15.5:0/1728094669 >> 172.21.15.5:0/1728094669 conn(0x7ff91c0b1830 msgr2=0x7ff91c0b0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:45.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:45.300+0000 7ff923db6640 1 -- 172.21.15.5:0/1728094669 shutdown_connections 2024-03-24T10:44:45.301 INFO:teuthology.misc.health.smithi005.stderr:2024-03-24T10:44:45.300+0000 7ff923db6640 1 -- 172.21.15.5:0/1728094669 wait complete. 2024-03-24T10:44:45.310 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-24T10:44:45.310 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-24T10:44:45.310 DEBUG:tasks.ceph_manager:running mds.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-24T10:44:45.310 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T10:44:45.310 DEBUG:teuthology.orchestra.run.smithi005:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i a 2024-03-24T10:44:45.313 INFO:tasks.ceph.mds.a:Started 2024-03-24T10:44:45.313 DEBUG:tasks.ceph_manager:running mds.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-24T10:44:45.313 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T10:44:45.313 DEBUG:teuthology.orchestra.run.smithi005:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i c 2024-03-24T10:44:45.316 INFO:tasks.ceph.mds.c:Started 2024-03-24T10:44:45.316 DEBUG:tasks.ceph_manager:running mds.e under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.e.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-24T10:44:45.316 INFO:tasks.ceph.mds.e:Restarting daemon 2024-03-24T10:44:45.316 DEBUG:teuthology.orchestra.run.smithi005:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.e.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i e 2024-03-24T10:44:45.318 INFO:tasks.ceph.mds.e:Started 2024-03-24T10:44:45.318 DEBUG:tasks.ceph_manager:running mds.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-24T10:44:45.319 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T10:44:45.319 DEBUG:teuthology.orchestra.run.smithi057:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i b 2024-03-24T10:44:45.321 INFO:tasks.ceph.mds.b:Started 2024-03-24T10:44:45.321 DEBUG:tasks.ceph_manager:running mds.d under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.d.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-24T10:44:45.321 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T10:44:45.322 DEBUG:teuthology.orchestra.run.smithi057:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.d.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i d 2024-03-24T10:44:45.323 INFO:tasks.ceph.mds.d:Started 2024-03-24T10:44:45.324 DEBUG:tasks.ceph_manager:running mds.f under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.f.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-24T10:44:45.324 INFO:tasks.ceph.mds.f:Restarting daemon 2024-03-24T10:44:45.324 DEBUG:teuthology.orchestra.run.smithi057:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.f.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i f 2024-03-24T10:44:45.326 INFO:tasks.ceph.mds.f:Started 2024-03-24T10:44:45.326 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-24T10:44:45.326 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:44:45.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.448+0000 7fc87771e640 1 Processor -- start 2024-03-24T10:44:45.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.448+0000 7fc87771e640 1 -- start start 2024-03-24T10:44:45.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.449+0000 7fc87771e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc870072640 0x7fc8700731d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:45.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.449+0000 7fc87771e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc8701544c0 0x7fc87015d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:45.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.449+0000 7fc87771e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc870073710 con 0x7fc8700c9650 2024-03-24T10:44:45.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.449+0000 7fc87771e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fc870070e00 con 0x7fc8701544c0 2024-03-24T10:44:45.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.449+0000 7fc87771e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc87015a590 con 0x7fc870072640 2024-03-24T10:44:45.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.450+0000 7fc874cb9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc870072640 0x7fc8700731d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:45.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.450+0000 7fc875cbb640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc8701544c0 0x7fc87015d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:45.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.450+0000 7fc8754ba640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fc8700c9650 0x7fc870153db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:41388/0 (socket says 172.21.15.5:41388) 2024-03-24T10:44:45.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.450+0000 7fc8754ba640 1 -- 172.21.15.5:0/9262795 learned_addr learned my addr 172.21.15.5:0/9262795 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:44:45.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.453+0000 7fc874cb9640 1 -- 172.21.15.5:0/9262795 >> v2:172.21.15.5:3300/0 conn(0x7fc8701544c0 msgr2=0x7fc87015d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:45.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.453+0000 7fc874cb9640 1 --2- 172.21.15.5:0/9262795 >> v2:172.21.15.5:3300/0 conn(0x7fc8701544c0 0x7fc87015d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:45.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.453+0000 7fc874cb9640 1 -- 172.21.15.5:0/9262795 >> v1:172.21.15.5:6789/0 conn(0x7fc8700c9650 legacy=0x7fc870153db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:45.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.453+0000 7fc874cb9640 1 -- 172.21.15.5:0/9262795 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc87015a870 con 0x7fc870072640 2024-03-24T10:44:45.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.456+0000 7fc874cb9640 1 --2- 172.21.15.5:0/9262795 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc870072640 0x7fc8700731d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc8600097a0 tx=0x7fc86002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=497ee9e6b061e21d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:45.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.461+0000 7fc867fff640 1 -- 172.21.15.5:0/9262795 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc860030610 con 0x7fc870072640 2024-03-24T10:44:45.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.461+0000 7fc867fff640 1 -- 172.21.15.5:0/9262795 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc860030770 con 0x7fc870072640 2024-03-24T10:44:45.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.461+0000 7fc867fff640 1 -- 172.21.15.5:0/9262795 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc860030b00 con 0x7fc870072640 2024-03-24T10:44:45.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.462+0000 7fc87771e640 1 -- 172.21.15.5:0/9262795 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc870072640 msgr2=0x7fc8700731d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:45.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.462+0000 7fc87771e640 1 --2- 172.21.15.5:0/9262795 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc870072640 0x7fc8700731d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc8600097a0 tx=0x7fc86002b960 comp rx=0 tx=0).stop 2024-03-24T10:44:45.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.462+0000 7fc87771e640 1 -- 172.21.15.5:0/9262795 shutdown_connections 2024-03-24T10:44:45.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.462+0000 7fc87771e640 1 --2- 172.21.15.5:0/9262795 >> v2:172.21.15.5:3300/0 conn(0x7fc8701544c0 0x7fc87015d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:45.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.462+0000 7fc87771e640 1 --2- 172.21.15.5:0/9262795 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc870072640 0x7fc8700731d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:45.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.462+0000 7fc87771e640 1 -- 172.21.15.5:0/9262795 >> 172.21.15.5:0/9262795 conn(0x7fc87006c240 msgr2=0x7fc87006c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:45.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.462+0000 7fc87771e640 1 -- 172.21.15.5:0/9262795 shutdown_connections 2024-03-24T10:44:45.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.462+0000 7fc87771e640 1 -- 172.21.15.5:0/9262795 wait complete. 2024-03-24T10:44:45.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.464+0000 7fc87771e640 1 Processor -- start 2024-03-24T10:44:45.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.464+0000 7fc87771e640 1 -- start start 2024-03-24T10:44:45.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.465+0000 7fc87771e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc870072640 0x7fc870156270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:45.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.465+0000 7fc87771e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8700c9650 0x7fc87014bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:45.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.465+0000 7fc87771e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc8701544c0 0x7fc8701f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:45.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.465+0000 7fc87771e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fc8701e7250 con 0x7fc8701544c0 2024-03-24T10:44:45.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.465+0000 7fc87771e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc870155540 con 0x7fc8700c9650 2024-03-24T10:44:45.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.465+0000 7fc87771e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc870155de0 con 0x7fc870072640 2024-03-24T10:44:45.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.466+0000 7fc8754ba640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc870072640 0x7fc870156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:45.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.466+0000 7fc875cbb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc8701544c0 0x7fc8701f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:45.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.466+0000 7fc874cb9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8700c9650 0x7fc87014bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:45.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.466+0000 7fc8754ba640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc870072640 0x7fc870156270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:45792/0 (socket says 172.21.15.5:45792) 2024-03-24T10:44:45.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.466+0000 7fc875cbb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc8701544c0 0x7fc8701f2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:47938/0 (socket says 172.21.15.5:47938) 2024-03-24T10:44:45.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.466+0000 7fc8754ba640 1 -- 172.21.15.5:0/858893917 learned_addr learned my addr 172.21.15.5:0/858893917 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:45.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.470+0000 7fc875cbb640 1 -- 172.21.15.5:0/858893917 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc870072640 msgr2=0x7fc870156270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:45.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.470+0000 7fc875cbb640 1 --2- 172.21.15.5:0/858893917 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc870072640 0x7fc870156270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:45.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.470+0000 7fc875cbb640 1 -- 172.21.15.5:0/858893917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8700c9650 msgr2=0x7fc87014bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:45.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.470+0000 7fc875cbb640 1 --2- 172.21.15.5:0/858893917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8700c9650 0x7fc87014bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:45.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.470+0000 7fc875cbb640 1 -- 172.21.15.5:0/858893917 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc860009400 con 0x7fc8701544c0 2024-03-24T10:44:45.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.472+0000 7fc875cbb640 1 --2- 172.21.15.5:0/858893917 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc8701544c0 0x7fc8701f2c90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc86800eb70 tx=0x7fc86800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:45.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.478+0000 7fc865ffb640 1 -- 172.21.15.5:0/858893917 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86800cd20 con 0x7fc8701544c0 2024-03-24T10:44:45.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.478+0000 7fc865ffb640 1 -- 172.21.15.5:0/858893917 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc86800f040 con 0x7fc8701544c0 2024-03-24T10:44:45.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.478+0000 7fc87771e640 1 -- 172.21.15.5:0/858893917 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8701f37c0 con 0x7fc8701544c0 2024-03-24T10:44:45.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.478+0000 7fc865ffb640 1 -- 172.21.15.5:0/858893917 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8680146c0 con 0x7fc8701544c0 2024-03-24T10:44:45.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.478+0000 7fc87771e640 1 -- 172.21.15.5:0/858893917 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc870155790 con 0x7fc8701544c0 2024-03-24T10:44:45.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.480+0000 7fc87771e640 1 -- 172.21.15.5:0/858893917 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8700688e0 con 0x7fc8701544c0 2024-03-24T10:44:45.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.512+0000 7fc865ffb640 1 -- 172.21.15.5:0/858893917 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 4) ==== 89724+0+0 (secure 0 0 0) 0x7fc868014820 con 0x7fc8701544c0 2024-03-24T10:44:45.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.513+0000 7fc865ffb640 1 --2- 172.21.15.5:0/858893917 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc85406b780 0x7fc85406dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:45.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.513+0000 7fc8754ba640 1 --2- 172.21.15.5:0/858893917 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc85406b780 0x7fc85406dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:45.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.513+0000 7fc8754ba640 1 --2- 172.21.15.5:0/858893917 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc85406b780 0x7fc85406dc40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc86c004e40 tx=0x7fc86c007f10 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:45.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.518+0000 7fc865ffb640 1 -- 172.21.15.5:0/858893917 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 4471+0+0 (secure 0 0 0) 0x7fc86800ce80 con 0x7fc8701544c0 2024-03-24T10:44:45.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:45.790+0000 7fc865ffb640 1 -- 172.21.15.5:0/858893917 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc868055d80 con 0x7fc8701544c0 2024-03-24T10:44:46.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.036+0000 7fc87771e640 1 -- 172.21.15.5:0/858893917 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc870155f20 con 0x7fc8701544c0 2024-03-24T10:44:46.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.239+0000 7fc865ffb640 1 -- 172.21.15.5:0/858893917 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+10858 (secure 0 0 0) 0x7fc868018020 con 0x7fc8701544c0 2024-03-24T10:44:46.240 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:44:46.240 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":14,"fsid":"7d9848db-2fe7-4d95-a6e2-b04f4f639ebc","created":"2024-03-24T10:44:30.435176+0000","modified":"2024-03-24T10:44:44.634047+0000","last_up_change":"2024-03-24T10:44:44.634047+0000","last_in_change":"2024-03-24T10:44:35.986391+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"50f590b3-f5ec-4e1b-ac46-25cab307a27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6825","nonce":1991336051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6827","nonce":1991336051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6831","nonce":1991336051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6829","nonce":1991336051}]},"public_addr":"172.21.15.5:6825/1991336051","cluster_addr":"172.21.15.5:6827/1991336051","heartbeat_back_addr":"172.21.15.5:6831/1991336051","heartbeat_front_addr":"172.21.15.5:6829/1991336051","state":["exists","up"]},{"osd":1,"uuid":"ccba9e93-c704-472b-a832-8d4eeb456c47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6801","nonce":2140667501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6803","nonce":2140667501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6807","nonce":2140667501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6805","nonce":2140667501}]},"public_addr":"172.21.15.5:6801/2140667501","cluster_addr":"172.21.15.5:6803/2140667501","heartbeat_back_addr":"172.21.15.5:6807/2140667501","heartbeat_front_addr":"172.21.15.5:6805/2140667501","state":["exists","up"]},{"osd":2,"uuid":"e614c5b1-6504-452e-82e0-95c7ca271737","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6817","nonce":90414679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6819","nonce":90414679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6823","nonce":90414679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6821","nonce":90414679}]},"public_addr":"172.21.15.5:6817/90414679","cluster_addr":"172.21.15.5:6819/90414679","heartbeat_back_addr":"172.21.15.5:6823/90414679","heartbeat_front_addr":"172.21.15.5:6821/90414679","state":["exists","up"]},{"osd":3,"uuid":"c60154d0-85e0-46a1-8273-4863bd8ad2dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6809","nonce":4212473119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6811","nonce":4212473119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6815","nonce":4212473119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6813","nonce":4212473119}]},"public_addr":"172.21.15.5:6809/4212473119","cluster_addr":"172.21.15.5:6811/4212473119","heartbeat_back_addr":"172.21.15.5:6815/4212473119","heartbeat_front_addr":"172.21.15.5:6813/4212473119","state":["exists","up"]},{"osd":4,"uuid":"21d74109-da3a-4fc0-b2d8-93e9ae3ed30c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6817","nonce":2193986505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6819","nonce":2193986505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6823","nonce":2193986505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6821","nonce":2193986505}]},"public_addr":"172.21.15.57:6817/2193986505","cluster_addr":"172.21.15.57:6819/2193986505","heartbeat_back_addr":"172.21.15.57:6823/2193986505","heartbeat_front_addr":"172.21.15.57:6821/2193986505","state":["exists","up"]},{"osd":5,"uuid":"5cf19cc7-215f-4d69-a30a-80543c0293e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6809","nonce":3044987466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6811","nonce":3044987466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6815","nonce":3044987466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6813","nonce":3044987466}]},"public_addr":"172.21.15.57:6809/3044987466","cluster_addr":"172.21.15.57:6811/3044987466","heartbeat_back_addr":"172.21.15.57:6815/3044987466","heartbeat_front_addr":"172.21.15.57:6813/3044987466","state":["exists","up"]},{"osd":6,"uuid":"a5e76c69-c3fc-47b2-8f92-e560cf503e45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6801","nonce":4290897445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6803","nonce":4290897445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6807","nonce":4290897445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6805","nonce":4290897445}]},"public_addr":"172.21.15.57:6801/4290897445","cluster_addr":"172.21.15.57:6803/4290897445","heartbeat_back_addr":"172.21.15.57:6807/4290897445","heartbeat_front_addr":"172.21.15.57:6805/4290897445","state":["exists","up"]},{"osd":7,"uuid":"47d0b873-e3ac-4f40-a372-681436e45d0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6825","nonce":212605698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6827","nonce":212605698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6831","nonce":212605698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6829","nonce":212605698}]},"public_addr":"172.21.15.57:6825/212605698","cluster_addr":"172.21.15.57:6827/212605698","heartbeat_back_addr":"172.21.15.57:6831/212605698","heartbeat_front_addr":"172.21.15.57:6829/212605698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.211237+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.277082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.190915+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:44:46.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.241+0000 7fc87771e640 1 -- 172.21.15.5:0/858893917 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc85406b780 msgr2=0x7fc85406dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:46.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.241+0000 7fc87771e640 1 --2- 172.21.15.5:0/858893917 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc85406b780 0x7fc85406dc40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc86c004e40 tx=0x7fc86c007f10 comp rx=0 tx=0).stop 2024-03-24T10:44:46.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.241+0000 7fc87771e640 1 -- 172.21.15.5:0/858893917 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc8701544c0 msgr2=0x7fc8701f2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:46.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.241+0000 7fc87771e640 1 --2- 172.21.15.5:0/858893917 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc8701544c0 0x7fc8701f2c90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc86800eb70 tx=0x7fc86800c490 comp rx=0 tx=0).stop 2024-03-24T10:44:46.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.242+0000 7fc87771e640 1 -- 172.21.15.5:0/858893917 shutdown_connections 2024-03-24T10:44:46.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.242+0000 7fc87771e640 1 --2- 172.21.15.5:0/858893917 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc85406b780 0x7fc85406dc40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:46.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.242+0000 7fc87771e640 1 --2- 172.21.15.5:0/858893917 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc8701544c0 0x7fc8701f2c90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:46.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.242+0000 7fc87771e640 1 --2- 172.21.15.5:0/858893917 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc8700c9650 0x7fc87014bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:46.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.242+0000 7fc87771e640 1 --2- 172.21.15.5:0/858893917 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc870072640 0x7fc870156270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:46.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.242+0000 7fc87771e640 1 -- 172.21.15.5:0/858893917 >> 172.21.15.5:0/858893917 conn(0x7fc87006c240 msgr2=0x7fc870158b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:46.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.242+0000 7fc87771e640 1 -- 172.21.15.5:0/858893917 shutdown_connections 2024-03-24T10:44:46.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.242+0000 7fc87771e640 1 -- 172.21.15.5:0/858893917 wait complete. 2024-03-24T10:44:46.256 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-03-24T10:44:46.257 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:44:46.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.415+0000 7f473080e640 1 Processor -- start 2024-03-24T10:44:46.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.415+0000 7f473080e640 1 -- start start 2024-03-24T10:44:46.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.416+0000 7f473080e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f472c0c8c30 0x7f472c0c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:46.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.416+0000 7f473080e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f472c15be90 0x7f472c162a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:46.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.416+0000 7f473080e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f472c0c8850 con 0x7f472c0c92f0 2024-03-24T10:44:46.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.416+0000 7f473080e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f472c0c9c10 con 0x7f472c15be90 2024-03-24T10:44:46.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.416+0000 7f473080e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f472c152560 con 0x7f472c0c8c30 2024-03-24T10:44:46.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.434+0000 7f4729d9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f472c0c8c30 0x7f472c0c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:46.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.473+0000 7f472a59c640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f472c0c92f0 0x7f472c15b780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:41400/0 (socket says 172.21.15.5:41400) 2024-03-24T10:44:46.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.473+0000 7f472a59c640 1 -- 172.21.15.5:0/2217199421 learned_addr learned my addr 172.21.15.5:0/2217199421 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:44:46.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.478+0000 7f472959a640 1 -- 172.21.15.5:0/2217199421 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2574351981 0 0) 0x7f472c0c8850 con 0x7f472c0c92f0 2024-03-24T10:44:46.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.478+0000 7f472959a640 1 -- 172.21.15.5:0/2217199421 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4714003520 con 0x7f472c0c92f0 2024-03-24T10:44:46.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.482+0000 7f472959a640 1 -- 172.21.15.5:0/2217199421 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3018922575 0 0) 0x7f4714003520 con 0x7f472c0c92f0 2024-03-24T10:44:46.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.482+0000 7f472959a640 1 -- 172.21.15.5:0/2217199421 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f472c0c8c30 msgr2=0x7f472c0c96d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:46.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.482+0000 7f472959a640 1 --2- 172.21.15.5:0/2217199421 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f472c0c8c30 0x7f472c0c96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:46.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.482+0000 7f472959a640 1 -- 172.21.15.5:0/2217199421 >> v2:172.21.15.57:3300/0 conn(0x7f472c15be90 msgr2=0x7f472c162a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:46.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.482+0000 7f472959a640 1 --2- 172.21.15.5:0/2217199421 >> v2:172.21.15.57:3300/0 conn(0x7f472c15be90 0x7f472c162a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:46.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.482+0000 7f472959a640 1 -- 172.21.15.5:0/2217199421 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f472c0b3b80 con 0x7f472c0c92f0 2024-03-24T10:44:46.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.482+0000 7f472959a640 1 -- 172.21.15.5:0/2217199421 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 100681612 0 0) 0x7f471c001670 con 0x7f472c0c92f0 2024-03-24T10:44:46.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.482+0000 7f472959a640 1 -- 172.21.15.5:0/2217199421 >> v1:172.21.15.5:6789/0 conn(0x7f472c0c92f0 legacy=0x7f472c15b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:46.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.482+0000 7f472959a640 1 --2- 172.21.15.5:0/2217199421 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4714003970 0x7f4714003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:46.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.482+0000 7f472959a640 1 --2- 172.21.15.5:0/2217199421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47140047c0 0x7f4714004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:46.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.482+0000 7f472959a640 1 --2- 172.21.15.5:0/2217199421 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f472c0c8c30 0x7f471400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:46.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.482+0000 7f472959a640 1 -- 172.21.15.5:0/2217199421 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f472c0c9c10 con 0x7f4714003970 2024-03-24T10:44:46.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.482+0000 7f472959a640 1 -- 172.21.15.5:0/2217199421 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f472c152560 con 0x7f47140047c0 2024-03-24T10:44:46.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.482+0000 7f472959a640 1 -- 172.21.15.5:0/2217199421 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f47140055d0 con 0x7f472c0c8c30 2024-03-24T10:44:46.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.487+0000 7f472a59c640 1 --2- 172.21.15.5:0/2217199421 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4714003970 0x7f4714003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:46.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.490+0000 7f472a59c640 1 -- 172.21.15.5:0/2217199421 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f472c0c8c30 msgr2=0x7f471400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:46.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.490+0000 7f472a59c640 1 --2- 172.21.15.5:0/2217199421 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f472c0c8c30 0x7f471400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:46.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.491+0000 7f472a59c640 1 -- 172.21.15.5:0/2217199421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47140047c0 msgr2=0x7f4714004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:46.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.491+0000 7f472a59c640 1 --2- 172.21.15.5:0/2217199421 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f47140047c0 0x7f4714004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:46.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.491+0000 7f472a59c640 1 -- 172.21.15.5:0/2217199421 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4714003520 con 0x7f4714003970 2024-03-24T10:44:46.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.493+0000 7f472a59c640 1 --2- 172.21.15.5:0/2217199421 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4714003970 0x7f4714003d90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f471c00c160 tx=0x7f47140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=704ea32434384fd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:46.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.496+0000 7f472959a640 1 -- 172.21.15.5:0/2217199421 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471c005ad0 con 0x7f4714003970 2024-03-24T10:44:46.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.497+0000 7f472959a640 1 -- 172.21.15.5:0/2217199421 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f471c005c30 con 0x7f4714003970 2024-03-24T10:44:46.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.498+0000 7f472959a640 1 -- 172.21.15.5:0/2217199421 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471c02c810 con 0x7f4714003970 2024-03-24T10:44:46.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.498+0000 7f473080e640 1 -- 172.21.15.5:0/2217199421 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4714003970 msgr2=0x7f4714003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:46.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.498+0000 7f473080e640 1 --2- 172.21.15.5:0/2217199421 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4714003970 0x7f4714003d90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f471c00c160 tx=0x7f47140057f0 comp rx=0 tx=0).stop 2024-03-24T10:44:46.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.498+0000 7f472ad9d640 1 -- 172.21.15.5:0/2217199421 reap_dead start 2024-03-24T10:44:46.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.499+0000 7f473080e640 1 -- 172.21.15.5:0/2217199421 shutdown_connections 2024-03-24T10:44:46.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.499+0000 7f473080e640 1 -- 172.21.15.5:0/2217199421 >> 172.21.15.5:0/2217199421 conn(0x7f472c06f820 msgr2=0x7f472c061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:46.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.499+0000 7f473080e640 1 -- 172.21.15.5:0/2217199421 shutdown_connections 2024-03-24T10:44:46.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.499+0000 7f473080e640 1 -- 172.21.15.5:0/2217199421 wait complete. 2024-03-24T10:44:46.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.501+0000 7f473080e640 1 Processor -- start 2024-03-24T10:44:46.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.501+0000 7f473080e640 1 -- start start 2024-03-24T10:44:46.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.502+0000 7f473080e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4714003970 0x7f472c1ee930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:46.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.502+0000 7f473080e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f472c14df20 0x7f472c0ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:46.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.502+0000 7f473080e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f472c1f1480 0x7f472c1481c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:46.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.502+0000 7f473080e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f472c1eee70 con 0x7f472c14df20 2024-03-24T10:44:46.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.502+0000 7f473080e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f472c1497f0 con 0x7f4714003970 2024-03-24T10:44:46.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.502+0000 7f473080e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f472c1f1a30 con 0x7f472c1f1480 2024-03-24T10:44:46.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.503+0000 7f4729d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f472c14df20 0x7f472c0ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:46.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.503+0000 7f4729d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f472c14df20 0x7f472c0ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:47964/0 (socket says 172.21.15.5:47964) 2024-03-24T10:44:46.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.503+0000 7f4729d9b640 1 -- 172.21.15.5:0/3454292211 learned_addr learned my addr 172.21.15.5:0/3454292211 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:46.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.507+0000 7f4729d9b640 1 -- 172.21.15.5:0/3454292211 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f472c1f1480 msgr2=0x7f472c1481c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:46.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.507+0000 7f4729d9b640 1 --2- 172.21.15.5:0/3454292211 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f472c1f1480 0x7f472c1481c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:46.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.507+0000 7f4729d9b640 1 -- 172.21.15.5:0/3454292211 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4714003970 msgr2=0x7f472c1ee930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:46.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.507+0000 7f4729d9b640 1 --2- 172.21.15.5:0/3454292211 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4714003970 0x7f472c1ee930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:46.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.507+0000 7f4729d9b640 1 -- 172.21.15.5:0/3454292211 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f472c1f1d10 con 0x7f472c14df20 2024-03-24T10:44:46.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.509+0000 7f4729d9b640 1 --2- 172.21.15.5:0/3454292211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f472c14df20 0x7f472c0ca540 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f472000c7c0 tx=0x7f472000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:46.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.516+0000 7f471affd640 1 -- 172.21.15.5:0/3454292211 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4720007820 con 0x7f472c14df20 2024-03-24T10:44:46.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.516+0000 7f471affd640 1 -- 172.21.15.5:0/3454292211 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4720007980 con 0x7f472c14df20 2024-03-24T10:44:46.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.516+0000 7f473080e640 1 -- 172.21.15.5:0/3454292211 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f472c1f1ff0 con 0x7f472c14df20 2024-03-24T10:44:46.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.516+0000 7f471affd640 1 -- 172.21.15.5:0/3454292211 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4720007d30 con 0x7f472c14df20 2024-03-24T10:44:46.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.516+0000 7f473080e640 1 -- 172.21.15.5:0/3454292211 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f472c149480 con 0x7f472c14df20 2024-03-24T10:44:46.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.518+0000 7f473080e640 1 -- 172.21.15.5:0/3454292211 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f472c149640 con 0x7f472c14df20 2024-03-24T10:44:46.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.551+0000 7f471affd640 1 -- 172.21.15.5:0/3454292211 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 4) ==== 89724+0+0 (secure 0 0 0) 0x7f4720024480 con 0x7f472c14df20 2024-03-24T10:44:46.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.551+0000 7f471affd640 1 --2- 172.21.15.5:0/3454292211 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f470c06b780 0x7f470c06dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:46.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.551+0000 7f472a59c640 1 --2- 172.21.15.5:0/3454292211 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f470c06b780 0x7f470c06dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:46.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.552+0000 7f472a59c640 1 --2- 172.21.15.5:0/3454292211 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f470c06b780 0x7f470c06dc40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f471c001050 tx=0x7f471c02e000 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:46.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.823+0000 7f471affd640 1 -- 172.21.15.5:0/3454292211 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4846+0+0 (secure 0 0 0) 0x7f4720059540 con 0x7f472c14df20 2024-03-24T10:44:46.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:46.823+0000 7f471affd640 1 -- 172.21.15.5:0/3454292211 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f47200550e0 con 0x7f472c14df20 2024-03-24T10:44:47.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:47.133+0000 7f473080e640 1 -- 172.21.15.5:0/3454292211 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f472c14e3b0 con 0x7f472c14df20 2024-03-24T10:44:47.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:47.186+0000 7f471affd640 1 -- 172.21.15.5:0/3454292211 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f472001f090 con 0x7f472c14df20 2024-03-24T10:44:47.187 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:44:47.187 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-24T10:44:47.187 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 1 2024-03-24T10:44:47.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:47.189+0000 7f473080e640 1 -- 172.21.15.5:0/3454292211 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f470c06b780 msgr2=0x7f470c06dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:47.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:47.189+0000 7f473080e640 1 --2- 172.21.15.5:0/3454292211 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f470c06b780 0x7f470c06dc40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f471c001050 tx=0x7f471c02e000 comp rx=0 tx=0).stop 2024-03-24T10:44:47.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:47.189+0000 7f473080e640 1 -- 172.21.15.5:0/3454292211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f472c14df20 msgr2=0x7f472c0ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:47.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:47.189+0000 7f473080e640 1 --2- 172.21.15.5:0/3454292211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f472c14df20 0x7f472c0ca540 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f472000c7c0 tx=0x7f472000cc80 comp rx=0 tx=0).stop 2024-03-24T10:44:47.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:47.189+0000 7f473080e640 1 -- 172.21.15.5:0/3454292211 shutdown_connections 2024-03-24T10:44:47.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:47.189+0000 7f473080e640 1 --2- 172.21.15.5:0/3454292211 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f470c06b780 0x7f470c06dc40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:47.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:47.189+0000 7f473080e640 1 --2- 172.21.15.5:0/3454292211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f472c14df20 0x7f472c0ca540 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f472000c7c0 tx=0x7f472000cc80 comp rx=0 tx=0).stop 2024-03-24T10:44:47.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:47.189+0000 7f473080e640 1 --2- 172.21.15.5:0/3454292211 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f472c1f1480 0x7f472c1481c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:47.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:47.189+0000 7f473080e640 1 --2- 172.21.15.5:0/3454292211 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4714003970 0x7f472c1ee930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:47.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:47.189+0000 7f473080e640 1 -- 172.21.15.5:0/3454292211 >> 172.21.15.5:0/3454292211 conn(0x7f472c06f820 msgr2=0x7f472c0b32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:47.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:47.190+0000 7f473080e640 1 -- 172.21.15.5:0/3454292211 shutdown_connections 2024-03-24T10:44:47.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:47.190+0000 7f473080e640 1 -- 172.21.15.5:0/3454292211 wait complete. 2024-03-24T10:44:49.207 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:44:49.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.365+0000 7f0b784cd640 1 Processor -- start 2024-03-24T10:44:49.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.365+0000 7f0b784cd640 1 -- start start 2024-03-24T10:44:49.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.366+0000 7f0b784cd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0b7006bf50 0x7f0b7006c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.366+0000 7f0b784cd640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b700cb2f0 con 0x7f0b701543a0 2024-03-24T10:44:49.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.366+0000 7f0b784cd640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b7015b070 con 0x7f0b7015ac00 2024-03-24T10:44:49.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.366+0000 7f0b784cd640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f0b700c6750 con 0x7f0b7006bf50 2024-03-24T10:44:49.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.367+0000 7f0b76269640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f0b701543a0 0x7f0b7006d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:41408/0 (socket says 172.21.15.5:41408) 2024-03-24T10:44:49.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.367+0000 7f0b76269640 1 -- 172.21.15.5:0/165140463 learned_addr learned my addr 172.21.15.5:0/165140463 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:44:49.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.367+0000 7f0b76a6a640 1 --2- 172.21.15.5:0/165140463 >> v2:172.21.15.5:3300/0 conn(0x7f0b7006bf50 0x7f0b7006c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.374+0000 7f0b76a6a640 1 -- 172.21.15.5:0/165140463 >> v1:172.21.15.57:6789/0 conn(0x7f0b7015ac00 legacy=0x7f0b700cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:49.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.374+0000 7f0b76a6a640 1 -- 172.21.15.5:0/165140463 >> v1:172.21.15.5:6789/0 conn(0x7f0b701543a0 legacy=0x7f0b7006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:49.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.374+0000 7f0b76a6a640 1 -- 172.21.15.5:0/165140463 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b700ca7e0 con 0x7f0b7006bf50 2024-03-24T10:44:49.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.379+0000 7f0b76a6a640 1 --2- 172.21.15.5:0/165140463 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0b7006bf50 0x7f0b7006c330 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009d90 tx=0x7f0b6802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cef06b27585794ad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:49.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.385+0000 7f0b75267640 1 -- 172.21.15.5:0/165140463 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6802cc60 con 0x7f0b7006bf50 2024-03-24T10:44:49.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.385+0000 7f0b75267640 1 -- 172.21.15.5:0/165140463 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b6802cdc0 con 0x7f0b7006bf50 2024-03-24T10:44:49.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.385+0000 7f0b75267640 1 -- 172.21.15.5:0/165140463 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b68034660 con 0x7f0b7006bf50 2024-03-24T10:44:49.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.385+0000 7f0b784cd640 1 -- 172.21.15.5:0/165140463 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0b7006bf50 msgr2=0x7f0b7006c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:49.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.385+0000 7f0b784cd640 1 --2- 172.21.15.5:0/165140463 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0b7006bf50 0x7f0b7006c330 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009d90 tx=0x7f0b6802beb0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.385+0000 7f0b784cd640 1 -- 172.21.15.5:0/165140463 shutdown_connections 2024-03-24T10:44:49.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.385+0000 7f0b784cd640 1 --2- 172.21.15.5:0/165140463 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0b7006bf50 0x7f0b7006c330 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.385+0000 7f0b784cd640 1 -- 172.21.15.5:0/165140463 >> 172.21.15.5:0/165140463 conn(0x7f0b70072ef0 msgr2=0x7f0b70061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:49.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.386+0000 7f0b784cd640 1 -- 172.21.15.5:0/165140463 shutdown_connections 2024-03-24T10:44:49.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.386+0000 7f0b784cd640 1 -- 172.21.15.5:0/165140463 wait complete. 2024-03-24T10:44:49.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.387+0000 7f0b784cd640 1 Processor -- start 2024-03-24T10:44:49.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.387+0000 7f0b784cd640 1 -- start start 2024-03-24T10:44:49.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.388+0000 7f0b784cd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0b7006bf50 0x7f0b701f3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.388+0000 7f0b784cd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b701543a0 0x7f0b700ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.388+0000 7f0b784cd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0b7015ac00 0x7f0b7015d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.388+0000 7f0b784cd640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7015c490 con 0x7f0b7006bf50 2024-03-24T10:44:49.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.388+0000 7f0b784cd640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7015c750 con 0x7f0b701543a0 2024-03-24T10:44:49.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.388+0000 7f0b784cd640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0b701f43f0 con 0x7f0b7015ac00 2024-03-24T10:44:49.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.388+0000 7f0b76269640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0b7006bf50 0x7f0b701f3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.389+0000 7f0b76269640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0b7006bf50 0x7f0b701f3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48008/0 (socket says 172.21.15.5:48008) 2024-03-24T10:44:49.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.389+0000 7f0b76269640 1 -- 172.21.15.5:0/4192111562 learned_addr learned my addr 172.21.15.5:0/4192111562 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:49.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.394+0000 7f0b76269640 1 -- 172.21.15.5:0/4192111562 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0b7015ac00 msgr2=0x7f0b7015d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:49.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.394+0000 7f0b76269640 1 --2- 172.21.15.5:0/4192111562 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0b7015ac00 0x7f0b7015d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.394+0000 7f0b76269640 1 -- 172.21.15.5:0/4192111562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b701543a0 msgr2=0x7f0b700ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:49.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.394+0000 7f0b76269640 1 --2- 172.21.15.5:0/4192111562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b701543a0 0x7f0b700ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:49.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.394+0000 7f0b76269640 1 -- 172.21.15.5:0/4192111562 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b68009a70 con 0x7f0b7006bf50 2024-03-24T10:44:49.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.397+0000 7f0b76269640 1 --2- 172.21.15.5:0/4192111562 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0b7006bf50 0x7f0b701f3eb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00c430 tx=0x7f0b6c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:49.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.402+0000 7f0b66ffd640 1 -- 172.21.15.5:0/4192111562 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c005ce0 con 0x7f0b7006bf50 2024-03-24T10:44:49.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.402+0000 7f0b66ffd640 1 -- 172.21.15.5:0/4192111562 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b6c005e40 con 0x7f0b7006bf50 2024-03-24T10:44:49.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.402+0000 7f0b66ffd640 1 -- 172.21.15.5:0/4192111562 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c0072f0 con 0x7f0b7006bf50 2024-03-24T10:44:49.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.402+0000 7f0b784cd640 1 -- 172.21.15.5:0/4192111562 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b700b4be0 con 0x7f0b7006bf50 2024-03-24T10:44:49.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.402+0000 7f0b784cd640 1 -- 172.21.15.5:0/4192111562 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b7015c9c0 con 0x7f0b7006bf50 2024-03-24T10:44:49.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.403+0000 7f0b784cd640 1 -- 172.21.15.5:0/4192111562 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b700681d0 con 0x7f0b7006bf50 2024-03-24T10:44:49.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.439+0000 7f0b66ffd640 1 -- 172.21.15.5:0/4192111562 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 4) ==== 89724+0+0 (secure 0 0 0) 0x7f0b6c007450 con 0x7f0b7006bf50 2024-03-24T10:44:49.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.440+0000 7f0b66ffd640 1 --2- 172.21.15.5:0/4192111562 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0b5406b800 0x7f0b5406dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:49.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.440+0000 7f0b75a68640 1 --2- 172.21.15.5:0/4192111562 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0b5406b800 0x7f0b5406dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:49.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.440+0000 7f0b75a68640 1 --2- 172.21.15.5:0/4192111562 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0b5406b800 0x7f0b5406dcc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0b70061e90 tx=0x7f0b60006ba0 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:49.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.446+0000 7f0b66ffd640 1 -- 172.21.15.5:0/4192111562 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f0b6c058ab0 con 0x7f0b7006bf50 2024-03-24T10:44:49.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:49.740+0000 7f0b66ffd640 1 -- 172.21.15.5:0/4192111562 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0b6c054650 con 0x7f0b7006bf50 2024-03-24T10:44:50.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:50.217+0000 7f0b784cd640 1 -- 172.21.15.5:0/4192111562 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b7015e4f0 con 0x7f0b7006bf50 2024-03-24T10:44:50.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:50.418+0000 7f0b66ffd640 1 -- 172.21.15.5:0/4192111562 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0b6c054a20 con 0x7f0b7006bf50 2024-03-24T10:44:50.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:50.437+0000 7f0b66ffd640 1 -- 172.21.15.5:0/4192111562 <== mon.0 v2:172.21.15.5:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f0b6c07c1d0 con 0x7f0b7006bf50 2024-03-24T10:44:50.438 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:44:50.438 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-24T10:44:50.438 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 1 2024-03-24T10:44:50.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:50.440+0000 7f0b784cd640 1 -- 172.21.15.5:0/4192111562 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0b5406b800 msgr2=0x7f0b5406dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:50.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:50.440+0000 7f0b784cd640 1 --2- 172.21.15.5:0/4192111562 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0b5406b800 0x7f0b5406dcc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0b70061e90 tx=0x7f0b60006ba0 comp rx=0 tx=0).stop 2024-03-24T10:44:50.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:50.440+0000 7f0b784cd640 1 -- 172.21.15.5:0/4192111562 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0b7006bf50 msgr2=0x7f0b701f3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:50.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:50.440+0000 7f0b784cd640 1 --2- 172.21.15.5:0/4192111562 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0b7006bf50 0x7f0b701f3eb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00c430 tx=0x7f0b6c00c8f0 comp rx=0 tx=0).stop 2024-03-24T10:44:50.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:50.440+0000 7f0b784cd640 1 -- 172.21.15.5:0/4192111562 shutdown_connections 2024-03-24T10:44:50.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:50.440+0000 7f0b784cd640 1 --2- 172.21.15.5:0/4192111562 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0b5406b800 0x7f0b5406dcc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:50.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:50.440+0000 7f0b784cd640 1 --2- 172.21.15.5:0/4192111562 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0b7015ac00 0x7f0b7015d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:50.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:50.440+0000 7f0b784cd640 1 --2- 172.21.15.5:0/4192111562 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0b701543a0 0x7f0b700ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:50.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:50.440+0000 7f0b784cd640 1 --2- 172.21.15.5:0/4192111562 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0b7006bf50 0x7f0b701f3eb0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:50.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:50.440+0000 7f0b784cd640 1 -- 172.21.15.5:0/4192111562 >> 172.21.15.5:0/4192111562 conn(0x7f0b70072ef0 msgr2=0x7f0b701524d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:50.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:50.440+0000 7f0b784cd640 1 -- 172.21.15.5:0/4192111562 shutdown_connections 2024-03-24T10:44:50.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:50.440+0000 7f0b784cd640 1 -- 172.21.15.5:0/4192111562 wait complete. 2024-03-24T10:44:51.913 INFO:tasks.ceph.mds.e.smithi005.stdout:starting mds.e at 2024-03-24T10:44:51.977 INFO:tasks.ceph.mds.c.smithi005.stdout:starting mds.c at 2024-03-24T10:44:52.099 INFO:tasks.ceph.mds.d.smithi057.stdout:starting mds.d at 2024-03-24T10:44:52.105 INFO:tasks.ceph.mds.b.smithi057.stdout:starting mds.b at 2024-03-24T10:44:52.106 INFO:tasks.ceph.mds.f.smithi057.stdout:starting mds.f at 2024-03-24T10:44:52.190 INFO:tasks.ceph.mds.a.smithi005.stdout:starting mds.a at 2024-03-24T10:44:52.455 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:44:52.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.562+0000 7fef9d793640 1 Processor -- start 2024-03-24T10:44:52.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.562+0000 7fef9d793640 1 -- start start 2024-03-24T10:44:52.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.564+0000 7fef9d793640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fef9815ac00 0x7fef980cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:52.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.564+0000 7fef9d793640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fef980ccaa0 0x7fef9806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:52.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.564+0000 7fef9d793640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef980cb2f0 con 0x7fef981543a0 2024-03-24T10:44:52.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.564+0000 7fef9d793640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fef980c6750 con 0x7fef980ccaa0 2024-03-24T10:44:52.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.564+0000 7fef9d793640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fef980cc380 con 0x7fef9815ac00 2024-03-24T10:44:52.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.572+0000 7fef977fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fef980ccaa0 0x7fef9806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:52.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.573+0000 7fef977fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fef980ccaa0 0x7fef9806bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54646/0 (socket says 172.21.15.5:54646) 2024-03-24T10:44:52.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.573+0000 7fef977fe640 1 -- 172.21.15.5:0/2081264845 learned_addr learned my addr 172.21.15.5:0/2081264845 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:52.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.576+0000 7fef977fe640 1 -- 172.21.15.5:0/2081264845 >> v2:172.21.15.57:3300/0 conn(0x7fef9815ac00 msgr2=0x7fef980cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:52.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.576+0000 7fef977fe640 1 --2- 172.21.15.5:0/2081264845 >> v2:172.21.15.57:3300/0 conn(0x7fef9815ac00 0x7fef980cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:52.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.576+0000 7fef977fe640 1 -- 172.21.15.5:0/2081264845 >> v1:172.21.15.57:6789/0 conn(0x7fef981543a0 legacy=0x7fef9806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:52.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.576+0000 7fef977fe640 1 -- 172.21.15.5:0/2081264845 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef9815b070 con 0x7fef980ccaa0 2024-03-24T10:44:52.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.578+0000 7fef977fe640 1 --2- 172.21.15.5:0/2081264845 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef980ccaa0 0x7fef9806bf50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fef980c6990 tx=0x7fef9002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f72b0948f834d9c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:52.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.582+0000 7fef967fc640 1 -- 172.21.15.5:0/2081264845 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9002ca80 con 0x7fef980ccaa0 2024-03-24T10:44:52.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.582+0000 7fef967fc640 1 -- 172.21.15.5:0/2081264845 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef9002cbe0 con 0x7fef980ccaa0 2024-03-24T10:44:52.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.582+0000 7fef967fc640 1 -- 172.21.15.5:0/2081264845 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef900343f0 con 0x7fef980ccaa0 2024-03-24T10:44:52.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.583+0000 7fef9d793640 1 -- 172.21.15.5:0/2081264845 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef980ccaa0 msgr2=0x7fef9806bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:52.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.583+0000 7fef9d793640 1 --2- 172.21.15.5:0/2081264845 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef980ccaa0 0x7fef9806bf50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fef980c6990 tx=0x7fef9002bee0 comp rx=0 tx=0).stop 2024-03-24T10:44:52.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.583+0000 7fef9d793640 1 -- 172.21.15.5:0/2081264845 shutdown_connections 2024-03-24T10:44:52.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.583+0000 7fef9d793640 1 --2- 172.21.15.5:0/2081264845 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef980ccaa0 0x7fef9806bf50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:52.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.583+0000 7fef9d793640 1 --2- 172.21.15.5:0/2081264845 >> v2:172.21.15.57:3300/0 conn(0x7fef9815ac00 0x7fef980cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:52.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.583+0000 7fef9d793640 1 -- 172.21.15.5:0/2081264845 >> 172.21.15.5:0/2081264845 conn(0x7fef98072ef0 msgr2=0x7fef98061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:52.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.583+0000 7fef9d793640 1 -- 172.21.15.5:0/2081264845 shutdown_connections 2024-03-24T10:44:52.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.583+0000 7fef9d793640 1 -- 172.21.15.5:0/2081264845 wait complete. 2024-03-24T10:44:52.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.584+0000 7fef9d793640 1 Processor -- start 2024-03-24T10:44:52.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.584+0000 7fef9d793640 1 -- start start 2024-03-24T10:44:52.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.585+0000 7fef9d793640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef980ccaa0 0x7fef98157460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:52.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.585+0000 7fef9d793640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef981543a0 0x7fef98159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:52.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.585+0000 7fef9d793640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fef9815ac00 0x7fef981eff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:52.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.585+0000 7fef9d793640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fef9815d3a0 con 0x7fef980ccaa0 2024-03-24T10:44:52.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.585+0000 7fef9d793640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fef9815d660 con 0x7fef981543a0 2024-03-24T10:44:52.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.585+0000 7fef9d793640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fef9815d940 con 0x7fef9815ac00 2024-03-24T10:44:52.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.586+0000 7fef96ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef980ccaa0 0x7fef98157460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:52.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.586+0000 7fef96ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef980ccaa0 0x7fef98157460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54656/0 (socket says 172.21.15.5:54656) 2024-03-24T10:44:52.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.586+0000 7fef96ffd640 1 -- 172.21.15.5:0/956362638 learned_addr learned my addr 172.21.15.5:0/956362638 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:52.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.589+0000 7fef96ffd640 1 -- 172.21.15.5:0/956362638 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fef9815ac00 msgr2=0x7fef981eff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:52.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.589+0000 7fef96ffd640 1 --2- 172.21.15.5:0/956362638 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fef9815ac00 0x7fef981eff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:52.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.589+0000 7fef96ffd640 1 -- 172.21.15.5:0/956362638 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef981543a0 msgr2=0x7fef98159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:52.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.589+0000 7fef96ffd640 1 --2- 172.21.15.5:0/956362638 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef981543a0 0x7fef98159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:52.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.589+0000 7fef96ffd640 1 -- 172.21.15.5:0/956362638 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef90009b30 con 0x7fef980ccaa0 2024-03-24T10:44:52.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.596+0000 7fef96ffd640 1 --2- 172.21.15.5:0/956362638 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef980ccaa0 0x7fef98157460 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fef8800c410 tx=0x7fef8800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:52.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.600+0000 7fef8f7fe640 1 -- 172.21.15.5:0/956362638 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef880035f0 con 0x7fef980ccaa0 2024-03-24T10:44:52.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.600+0000 7fef8f7fe640 1 -- 172.21.15.5:0/956362638 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef88003750 con 0x7fef980ccaa0 2024-03-24T10:44:52.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.600+0000 7fef8f7fe640 1 -- 172.21.15.5:0/956362638 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef88003b40 con 0x7fef980ccaa0 2024-03-24T10:44:52.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.600+0000 7fef9d793640 1 -- 172.21.15.5:0/956362638 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef9815dc20 con 0x7fef980ccaa0 2024-03-24T10:44:52.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.600+0000 7fef9d793640 1 -- 172.21.15.5:0/956362638 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef9815b590 con 0x7fef980ccaa0 2024-03-24T10:44:52.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.601+0000 7fef9d793640 1 -- 172.21.15.5:0/956362638 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef9815a8b0 con 0x7fef980ccaa0 2024-03-24T10:44:52.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.619+0000 7fef8f7fe640 1 -- 172.21.15.5:0/956362638 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fef88003ca0 con 0x7fef980ccaa0 2024-03-24T10:44:52.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.620+0000 7fef8f7fe640 1 --2- 172.21.15.5:0/956362638 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fef7806b860 0x7fef7806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:52.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.620+0000 7fef8ffff640 1 --2- 172.21.15.5:0/956362638 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fef7806b860 0x7fef7806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:52.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.620+0000 7fef8ffff640 1 --2- 172.21.15.5:0/956362638 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fef7806b860 0x7fef7806dd20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fef980cf9e0 tx=0x7fef80006bf0 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:52.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.623+0000 7fef8f7fe640 1 -- 172.21.15.5:0/956362638 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fef88058b70 con 0x7fef980ccaa0 2024-03-24T10:44:52.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:52.908+0000 7fef8f7fe640 1 -- 172.21.15.5:0/956362638 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fef88054710 con 0x7fef980ccaa0 2024-03-24T10:44:53.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:53.130+0000 7fef9d793640 1 -- 172.21.15.5:0/956362638 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef980681d0 con 0x7fef980ccaa0 2024-03-24T10:44:53.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:53.143+0000 7fef8f7fe640 1 -- 172.21.15.5:0/956362638 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1389 (secure 0 0 0) 0x7fef88017070 con 0x7fef980ccaa0 2024-03-24T10:44:53.143 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:44:53.143 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4252,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-24T10:44:53.143 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 2 2024-03-24T10:44:53.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:53.144+0000 7fef9d793640 1 -- 172.21.15.5:0/956362638 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fef7806b860 msgr2=0x7fef7806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:53.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:53.144+0000 7fef9d793640 1 --2- 172.21.15.5:0/956362638 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fef7806b860 0x7fef7806dd20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fef980cf9e0 tx=0x7fef80006bf0 comp rx=0 tx=0).stop 2024-03-24T10:44:53.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:53.145+0000 7fef9d793640 1 -- 172.21.15.5:0/956362638 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef980ccaa0 msgr2=0x7fef98157460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:53.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:53.145+0000 7fef9d793640 1 --2- 172.21.15.5:0/956362638 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef980ccaa0 0x7fef98157460 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fef8800c410 tx=0x7fef8800c8d0 comp rx=0 tx=0).stop 2024-03-24T10:44:53.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:53.145+0000 7fef9d793640 1 -- 172.21.15.5:0/956362638 shutdown_connections 2024-03-24T10:44:53.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:53.145+0000 7fef9d793640 1 --2- 172.21.15.5:0/956362638 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fef7806b860 0x7fef7806dd20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:53.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:53.145+0000 7fef9d793640 1 --2- 172.21.15.5:0/956362638 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fef9815ac00 0x7fef981eff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:53.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:53.145+0000 7fef9d793640 1 --2- 172.21.15.5:0/956362638 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fef981543a0 0x7fef98159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:53.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:53.145+0000 7fef9d793640 1 --2- 172.21.15.5:0/956362638 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fef980ccaa0 0x7fef98157460 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:53.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:53.145+0000 7fef9d793640 1 -- 172.21.15.5:0/956362638 >> 172.21.15.5:0/956362638 conn(0x7fef98072ef0 msgr2=0x7fef98150540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:53.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:53.145+0000 7fef9d793640 1 -- 172.21.15.5:0/956362638 shutdown_connections 2024-03-24T10:44:53.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:53.145+0000 7fef9d793640 1 -- 172.21.15.5:0/956362638 wait complete. 2024-03-24T10:44:55.156 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:44:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.266+0000 7ff2dfec9640 1 Processor -- start 2024-03-24T10:44:55.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.266+0000 7ff2dfec9640 1 -- start start 2024-03-24T10:44:55.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.267+0000 7ff2dfec9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff2d815ac00 0x7ff2d80cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:55.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.267+0000 7ff2dfec9640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff2d80ccaa0 0x7ff2d806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:55.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.267+0000 7ff2dfec9640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2d80cb2f0 con 0x7ff2d81543a0 2024-03-24T10:44:55.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.267+0000 7ff2dfec9640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7ff2d80c6750 con 0x7ff2d80ccaa0 2024-03-24T10:44:55.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.267+0000 7ff2dfec9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff2d80cc380 con 0x7ff2d815ac00 2024-03-24T10:44:55.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.268+0000 7ff2de466640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff2d80ccaa0 0x7ff2d806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:55.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.268+0000 7ff2dd464640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff2d815ac00 0x7ff2d80cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:55.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.268+0000 7ff2dd464640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff2d815ac00 0x7ff2d80cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:46128/0 (socket says 172.21.15.5:46128) 2024-03-24T10:44:55.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.268+0000 7ff2dd464640 1 -- 172.21.15.5:0/5797749 learned_addr learned my addr 172.21.15.5:0/5797749 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:55.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.271+0000 7ff2de466640 1 -- 172.21.15.5:0/5797749 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff2d815ac00 msgr2=0x7ff2d80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:55.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.271+0000 7ff2de466640 1 --2- 172.21.15.5:0/5797749 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff2d815ac00 0x7ff2d80cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:55.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.271+0000 7ff2de466640 1 -- 172.21.15.5:0/5797749 >> v1:172.21.15.5:6789/0 conn(0x7ff2d81543a0 legacy=0x7ff2d806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:55.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.271+0000 7ff2de466640 1 -- 172.21.15.5:0/5797749 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2d815b070 con 0x7ff2d80ccaa0 2024-03-24T10:44:55.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.271+0000 7ff2dd464640 1 --2- 172.21.15.5:0/5797749 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff2d815ac00 0x7ff2d80cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:44:55.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.274+0000 7ff2de466640 1 --2- 172.21.15.5:0/5797749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff2d80ccaa0 0x7ff2d806bf50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8009d90 tx=0x7ff2c802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54aa28038bac014 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.278+0000 7ff2dcc63640 1 -- 172.21.15.5:0/5797749 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c802cc50 con 0x7ff2d80ccaa0 2024-03-24T10:44:55.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.278+0000 7ff2dcc63640 1 -- 172.21.15.5:0/5797749 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2c802cdb0 con 0x7ff2d80ccaa0 2024-03-24T10:44:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.278+0000 7ff2dcc63640 1 -- 172.21.15.5:0/5797749 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c8034660 con 0x7ff2d80ccaa0 2024-03-24T10:44:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.279+0000 7ff2dfec9640 1 -- 172.21.15.5:0/5797749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff2d80ccaa0 msgr2=0x7ff2d806bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.279+0000 7ff2dfec9640 1 --2- 172.21.15.5:0/5797749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff2d80ccaa0 0x7ff2d806bf50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8009d90 tx=0x7ff2c802bea0 comp rx=0 tx=0).stop 2024-03-24T10:44:55.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.279+0000 7ff2dfec9640 1 -- 172.21.15.5:0/5797749 shutdown_connections 2024-03-24T10:44:55.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.279+0000 7ff2dfec9640 1 --2- 172.21.15.5:0/5797749 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff2d80ccaa0 0x7ff2d806bf50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:55.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.279+0000 7ff2dfec9640 1 --2- 172.21.15.5:0/5797749 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff2d815ac00 0x7ff2d80cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:55.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.279+0000 7ff2dfec9640 1 -- 172.21.15.5:0/5797749 >> 172.21.15.5:0/5797749 conn(0x7ff2d8072ef0 msgr2=0x7ff2d8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:55.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.279+0000 7ff2dfec9640 1 -- 172.21.15.5:0/5797749 shutdown_connections 2024-03-24T10:44:55.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.279+0000 7ff2dfec9640 1 -- 172.21.15.5:0/5797749 wait complete. 2024-03-24T10:44:55.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.280+0000 7ff2dfec9640 1 Processor -- start 2024-03-24T10:44:55.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.280+0000 7ff2dfec9640 1 -- start start 2024-03-24T10:44:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.281+0000 7ff2dfec9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff2d80ccaa0 0x7ff2d80ce650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.281+0000 7ff2dfec9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff2d81543a0 0x7ff2d80c5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.281+0000 7ff2dfec9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2d815ac00 0x7ff2d80c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.281+0000 7ff2dfec9640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d8153c50 con 0x7ff2d81543a0 2024-03-24T10:44:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.281+0000 7ff2dfec9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d814fcb0 con 0x7ff2d815ac00 2024-03-24T10:44:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.281+0000 7ff2dfec9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff2d80b4f40 con 0x7ff2d80ccaa0 2024-03-24T10:44:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.282+0000 7ff2dd464640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff2d81543a0 0x7ff2d80c5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.282+0000 7ff2ddc65640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff2d80ccaa0 0x7ff2d80ce650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:55.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.282+0000 7ff2de466640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2d815ac00 0x7ff2d80c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:55.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.282+0000 7ff2dd464640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff2d81543a0 0x7ff2d80c5000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54662/0 (socket says 172.21.15.5:54662) 2024-03-24T10:44:55.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.282+0000 7ff2dd464640 1 -- 172.21.15.5:0/2704106778 learned_addr learned my addr 172.21.15.5:0/2704106778 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:55.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.285+0000 7ff2dd464640 1 -- 172.21.15.5:0/2704106778 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff2d80ccaa0 msgr2=0x7ff2d80ce650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:55.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.285+0000 7ff2dd464640 1 --2- 172.21.15.5:0/2704106778 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff2d80ccaa0 0x7ff2d80ce650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:55.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.285+0000 7ff2dd464640 1 -- 172.21.15.5:0/2704106778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2d815ac00 msgr2=0x7ff2d80c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:55.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.285+0000 7ff2dd464640 1 --2- 172.21.15.5:0/2704106778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2d815ac00 0x7ff2d80c5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:55.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.286+0000 7ff2dd464640 1 -- 172.21.15.5:0/2704106778 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2cc0094d0 con 0x7ff2d81543a0 2024-03-24T10:44:55.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.288+0000 7ff2dd464640 1 --2- 172.21.15.5:0/2704106778 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff2d81543a0 0x7ff2d80c5000 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00e9b0 tx=0x7ff2cc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:55.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.290+0000 7ff2d67fc640 1 -- 172.21.15.5:0/2704106778 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc00c9a0 con 0x7ff2d81543a0 2024-03-24T10:44:55.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.290+0000 7ff2dfec9640 1 -- 172.21.15.5:0/2704106778 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2c8009a70 con 0x7ff2d81543a0 2024-03-24T10:44:55.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.290+0000 7ff2dfec9640 1 -- 172.21.15.5:0/2704106778 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2d8150530 con 0x7ff2d81543a0 2024-03-24T10:44:55.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.291+0000 7ff2dfec9640 1 -- 172.21.15.5:0/2704106778 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2d814e410 con 0x7ff2d81543a0 2024-03-24T10:44:55.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.293+0000 7ff2d67fc640 1 -- 172.21.15.5:0/2704106778 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2cc00cb00 con 0x7ff2d81543a0 2024-03-24T10:44:55.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.293+0000 7ff2d67fc640 1 -- 172.21.15.5:0/2704106778 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc00ce70 con 0x7ff2d81543a0 2024-03-24T10:44:55.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.565+0000 7ff2d67fc640 1 -- 172.21.15.5:0/2704106778 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff2cc004020 con 0x7ff2d81543a0 2024-03-24T10:44:55.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.565+0000 7ff2d67fc640 1 --2- 172.21.15.5:0/2704106778 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff2c406b810 0x7ff2c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:55.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.565+0000 7ff2ddc65640 1 --2- 172.21.15.5:0/2704106778 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff2c406b810 0x7ff2c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:55.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.565+0000 7ff2ddc65640 1 --2- 172.21.15.5:0/2704106778 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff2c406b810 0x7ff2c406dcd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff2d0001780 tx=0x7ff2d0009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:55.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.578+0000 7ff2d67fc640 1 -- 172.21.15.5:0/2704106778 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7ff2cc058b20 con 0x7ff2d81543a0 2024-03-24T10:44:55.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.578+0000 7ff2d67fc640 1 -- 172.21.15.5:0/2704106778 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff2cc054dc0 con 0x7ff2d81543a0 2024-03-24T10:44:55.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.814+0000 7ff2dfec9640 1 -- 172.21.15.5:0/2704106778 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2d81591f0 con 0x7ff2d81543a0 2024-03-24T10:44:55.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.826+0000 7ff2d67fc640 1 -- 172.21.15.5:0/2704106778 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+5497 (secure 0 0 0) 0x7ff2cc017020 con 0x7ff2d81543a0 2024-03-24T10:44:55.827 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:44:55.827 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4251,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4252,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4254,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/1426147484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":1426147484},{"type":"v1","addr":"172.21.15.57:6837","nonce":1426147484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4258,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-24T10:44:55.828 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 3 2024-03-24T10:44:55.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.828+0000 7ff2dfec9640 1 -- 172.21.15.5:0/2704106778 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff2c406b810 msgr2=0x7ff2c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.828+0000 7ff2dfec9640 1 --2- 172.21.15.5:0/2704106778 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff2c406b810 0x7ff2c406dcd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff2d0001780 tx=0x7ff2d0009040 comp rx=0 tx=0).stop 2024-03-24T10:44:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.828+0000 7ff2dfec9640 1 -- 172.21.15.5:0/2704106778 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff2d81543a0 msgr2=0x7ff2d80c5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.828+0000 7ff2dfec9640 1 --2- 172.21.15.5:0/2704106778 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff2d81543a0 0x7ff2d80c5000 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc00e9b0 tx=0x7ff2cc00ee70 comp rx=0 tx=0).stop 2024-03-24T10:44:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.828+0000 7ff2dfec9640 1 -- 172.21.15.5:0/2704106778 shutdown_connections 2024-03-24T10:44:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.828+0000 7ff2dfec9640 1 --2- 172.21.15.5:0/2704106778 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff2c406b810 0x7ff2c406dcd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.828+0000 7ff2dfec9640 1 --2- 172.21.15.5:0/2704106778 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff2d81543a0 0x7ff2d80c5000 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.828+0000 7ff2dfec9640 1 --2- 172.21.15.5:0/2704106778 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff2d815ac00 0x7ff2d80c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.828+0000 7ff2dfec9640 1 --2- 172.21.15.5:0/2704106778 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff2d80ccaa0 0x7ff2d80ce650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.828+0000 7ff2dfec9640 1 -- 172.21.15.5:0/2704106778 >> 172.21.15.5:0/2704106778 conn(0x7ff2d8072ef0 msgr2=0x7ff2d81584a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.828+0000 7ff2dfec9640 1 -- 172.21.15.5:0/2704106778 shutdown_connections 2024-03-24T10:44:55.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.828+0000 7ff2dfec9640 1 -- 172.21.15.5:0/2704106778 wait complete. 2024-03-24T10:44:55.841 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:44:55.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.940+0000 7f834cc07640 1 Processor -- start 2024-03-24T10:44:55.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.940+0000 7f834cc07640 1 -- start start 2024-03-24T10:44:55.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.941+0000 7f834cc07640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f834815a310 0x7f834806c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:55.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.941+0000 7f834cc07640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83480cb0c0 con 0x7f83480cace0 2024-03-24T10:44:55.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.941+0000 7f834cc07640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f834806cf70 con 0x7f834806c780 2024-03-24T10:44:55.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.941+0000 7f834cc07640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f834806f110 con 0x7f834815a310 2024-03-24T10:44:55.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.942+0000 7f8345d9b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f834815a310 0x7f834806c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:55.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.942+0000 7f8346d9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f834806c780 0x7f834806f530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:58184/0 (socket says 172.21.15.5:58184) 2024-03-24T10:44:55.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.942+0000 7f8346d9d640 1 -- 172.21.15.5:0/2503485127 learned_addr learned my addr 172.21.15.5:0/2503485127 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:44:55.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.944+0000 7f834559a640 1 -- 172.21.15.5:0/2503485127 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 269814046 0 0) 0x7f834806cf70 con 0x7f834806c780 2024-03-24T10:44:55.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.944+0000 7f834559a640 1 -- 172.21.15.5:0/2503485127 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8334003520 con 0x7f834806c780 2024-03-24T10:44:55.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.945+0000 7f8345d9b640 1 -- 172.21.15.5:0/2503485127 >> v1:172.21.15.57:6789/0 conn(0x7f834806c780 legacy=0x7f834806f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:55.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.945+0000 7f8345d9b640 1 -- 172.21.15.5:0/2503485127 >> v1:172.21.15.5:6789/0 conn(0x7f83480cace0 legacy=0x7f83481432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:55.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.945+0000 7f8345d9b640 1 -- 172.21.15.5:0/2503485127 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f834815a7a0 con 0x7f834815a310 2024-03-24T10:44:55.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.949+0000 7f8345d9b640 1 --2- 172.21.15.5:0/2503485127 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834815a310 0x7f834806c240 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f83300097a0 tx=0x7f833002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed52520539c80a03 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:55.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.953+0000 7f834559a640 1 -- 172.21.15.5:0/2503485127 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833002f810 con 0x7f834815a310 2024-03-24T10:44:55.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.953+0000 7f834559a640 1 -- 172.21.15.5:0/2503485127 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f833002f970 con 0x7f834815a310 2024-03-24T10:44:55.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.953+0000 7f834559a640 1 -- 172.21.15.5:0/2503485127 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833002fd00 con 0x7f834815a310 2024-03-24T10:44:55.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.954+0000 7f834cc07640 1 -- 172.21.15.5:0/2503485127 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834815a310 msgr2=0x7f834806c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:55.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.954+0000 7f834cc07640 1 --2- 172.21.15.5:0/2503485127 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834815a310 0x7f834806c240 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f83300097a0 tx=0x7f833002b9f0 comp rx=0 tx=0).stop 2024-03-24T10:44:55.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.954+0000 7f834cc07640 1 -- 172.21.15.5:0/2503485127 shutdown_connections 2024-03-24T10:44:55.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.954+0000 7f834cc07640 1 --2- 172.21.15.5:0/2503485127 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f834815a310 0x7f834806c240 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:55.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.954+0000 7f834cc07640 1 -- 172.21.15.5:0/2503485127 >> 172.21.15.5:0/2503485127 conn(0x7f8348071240 msgr2=0x7f8348071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:55.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.954+0000 7f834cc07640 1 -- 172.21.15.5:0/2503485127 shutdown_connections 2024-03-24T10:44:55.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.954+0000 7f834cc07640 1 -- 172.21.15.5:0/2503485127 wait complete. 2024-03-24T10:44:55.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.956+0000 7f834cc07640 1 Processor -- start 2024-03-24T10:44:55.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.956+0000 7f834cc07640 1 -- start start 2024-03-24T10:44:55.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.956+0000 7f834cc07640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f834806c780 0x7f83481f37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:55.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.956+0000 7f834cc07640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f83480cace0 0x7f83480add20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:55.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.956+0000 7f834cc07640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f83480ae260 0x7f83480af900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:55.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.956+0000 7f834cc07640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f83481f3d10 con 0x7f83480cace0 2024-03-24T10:44:55.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.956+0000 7f834cc07640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f83480cbba0 con 0x7f834806c780 2024-03-24T10:44:55.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.956+0000 7f834cc07640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f83480cbe60 con 0x7f83480ae260 2024-03-24T10:44:55.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.957+0000 7f8345d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f83480cace0 0x7f83480add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:55.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.957+0000 7f834659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f834806c780 0x7f83481f37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:55.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.957+0000 7f8346d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f83480ae260 0x7f83480af900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:55.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.957+0000 7f8345d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f83480cace0 0x7f83480add20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54672/0 (socket says 172.21.15.5:54672) 2024-03-24T10:44:55.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.957+0000 7f834659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f834806c780 0x7f83481f37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:56410/0 (socket says 172.21.15.5:56410) 2024-03-24T10:44:55.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.957+0000 7f8345d9b640 1 -- 172.21.15.5:0/947588881 learned_addr learned my addr 172.21.15.5:0/947588881 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:55.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.960+0000 7f8345d9b640 1 -- 172.21.15.5:0/947588881 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f83480ae260 msgr2=0x7f83480af900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:55.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.960+0000 7f8345d9b640 1 --2- 172.21.15.5:0/947588881 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f83480ae260 0x7f83480af900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:55.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.960+0000 7f8345d9b640 1 -- 172.21.15.5:0/947588881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f834806c780 msgr2=0x7f83481f37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:55.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.960+0000 7f8345d9b640 1 --2- 172.21.15.5:0/947588881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f834806c780 0x7f83481f37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:55.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.960+0000 7f8345d9b640 1 -- 172.21.15.5:0/947588881 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8330009400 con 0x7f83480cace0 2024-03-24T10:44:55.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.962+0000 7f8345d9b640 1 --2- 172.21.15.5:0/947588881 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f83480cace0 0x7f83480add20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f83300098c0 tx=0x7f83300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:55.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.966+0000 7f833effd640 1 -- 172.21.15.5:0/947588881 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833003b040 con 0x7f83480cace0 2024-03-24T10:44:55.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.966+0000 7f833effd640 1 -- 172.21.15.5:0/947588881 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83300378f0 con 0x7f83480cace0 2024-03-24T10:44:55.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.966+0000 7f833effd640 1 -- 172.21.15.5:0/947588881 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8330037ca0 con 0x7f83480cace0 2024-03-24T10:44:55.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.966+0000 7f834cc07640 1 -- 172.21.15.5:0/947588881 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83480cc140 con 0x7f83480cace0 2024-03-24T10:44:55.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.966+0000 7f834cc07640 1 -- 172.21.15.5:0/947588881 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83480af100 con 0x7f83480cace0 2024-03-24T10:44:55.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:55.967+0000 7f834cc07640 1 -- 172.21.15.5:0/947588881 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83480ae670 con 0x7f83480cace0 2024-03-24T10:44:56.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.233+0000 7f833effd640 1 -- 172.21.15.5:0/947588881 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8330042480 con 0x7f83480cace0 2024-03-24T10:44:56.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.233+0000 7f833effd640 1 --2- 172.21.15.5:0/947588881 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f832806b890 0x7f832806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:56.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.234+0000 7f834659c640 1 --2- 172.21.15.5:0/947588881 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f832806b890 0x7f832806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:56.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.234+0000 7f834659c640 1 --2- 172.21.15.5:0/947588881 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f832806b890 0x7f832806dd50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8340001780 tx=0x7f8340009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:56.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.247+0000 7f833effd640 1 -- 172.21.15.5:0/947588881 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f833007c0d0 con 0x7f83480cace0 2024-03-24T10:44:56.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.248+0000 7f833effd640 1 -- 172.21.15.5:0/947588881 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8330078370 con 0x7f83480cace0 2024-03-24T10:44:56.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.430+0000 7f834cc07640 1 -- 172.21.15.5:0/947588881 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f83480681d0 con 0x7f83480cace0 2024-03-24T10:44:56.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.518+0000 7f833effd640 1 -- 172.21.15.5:0/947588881 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12764 (secure 0 0 0) 0x7f833007ba30 con 0x7f83480cace0 2024-03-24T10:44:56.520 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:44:56.520 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":17,"fsid":"7d9848db-2fe7-4d95-a6e2-b04f4f639ebc","created":"2024-03-24T10:44:30.435176+0000","modified":"2024-03-24T10:44:49.048115+0000","last_up_change":"2024-03-24T10:44:44.634047+0000","last_in_change":"2024-03-24T10:44:35.986391+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:44:45.829320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50f590b3-f5ec-4e1b-ac46-25cab307a27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6825","nonce":1991336051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6827","nonce":1991336051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6831","nonce":1991336051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6829","nonce":1991336051}]},"public_addr":"172.21.15.5:6825/1991336051","cluster_addr":"172.21.15.5:6827/1991336051","heartbeat_back_addr":"172.21.15.5:6831/1991336051","heartbeat_front_addr":"172.21.15.5:6829/1991336051","state":["exists","up"]},{"osd":1,"uuid":"ccba9e93-c704-472b-a832-8d4eeb456c47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6801","nonce":2140667501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6803","nonce":2140667501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6807","nonce":2140667501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6805","nonce":2140667501}]},"public_addr":"172.21.15.5:6801/2140667501","cluster_addr":"172.21.15.5:6803/2140667501","heartbeat_back_addr":"172.21.15.5:6807/2140667501","heartbeat_front_addr":"172.21.15.5:6805/2140667501","state":["exists","up"]},{"osd":2,"uuid":"e614c5b1-6504-452e-82e0-95c7ca271737","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6817","nonce":90414679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6819","nonce":90414679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6823","nonce":90414679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6821","nonce":90414679}]},"public_addr":"172.21.15.5:6817/90414679","cluster_addr":"172.21.15.5:6819/90414679","heartbeat_back_addr":"172.21.15.5:6823/90414679","heartbeat_front_addr":"172.21.15.5:6821/90414679","state":["exists","up"]},{"osd":3,"uuid":"c60154d0-85e0-46a1-8273-4863bd8ad2dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6809","nonce":4212473119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6811","nonce":4212473119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6815","nonce":4212473119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6813","nonce":4212473119}]},"public_addr":"172.21.15.5:6809/4212473119","cluster_addr":"172.21.15.5:6811/4212473119","heartbeat_back_addr":"172.21.15.5:6815/4212473119","heartbeat_front_addr":"172.21.15.5:6813/4212473119","state":["exists","up"]},{"osd":4,"uuid":"21d74109-da3a-4fc0-b2d8-93e9ae3ed30c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6817","nonce":2193986505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6819","nonce":2193986505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6823","nonce":2193986505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6821","nonce":2193986505}]},"public_addr":"172.21.15.57:6817/2193986505","cluster_addr":"172.21.15.57:6819/2193986505","heartbeat_back_addr":"172.21.15.57:6823/2193986505","heartbeat_front_addr":"172.21.15.57:6821/2193986505","state":["exists","up"]},{"osd":5,"uuid":"5cf19cc7-215f-4d69-a30a-80543c0293e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6809","nonce":3044987466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6811","nonce":3044987466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6815","nonce":3044987466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6813","nonce":3044987466}]},"public_addr":"172.21.15.57:6809/3044987466","cluster_addr":"172.21.15.57:6811/3044987466","heartbeat_back_addr":"172.21.15.57:6815/3044987466","heartbeat_front_addr":"172.21.15.57:6813/3044987466","state":["exists","up"]},{"osd":6,"uuid":"a5e76c69-c3fc-47b2-8f92-e560cf503e45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6801","nonce":4290897445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6803","nonce":4290897445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6807","nonce":4290897445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6805","nonce":4290897445}]},"public_addr":"172.21.15.57:6801/4290897445","cluster_addr":"172.21.15.57:6803/4290897445","heartbeat_back_addr":"172.21.15.57:6807/4290897445","heartbeat_front_addr":"172.21.15.57:6805/4290897445","state":["exists","up"]},{"osd":7,"uuid":"47d0b873-e3ac-4f40-a372-681436e45d0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6825","nonce":212605698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6827","nonce":212605698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6831","nonce":212605698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6829","nonce":212605698}]},"public_addr":"172.21.15.57:6825/212605698","cluster_addr":"172.21.15.57:6827/212605698","heartbeat_back_addr":"172.21.15.57:6831/212605698","heartbeat_front_addr":"172.21.15.57:6829/212605698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.211237+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.192822+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.273429+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.199212+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.277082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.218768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.190915+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.225294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:44:56.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.521+0000 7f834cc07640 1 -- 172.21.15.5:0/947588881 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f832806b890 msgr2=0x7f832806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:56.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.521+0000 7f834cc07640 1 --2- 172.21.15.5:0/947588881 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f832806b890 0x7f832806dd50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8340001780 tx=0x7f8340009040 comp rx=0 tx=0).stop 2024-03-24T10:44:56.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.521+0000 7f834cc07640 1 -- 172.21.15.5:0/947588881 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f83480cace0 msgr2=0x7f83480add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:56.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.521+0000 7f834cc07640 1 --2- 172.21.15.5:0/947588881 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f83480cace0 0x7f83480add20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f83300098c0 tx=0x7f83300046c0 comp rx=0 tx=0).stop 2024-03-24T10:44:56.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.521+0000 7f834cc07640 1 -- 172.21.15.5:0/947588881 shutdown_connections 2024-03-24T10:44:56.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.521+0000 7f834cc07640 1 --2- 172.21.15.5:0/947588881 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f832806b890 0x7f832806dd50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:56.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.521+0000 7f834cc07640 1 --2- 172.21.15.5:0/947588881 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f83480cace0 0x7f83480add20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:56.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.521+0000 7f834cc07640 1 --2- 172.21.15.5:0/947588881 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f83480ae260 0x7f83480af900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:56.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.521+0000 7f834cc07640 1 --2- 172.21.15.5:0/947588881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f834806c780 0x7f83481f37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:56.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.521+0000 7f834cc07640 1 -- 172.21.15.5:0/947588881 >> 172.21.15.5:0/947588881 conn(0x7f8348071240 msgr2=0x7f83480b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:56.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.521+0000 7f834cc07640 1 -- 172.21.15.5:0/947588881 shutdown_connections 2024-03-24T10:44:56.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.521+0000 7f834cc07640 1 -- 172.21.15.5:0/947588881 wait complete. 2024-03-24T10:44:56.534 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:44:45.829320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:44:56.535 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:44:56.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.638+0000 7f61c9db5640 1 Processor -- start 2024-03-24T10:44:56.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.638+0000 7f61c9db5640 1 -- start start 2024-03-24T10:44:56.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.639+0000 7f61c9db5640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f61c40cca90 0x7f61c406bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:56.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.639+0000 7f61c9db5640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61c40ce080 0x7f61c40ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:56.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.639+0000 7f61c9db5640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61c40cd910 con 0x7f61c415ac40 2024-03-24T10:44:56.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.639+0000 7f61c9db5640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f61c40cea50 con 0x7f61c40cca90 2024-03-24T10:44:56.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.639+0000 7f61c9db5640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f61c40cdbf0 con 0x7f61c40ce080 2024-03-24T10:44:56.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.639+0000 7f61c37fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f61c40cca90 0x7f61c406bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:56.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.639+0000 7f61c3fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61c40ce080 0x7f61c40ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:56.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.640+0000 7f61c3fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61c40ce080 0x7f61c40ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:46158/0 (socket says 172.21.15.5:46158) 2024-03-24T10:44:56.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.640+0000 7f61c3fff640 1 -- 172.21.15.5:0/2199806402 learned_addr learned my addr 172.21.15.5:0/2199806402 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:56.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.640+0000 7f61c2ffd640 1 --1- 172.21.15.5:0/2199806402 >> v1:172.21.15.5:6789/0 conn(0x7f61c415ac40 0x7f61c40ca4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:53500/0 (socket says 172.21.15.5:53500) 2024-03-24T10:44:56.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.642+0000 7f61c3fff640 1 -- 172.21.15.5:0/2199806402 >> v2:172.21.15.5:3300/0 conn(0x7f61c40cca90 msgr2=0x7f61c406bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:56.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.642+0000 7f61c3fff640 1 --2- 172.21.15.5:0/2199806402 >> v2:172.21.15.5:3300/0 conn(0x7f61c40cca90 0x7f61c406bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:56.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.642+0000 7f61c3fff640 1 -- 172.21.15.5:0/2199806402 >> v1:172.21.15.5:6789/0 conn(0x7f61c415ac40 legacy=0x7f61c40ca4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:56.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.642+0000 7f61c3fff640 1 -- 172.21.15.5:0/2199806402 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c40cce70 con 0x7f61c40ce080 2024-03-24T10:44:56.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.645+0000 7f61c3fff640 1 --2- 172.21.15.5:0/2199806402 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61c40ce080 0x7f61c40ce510 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f61bc009e50 tx=0x7f61bc02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a95a7d6642562c63 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:56.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.649+0000 7f61c27fc640 1 -- 172.21.15.5:0/2199806402 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61bc02ca70 con 0x7f61c40ce080 2024-03-24T10:44:56.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.649+0000 7f61c27fc640 1 -- 172.21.15.5:0/2199806402 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61bc02cbd0 con 0x7f61c40ce080 2024-03-24T10:44:56.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.649+0000 7f61c27fc640 1 -- 172.21.15.5:0/2199806402 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61bc0343f0 con 0x7f61c40ce080 2024-03-24T10:44:56.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.650+0000 7f61c9db5640 1 -- 172.21.15.5:0/2199806402 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61c40ce080 msgr2=0x7f61c40ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:56.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.650+0000 7f61c9db5640 1 --2- 172.21.15.5:0/2199806402 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61c40ce080 0x7f61c40ce510 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f61bc009e50 tx=0x7f61bc02bed0 comp rx=0 tx=0).stop 2024-03-24T10:44:56.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.650+0000 7f61c9db5640 1 -- 172.21.15.5:0/2199806402 shutdown_connections 2024-03-24T10:44:56.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.650+0000 7f61c9db5640 1 --2- 172.21.15.5:0/2199806402 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61c40ce080 0x7f61c40ce510 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:56.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.650+0000 7f61c9db5640 1 --2- 172.21.15.5:0/2199806402 >> v2:172.21.15.5:3300/0 conn(0x7f61c40cca90 0x7f61c406bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:56.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.650+0000 7f61c9db5640 1 -- 172.21.15.5:0/2199806402 >> 172.21.15.5:0/2199806402 conn(0x7f61c40b16c0 msgr2=0x7f61c40b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:56.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.650+0000 7f61c9db5640 1 -- 172.21.15.5:0/2199806402 shutdown_connections 2024-03-24T10:44:56.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.650+0000 7f61c9db5640 1 -- 172.21.15.5:0/2199806402 wait complete. 2024-03-24T10:44:56.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.651+0000 7f61c9db5640 1 Processor -- start 2024-03-24T10:44:56.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.651+0000 7f61c9db5640 1 -- start start 2024-03-24T10:44:56.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.652+0000 7f61c9db5640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61c40cca90 0x7f61c40c49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:56.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.652+0000 7f61c9db5640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61c40ce080 0x7f61c4157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:56.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.652+0000 7f61c9db5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61c415ac40 0x7f61c415c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:56.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.652+0000 7f61c9db5640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f61c41f0cc0 con 0x7f61c40ce080 2024-03-24T10:44:56.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.652+0000 7f61c9db5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f61c415e2c0 con 0x7f61c415ac40 2024-03-24T10:44:56.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.652+0000 7f61c9db5640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f61c40c3ba0 con 0x7f61c40cca90 2024-03-24T10:44:56.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.652+0000 7f61c2ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61c40ce080 0x7f61c4157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:56.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.652+0000 7f61c37fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61c40cca90 0x7f61c40c49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:56.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.653+0000 7f61c3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61c415ac40 0x7f61c415c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:56.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.653+0000 7f61c2ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61c40ce080 0x7f61c4157320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54692/0 (socket says 172.21.15.5:54692) 2024-03-24T10:44:56.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.653+0000 7f61c2ffd640 1 -- 172.21.15.5:0/1275290831 learned_addr learned my addr 172.21.15.5:0/1275290831 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:56.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.656+0000 7f61c2ffd640 1 -- 172.21.15.5:0/1275290831 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61c40cca90 msgr2=0x7f61c40c49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:56.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.656+0000 7f61c2ffd640 1 --2- 172.21.15.5:0/1275290831 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61c40cca90 0x7f61c40c49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:56.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.656+0000 7f61c2ffd640 1 -- 172.21.15.5:0/1275290831 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61c415ac40 msgr2=0x7f61c415c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:56.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.656+0000 7f61c2ffd640 1 --2- 172.21.15.5:0/1275290831 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61c415ac40 0x7f61c415c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:56.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.656+0000 7f61c2ffd640 1 -- 172.21.15.5:0/1275290831 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61bc009b30 con 0x7f61c40ce080 2024-03-24T10:44:56.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.658+0000 7f61c2ffd640 1 --2- 172.21.15.5:0/1275290831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61c40ce080 0x7f61c4157320 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f61b400c400 tx=0x7f61b400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:56.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.660+0000 7f61abfff640 1 -- 172.21.15.5:0/1275290831 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b40074e0 con 0x7f61c40ce080 2024-03-24T10:44:56.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.660+0000 7f61c9db5640 1 -- 172.21.15.5:0/1275290831 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61c40c3660 con 0x7f61c40ce080 2024-03-24T10:44:56.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.660+0000 7f61c9db5640 1 -- 172.21.15.5:0/1275290831 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61c40c38c0 con 0x7f61c40ce080 2024-03-24T10:44:56.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.661+0000 7f61c9db5640 1 -- 172.21.15.5:0/1275290831 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61c4073480 con 0x7f61c40ce080 2024-03-24T10:44:56.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.662+0000 7f61abfff640 1 -- 172.21.15.5:0/1275290831 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61b4007640 con 0x7f61c40ce080 2024-03-24T10:44:56.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.662+0000 7f61abfff640 1 -- 172.21.15.5:0/1275290831 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b40049e0 con 0x7f61c40ce080 2024-03-24T10:44:56.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.682+0000 7f61abfff640 1 -- 172.21.15.5:0/1275290831 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f61b4004b40 con 0x7f61c40ce080 2024-03-24T10:44:56.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.682+0000 7f61abfff640 1 --2- 172.21.15.5:0/1275290831 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f61b006b810 0x7f61b006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:56.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.683+0000 7f61c37fe640 1 --2- 172.21.15.5:0/1275290831 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f61b006b810 0x7f61b006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:56.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.683+0000 7f61c37fe640 1 --2- 172.21.15.5:0/1275290831 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f61b006b810 0x7f61b006dcd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f61b8009aa0 tx=0x7f61b8009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:56.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.943+0000 7f61abfff640 1 -- 172.21.15.5:0/1275290831 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f61b4058a70 con 0x7f61c40ce080 2024-03-24T10:44:56.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:56.943+0000 7f61abfff640 1 -- 172.21.15.5:0/1275290831 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f61b4054d10 con 0x7f61c40ce080 2024-03-24T10:44:57.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.135+0000 7f61c9db5640 1 -- 172.21.15.5:0/1275290831 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f61c406be70 con 0x7f61c40ce080 2024-03-24T10:44:57.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.164+0000 7f61abfff640 1 -- 172.21.15.5:0/1275290831 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f61b40583d0 con 0x7f61c40ce080 2024-03-24T10:44:57.166 INFO:teuthology.orchestra.run.smithi005.stdout:pg_num: 1 2024-03-24T10:44:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.166+0000 7f61c9db5640 1 -- 172.21.15.5:0/1275290831 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f61b006b810 msgr2=0x7f61b006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.166+0000 7f61c9db5640 1 --2- 172.21.15.5:0/1275290831 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f61b006b810 0x7f61b006dcd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f61b8009aa0 tx=0x7f61b8009190 comp rx=0 tx=0).stop 2024-03-24T10:44:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.166+0000 7f61c9db5640 1 -- 172.21.15.5:0/1275290831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61c40ce080 msgr2=0x7f61c4157320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.166+0000 7f61c9db5640 1 --2- 172.21.15.5:0/1275290831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61c40ce080 0x7f61c4157320 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f61b400c400 tx=0x7f61b400c8c0 comp rx=0 tx=0).stop 2024-03-24T10:44:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.166+0000 7f61c9db5640 1 -- 172.21.15.5:0/1275290831 shutdown_connections 2024-03-24T10:44:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.166+0000 7f61c9db5640 1 --2- 172.21.15.5:0/1275290831 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f61b006b810 0x7f61b006dcd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.166+0000 7f61c9db5640 1 --2- 172.21.15.5:0/1275290831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61c40ce080 0x7f61c4157320 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.166+0000 7f61c9db5640 1 --2- 172.21.15.5:0/1275290831 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61c415ac40 0x7f61c415c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.166+0000 7f61c9db5640 1 --2- 172.21.15.5:0/1275290831 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61c40cca90 0x7f61c40c49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.166+0000 7f61c9db5640 1 -- 172.21.15.5:0/1275290831 >> 172.21.15.5:0/1275290831 conn(0x7f61c40b16c0 msgr2=0x7f61c4157f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.167+0000 7f61c9db5640 1 -- 172.21.15.5:0/1275290831 shutdown_connections 2024-03-24T10:44:57.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.167+0000 7f61c9db5640 1 -- 172.21.15.5:0/1275290831 wait complete. 2024-03-24T10:44:57.178 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T10:44:57.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.280+0000 7f4f49a6f640 1 Processor -- start 2024-03-24T10:44:57.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.280+0000 7f4f49a6f640 1 -- start start 2024-03-24T10:44:57.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.281+0000 7f4f49a6f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4f440cee70 0x7f4f440cba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:57.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.281+0000 7f4f49a6f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4f440cbf50 0x7f4f4415ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:57.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.281+0000 7f4f49a6f640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f4406c5a0 con 0x7f4f440ca490 2024-03-24T10:44:57.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.281+0000 7f4f49a6f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f4f440cc3e0 con 0x7f4f440cee70 2024-03-24T10:44:57.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.281+0000 7f4f49a6f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4f44154c70 con 0x7f4f440cbf50 2024-03-24T10:44:57.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.281+0000 7f4f42ffd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f4f440ca490 0x7f4f4415a480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:53514/0 (socket says 172.21.15.5:53514) 2024-03-24T10:44:57.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.281+0000 7f4f42ffd640 1 -- 172.21.15.5:0/1259599143 learned_addr learned my addr 172.21.15.5:0/1259599143 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:44:57.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.282+0000 7f4f43fff640 1 --2- 172.21.15.5:0/1259599143 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4f440cbf50 0x7f4f4415ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:57.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.282+0000 7f4f437fe640 1 --2- 172.21.15.5:0/1259599143 >> v2:172.21.15.57:3300/0 conn(0x7f4f440cee70 0x7f4f440cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:57.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.284+0000 7f4f427fc640 1 -- 172.21.15.5:0/1259599143 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2025910037 0 0) 0x7f4f4406c5a0 con 0x7f4f440ca490 2024-03-24T10:44:57.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.284+0000 7f4f427fc640 1 -- 172.21.15.5:0/1259599143 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f2c003520 con 0x7f4f440ca490 2024-03-24T10:44:57.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.285+0000 7f4f437fe640 1 -- 172.21.15.5:0/1259599143 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4f440cbf50 msgr2=0x7f4f4415ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:57.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.285+0000 7f4f437fe640 1 --2- 172.21.15.5:0/1259599143 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4f440cbf50 0x7f4f4415ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:57.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.285+0000 7f4f437fe640 1 -- 172.21.15.5:0/1259599143 >> v1:172.21.15.5:6789/0 conn(0x7f4f440ca490 legacy=0x7f4f4415a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:57.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.285+0000 7f4f437fe640 1 -- 172.21.15.5:0/1259599143 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f4406c860 con 0x7f4f440cee70 2024-03-24T10:44:57.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.287+0000 7f4f437fe640 1 --2- 172.21.15.5:0/1259599143 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f440cee70 0x7f4f440cba10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4f340097d0 tx=0x7f4f3402b930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4e60882a0dc97ae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:57.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.291+0000 7f4f427fc640 1 -- 172.21.15.5:0/1259599143 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f34031890 con 0x7f4f440cee70 2024-03-24T10:44:57.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.291+0000 7f4f427fc640 1 -- 172.21.15.5:0/1259599143 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f340319f0 con 0x7f4f440cee70 2024-03-24T10:44:57.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.291+0000 7f4f427fc640 1 -- 172.21.15.5:0/1259599143 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f34031d80 con 0x7f4f440cee70 2024-03-24T10:44:57.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.292+0000 7f4f49a6f640 1 -- 172.21.15.5:0/1259599143 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f440cee70 msgr2=0x7f4f440cba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:57.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.292+0000 7f4f49a6f640 1 --2- 172.21.15.5:0/1259599143 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f440cee70 0x7f4f440cba10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4f340097d0 tx=0x7f4f3402b930 comp rx=0 tx=0).stop 2024-03-24T10:44:57.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.292+0000 7f4f49a6f640 1 -- 172.21.15.5:0/1259599143 shutdown_connections 2024-03-24T10:44:57.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.292+0000 7f4f49a6f640 1 --2- 172.21.15.5:0/1259599143 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4f440cbf50 0x7f4f4415ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:57.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.292+0000 7f4f49a6f640 1 --2- 172.21.15.5:0/1259599143 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f440cee70 0x7f4f440cba10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:57.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.292+0000 7f4f49a6f640 1 -- 172.21.15.5:0/1259599143 >> 172.21.15.5:0/1259599143 conn(0x7f4f440cfb10 msgr2=0x7f4f440cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:57.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.292+0000 7f4f49a6f640 1 -- 172.21.15.5:0/1259599143 shutdown_connections 2024-03-24T10:44:57.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.292+0000 7f4f49a6f640 1 -- 172.21.15.5:0/1259599143 wait complete. 2024-03-24T10:44:57.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.293+0000 7f4f49a6f640 1 Processor -- start 2024-03-24T10:44:57.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.293+0000 7f4f49a6f640 1 -- start start 2024-03-24T10:44:57.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.294+0000 7f4f49a6f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f440ca490 0x7f4f441f0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:57.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.294+0000 7f4f49a6f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4f440cbf50 0x7f4f441560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:57.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.294+0000 7f4f49a6f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f440cee70 0x7f4f4415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:57.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.294+0000 7f4f49a6f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f4f44151a10 con 0x7f4f440ca490 2024-03-24T10:44:57.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.294+0000 7f4f49a6f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4f441582f0 con 0x7f4f440cee70 2024-03-24T10:44:57.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.294+0000 7f4f49a6f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4f441585d0 con 0x7f4f440cbf50 2024-03-24T10:44:57.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.294+0000 7f4f437fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f440ca490 0x7f4f441f0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:57.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.294+0000 7f4f42ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4f440cbf50 0x7f4f441560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:57.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.295+0000 7f4f437fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f440ca490 0x7f4f441f0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54700/0 (socket says 172.21.15.5:54700) 2024-03-24T10:44:57.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.295+0000 7f4f437fe640 1 -- 172.21.15.5:0/3183977021 learned_addr learned my addr 172.21.15.5:0/3183977021 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:57.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.295+0000 7f4f43fff640 1 --2- 172.21.15.5:0/3183977021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f440cee70 0x7f4f4415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:57.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.298+0000 7f4f437fe640 1 -- 172.21.15.5:0/3183977021 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4f440cbf50 msgr2=0x7f4f441560b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:57.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.298+0000 7f4f437fe640 1 --2- 172.21.15.5:0/3183977021 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4f440cbf50 0x7f4f441560b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:57.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.298+0000 7f4f437fe640 1 -- 172.21.15.5:0/3183977021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f440cee70 msgr2=0x7f4f4415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:57.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.298+0000 7f4f437fe640 1 --2- 172.21.15.5:0/3183977021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f440cee70 0x7f4f4415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:57.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.298+0000 7f4f437fe640 1 -- 172.21.15.5:0/3183977021 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f34009400 con 0x7f4f440ca490 2024-03-24T10:44:57.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.300+0000 7f4f437fe640 1 --2- 172.21.15.5:0/3183977021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f440ca490 0x7f4f441f0ab0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4f34034040 tx=0x7f4f340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:57.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.304+0000 7f4f2bfff640 1 -- 172.21.15.5:0/3183977021 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3403aba0 con 0x7f4f440ca490 2024-03-24T10:44:57.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.304+0000 7f4f2bfff640 1 -- 172.21.15.5:0/3183977021 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f3403ad20 con 0x7f4f440ca490 2024-03-24T10:44:57.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.304+0000 7f4f2bfff640 1 -- 172.21.15.5:0/3183977021 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f34044640 con 0x7f4f440ca490 2024-03-24T10:44:57.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.304+0000 7f4f49a6f640 1 -- 172.21.15.5:0/3183977021 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f441588b0 con 0x7f4f440ca490 2024-03-24T10:44:57.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.304+0000 7f4f49a6f640 1 -- 172.21.15.5:0/3183977021 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f44158e70 con 0x7f4f440ca490 2024-03-24T10:44:57.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.305+0000 7f4f49a6f640 1 -- 172.21.15.5:0/3183977021 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f440c4070 con 0x7f4f440ca490 2024-03-24T10:44:57.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.324+0000 7f4f2bfff640 1 -- 172.21.15.5:0/3183977021 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4f3404d440 con 0x7f4f440ca490 2024-03-24T10:44:57.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.325+0000 7f4f2bfff640 1 --2- 172.21.15.5:0/3183977021 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4f3006b860 0x7f4f3006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:57.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.325+0000 7f4f42ffd640 1 --2- 172.21.15.5:0/3183977021 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4f3006b860 0x7f4f3006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:57.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.325+0000 7f4f42ffd640 1 --2- 172.21.15.5:0/3183977021 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4f3006b860 0x7f4f3006dd20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4f440c2b90 tx=0x7f4f38009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:57.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.327+0000 7f4f2bfff640 1 -- 172.21.15.5:0/3183977021 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f4f340447a0 con 0x7f4f440ca490 2024-03-24T10:44:57.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.599+0000 7f4f2bfff640 1 -- 172.21.15.5:0/3183977021 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4f34077540 con 0x7f4f440ca490 2024-03-24T10:44:57.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.782+0000 7f4f49a6f640 1 -- 172.21.15.5:0/3183977021 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f4f44068a10 con 0x7f4f440ca490 2024-03-24T10:44:57.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.789+0000 7f4f2bfff640 1 -- 172.21.15.5:0/3183977021 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7f4f34035020 con 0x7f4f440ca490 2024-03-24T10:44:57.790 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:44:57.790 INFO:teuthology.orchestra.run.smithi005.stdout:[ 2024-03-24T10:44:57.790 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-03-24T10:44:57.790 INFO:teuthology.orchestra.run.smithi005.stdout: "poolnum": 1, 2024-03-24T10:44:57.791 INFO:teuthology.orchestra.run.smithi005.stdout: "poolname": ".mgr" 2024-03-24T10:44:57.791 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-24T10:44:57.791 INFO:teuthology.orchestra.run.smithi005.stdout:] 2024-03-24T10:44:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.791+0000 7f4f49a6f640 1 -- 172.21.15.5:0/3183977021 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4f3006b860 msgr2=0x7f4f3006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.791+0000 7f4f49a6f640 1 --2- 172.21.15.5:0/3183977021 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4f3006b860 0x7f4f3006dd20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4f440c2b90 tx=0x7f4f38009040 comp rx=0 tx=0).stop 2024-03-24T10:44:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.791+0000 7f4f49a6f640 1 -- 172.21.15.5:0/3183977021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f440ca490 msgr2=0x7f4f441f0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.791+0000 7f4f49a6f640 1 --2- 172.21.15.5:0/3183977021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f440ca490 0x7f4f441f0ab0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4f34034040 tx=0x7f4f340046c0 comp rx=0 tx=0).stop 2024-03-24T10:44:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.791+0000 7f4f49a6f640 1 -- 172.21.15.5:0/3183977021 shutdown_connections 2024-03-24T10:44:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.791+0000 7f4f49a6f640 1 --2- 172.21.15.5:0/3183977021 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4f3006b860 0x7f4f3006dd20 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.791+0000 7f4f49a6f640 1 --2- 172.21.15.5:0/3183977021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4f440cee70 0x7f4f4415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.791+0000 7f4f49a6f640 1 --2- 172.21.15.5:0/3183977021 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4f440cbf50 0x7f4f441560b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.791+0000 7f4f49a6f640 1 --2- 172.21.15.5:0/3183977021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4f440ca490 0x7f4f441f0ab0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.791+0000 7f4f49a6f640 1 -- 172.21.15.5:0/3183977021 >> 172.21.15.5:0/3183977021 conn(0x7f4f440cfb10 msgr2=0x7f4f440b4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.791+0000 7f4f49a6f640 1 -- 172.21.15.5:0/3183977021 shutdown_connections 2024-03-24T10:44:57.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.791+0000 7f4f49a6f640 1 -- 172.21.15.5:0/3183977021 wait complete. 2024-03-24T10:44:57.803 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T10:44:57.803 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T10:44:57.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.906+0000 7f67083b6640 1 Processor -- start 2024-03-24T10:44:57.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.906+0000 7f67083b6640 1 -- start start 2024-03-24T10:44:57.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.907+0000 7f67083b6640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f67000c7aa0 0x7f67000c2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:57.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.907+0000 7f67083b6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6700072240 0x7f67001508a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:57.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.907+0000 7f67083b6640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6700071db0 con 0x7f67000c76c0 2024-03-24T10:44:57.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.907+0000 7f67083b6640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f67000c3ce0 con 0x7f67000c7aa0 2024-03-24T10:44:57.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.907+0000 7f67083b6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f67000c3740 con 0x7f6700072240 2024-03-24T10:44:57.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.908+0000 7f6705951640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f67000c7aa0 0x7f67000c2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:57.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.908+0000 7f6706953640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6700072240 0x7f67001508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:57.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.908+0000 7f6705951640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f67000c7aa0 0x7f67000c2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54706/0 (socket says 172.21.15.5:54706) 2024-03-24T10:44:57.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.908+0000 7f6705951640 1 -- 172.21.15.5:0/975185300 learned_addr learned my addr 172.21.15.5:0/975185300 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:57.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.911+0000 7f6706953640 1 -- 172.21.15.5:0/975185300 >> v2:172.21.15.5:3300/0 conn(0x7f67000c7aa0 msgr2=0x7f67000c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:57.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.911+0000 7f6706953640 1 --2- 172.21.15.5:0/975185300 >> v2:172.21.15.5:3300/0 conn(0x7f67000c7aa0 0x7f67000c2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:57.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.911+0000 7f6706953640 1 -- 172.21.15.5:0/975185300 >> v1:172.21.15.5:6789/0 conn(0x7f67000c76c0 legacy=0x7f670015a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:57.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.911+0000 7f6706953640 1 -- 172.21.15.5:0/975185300 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67000c3a00 con 0x7f6700072240 2024-03-24T10:44:57.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.913+0000 7f6706953640 1 --2- 172.21.15.5:0/975185300 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6700072240 0x7f67001508a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f66f0009d90 tx=0x7f66f002bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1011d3b5447d651a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:57.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.917+0000 7f6705150640 1 -- 172.21.15.5:0/975185300 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f002cce0 con 0x7f6700072240 2024-03-24T10:44:57.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.917+0000 7f6705150640 1 -- 172.21.15.5:0/975185300 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66f002ce40 con 0x7f6700072240 2024-03-24T10:44:57.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.917+0000 7f6705150640 1 -- 172.21.15.5:0/975185300 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f0034660 con 0x7f6700072240 2024-03-24T10:44:57.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.917+0000 7f67083b6640 1 -- 172.21.15.5:0/975185300 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6700072240 msgr2=0x7f67001508a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:57.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.917+0000 7f67083b6640 1 --2- 172.21.15.5:0/975185300 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6700072240 0x7f67001508a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f66f0009d90 tx=0x7f66f002bf30 comp rx=0 tx=0).stop 2024-03-24T10:44:57.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.917+0000 7f67083b6640 1 -- 172.21.15.5:0/975185300 shutdown_connections 2024-03-24T10:44:57.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.917+0000 7f67083b6640 1 --2- 172.21.15.5:0/975185300 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6700072240 0x7f67001508a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:57.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.917+0000 7f67083b6640 1 --2- 172.21.15.5:0/975185300 >> v2:172.21.15.5:3300/0 conn(0x7f67000c7aa0 0x7f67000c2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:57.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.917+0000 7f67083b6640 1 -- 172.21.15.5:0/975185300 >> 172.21.15.5:0/975185300 conn(0x7f670006f180 msgr2=0x7f670006f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:57.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.918+0000 7f67083b6640 1 -- 172.21.15.5:0/975185300 shutdown_connections 2024-03-24T10:44:57.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.918+0000 7f67083b6640 1 -- 172.21.15.5:0/975185300 wait complete. 2024-03-24T10:44:57.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.919+0000 7f67083b6640 1 Processor -- start 2024-03-24T10:44:57.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.919+0000 7f67083b6640 1 -- start start 2024-03-24T10:44:57.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.919+0000 7f67083b6640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6700072240 0x7f6700152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:57.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.919+0000 7f67083b6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f67000c76c0 0x7f670014bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:57.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.919+0000 7f67083b6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67000c7aa0 0x7f67001578e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:57.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.919+0000 7f67083b6640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f670014b600 con 0x7f6700072240 2024-03-24T10:44:57.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.919+0000 7f67083b6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6700153100 con 0x7f67000c7aa0 2024-03-24T10:44:57.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.919+0000 7f67083b6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f6700151450 con 0x7f67000c76c0 2024-03-24T10:44:57.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.920+0000 7f6706152640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6700072240 0x7f6700152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:57.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.920+0000 7f6705951640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f67000c76c0 0x7f670014bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:57.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.920+0000 7f6706152640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6700072240 0x7f6700152bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54716/0 (socket says 172.21.15.5:54716) 2024-03-24T10:44:57.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.920+0000 7f6706152640 1 -- 172.21.15.5:0/1158662287 learned_addr learned my addr 172.21.15.5:0/1158662287 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:57.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.920+0000 7f6706953640 1 --2- 172.21.15.5:0/1158662287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67000c7aa0 0x7f67001578e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:57.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.924+0000 7f6706152640 1 -- 172.21.15.5:0/1158662287 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f67000c76c0 msgr2=0x7f670014bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:57.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.924+0000 7f6706152640 1 --2- 172.21.15.5:0/1158662287 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f67000c76c0 0x7f670014bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:57.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.924+0000 7f6706152640 1 -- 172.21.15.5:0/1158662287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67000c7aa0 msgr2=0x7f67001578e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:57.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.924+0000 7f6706152640 1 --2- 172.21.15.5:0/1158662287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67000c7aa0 0x7f67001578e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:57.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.924+0000 7f6706152640 1 -- 172.21.15.5:0/1158662287 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f0009a70 con 0x7f6700072240 2024-03-24T10:44:57.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.926+0000 7f6706152640 1 --2- 172.21.15.5:0/1158662287 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6700072240 0x7f6700152bc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f66f800c400 tx=0x7f66f800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:57.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.929+0000 7f66feffd640 1 -- 172.21.15.5:0/1158662287 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f80052d0 con 0x7f6700072240 2024-03-24T10:44:57.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.929+0000 7f66feffd640 1 -- 172.21.15.5:0/1158662287 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66f8005450 con 0x7f6700072240 2024-03-24T10:44:57.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.929+0000 7f67083b6640 1 -- 172.21.15.5:0/1158662287 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6700151fb0 con 0x7f6700072240 2024-03-24T10:44:57.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.929+0000 7f66feffd640 1 -- 172.21.15.5:0/1158662287 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f8007260 con 0x7f6700072240 2024-03-24T10:44:57.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.929+0000 7f67083b6640 1 -- 172.21.15.5:0/1158662287 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67001519f0 con 0x7f6700072240 2024-03-24T10:44:57.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:57.930+0000 7f67083b6640 1 -- 172.21.15.5:0/1158662287 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f670014b460 con 0x7f6700072240 2024-03-24T10:44:58.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:58.197+0000 7f66feffd640 1 -- 172.21.15.5:0/1158662287 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f66f80073c0 con 0x7f6700072240 2024-03-24T10:44:58.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:58.198+0000 7f66feffd640 1 --2- 172.21.15.5:0/1158662287 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f66ec06b860 0x7f66ec06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:58.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:58.198+0000 7f6705951640 1 --2- 172.21.15.5:0/1158662287 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f66ec06b860 0x7f66ec06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:58.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:58.198+0000 7f6705951640 1 --2- 172.21.15.5:0/1158662287 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f66ec06b860 0x7f66ec06dd20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f67000c4700 tx=0x7f66f400b450 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:58.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:58.212+0000 7f66feffd640 1 -- 172.21.15.5:0/1158662287 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f66f8007670 con 0x7f6700072240 2024-03-24T10:44:58.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:58.212+0000 7f66feffd640 1 -- 172.21.15.5:0/1158662287 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f66f8054700 con 0x7f6700072240 2024-03-24T10:44:58.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:58.426+0000 7f67083b6640 1 -- 172.21.15.5:0/1158662287 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f670014b460 con 0x7f6700072240 2024-03-24T10:44:59.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.487+0000 7f66feffd640 1 -- 172.21.15.5:0/1158662287 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7f670014b460 con 0x7f6700072240 2024-03-24T10:44:59.489 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'cephfs_metadata' created 2024-03-24T10:44:59.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.490+0000 7f67083b6640 1 -- 172.21.15.5:0/1158662287 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f66ec06b860 msgr2=0x7f66ec06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:59.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.490+0000 7f67083b6640 1 --2- 172.21.15.5:0/1158662287 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f66ec06b860 0x7f66ec06dd20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f67000c4700 tx=0x7f66f400b450 comp rx=0 tx=0).stop 2024-03-24T10:44:59.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.490+0000 7f67083b6640 1 -- 172.21.15.5:0/1158662287 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6700072240 msgr2=0x7f6700152bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:59.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.490+0000 7f67083b6640 1 --2- 172.21.15.5:0/1158662287 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6700072240 0x7f6700152bc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f66f800c400 tx=0x7f66f800c8c0 comp rx=0 tx=0).stop 2024-03-24T10:44:59.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.490+0000 7f67083b6640 1 -- 172.21.15.5:0/1158662287 shutdown_connections 2024-03-24T10:44:59.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.490+0000 7f67083b6640 1 --2- 172.21.15.5:0/1158662287 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f66ec06b860 0x7f66ec06dd20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:59.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.490+0000 7f67083b6640 1 --2- 172.21.15.5:0/1158662287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f67000c7aa0 0x7f67001578e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:59.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.490+0000 7f67083b6640 1 --2- 172.21.15.5:0/1158662287 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f67000c76c0 0x7f670014bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:59.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.490+0000 7f67083b6640 1 --2- 172.21.15.5:0/1158662287 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6700072240 0x7f6700152bc0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:59.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.490+0000 7f67083b6640 1 -- 172.21.15.5:0/1158662287 >> 172.21.15.5:0/1158662287 conn(0x7f670006f180 msgr2=0x7f67000c6230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:59.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.490+0000 7f67083b6640 1 -- 172.21.15.5:0/1158662287 shutdown_connections 2024-03-24T10:44:59.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.490+0000 7f67083b6640 1 -- 172.21.15.5:0/1158662287 wait complete. 2024-03-24T10:44:59.502 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T10:44:59.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.622+0000 7efc64eda640 1 Processor -- start 2024-03-24T10:44:59.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.622+0000 7efc64eda640 1 -- start start 2024-03-24T10:44:59.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.623+0000 7efc64eda640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efc60152610 0x7efc60158b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:59.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.623+0000 7efc64eda640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7efc600c9560 0x7efc6015bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:59.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.623+0000 7efc64eda640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7efc600c9f60 0x7efc600c8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:59.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.623+0000 7efc64eda640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7efc601590c0 con 0x7efc600c9f60 2024-03-24T10:44:59.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.623+0000 7efc64eda640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7efc6015ba80 con 0x7efc600c9560 2024-03-24T10:44:59.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.623+0000 7efc64eda640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7efc600ca530 con 0x7efc60152610 2024-03-24T10:44:59.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.624+0000 7efc5e59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efc60152610 0x7efc60158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:59.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.624+0000 7efc5dd9b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7efc600c9560 0x7efc6015bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:59.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.624+0000 7efc5dd9b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7efc600c9560 0x7efc6015bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:56444/0 (socket says 172.21.15.5:56444) 2024-03-24T10:44:59.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.624+0000 7efc5dd9b640 1 -- 172.21.15.5:0/3515756794 learned_addr learned my addr 172.21.15.5:0/3515756794 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:59.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.626+0000 7efc5dd9b640 1 -- 172.21.15.5:0/3515756794 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efc60152610 msgr2=0x7efc60158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:59.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.626+0000 7efc5dd9b640 1 --2- 172.21.15.5:0/3515756794 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efc60152610 0x7efc60158b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:59.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.626+0000 7efc5dd9b640 1 -- 172.21.15.5:0/3515756794 >> v2:172.21.15.5:3300/0 conn(0x7efc600c9f60 msgr2=0x7efc600c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:44:59.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.626+0000 7efc5dd9b640 1 --2- 172.21.15.5:0/3515756794 >> v2:172.21.15.5:3300/0 conn(0x7efc600c9f60 0x7efc600c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:59.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.626+0000 7efc5dd9b640 1 -- 172.21.15.5:0/3515756794 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc600ca810 con 0x7efc600c9560 2024-03-24T10:44:59.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.628+0000 7efc5dd9b640 1 --2- 172.21.15.5:0/3515756794 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc600c9560 0x7efc6015bf10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efc60070490 tx=0x7efc4802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f32e122c5ea1d1f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:59.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.632+0000 7efc5d59a640 1 -- 172.21.15.5:0/3515756794 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc48030860 con 0x7efc600c9560 2024-03-24T10:44:59.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.632+0000 7efc5d59a640 1 -- 172.21.15.5:0/3515756794 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc480309c0 con 0x7efc600c9560 2024-03-24T10:44:59.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.632+0000 7efc5d59a640 1 -- 172.21.15.5:0/3515756794 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc48030d50 con 0x7efc600c9560 2024-03-24T10:44:59.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.632+0000 7efc64eda640 1 -- 172.21.15.5:0/3515756794 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc600c9560 msgr2=0x7efc6015bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:59.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.632+0000 7efc64eda640 1 --2- 172.21.15.5:0/3515756794 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc600c9560 0x7efc6015bf10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efc60070490 tx=0x7efc4802b9a0 comp rx=0 tx=0).stop 2024-03-24T10:44:59.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.632+0000 7efc64eda640 1 -- 172.21.15.5:0/3515756794 shutdown_connections 2024-03-24T10:44:59.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.632+0000 7efc64eda640 1 --2- 172.21.15.5:0/3515756794 >> v2:172.21.15.5:3300/0 conn(0x7efc600c9f60 0x7efc600c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:59.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.632+0000 7efc64eda640 1 --2- 172.21.15.5:0/3515756794 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc600c9560 0x7efc6015bf10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:59.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.632+0000 7efc64eda640 1 --2- 172.21.15.5:0/3515756794 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efc60152610 0x7efc60158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:59.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.632+0000 7efc64eda640 1 -- 172.21.15.5:0/3515756794 >> 172.21.15.5:0/3515756794 conn(0x7efc600711f0 msgr2=0x7efc600715d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:44:59.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.632+0000 7efc64eda640 1 -- 172.21.15.5:0/3515756794 shutdown_connections 2024-03-24T10:44:59.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.633+0000 7efc64eda640 1 -- 172.21.15.5:0/3515756794 wait complete. 2024-03-24T10:44:59.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.634+0000 7efc64eda640 1 Processor -- start 2024-03-24T10:44:59.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.634+0000 7efc64eda640 1 -- start start 2024-03-24T10:44:59.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.634+0000 7efc64eda640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc600c9560 0x7efc6006d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:59.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.634+0000 7efc64eda640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efc600c9f60 0x7efc601f0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:59.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.634+0000 7efc64eda640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc60152610 0x7efc601570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:59.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.634+0000 7efc64eda640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7efc600ce930 con 0x7efc60152610 2024-03-24T10:44:59.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.634+0000 7efc64eda640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7efc600c77a0 con 0x7efc600c9560 2024-03-24T10:44:59.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.634+0000 7efc64eda640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7efc601f36c0 con 0x7efc600c9f60 2024-03-24T10:44:59.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.635+0000 7efc5e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc600c9560 0x7efc6006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:59.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.635+0000 7efc5e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc600c9560 0x7efc6006d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:56446/0 (socket says 172.21.15.5:56446) 2024-03-24T10:44:59.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.635+0000 7efc5e59c640 1 -- 172.21.15.5:0/3245433100 learned_addr learned my addr 172.21.15.5:0/3245433100 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:44:59.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.636+0000 7efc5dd9b640 1 --2- 172.21.15.5:0/3245433100 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efc600c9f60 0x7efc601f0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:59.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.638+0000 7efc5e59c640 1 -- 172.21.15.5:0/3245433100 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efc600c9f60 msgr2=0x7efc601f0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:59.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.638+0000 7efc5e59c640 1 --2- 172.21.15.5:0/3245433100 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efc600c9f60 0x7efc601f0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:59.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.638+0000 7efc5e59c640 1 -- 172.21.15.5:0/3245433100 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc60152610 msgr2=0x7efc601570d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:44:59.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.638+0000 7efc5e59c640 1 --2- 172.21.15.5:0/3245433100 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc60152610 0x7efc601570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:44:59.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.638+0000 7efc5e59c640 1 -- 172.21.15.5:0/3245433100 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc480094d0 con 0x7efc600c9560 2024-03-24T10:44:59.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.641+0000 7efc5e59c640 1 --2- 172.21.15.5:0/3245433100 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc600c9560 0x7efc6006d300 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efc5800e9e0 tx=0x7efc5800eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:59.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.644+0000 7efc56ffd640 1 -- 172.21.15.5:0/3245433100 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc58009670 con 0x7efc600c9560 2024-03-24T10:44:59.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.644+0000 7efc56ffd640 1 -- 172.21.15.5:0/3245433100 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc5800c9d0 con 0x7efc600c9560 2024-03-24T10:44:59.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.644+0000 7efc56ffd640 1 -- 172.21.15.5:0/3245433100 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc5800cd80 con 0x7efc600c9560 2024-03-24T10:44:59.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.644+0000 7efc64eda640 1 -- 172.21.15.5:0/3245433100 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc601ef6a0 con 0x7efc600c9560 2024-03-24T10:44:59.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.644+0000 7efc64eda640 1 -- 172.21.15.5:0/3245433100 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc601ef960 con 0x7efc600c9560 2024-03-24T10:44:59.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.645+0000 7efc64eda640 1 -- 172.21.15.5:0/3245433100 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc600688e0 con 0x7efc600c9560 2024-03-24T10:44:59.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.665+0000 7efc56ffd640 1 -- 172.21.15.5:0/3245433100 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efc58013470 con 0x7efc600c9560 2024-03-24T10:44:59.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.665+0000 7efc56ffd640 1 --2- 172.21.15.5:0/3245433100 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7efc4006b890 0x7efc4006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:44:59.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.665+0000 7efc5dd9b640 1 --2- 172.21.15.5:0/3245433100 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7efc4006b890 0x7efc4006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:44:59.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.666+0000 7efc5dd9b640 1 --2- 172.21.15.5:0/3245433100 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7efc4006b890 0x7efc4006dd50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efc48036000 tx=0x7efc48008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:44:59.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.667+0000 7efc56ffd640 1 -- 172.21.15.5:0/3245433100 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7efc580593c0 con 0x7efc600c9560 2024-03-24T10:44:59.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:44:59.930+0000 7efc56ffd640 1 -- 172.21.15.5:0/3245433100 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efc58055660 con 0x7efc600c9560 2024-03-24T10:45:00.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.113+0000 7efc64eda640 1 -- 172.21.15.5:0/3245433100 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7efc600c9200 con 0x7efc600c9560 2024-03-24T10:45:00.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.651+0000 7efc56ffd640 1 -- 172.21.15.5:0/3245433100 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) ==== 170+0+0 (secure 0 0 0) 0x7efc580170d0 con 0x7efc600c9560 2024-03-24T10:45:00.653 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'cephfs_data' created 2024-03-24T10:45:00.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.654+0000 7efc64eda640 1 -- 172.21.15.5:0/3245433100 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7efc4006b890 msgr2=0x7efc4006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:00.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.654+0000 7efc64eda640 1 --2- 172.21.15.5:0/3245433100 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7efc4006b890 0x7efc4006dd50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efc48036000 tx=0x7efc48008030 comp rx=0 tx=0).stop 2024-03-24T10:45:00.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.654+0000 7efc64eda640 1 -- 172.21.15.5:0/3245433100 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc600c9560 msgr2=0x7efc6006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:00.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.654+0000 7efc64eda640 1 --2- 172.21.15.5:0/3245433100 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc600c9560 0x7efc6006d300 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efc5800e9e0 tx=0x7efc5800eea0 comp rx=0 tx=0).stop 2024-03-24T10:45:00.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.654+0000 7efc64eda640 1 -- 172.21.15.5:0/3245433100 shutdown_connections 2024-03-24T10:45:00.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.654+0000 7efc64eda640 1 --2- 172.21.15.5:0/3245433100 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7efc4006b890 0x7efc4006dd50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:00.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.654+0000 7efc64eda640 1 --2- 172.21.15.5:0/3245433100 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efc60152610 0x7efc601570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:00.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.654+0000 7efc64eda640 1 --2- 172.21.15.5:0/3245433100 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efc600c9f60 0x7efc601f0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:00.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.654+0000 7efc64eda640 1 --2- 172.21.15.5:0/3245433100 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efc600c9560 0x7efc6006d300 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:00.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.654+0000 7efc64eda640 1 -- 172.21.15.5:0/3245433100 >> 172.21.15.5:0/3245433100 conn(0x7efc600711f0 msgr2=0x7efc601e2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:00.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.654+0000 7efc64eda640 1 -- 172.21.15.5:0/3245433100 shutdown_connections 2024-03-24T10:45:00.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.654+0000 7efc64eda640 1 -- 172.21.15.5:0/3245433100 wait complete. 2024-03-24T10:45:00.665 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T10:45:00.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.771+0000 7ff4becdf640 1 Processor -- start 2024-03-24T10:45:00.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.771+0000 7ff4becdf640 1 -- start start 2024-03-24T10:45:00.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.771+0000 7ff4becdf640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff4b80b0580 0x7ff4b80d0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:00.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.771+0000 7ff4becdf640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4b80d0660 0x7ff4b815de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:00.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.771+0000 7ff4becdf640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4b80b14d0 con 0x7ff4b81547a0 2024-03-24T10:45:00.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.771+0000 7ff4becdf640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7ff4b80b11f0 con 0x7ff4b80b0580 2024-03-24T10:45:00.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.771+0000 7ff4becdf640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b80ca5f0 con 0x7ff4b80d0660 2024-03-24T10:45:00.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.812+0000 7ff4bd27c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4b80d0660 0x7ff4b815de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:00.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.813+0000 7ff4bd27c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4b80d0660 0x7ff4b815de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:46254/0 (socket says 172.21.15.5:46254) 2024-03-24T10:45:00.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.813+0000 7ff4bd27c640 1 -- 172.21.15.5:0/2307372538 learned_addr learned my addr 172.21.15.5:0/2307372538 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:00.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.815+0000 7ff4bd27c640 1 -- 172.21.15.5:0/2307372538 >> v2:172.21.15.57:3300/0 conn(0x7ff4b80b0580 msgr2=0x7ff4b80d0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:00.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.815+0000 7ff4bd27c640 1 --2- 172.21.15.5:0/2307372538 >> v2:172.21.15.57:3300/0 conn(0x7ff4b80b0580 0x7ff4b80d0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:00.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.815+0000 7ff4bd27c640 1 -- 172.21.15.5:0/2307372538 >> v1:172.21.15.57:6789/0 conn(0x7ff4b81547a0 legacy=0x7ff4b815ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:00.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.815+0000 7ff4bd27c640 1 -- 172.21.15.5:0/2307372538 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b80ca8d0 con 0x7ff4b80d0660 2024-03-24T10:45:00.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.818+0000 7ff4bd27c640 1 --2- 172.21.15.5:0/2307372538 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4b80d0660 0x7ff4b815de40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80af820 tx=0x7ff4b002bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b0e721a56913aa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:00.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.822+0000 7ff4b77fe640 1 -- 172.21.15.5:0/2307372538 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b002caf0 con 0x7ff4b80d0660 2024-03-24T10:45:00.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.822+0000 7ff4b77fe640 1 -- 172.21.15.5:0/2307372538 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b002cc50 con 0x7ff4b80d0660 2024-03-24T10:45:00.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.822+0000 7ff4b77fe640 1 -- 172.21.15.5:0/2307372538 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b00343f0 con 0x7ff4b80d0660 2024-03-24T10:45:00.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.822+0000 7ff4becdf640 1 -- 172.21.15.5:0/2307372538 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4b80d0660 msgr2=0x7ff4b815de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:00.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.822+0000 7ff4becdf640 1 --2- 172.21.15.5:0/2307372538 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4b80d0660 0x7ff4b815de40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80af820 tx=0x7ff4b002bf50 comp rx=0 tx=0).stop 2024-03-24T10:45:00.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.823+0000 7ff4becdf640 1 -- 172.21.15.5:0/2307372538 shutdown_connections 2024-03-24T10:45:00.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.823+0000 7ff4becdf640 1 --2- 172.21.15.5:0/2307372538 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4b80d0660 0x7ff4b815de40 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:00.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.823+0000 7ff4becdf640 1 --2- 172.21.15.5:0/2307372538 >> v2:172.21.15.57:3300/0 conn(0x7ff4b80b0580 0x7ff4b80d0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:00.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.823+0000 7ff4becdf640 1 -- 172.21.15.5:0/2307372538 >> 172.21.15.5:0/2307372538 conn(0x7ff4b8061f70 msgr2=0x7ff4b8062380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:00.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.823+0000 7ff4becdf640 1 -- 172.21.15.5:0/2307372538 shutdown_connections 2024-03-24T10:45:00.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.823+0000 7ff4becdf640 1 -- 172.21.15.5:0/2307372538 wait complete. 2024-03-24T10:45:00.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.824+0000 7ff4becdf640 1 Processor -- start 2024-03-24T10:45:00.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.824+0000 7ff4becdf640 1 -- start start 2024-03-24T10:45:00.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.825+0000 7ff4becdf640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4b80b0580 0x7ff4b815cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:00.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.825+0000 7ff4becdf640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4b80d0660 0x7ff4b8159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:00.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.825+0000 7ff4becdf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4b81547a0 0x7ff4b8147d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:00.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.825+0000 7ff4becdf640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b806d950 con 0x7ff4b80b0580 2024-03-24T10:45:00.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.825+0000 7ff4becdf640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b815c750 con 0x7ff4b81547a0 2024-03-24T10:45:00.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.825+0000 7ff4becdf640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b815d120 con 0x7ff4b80d0660 2024-03-24T10:45:00.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.825+0000 7ff4b7fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4b80d0660 0x7ff4b8159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:00.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.825+0000 7ff4b7fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4b80d0660 0x7ff4b8159d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:46262/0 (socket says 172.21.15.5:46262) 2024-03-24T10:45:00.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.825+0000 7ff4b7fff640 1 -- 172.21.15.5:0/3099695791 learned_addr learned my addr 172.21.15.5:0/3099695791 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:00.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.829+0000 7ff4b7fff640 1 -- 172.21.15.5:0/3099695791 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4b81547a0 msgr2=0x7ff4b8147d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:00.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.829+0000 7ff4b7fff640 1 --2- 172.21.15.5:0/3099695791 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4b81547a0 0x7ff4b8147d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:00.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.829+0000 7ff4b7fff640 1 -- 172.21.15.5:0/3099695791 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4b80b0580 msgr2=0x7ff4b815cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:00.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.829+0000 7ff4b7fff640 1 --2- 172.21.15.5:0/3099695791 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4b80b0580 0x7ff4b815cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:00.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.829+0000 7ff4b7fff640 1 -- 172.21.15.5:0/3099695791 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b0009b30 con 0x7ff4b80d0660 2024-03-24T10:45:00.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.831+0000 7ff4b7fff640 1 --2- 172.21.15.5:0/3099695791 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4b80d0660 0x7ff4b8159d00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400b410 tx=0x7ff4a400b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:00.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.835+0000 7ff4b57fa640 1 -- 172.21.15.5:0/3099695791 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4011830 con 0x7ff4b80d0660 2024-03-24T10:45:00.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.835+0000 7ff4b57fa640 1 -- 172.21.15.5:0/3099695791 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4a40119b0 con 0x7ff4b80d0660 2024-03-24T10:45:00.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.835+0000 7ff4b57fa640 1 -- 172.21.15.5:0/3099695791 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4011d60 con 0x7ff4b80d0660 2024-03-24T10:45:00.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.835+0000 7ff4becdf640 1 -- 172.21.15.5:0/3099695791 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4b814afa0 con 0x7ff4b80d0660 2024-03-24T10:45:00.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.835+0000 7ff4becdf640 1 -- 172.21.15.5:0/3099695791 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4b814acc0 con 0x7ff4b80d0660 2024-03-24T10:45:00.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.836+0000 7ff4becdf640 1 -- 172.21.15.5:0/3099695791 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4b80688e0 con 0x7ff4b80d0660 2024-03-24T10:45:00.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.855+0000 7ff4b57fa640 1 -- 172.21.15.5:0/3099695791 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff4a401a400 con 0x7ff4b80d0660 2024-03-24T10:45:00.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.856+0000 7ff4b57fa640 1 --2- 172.21.15.5:0/3099695791 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff49c06b860 0x7ff49c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:00.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.856+0000 7ff4bca7b640 1 --2- 172.21.15.5:0/3099695791 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff49c06b860 0x7ff49c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:00.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.856+0000 7ff4bca7b640 1 --2- 172.21.15.5:0/3099695791 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff49c06b860 0x7ff49c06dd20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac001560 tx=0x7ff4ac007990 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:00.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:00.858+0000 7ff4b57fa640 1 -- 172.21.15.5:0/3099695791 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7ff4a4059210 con 0x7ff4b80d0660 2024-03-24T10:45:01.104 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:45:01.106 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:45:01.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:01.116+0000 7ff4b57fa640 1 -- 172.21.15.5:0/3099695791 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff4a40554b0 con 0x7ff4b80d0660 2024-03-24T10:45:01.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:01.340+0000 7ff4becdf640 1 -- 172.21.15.5:0/3099695791 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7ff4b80b5b50 con 0x7ff4b80d0660 2024-03-24T10:45:02.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.153+0000 7ff4b57fa640 1 -- 172.21.15.5:0/3099695791 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7ff4a4015070 con 0x7ff4b80d0660 2024-03-24T10:45:02.154 INFO:teuthology.orchestra.run.smithi005.stderr:new fs with metadata pool 2 and data pool 3 2024-03-24T10:45:02.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.155+0000 7ff4becdf640 1 -- 172.21.15.5:0/3099695791 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff49c06b860 msgr2=0x7ff49c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:02.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.155+0000 7ff4becdf640 1 --2- 172.21.15.5:0/3099695791 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff49c06b860 0x7ff49c06dd20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac001560 tx=0x7ff4ac007990 comp rx=0 tx=0).stop 2024-03-24T10:45:02.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.155+0000 7ff4becdf640 1 -- 172.21.15.5:0/3099695791 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4b80d0660 msgr2=0x7ff4b8159d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:02.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.155+0000 7ff4becdf640 1 --2- 172.21.15.5:0/3099695791 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4b80d0660 0x7ff4b8159d00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400b410 tx=0x7ff4a400b8d0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.155+0000 7ff4becdf640 1 -- 172.21.15.5:0/3099695791 shutdown_connections 2024-03-24T10:45:02.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.155+0000 7ff4becdf640 1 --2- 172.21.15.5:0/3099695791 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff49c06b860 0x7ff49c06dd20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.155+0000 7ff4becdf640 1 --2- 172.21.15.5:0/3099695791 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4b81547a0 0x7ff4b8147d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.155+0000 7ff4becdf640 1 --2- 172.21.15.5:0/3099695791 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4b80d0660 0x7ff4b8159d00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.155+0000 7ff4becdf640 1 --2- 172.21.15.5:0/3099695791 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4b80b0580 0x7ff4b815cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.155+0000 7ff4becdf640 1 -- 172.21.15.5:0/3099695791 >> 172.21.15.5:0/3099695791 conn(0x7ff4b8061f70 msgr2=0x7ff4b80cc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:02.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.155+0000 7ff4becdf640 1 -- 172.21.15.5:0/3099695791 shutdown_connections 2024-03-24T10:45:02.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.156+0000 7ff4becdf640 1 -- 172.21.15.5:0/3099695791 wait complete. 2024-03-24T10:45:02.166 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:45:02.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.272+0000 7fa31c2d4640 1 Processor -- start 2024-03-24T10:45:02.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.272+0000 7fa31c2d4640 1 -- start start 2024-03-24T10:45:02.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.273+0000 7fa31c2d4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3140c92f0 0x7fa3140c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.273+0000 7fa31c2d4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa3140c8c30 0x7fa31415b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.273+0000 7fa31c2d4640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa3140c9e30 0x7fa31415bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.273+0000 7fa31c2d4640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fa3140c9c10 con 0x7fa3140c9e30 2024-03-24T10:45:02.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.273+0000 7fa31c2d4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fa3140c8850 con 0x7fa3140c8c30 2024-03-24T10:45:02.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.273+0000 7fa31c2d4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fa314152560 con 0x7fa3140c92f0 2024-03-24T10:45:02.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.274+0000 7fa31a070640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3140c92f0 0x7fa3140c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:02.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.274+0000 7fa31986f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa3140c8c30 0x7fa31415b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:02.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.274+0000 7fa31a070640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3140c92f0 0x7fa3140c96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:46264/0 (socket says 172.21.15.5:46264) 2024-03-24T10:45:02.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.274+0000 7fa31a070640 1 -- 172.21.15.5:0/811666982 learned_addr learned my addr 172.21.15.5:0/811666982 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:02.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.276+0000 7fa31a070640 1 -- 172.21.15.5:0/811666982 >> v2:172.21.15.57:3300/0 conn(0x7fa3140c8c30 msgr2=0x7fa31415b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:02.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.276+0000 7fa31a070640 1 --2- 172.21.15.5:0/811666982 >> v2:172.21.15.57:3300/0 conn(0x7fa3140c8c30 0x7fa31415b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.276+0000 7fa31a070640 1 -- 172.21.15.5:0/811666982 >> v2:172.21.15.5:3300/0 conn(0x7fa3140c9e30 msgr2=0x7fa31415bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:02.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.276+0000 7fa31a070640 1 --2- 172.21.15.5:0/811666982 >> v2:172.21.15.5:3300/0 conn(0x7fa3140c9e30 0x7fa31415bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.276+0000 7fa31a070640 1 -- 172.21.15.5:0/811666982 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3140b3b80 con 0x7fa3140c92f0 2024-03-24T10:45:02.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.278+0000 7fa31a070640 1 --2- 172.21.15.5:0/811666982 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3140c92f0 0x7fa3140c96d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa3140cb600 tx=0x7fa31002b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42734cf1b5b51d11 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:02.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.282+0000 7fa31906e640 1 -- 172.21.15.5:0/811666982 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa310030860 con 0x7fa3140c92f0 2024-03-24T10:45:02.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.282+0000 7fa31906e640 1 -- 172.21.15.5:0/811666982 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3100309c0 con 0x7fa3140c92f0 2024-03-24T10:45:02.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.282+0000 7fa31906e640 1 -- 172.21.15.5:0/811666982 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa310030d50 con 0x7fa3140c92f0 2024-03-24T10:45:02.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.282+0000 7fa31c2d4640 1 -- 172.21.15.5:0/811666982 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3140c92f0 msgr2=0x7fa3140c96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:02.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.282+0000 7fa31c2d4640 1 --2- 172.21.15.5:0/811666982 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3140c92f0 0x7fa3140c96d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa3140cb600 tx=0x7fa31002b9a0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.282+0000 7fa31c2d4640 1 -- 172.21.15.5:0/811666982 shutdown_connections 2024-03-24T10:45:02.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.282+0000 7fa31c2d4640 1 --2- 172.21.15.5:0/811666982 >> v2:172.21.15.5:3300/0 conn(0x7fa3140c9e30 0x7fa31415bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.282+0000 7fa31c2d4640 1 --2- 172.21.15.5:0/811666982 >> v2:172.21.15.57:3300/0 conn(0x7fa3140c8c30 0x7fa31415b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.282+0000 7fa31c2d4640 1 --2- 172.21.15.5:0/811666982 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3140c92f0 0x7fa3140c96d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.282+0000 7fa31c2d4640 1 -- 172.21.15.5:0/811666982 >> 172.21.15.5:0/811666982 conn(0x7fa31406f820 msgr2=0x7fa314061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:02.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.282+0000 7fa31c2d4640 1 -- 172.21.15.5:0/811666982 shutdown_connections 2024-03-24T10:45:02.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.283+0000 7fa31c2d4640 1 -- 172.21.15.5:0/811666982 wait complete. 2024-03-24T10:45:02.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.284+0000 7fa31c2d4640 1 Processor -- start 2024-03-24T10:45:02.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.284+0000 7fa31c2d4640 1 -- start start 2024-03-24T10:45:02.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.284+0000 7fa31c2d4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3140c8c30 0x7fa3140ca580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.284+0000 7fa31c2d4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3140c92f0 0x7fa3141eeea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.284+0000 7fa31c2d4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3140c9e30 0x7fa3141ef3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.284+0000 7fa31c2d4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa3141f0d10 con 0x7fa3140c8c30 2024-03-24T10:45:02.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.284+0000 7fa31c2d4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa3141f0ff0 con 0x7fa3140c92f0 2024-03-24T10:45:02.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.284+0000 7fa31c2d4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fa3141f12d0 con 0x7fa3140c9e30 2024-03-24T10:45:02.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.285+0000 7fa31a871640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3140c9e30 0x7fa3141ef3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:02.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.285+0000 7fa31986f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3140c92f0 0x7fa3141eeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:02.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.285+0000 7fa31a871640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3140c9e30 0x7fa3141ef3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:46266/0 (socket says 172.21.15.5:46266) 2024-03-24T10:45:02.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.285+0000 7fa31a871640 1 -- 172.21.15.5:0/4113608660 learned_addr learned my addr 172.21.15.5:0/4113608660 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:02.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.288+0000 7fa31a871640 1 -- 172.21.15.5:0/4113608660 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3140c92f0 msgr2=0x7fa3141eeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:02.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.288+0000 7fa31a871640 1 --2- 172.21.15.5:0/4113608660 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3140c92f0 0x7fa3141eeea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.288+0000 7fa31a871640 1 -- 172.21.15.5:0/4113608660 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3140c8c30 msgr2=0x7fa3140ca580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:02.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.288+0000 7fa31a871640 1 --2- 172.21.15.5:0/4113608660 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3140c8c30 0x7fa3140ca580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.288+0000 7fa31a871640 1 -- 172.21.15.5:0/4113608660 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3100094d0 con 0x7fa3140c9e30 2024-03-24T10:45:02.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.288+0000 7fa31986f640 1 --2- 172.21.15.5:0/4113608660 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3140c92f0 0x7fa3141eeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:02.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.291+0000 7fa31a871640 1 --2- 172.21.15.5:0/4113608660 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3140c9e30 0x7fa3141ef3e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa30c00b9e0 tx=0x7fa30c00bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:02.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.294+0000 7fa30affd640 1 -- 172.21.15.5:0/4113608660 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30c00cb20 con 0x7fa3140c9e30 2024-03-24T10:45:02.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.294+0000 7fa30affd640 1 -- 172.21.15.5:0/4113608660 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa30c00cc80 con 0x7fa3140c9e30 2024-03-24T10:45:02.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.294+0000 7fa31c2d4640 1 -- 172.21.15.5:0/4113608660 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3141f15b0 con 0x7fa3140c9e30 2024-03-24T10:45:02.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.294+0000 7fa30affd640 1 -- 172.21.15.5:0/4113608660 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30c014680 con 0x7fa3140c9e30 2024-03-24T10:45:02.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.294+0000 7fa31c2d4640 1 -- 172.21.15.5:0/4113608660 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3140caac0 con 0x7fa3140c9e30 2024-03-24T10:45:02.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.296+0000 7fa31c2d4640 1 -- 172.21.15.5:0/4113608660 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3140681d0 con 0x7fa3140c9e30 2024-03-24T10:45:02.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.319+0000 7fa30affd640 1 -- 172.21.15.5:0/4113608660 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa30c01d3b0 con 0x7fa3140c9e30 2024-03-24T10:45:02.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.320+0000 7fa30affd640 1 --2- 172.21.15.5:0/4113608660 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa2f806b840 0x7fa2f806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:02.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.320+0000 7fa31a070640 1 --2- 172.21.15.5:0/4113608660 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa2f806b840 0x7fa2f806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:02.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.320+0000 7fa31a070640 1 --2- 172.21.15.5:0/4113608660 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa2f806b840 0x7fa2f806dd00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa310036000 tx=0x7fa31003b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:02.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.571+0000 7fa30affd640 1 -- 172.21.15.5:0/4113608660 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa30c00ce30 con 0x7fa3140c9e30 2024-03-24T10:45:02.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.594+0000 7fa30affd640 1 -- 172.21.15.5:0/4113608660 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa30c054f70 con 0x7fa3140c9e30 2024-03-24T10:45:02.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.780+0000 7fa31c2d4640 1 -- 172.21.15.5:0/4113608660 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa314156630 con 0x7fa3140c9e30 2024-03-24T10:45:02.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.901+0000 7fa30affd640 1 -- 172.21.15.5:0/4113608660 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16498 (secure 0 0 0) 0x7fa30c019020 con 0x7fa3140c9e30 2024-03-24T10:45:02.902 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:45:02.902 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":20,"fsid":"7d9848db-2fe7-4d95-a6e2-b04f4f639ebc","created":"2024-03-24T10:44:30.435176+0000","modified":"2024-03-24T10:45:01.374174+0000","last_up_change":"2024-03-24T10:44:44.634047+0000","last_in_change":"2024-03-24T10:44:35.986391+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:44:45.829320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:44:58.435543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:45:00.129313+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50f590b3-f5ec-4e1b-ac46-25cab307a27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6825","nonce":1991336051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6827","nonce":1991336051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6831","nonce":1991336051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6829","nonce":1991336051}]},"public_addr":"172.21.15.5:6825/1991336051","cluster_addr":"172.21.15.5:6827/1991336051","heartbeat_back_addr":"172.21.15.5:6831/1991336051","heartbeat_front_addr":"172.21.15.5:6829/1991336051","state":["exists","up"]},{"osd":1,"uuid":"ccba9e93-c704-472b-a832-8d4eeb456c47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6801","nonce":2140667501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6803","nonce":2140667501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6807","nonce":2140667501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6805","nonce":2140667501}]},"public_addr":"172.21.15.5:6801/2140667501","cluster_addr":"172.21.15.5:6803/2140667501","heartbeat_back_addr":"172.21.15.5:6807/2140667501","heartbeat_front_addr":"172.21.15.5:6805/2140667501","state":["exists","up"]},{"osd":2,"uuid":"e614c5b1-6504-452e-82e0-95c7ca271737","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6817","nonce":90414679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6819","nonce":90414679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6823","nonce":90414679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6821","nonce":90414679}]},"public_addr":"172.21.15.5:6817/90414679","cluster_addr":"172.21.15.5:6819/90414679","heartbeat_back_addr":"172.21.15.5:6823/90414679","heartbeat_front_addr":"172.21.15.5:6821/90414679","state":["exists","up"]},{"osd":3,"uuid":"c60154d0-85e0-46a1-8273-4863bd8ad2dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6809","nonce":4212473119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6811","nonce":4212473119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6815","nonce":4212473119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6813","nonce":4212473119}]},"public_addr":"172.21.15.5:6809/4212473119","cluster_addr":"172.21.15.5:6811/4212473119","heartbeat_back_addr":"172.21.15.5:6815/4212473119","heartbeat_front_addr":"172.21.15.5:6813/4212473119","state":["exists","up"]},{"osd":4,"uuid":"21d74109-da3a-4fc0-b2d8-93e9ae3ed30c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6817","nonce":2193986505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6819","nonce":2193986505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6823","nonce":2193986505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6821","nonce":2193986505}]},"public_addr":"172.21.15.57:6817/2193986505","cluster_addr":"172.21.15.57:6819/2193986505","heartbeat_back_addr":"172.21.15.57:6823/2193986505","heartbeat_front_addr":"172.21.15.57:6821/2193986505","state":["exists","up"]},{"osd":5,"uuid":"5cf19cc7-215f-4d69-a30a-80543c0293e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6809","nonce":3044987466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6811","nonce":3044987466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6815","nonce":3044987466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6813","nonce":3044987466}]},"public_addr":"172.21.15.57:6809/3044987466","cluster_addr":"172.21.15.57:6811/3044987466","heartbeat_back_addr":"172.21.15.57:6815/3044987466","heartbeat_front_addr":"172.21.15.57:6813/3044987466","state":["exists","up"]},{"osd":6,"uuid":"a5e76c69-c3fc-47b2-8f92-e560cf503e45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6801","nonce":4290897445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6803","nonce":4290897445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6807","nonce":4290897445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6805","nonce":4290897445}]},"public_addr":"172.21.15.57:6801/4290897445","cluster_addr":"172.21.15.57:6803/4290897445","heartbeat_back_addr":"172.21.15.57:6807/4290897445","heartbeat_front_addr":"172.21.15.57:6805/4290897445","state":["exists","up"]},{"osd":7,"uuid":"47d0b873-e3ac-4f40-a372-681436e45d0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6825","nonce":212605698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6827","nonce":212605698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6831","nonce":212605698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6829","nonce":212605698}]},"public_addr":"172.21.15.57:6825/212605698","cluster_addr":"172.21.15.57:6827/212605698","heartbeat_back_addr":"172.21.15.57:6831/212605698","heartbeat_front_addr":"172.21.15.57:6829/212605698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.211237+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.192822+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.273429+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.199212+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.277082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.218768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.190915+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.225294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:45:02.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.903+0000 7fa31c2d4640 1 -- 172.21.15.5:0/4113608660 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa2f806b840 msgr2=0x7fa2f806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:02.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.903+0000 7fa31c2d4640 1 --2- 172.21.15.5:0/4113608660 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa2f806b840 0x7fa2f806dd00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa310036000 tx=0x7fa31003b040 comp rx=0 tx=0).stop 2024-03-24T10:45:02.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.903+0000 7fa31c2d4640 1 -- 172.21.15.5:0/4113608660 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3140c9e30 msgr2=0x7fa3141ef3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:02.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.903+0000 7fa31c2d4640 1 --2- 172.21.15.5:0/4113608660 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3140c9e30 0x7fa3141ef3e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa30c00b9e0 tx=0x7fa30c00bea0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.903+0000 7fa31c2d4640 1 -- 172.21.15.5:0/4113608660 shutdown_connections 2024-03-24T10:45:02.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.903+0000 7fa31c2d4640 1 --2- 172.21.15.5:0/4113608660 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa2f806b840 0x7fa2f806dd00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.903+0000 7fa31c2d4640 1 --2- 172.21.15.5:0/4113608660 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa3140c9e30 0x7fa3141ef3e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.903+0000 7fa31c2d4640 1 --2- 172.21.15.5:0/4113608660 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa3140c92f0 0x7fa3141eeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.903+0000 7fa31c2d4640 1 --2- 172.21.15.5:0/4113608660 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa3140c8c30 0x7fa3140ca580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:02.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.903+0000 7fa31c2d4640 1 -- 172.21.15.5:0/4113608660 >> 172.21.15.5:0/4113608660 conn(0x7fa31406f820 msgr2=0x7fa3141579b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:02.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.903+0000 7fa31c2d4640 1 -- 172.21.15.5:0/4113608660 shutdown_connections 2024-03-24T10:45:02.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:02.903+0000 7fa31c2d4640 1 -- 172.21.15.5:0/4113608660 wait complete. 2024-03-24T10:45:02.916 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:45:03.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.022+0000 7fe508b6c640 1 Processor -- start 2024-03-24T10:45:03.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.022+0000 7fe508b6c640 1 -- start start 2024-03-24T10:45:03.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.023+0000 7fe508b6c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe5041543a0 0x7fe50415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.023+0000 7fe508b6c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe50406bf50 0x7fe50406c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.023+0000 7fe508b6c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5040cb2f0 con 0x7fe50406d4f0 2024-03-24T10:45:03.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.023+0000 7fe508b6c640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fe5040c6750 con 0x7fe50406bf50 2024-03-24T10:45:03.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.023+0000 7fe508b6c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe5040cc380 con 0x7fe5041543a0 2024-03-24T10:45:03.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.024+0000 7fe501d9b640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fe50406d4f0 0x7fe5040cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:34270/0 (socket says 172.21.15.5:34270) 2024-03-24T10:45:03.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.024+0000 7fe501d9b640 1 -- 172.21.15.5:0/1931788271 learned_addr learned my addr 172.21.15.5:0/1931788271 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:03.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.024+0000 7fe50259c640 1 --2- 172.21.15.5:0/1931788271 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe5041543a0 0x7fe50415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.026+0000 7fe50259c640 1 -- 172.21.15.5:0/1931788271 >> v2:172.21.15.5:3300/0 conn(0x7fe50406bf50 msgr2=0x7fe50406c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:03.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.026+0000 7fe50259c640 1 --2- 172.21.15.5:0/1931788271 >> v2:172.21.15.5:3300/0 conn(0x7fe50406bf50 0x7fe50406c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.026+0000 7fe50259c640 1 -- 172.21.15.5:0/1931788271 >> v1:172.21.15.57:6789/0 conn(0x7fe50406d4f0 legacy=0x7fe5040cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:03.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.026+0000 7fe50259c640 1 -- 172.21.15.5:0/1931788271 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5040ca7e0 con 0x7fe5041543a0 2024-03-24T10:45:03.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.029+0000 7fe50259c640 1 --2- 172.21.15.5:0/1931788271 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe5041543a0 0x7fe50415ac00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc0097a0 tx=0x7fe4fc02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66e73d5e134088f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:03.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.035+0000 7fe50159a640 1 -- 172.21.15.5:0/1931788271 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc030610 con 0x7fe5041543a0 2024-03-24T10:45:03.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.035+0000 7fe50159a640 1 -- 172.21.15.5:0/1931788271 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4fc030770 con 0x7fe5041543a0 2024-03-24T10:45:03.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.035+0000 7fe50159a640 1 -- 172.21.15.5:0/1931788271 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc030b00 con 0x7fe5041543a0 2024-03-24T10:45:03.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.035+0000 7fe508b6c640 1 -- 172.21.15.5:0/1931788271 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe5041543a0 msgr2=0x7fe50415ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:03.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.035+0000 7fe508b6c640 1 --2- 172.21.15.5:0/1931788271 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe5041543a0 0x7fe50415ac00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc0097a0 tx=0x7fe4fc02b960 comp rx=0 tx=0).stop 2024-03-24T10:45:03.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.035+0000 7fe508b6c640 1 -- 172.21.15.5:0/1931788271 shutdown_connections 2024-03-24T10:45:03.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.035+0000 7fe508b6c640 1 --2- 172.21.15.5:0/1931788271 >> v2:172.21.15.5:3300/0 conn(0x7fe50406bf50 0x7fe50406c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.036+0000 7fe508b6c640 1 --2- 172.21.15.5:0/1931788271 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe5041543a0 0x7fe50415ac00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.036+0000 7fe508b6c640 1 -- 172.21.15.5:0/1931788271 >> 172.21.15.5:0/1931788271 conn(0x7fe504072ef0 msgr2=0x7fe504061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:03.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.036+0000 7fe508b6c640 1 -- 172.21.15.5:0/1931788271 shutdown_connections 2024-03-24T10:45:03.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.036+0000 7fe508b6c640 1 -- 172.21.15.5:0/1931788271 wait complete. 2024-03-24T10:45:03.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.037+0000 7fe508b6c640 1 Processor -- start 2024-03-24T10:45:03.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.037+0000 7fe508b6c640 1 -- start start 2024-03-24T10:45:03.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.038+0000 7fe508b6c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe50406bf50 0x7fe50415d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.038+0000 7fe508b6c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe50406d4f0 0x7fe50414a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.038+0000 7fe508b6c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5041543a0 0x7fe5041f01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.038+0000 7fe508b6c640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fe50415cd30 con 0x7fe50406bf50 2024-03-24T10:45:03.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.038+0000 7fe508b6c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe5040c97b0 con 0x7fe5041543a0 2024-03-24T10:45:03.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.038+0000 7fe508b6c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe5040c75e0 con 0x7fe50406d4f0 2024-03-24T10:45:03.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.040+0000 7fe501d9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe50406d4f0 0x7fe50414a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.041+0000 7fe501d9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe50406d4f0 0x7fe50414a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:39206/0 (socket says 172.21.15.5:39206) 2024-03-24T10:45:03.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.041+0000 7fe501d9b640 1 -- 172.21.15.5:0/881178092 learned_addr learned my addr 172.21.15.5:0/881178092 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:03.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.044+0000 7fe501d9b640 1 -- 172.21.15.5:0/881178092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5041543a0 msgr2=0x7fe5041f01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:03.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.044+0000 7fe501d9b640 1 --2- 172.21.15.5:0/881178092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5041543a0 0x7fe5041f01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.044+0000 7fe501d9b640 1 -- 172.21.15.5:0/881178092 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe50406bf50 msgr2=0x7fe50415d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:03.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.044+0000 7fe501d9b640 1 --2- 172.21.15.5:0/881178092 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe50406bf50 0x7fe50415d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.044+0000 7fe501d9b640 1 -- 172.21.15.5:0/881178092 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4fc009400 con 0x7fe50406d4f0 2024-03-24T10:45:03.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.047+0000 7fe501d9b640 1 --2- 172.21.15.5:0/881178092 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe50406d4f0 0x7fe50414a640 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec00c400 tx=0x7fe4ec00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:03.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.099+0000 7fe4faffd640 1 -- 172.21.15.5:0/881178092 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec005ce0 con 0x7fe50406d4f0 2024-03-24T10:45:03.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.099+0000 7fe4faffd640 1 -- 172.21.15.5:0/881178092 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4ec005e40 con 0x7fe50406d4f0 2024-03-24T10:45:03.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.099+0000 7fe508b6c640 1 -- 172.21.15.5:0/881178092 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe50414ab80 con 0x7fe50406d4f0 2024-03-24T10:45:03.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.099+0000 7fe4faffd640 1 -- 172.21.15.5:0/881178092 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec0072d0 con 0x7fe50406d4f0 2024-03-24T10:45:03.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.099+0000 7fe508b6c640 1 -- 172.21.15.5:0/881178092 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe50415e250 con 0x7fe50406d4f0 2024-03-24T10:45:03.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.100+0000 7fe508b6c640 1 -- 172.21.15.5:0/881178092 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5040cb110 con 0x7fe50406d4f0 2024-03-24T10:45:03.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.372+0000 7fe4faffd640 1 -- 172.21.15.5:0/881178092 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe4ec0074f0 con 0x7fe50406d4f0 2024-03-24T10:45:03.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.373+0000 7fe4faffd640 1 --2- 172.21.15.5:0/881178092 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe4e406b860 0x7fe4e406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.373+0000 7fe50259c640 1 --2- 172.21.15.5:0/881178092 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe4e406b860 0x7fe4e406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.373+0000 7fe50259c640 1 --2- 172.21.15.5:0/881178092 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe4e406b860 0x7fe4e406dd20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc036000 tx=0x7fe4fc03b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:03.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.375+0000 7fe4faffd640 1 -- 172.21.15.5:0/881178092 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe4ec058af0 con 0x7fe50406d4f0 2024-03-24T10:45:03.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.386+0000 7fe4faffd640 1 -- 172.21.15.5:0/881178092 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe4ec054690 con 0x7fe50406d4f0 2024-03-24T10:45:03.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.567+0000 7fe508b6c640 1 -- 172.21.15.5:0/881178092 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe5040c98f0 con 0x7fe50406d4f0 2024-03-24T10:45:03.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.605+0000 7fe4faffd640 1 -- 172.21.15.5:0/881178092 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16485 (secure 0 0 0) 0x7fe5040c98f0 con 0x7fe50406d4f0 2024-03-24T10:45:03.606 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:45:03.606 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":21,"fsid":"7d9848db-2fe7-4d95-a6e2-b04f4f639ebc","created":"2024-03-24T10:44:30.435176+0000","modified":"2024-03-24T10:45:02.781257+0000","last_up_change":"2024-03-24T10:44:44.634047+0000","last_in_change":"2024-03-24T10:44:35.986391+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:44:45.829320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:44:58.435543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:45:00.129313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50f590b3-f5ec-4e1b-ac46-25cab307a27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6825","nonce":1991336051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6827","nonce":1991336051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6831","nonce":1991336051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6829","nonce":1991336051}]},"public_addr":"172.21.15.5:6825/1991336051","cluster_addr":"172.21.15.5:6827/1991336051","heartbeat_back_addr":"172.21.15.5:6831/1991336051","heartbeat_front_addr":"172.21.15.5:6829/1991336051","state":["exists","up"]},{"osd":1,"uuid":"ccba9e93-c704-472b-a832-8d4eeb456c47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6801","nonce":2140667501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6803","nonce":2140667501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6807","nonce":2140667501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6805","nonce":2140667501}]},"public_addr":"172.21.15.5:6801/2140667501","cluster_addr":"172.21.15.5:6803/2140667501","heartbeat_back_addr":"172.21.15.5:6807/2140667501","heartbeat_front_addr":"172.21.15.5:6805/2140667501","state":["exists","up"]},{"osd":2,"uuid":"e614c5b1-6504-452e-82e0-95c7ca271737","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6817","nonce":90414679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6819","nonce":90414679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6823","nonce":90414679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6821","nonce":90414679}]},"public_addr":"172.21.15.5:6817/90414679","cluster_addr":"172.21.15.5:6819/90414679","heartbeat_back_addr":"172.21.15.5:6823/90414679","heartbeat_front_addr":"172.21.15.5:6821/90414679","state":["exists","up"]},{"osd":3,"uuid":"c60154d0-85e0-46a1-8273-4863bd8ad2dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6809","nonce":4212473119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6811","nonce":4212473119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6815","nonce":4212473119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6813","nonce":4212473119}]},"public_addr":"172.21.15.5:6809/4212473119","cluster_addr":"172.21.15.5:6811/4212473119","heartbeat_back_addr":"172.21.15.5:6815/4212473119","heartbeat_front_addr":"172.21.15.5:6813/4212473119","state":["exists","up"]},{"osd":4,"uuid":"21d74109-da3a-4fc0-b2d8-93e9ae3ed30c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6817","nonce":2193986505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6819","nonce":2193986505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6823","nonce":2193986505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6821","nonce":2193986505}]},"public_addr":"172.21.15.57:6817/2193986505","cluster_addr":"172.21.15.57:6819/2193986505","heartbeat_back_addr":"172.21.15.57:6823/2193986505","heartbeat_front_addr":"172.21.15.57:6821/2193986505","state":["exists","up"]},{"osd":5,"uuid":"5cf19cc7-215f-4d69-a30a-80543c0293e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6809","nonce":3044987466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6811","nonce":3044987466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6815","nonce":3044987466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6813","nonce":3044987466}]},"public_addr":"172.21.15.57:6809/3044987466","cluster_addr":"172.21.15.57:6811/3044987466","heartbeat_back_addr":"172.21.15.57:6815/3044987466","heartbeat_front_addr":"172.21.15.57:6813/3044987466","state":["exists","up"]},{"osd":6,"uuid":"a5e76c69-c3fc-47b2-8f92-e560cf503e45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6801","nonce":4290897445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6803","nonce":4290897445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6807","nonce":4290897445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6805","nonce":4290897445}]},"public_addr":"172.21.15.57:6801/4290897445","cluster_addr":"172.21.15.57:6803/4290897445","heartbeat_back_addr":"172.21.15.57:6807/4290897445","heartbeat_front_addr":"172.21.15.57:6805/4290897445","state":["exists","up"]},{"osd":7,"uuid":"47d0b873-e3ac-4f40-a372-681436e45d0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6825","nonce":212605698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6827","nonce":212605698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6831","nonce":212605698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6829","nonce":212605698}]},"public_addr":"172.21.15.57:6825/212605698","cluster_addr":"172.21.15.57:6827/212605698","heartbeat_back_addr":"172.21.15.57:6831/212605698","heartbeat_front_addr":"172.21.15.57:6829/212605698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.211237+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.192822+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.273429+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.199212+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.277082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.218768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.190915+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.225294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:45:03.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.607+0000 7fe508b6c640 1 -- 172.21.15.5:0/881178092 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe4e406b860 msgr2=0x7fe4e406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:03.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.607+0000 7fe508b6c640 1 --2- 172.21.15.5:0/881178092 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe4e406b860 0x7fe4e406dd20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc036000 tx=0x7fe4fc03b040 comp rx=0 tx=0).stop 2024-03-24T10:45:03.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.607+0000 7fe508b6c640 1 -- 172.21.15.5:0/881178092 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe50406d4f0 msgr2=0x7fe50414a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:03.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.607+0000 7fe508b6c640 1 --2- 172.21.15.5:0/881178092 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe50406d4f0 0x7fe50414a640 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec00c400 tx=0x7fe4ec00c8c0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.607+0000 7fe508b6c640 1 -- 172.21.15.5:0/881178092 shutdown_connections 2024-03-24T10:45:03.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.607+0000 7fe508b6c640 1 --2- 172.21.15.5:0/881178092 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe4e406b860 0x7fe4e406dd20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.607+0000 7fe508b6c640 1 --2- 172.21.15.5:0/881178092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe5041543a0 0x7fe5041f01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.607+0000 7fe508b6c640 1 --2- 172.21.15.5:0/881178092 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe50406d4f0 0x7fe50414a640 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.607+0000 7fe508b6c640 1 --2- 172.21.15.5:0/881178092 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe50406bf50 0x7fe50415d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.607+0000 7fe508b6c640 1 -- 172.21.15.5:0/881178092 >> 172.21.15.5:0/881178092 conn(0x7fe504072ef0 msgr2=0x7fe5040cf1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:03.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.607+0000 7fe508b6c640 1 -- 172.21.15.5:0/881178092 shutdown_connections 2024-03-24T10:45:03.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.607+0000 7fe508b6c640 1 -- 172.21.15.5:0/881178092 wait complete. 2024-03-24T10:45:03.619 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T10:45:03.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.723+0000 7f4b29e9a640 1 Processor -- start 2024-03-24T10:45:03.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.723+0000 7f4b29e9a640 1 -- start start 2024-03-24T10:45:03.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.724+0000 7f4b29e9a640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4b24154610 0x7f4b240ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.724+0000 7f4b29e9a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4b240cb070 0x7f4b2406f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.724+0000 7f4b29e9a640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b2415a630 con 0x7f4b2415d8d0 2024-03-24T10:45:03.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.724+0000 7f4b29e9a640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f4b2406db00 con 0x7f4b24154610 2024-03-24T10:45:03.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.724+0000 7f4b29e9a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4b240cebe0 con 0x7f4b240cb070 2024-03-24T10:45:03.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.724+0000 7f4b22ffd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f4b2415d8d0 0x7f4b2415aac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59612/0 (socket says 172.21.15.5:59612) 2024-03-24T10:45:03.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.724+0000 7f4b22ffd640 1 -- 172.21.15.5:0/4215460511 learned_addr learned my addr 172.21.15.5:0/4215460511 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:03.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.725+0000 7f4b237fe640 1 --2- 172.21.15.5:0/4215460511 >> v2:172.21.15.57:3300/0 conn(0x7f4b24154610 0x7f4b240ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.725+0000 7f4b23fff640 1 --2- 172.21.15.5:0/4215460511 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4b240cb070 0x7f4b2406f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.727+0000 7f4b227fc640 1 -- 172.21.15.5:0/4215460511 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1805041701 0 0) 0x7f4b2415a630 con 0x7f4b2415d8d0 2024-03-24T10:45:03.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.727+0000 7f4b227fc640 1 -- 172.21.15.5:0/4215460511 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b0c003520 con 0x7f4b2415d8d0 2024-03-24T10:45:03.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.727+0000 7f4b237fe640 1 -- 172.21.15.5:0/4215460511 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4b240cb070 msgr2=0x7f4b2406f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:03.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.727+0000 7f4b237fe640 1 --2- 172.21.15.5:0/4215460511 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4b240cb070 0x7f4b2406f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.727+0000 7f4b237fe640 1 -- 172.21.15.5:0/4215460511 >> v1:172.21.15.5:6789/0 conn(0x7f4b2415d8d0 legacy=0x7f4b2415aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:03.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.727+0000 7f4b237fe640 1 -- 172.21.15.5:0/4215460511 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b241549f0 con 0x7f4b24154610 2024-03-24T10:45:03.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.731+0000 7f4b237fe640 1 --2- 172.21.15.5:0/4215460511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b24154610 0x7f4b240ce6a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4b140097a0 tx=0x7f4b1402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c0103efdd50bad1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:03.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.735+0000 7f4b227fc640 1 -- 172.21.15.5:0/4215460511 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1402e070 con 0x7f4b24154610 2024-03-24T10:45:03.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.735+0000 7f4b227fc640 1 -- 172.21.15.5:0/4215460511 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b140315a0 con 0x7f4b24154610 2024-03-24T10:45:03.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.735+0000 7f4b227fc640 1 -- 172.21.15.5:0/4215460511 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b14031930 con 0x7f4b24154610 2024-03-24T10:45:03.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.736+0000 7f4b29e9a640 1 -- 172.21.15.5:0/4215460511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b24154610 msgr2=0x7f4b240ce6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:03.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.736+0000 7f4b29e9a640 1 --2- 172.21.15.5:0/4215460511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b24154610 0x7f4b240ce6a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4b140097a0 tx=0x7f4b1402b840 comp rx=0 tx=0).stop 2024-03-24T10:45:03.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.736+0000 7f4b29e9a640 1 -- 172.21.15.5:0/4215460511 shutdown_connections 2024-03-24T10:45:03.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.736+0000 7f4b29e9a640 1 --2- 172.21.15.5:0/4215460511 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4b240cb070 0x7f4b2406f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.736+0000 7f4b29e9a640 1 --2- 172.21.15.5:0/4215460511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b24154610 0x7f4b240ce6a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.736+0000 7f4b29e9a640 1 -- 172.21.15.5:0/4215460511 >> 172.21.15.5:0/4215460511 conn(0x7f4b24070c90 msgr2=0x7f4b24071580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:03.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.736+0000 7f4b29e9a640 1 -- 172.21.15.5:0/4215460511 shutdown_connections 2024-03-24T10:45:03.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.736+0000 7f4b29e9a640 1 -- 172.21.15.5:0/4215460511 wait complete. 2024-03-24T10:45:03.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.737+0000 7f4b29e9a640 1 Processor -- start 2024-03-24T10:45:03.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.737+0000 7f4b29e9a640 1 -- start start 2024-03-24T10:45:03.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.738+0000 7f4b29e9a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4b240cb070 0x7f4b241f0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.738+0000 7f4b29e9a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b24154610 0x7f4b240adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.738+0000 7f4b29e9a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b2415d8d0 0x7f4b240b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:03.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.738+0000 7f4b29e9a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f4b240b1960 con 0x7f4b24154610 2024-03-24T10:45:03.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.738+0000 7f4b29e9a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4b241521d0 con 0x7f4b2415d8d0 2024-03-24T10:45:03.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.738+0000 7f4b29e9a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4b241f0fb0 con 0x7f4b240cb070 2024-03-24T10:45:03.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.739+0000 7f4b22ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b24154610 0x7f4b240adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.739+0000 7f4b23fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b2415d8d0 0x7f4b240b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.739+0000 7f4b237fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4b240cb070 0x7f4b241f0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:03.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.739+0000 7f4b23fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b2415d8d0 0x7f4b240b1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:34660/0 (socket says 172.21.15.5:34660) 2024-03-24T10:45:03.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.739+0000 7f4b23fff640 1 -- 172.21.15.5:0/3419165742 learned_addr learned my addr 172.21.15.5:0/3419165742 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:03.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.742+0000 7f4b23fff640 1 -- 172.21.15.5:0/3419165742 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4b240cb070 msgr2=0x7f4b241f0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:03.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.742+0000 7f4b23fff640 1 --2- 172.21.15.5:0/3419165742 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4b240cb070 0x7f4b241f0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.742+0000 7f4b23fff640 1 -- 172.21.15.5:0/3419165742 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b24154610 msgr2=0x7f4b240adb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:03.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.742+0000 7f4b23fff640 1 --2- 172.21.15.5:0/3419165742 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b24154610 0x7f4b240adb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:03.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.742+0000 7f4b23fff640 1 -- 172.21.15.5:0/3419165742 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b14009400 con 0x7f4b2415d8d0 2024-03-24T10:45:03.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.742+0000 7f4b237fe640 1 --2- 172.21.15.5:0/3419165742 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4b240cb070 0x7f4b241f0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:45:03.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.745+0000 7f4b23fff640 1 --2- 172.21.15.5:0/3419165742 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b2415d8d0 0x7f4b240b1d80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c009cb0 tx=0x7f4b1c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:03.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.748+0000 7f4b0bfff640 1 -- 172.21.15.5:0/3419165742 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c009b20 con 0x7f4b2415d8d0 2024-03-24T10:45:03.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.748+0000 7f4b0bfff640 1 -- 172.21.15.5:0/3419165742 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b1c00f040 con 0x7f4b2415d8d0 2024-03-24T10:45:03.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.748+0000 7f4b0bfff640 1 -- 172.21.15.5:0/3419165742 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c0146c0 con 0x7f4b2415d8d0 2024-03-24T10:45:03.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.748+0000 7f4b29e9a640 1 -- 172.21.15.5:0/3419165742 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b240b3000 con 0x7f4b2415d8d0 2024-03-24T10:45:03.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.748+0000 7f4b29e9a640 1 -- 172.21.15.5:0/3419165742 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b240b0fc0 con 0x7f4b2415d8d0 2024-03-24T10:45:03.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:03.749+0000 7f4b29e9a640 1 -- 172.21.15.5:0/3419165742 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b240b1aa0 con 0x7f4b2415d8d0 2024-03-24T10:45:04.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:04.016+0000 7f4b0bfff640 1 -- 172.21.15.5:0/3419165742 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4b1c014820 con 0x7f4b2415d8d0 2024-03-24T10:45:04.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:04.017+0000 7f4b0bfff640 1 --2- 172.21.15.5:0/3419165742 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4b1006b860 0x7f4b1006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:04.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:04.017+0000 7f4b237fe640 1 --2- 172.21.15.5:0/3419165742 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4b1006b860 0x7f4b1006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:04.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:04.017+0000 7f4b237fe640 1 --2- 172.21.15.5:0/3419165742 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4b1006b860 0x7f4b1006dd20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4b14036000 tx=0x7f4b1403d040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:04.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:04.022+0000 7f4b0bfff640 1 -- 172.21.15.5:0/3419165742 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4b1c059b50 con 0x7f4b2415d8d0 2024-03-24T10:45:04.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:04.033+0000 7f4b0bfff640 1 -- 172.21.15.5:0/3419165742 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4b1c055df0 con 0x7f4b2415d8d0 2024-03-24T10:45:04.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:04.257+0000 7f4b29e9a640 1 -- 172.21.15.5:0/3419165742 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f4b240b1aa0 con 0x7f4b2415d8d0 2024-03-24T10:45:05.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.428+0000 7f4b0bfff640 1 -- 172.21.15.5:0/3419165742 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7f4b240b1aa0 con 0x7f4b2415d8d0 2024-03-24T10:45:05.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.431+0000 7f4b29e9a640 1 -- 172.21.15.5:0/3419165742 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4b1006b860 msgr2=0x7f4b1006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.431+0000 7f4b29e9a640 1 --2- 172.21.15.5:0/3419165742 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4b1006b860 0x7f4b1006dd20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4b14036000 tx=0x7f4b1403d040 comp rx=0 tx=0).stop 2024-03-24T10:45:05.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.431+0000 7f4b29e9a640 1 -- 172.21.15.5:0/3419165742 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b2415d8d0 msgr2=0x7f4b240b1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.431+0000 7f4b29e9a640 1 --2- 172.21.15.5:0/3419165742 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b2415d8d0 0x7f4b240b1d80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c009cb0 tx=0x7f4b1c00c490 comp rx=0 tx=0).stop 2024-03-24T10:45:05.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.431+0000 7f4b29e9a640 1 -- 172.21.15.5:0/3419165742 shutdown_connections 2024-03-24T10:45:05.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.431+0000 7f4b29e9a640 1 --2- 172.21.15.5:0/3419165742 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4b1006b860 0x7f4b1006dd20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.431+0000 7f4b29e9a640 1 --2- 172.21.15.5:0/3419165742 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4b24154610 0x7f4b240adb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.431+0000 7f4b29e9a640 1 --2- 172.21.15.5:0/3419165742 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4b2415d8d0 0x7f4b240b1d80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.431+0000 7f4b29e9a640 1 --2- 172.21.15.5:0/3419165742 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4b240cb070 0x7f4b241f0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.431+0000 7f4b29e9a640 1 -- 172.21.15.5:0/3419165742 >> 172.21.15.5:0/3419165742 conn(0x7f4b24070c90 msgr2=0x7f4b24158830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.431+0000 7f4b29e9a640 1 -- 172.21.15.5:0/3419165742 shutdown_connections 2024-03-24T10:45:05.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.431+0000 7f4b29e9a640 1 -- 172.21.15.5:0/3419165742 wait complete. 2024-03-24T10:45:05.442 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300} 2024-03-24T10:45:05.442 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-24T10:45:05.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.546+0000 7f90c57af640 1 Processor -- start 2024-03-24T10:45:05.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.546+0000 7f90c57af640 1 -- start start 2024-03-24T10:45:05.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.546+0000 7f90c57af640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f90c0150f00 0x7f90c00c5390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.546+0000 7f90c57af640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90c0159ef0 0x7f90c00c7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.546+0000 7f90c57af640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90c015a340 con 0x7f90c0150b20 2024-03-24T10:45:05.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.546+0000 7f90c57af640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f90c00c2bb0 con 0x7f90c0150f00 2024-03-24T10:45:05.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.546+0000 7f90c57af640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f90c00c4f00 con 0x7f90c0159ef0 2024-03-24T10:45:05.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.547+0000 7f90be7fc640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f90c0150f00 0x7f90c00c5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.547+0000 7f90beffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f90c0150b20 0x7f90c00c2e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:34274/0 (socket says 172.21.15.5:34274) 2024-03-24T10:45:05.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.547+0000 7f90beffd640 1 -- 172.21.15.5:0/476745803 learned_addr learned my addr 172.21.15.5:0/476745803 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:05.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.547+0000 7f90bf7fe640 1 --2- 172.21.15.5:0/476745803 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90c0159ef0 0x7f90c00c7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.550+0000 7f90bdffb640 1 -- 172.21.15.5:0/476745803 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3401072247 0 0) 0x7f90b0001110 con 0x7f90c0150b20 2024-03-24T10:45:05.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.550+0000 7f90bdffb640 1 -- 172.21.15.5:0/476745803 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90a8005730 con 0x7f90c0150b20 2024-03-24T10:45:05.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.550+0000 7f90bf7fe640 1 -- 172.21.15.5:0/476745803 >> v2:172.21.15.5:3300/0 conn(0x7f90c0150f00 msgr2=0x7f90c00c5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:05.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.550+0000 7f90bf7fe640 1 --2- 172.21.15.5:0/476745803 >> v2:172.21.15.5:3300/0 conn(0x7f90c0150f00 0x7f90c00c5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.550+0000 7f90bf7fe640 1 -- 172.21.15.5:0/476745803 >> v1:172.21.15.57:6789/0 conn(0x7f90c0150b20 legacy=0x7f90c00c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.550+0000 7f90bf7fe640 1 -- 172.21.15.5:0/476745803 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90c00c2350 con 0x7f90c0159ef0 2024-03-24T10:45:05.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.552+0000 7f90bf7fe640 1 --2- 172.21.15.5:0/476745803 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90c0159ef0 0x7f90c00c7630 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f90b8009c80 tx=0x7f90b802be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db288661967e6f9f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.556+0000 7f90bdffb640 1 -- 172.21.15.5:0/476745803 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b802cbb0 con 0x7f90c0159ef0 2024-03-24T10:45:05.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.556+0000 7f90bdffb640 1 -- 172.21.15.5:0/476745803 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90b802cd10 con 0x7f90c0159ef0 2024-03-24T10:45:05.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.556+0000 7f90bdffb640 1 -- 172.21.15.5:0/476745803 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b8034660 con 0x7f90c0159ef0 2024-03-24T10:45:05.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.557+0000 7f90c57af640 1 -- 172.21.15.5:0/476745803 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90c0159ef0 msgr2=0x7f90c00c7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.557+0000 7f90c57af640 1 --2- 172.21.15.5:0/476745803 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90c0159ef0 0x7f90c00c7630 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f90b8009c80 tx=0x7f90b802be00 comp rx=0 tx=0).stop 2024-03-24T10:45:05.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.557+0000 7f90c57af640 1 -- 172.21.15.5:0/476745803 shutdown_connections 2024-03-24T10:45:05.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.557+0000 7f90c57af640 1 --2- 172.21.15.5:0/476745803 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90c0159ef0 0x7f90c00c7630 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.557+0000 7f90c57af640 1 --2- 172.21.15.5:0/476745803 >> v2:172.21.15.5:3300/0 conn(0x7f90c0150f00 0x7f90c00c5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.557+0000 7f90c57af640 1 -- 172.21.15.5:0/476745803 >> 172.21.15.5:0/476745803 conn(0x7f90c006f180 msgr2=0x7f90c006f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:05.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.557+0000 7f90c57af640 1 -- 172.21.15.5:0/476745803 shutdown_connections 2024-03-24T10:45:05.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.557+0000 7f90c57af640 1 -- 172.21.15.5:0/476745803 wait complete. 2024-03-24T10:45:05.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.558+0000 7f90c57af640 1 Processor -- start 2024-03-24T10:45:05.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.558+0000 7f90c57af640 1 -- start start 2024-03-24T10:45:05.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.559+0000 7f90c57af640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90c0150b20 0x7f90c0072fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.559+0000 7f90c57af640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90c0159ef0 0x7f90c0072860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.559+0000 7f90c57af640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90c00c99b0 0x7f90c01f3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.559+0000 7f90c57af640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f90c0148f10 con 0x7f90c0159ef0 2024-03-24T10:45:05.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.559+0000 7f90c57af640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f90c01efd30 con 0x7f90c0150b20 2024-03-24T10:45:05.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.559+0000 7f90c57af640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f90c0072da0 con 0x7f90c00c99b0 2024-03-24T10:45:05.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.560+0000 7f90be7fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90c0159ef0 0x7f90c0072860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.560+0000 7f90beffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90c0150b20 0x7f90c0072fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.560+0000 7f90bf7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90c00c99b0 0x7f90c01f3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.560+0000 7f90beffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90c0150b20 0x7f90c0072fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:34672/0 (socket says 172.21.15.5:34672) 2024-03-24T10:45:05.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.560+0000 7f90beffd640 1 -- 172.21.15.5:0/602362603 learned_addr learned my addr 172.21.15.5:0/602362603 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:05.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.563+0000 7f90be7fc640 1 -- 172.21.15.5:0/602362603 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90c00c99b0 msgr2=0x7f90c01f3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.563+0000 7f90be7fc640 1 --2- 172.21.15.5:0/602362603 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90c00c99b0 0x7f90c01f3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.563+0000 7f90be7fc640 1 -- 172.21.15.5:0/602362603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90c0150b20 msgr2=0x7f90c0072fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:05.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.563+0000 7f90be7fc640 1 --2- 172.21.15.5:0/602362603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90c0150b20 0x7f90c0072fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:05.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.563+0000 7f90be7fc640 1 -- 172.21.15.5:0/602362603 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b800d040 con 0x7f90c0159ef0 2024-03-24T10:45:05.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.566+0000 7f90be7fc640 1 --2- 172.21.15.5:0/602362603 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90c0159ef0 0x7f90c0072860 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f90b400e880 tx=0x7f90b400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.569+0000 7f90a77fe640 1 -- 172.21.15.5:0/602362603 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b400c940 con 0x7f90c0159ef0 2024-03-24T10:45:05.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.569+0000 7f90a77fe640 1 -- 172.21.15.5:0/602362603 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90b400caa0 con 0x7f90c0159ef0 2024-03-24T10:45:05.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.569+0000 7f90c57af640 1 -- 172.21.15.5:0/602362603 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90c01eff20 con 0x7f90c0159ef0 2024-03-24T10:45:05.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.569+0000 7f90a77fe640 1 -- 172.21.15.5:0/602362603 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b400ce50 con 0x7f90c0159ef0 2024-03-24T10:45:05.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.569+0000 7f90c57af640 1 -- 172.21.15.5:0/602362603 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90c01f04e0 con 0x7f90c0159ef0 2024-03-24T10:45:05.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.570+0000 7f90c57af640 1 -- 172.21.15.5:0/602362603 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90c00681d0 con 0x7f90c0159ef0 2024-03-24T10:45:05.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.838+0000 7f90a77fe640 1 -- 172.21.15.5:0/602362603 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f90b4013470 con 0x7f90c0159ef0 2024-03-24T10:45:05.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.838+0000 7f90a77fe640 1 --2- 172.21.15.5:0/602362603 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f90ac06b810 0x7f90ac06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:05.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.838+0000 7f90beffd640 1 --2- 172.21.15.5:0/602362603 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f90ac06b810 0x7f90ac06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:05.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.839+0000 7f90beffd640 1 --2- 172.21.15.5:0/602362603 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f90ac06b810 0x7f90ac06dcd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f90b0005880 tx=0x7f90b0007c10 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:05.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.853+0000 7f90a77fe640 1 -- 172.21.15.5:0/602362603 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f90b4058ec0 con 0x7f90c0159ef0 2024-03-24T10:45:05.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:05.853+0000 7f90a77fe640 1 -- 172.21.15.5:0/602362603 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f90b4055160 con 0x7f90c0159ef0 2024-03-24T10:45:06.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.076+0000 7f90c57af640 1 -- 172.21.15.5:0/602362603 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f90c00c7410 con 0x7f90c0159ef0 2024-03-24T10:45:06.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.511+0000 7f90a77fe640 1 -- 172.21.15.5:0/602362603 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v8) ==== 105+0+0 (secure 0 0 0) 0x7f90b40170b0 con 0x7f90c0159ef0 2024-03-24T10:45:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.514+0000 7f90c57af640 1 -- 172.21.15.5:0/602362603 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f90ac06b810 msgr2=0x7f90ac06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.514+0000 7f90c57af640 1 --2- 172.21.15.5:0/602362603 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f90ac06b810 0x7f90ac06dcd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f90b0005880 tx=0x7f90b0007c10 comp rx=0 tx=0).stop 2024-03-24T10:45:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.514+0000 7f90c57af640 1 -- 172.21.15.5:0/602362603 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90c0159ef0 msgr2=0x7f90c0072860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.514+0000 7f90c57af640 1 --2- 172.21.15.5:0/602362603 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90c0159ef0 0x7f90c0072860 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f90b400e880 tx=0x7f90b400ed40 comp rx=0 tx=0).stop 2024-03-24T10:45:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.514+0000 7f90c57af640 1 -- 172.21.15.5:0/602362603 shutdown_connections 2024-03-24T10:45:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.514+0000 7f90c57af640 1 --2- 172.21.15.5:0/602362603 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f90ac06b810 0x7f90ac06dcd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.514+0000 7f90c57af640 1 --2- 172.21.15.5:0/602362603 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90c0159ef0 0x7f90c0072860 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.514+0000 7f90c57af640 1 --2- 172.21.15.5:0/602362603 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90c00c99b0 0x7f90c01f3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.514+0000 7f90c57af640 1 --2- 172.21.15.5:0/602362603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90c0150b20 0x7f90c0072fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.514+0000 7f90c57af640 1 -- 172.21.15.5:0/602362603 >> 172.21.15.5:0/602362603 conn(0x7f90c006f180 msgr2=0x7f90c014e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:06.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.514+0000 7f90c57af640 1 -- 172.21.15.5:0/602362603 shutdown_connections 2024-03-24T10:45:06.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.514+0000 7f90c57af640 1 -- 172.21.15.5:0/602362603 wait complete. 2024-03-24T10:45:06.525 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T10:45:06.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.631+0000 7fe22e9d6640 1 Processor -- start 2024-03-24T10:45:06.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.631+0000 7fe22e9d6640 1 -- start start 2024-03-24T10:45:06.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.632+0000 7fe22e9d6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe228154640 0x7fe2280cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.632+0000 7fe22e9d6640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe22806c230 0x7fe22806c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.632+0000 7fe22e9d6640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2280cb6a0 con 0x7fe2280caba0 2024-03-24T10:45:06.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.632+0000 7fe22e9d6640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fe22815ae70 con 0x7fe22806c230 2024-03-24T10:45:06.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.632+0000 7fe22e9d6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe2280cc350 con 0x7fe228154640 2024-03-24T10:45:06.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.633+0000 7fe227fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe228154640 0x7fe2280cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.633+0000 7fe227fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe228154640 0x7fe2280cb8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:39242/0 (socket says 172.21.15.5:39242) 2024-03-24T10:45:06.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.633+0000 7fe227fff640 1 -- 172.21.15.5:0/3724349448 learned_addr learned my addr 172.21.15.5:0/3724349448 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:06.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.635+0000 7fe227fff640 1 -- 172.21.15.5:0/3724349448 >> v2:172.21.15.5:3300/0 conn(0x7fe22806c230 msgr2=0x7fe22806c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:06.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.635+0000 7fe227fff640 1 --2- 172.21.15.5:0/3724349448 >> v2:172.21.15.5:3300/0 conn(0x7fe22806c230 0x7fe22806c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.635+0000 7fe227fff640 1 -- 172.21.15.5:0/3724349448 >> v1:172.21.15.5:6789/0 conn(0x7fe2280caba0 legacy=0x7fe22815b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:06.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.635+0000 7fe227fff640 1 -- 172.21.15.5:0/3724349448 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2280cbde0 con 0x7fe228154640 2024-03-24T10:45:06.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.638+0000 7fe227fff640 1 --2- 172.21.15.5:0/3724349448 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe228154640 0x7fe2280cb8a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe218009870 tx=0x7fe21802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a166538a046e0e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:06.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.700+0000 7fe226ffd640 1 -- 172.21.15.5:0/3724349448 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe218031640 con 0x7fe228154640 2024-03-24T10:45:06.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.700+0000 7fe226ffd640 1 -- 172.21.15.5:0/3724349448 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2180317a0 con 0x7fe228154640 2024-03-24T10:45:06.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.700+0000 7fe226ffd640 1 -- 172.21.15.5:0/3724349448 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe218031b30 con 0x7fe228154640 2024-03-24T10:45:06.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.701+0000 7fe22e9d6640 1 -- 172.21.15.5:0/3724349448 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe228154640 msgr2=0x7fe2280cb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:06.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.701+0000 7fe22e9d6640 1 --2- 172.21.15.5:0/3724349448 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe228154640 0x7fe2280cb8a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe218009870 tx=0x7fe21802b8f0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.701+0000 7fe22e9d6640 1 -- 172.21.15.5:0/3724349448 shutdown_connections 2024-03-24T10:45:06.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.701+0000 7fe22e9d6640 1 --2- 172.21.15.5:0/3724349448 >> v2:172.21.15.5:3300/0 conn(0x7fe22806c230 0x7fe22806c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.701+0000 7fe22e9d6640 1 --2- 172.21.15.5:0/3724349448 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe228154640 0x7fe2280cb8a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.701+0000 7fe22e9d6640 1 -- 172.21.15.5:0/3724349448 >> 172.21.15.5:0/3724349448 conn(0x7fe228062180 msgr2=0x7fe2280cfb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:06.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.701+0000 7fe22e9d6640 1 -- 172.21.15.5:0/3724349448 shutdown_connections 2024-03-24T10:45:06.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.701+0000 7fe22e9d6640 1 -- 172.21.15.5:0/3724349448 wait complete. 2024-03-24T10:45:06.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.702+0000 7fe22e9d6640 1 Processor -- start 2024-03-24T10:45:06.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.702+0000 7fe22e9d6640 1 -- start start 2024-03-24T10:45:06.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.703+0000 7fe22e9d6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe22806c230 0x7fe2280c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.703+0000 7fe22e9d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2280caba0 0x7fe2280c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.703+0000 7fe22e9d6640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe228154640 0x7fe2280b4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.703+0000 7fe22e9d6640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fe2280c9e20 con 0x7fe228154640 2024-03-24T10:45:06.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.703+0000 7fe22e9d6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe2281557d0 con 0x7fe2280caba0 2024-03-24T10:45:06.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.703+0000 7fe22e9d6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe2281f0160 con 0x7fe22806c230 2024-03-24T10:45:06.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.704+0000 7fe227fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe22806c230 0x7fe2280c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.704+0000 7fe2277fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2280caba0 0x7fe2280c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.704+0000 7fe22cf73640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe228154640 0x7fe2280b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.704+0000 7fe227fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe22806c230 0x7fe2280c82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:39252/0 (socket says 172.21.15.5:39252) 2024-03-24T10:45:06.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.704+0000 7fe227fff640 1 -- 172.21.15.5:0/716608621 learned_addr learned my addr 172.21.15.5:0/716608621 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:06.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.707+0000 7fe227fff640 1 -- 172.21.15.5:0/716608621 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2280caba0 msgr2=0x7fe2280c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:06.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.707+0000 7fe227fff640 1 --2- 172.21.15.5:0/716608621 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2280caba0 0x7fe2280c9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.707+0000 7fe227fff640 1 -- 172.21.15.5:0/716608621 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe228154640 msgr2=0x7fe2280b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:06.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.707+0000 7fe227fff640 1 --2- 172.21.15.5:0/716608621 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe228154640 0x7fe2280b4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:06.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.707+0000 7fe227fff640 1 -- 172.21.15.5:0/716608621 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2180094d0 con 0x7fe22806c230 2024-03-24T10:45:06.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.709+0000 7fe227fff640 1 --2- 172.21.15.5:0/716608621 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe22806c230 0x7fe2280c82e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe218009990 tx=0x7fe2180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:06.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.713+0000 7fe224ff9640 1 -- 172.21.15.5:0/716608621 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21803bc70 con 0x7fe22806c230 2024-03-24T10:45:06.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.713+0000 7fe224ff9640 1 -- 172.21.15.5:0/716608621 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe21803bdf0 con 0x7fe22806c230 2024-03-24T10:45:06.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.713+0000 7fe22e9d6640 1 -- 172.21.15.5:0/716608621 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2281f1130 con 0x7fe22806c230 2024-03-24T10:45:06.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.713+0000 7fe224ff9640 1 -- 172.21.15.5:0/716608621 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe218045630 con 0x7fe22806c230 2024-03-24T10:45:06.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.713+0000 7fe22e9d6640 1 -- 172.21.15.5:0/716608621 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2281f0ca0 con 0x7fe22806c230 2024-03-24T10:45:06.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.714+0000 7fe22e9d6640 1 -- 172.21.15.5:0/716608621 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2280c8820 con 0x7fe22806c230 2024-03-24T10:45:06.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.979+0000 7fe224ff9640 1 -- 172.21.15.5:0/716608621 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe21802f050 con 0x7fe22806c230 2024-03-24T10:45:06.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.979+0000 7fe224ff9640 1 --2- 172.21.15.5:0/716608621 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe21406b860 0x7fe21406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:06.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.980+0000 7fe2277fe640 1 --2- 172.21.15.5:0/716608621 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe21406b860 0x7fe21406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:06.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.980+0000 7fe2277fe640 1 --2- 172.21.15.5:0/716608621 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe21406b860 0x7fe21406dd20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe2280c6880 tx=0x7fe21c009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:06.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.983+0000 7fe224ff9640 1 -- 172.21.15.5:0/716608621 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe21807ac50 con 0x7fe22806c230 2024-03-24T10:45:06.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:06.995+0000 7fe224ff9640 1 -- 172.21.15.5:0/716608621 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe218076ef0 con 0x7fe22806c230 2024-03-24T10:45:07.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.248+0000 7fe22e9d6640 1 -- 172.21.15.5:0/716608621 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fe2280681d0 con 0x7fe22806c230 2024-03-24T10:45:07.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.851+0000 7fe224ff9640 1 -- 172.21.15.5:0/716608621 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7fe21807a5b0 con 0x7fe22806c230 2024-03-24T10:45:07.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.853+0000 7fe22e9d6640 1 -- 172.21.15.5:0/716608621 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe21406b860 msgr2=0x7fe21406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:07.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.853+0000 7fe22e9d6640 1 --2- 172.21.15.5:0/716608621 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe21406b860 0x7fe21406dd20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe2280c6880 tx=0x7fe21c009040 comp rx=0 tx=0).stop 2024-03-24T10:45:07.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.853+0000 7fe22e9d6640 1 -- 172.21.15.5:0/716608621 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe22806c230 msgr2=0x7fe2280c82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:07.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.853+0000 7fe22e9d6640 1 --2- 172.21.15.5:0/716608621 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe22806c230 0x7fe2280c82e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe218009990 tx=0x7fe2180046c0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.853+0000 7fe22e9d6640 1 -- 172.21.15.5:0/716608621 shutdown_connections 2024-03-24T10:45:07.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.853+0000 7fe22e9d6640 1 --2- 172.21.15.5:0/716608621 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe21406b860 0x7fe21406dd20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.853+0000 7fe22e9d6640 1 --2- 172.21.15.5:0/716608621 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe228154640 0x7fe2280b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.853+0000 7fe22e9d6640 1 --2- 172.21.15.5:0/716608621 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe2280caba0 0x7fe2280c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.853+0000 7fe22e9d6640 1 --2- 172.21.15.5:0/716608621 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe22806c230 0x7fe2280c82e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.853+0000 7fe22e9d6640 1 -- 172.21.15.5:0/716608621 >> 172.21.15.5:0/716608621 conn(0x7fe228062180 msgr2=0x7fe2280cc520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:07.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.853+0000 7fe22e9d6640 1 -- 172.21.15.5:0/716608621 shutdown_connections 2024-03-24T10:45:07.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.853+0000 7fe22e9d6640 1 -- 172.21.15.5:0/716608621 wait complete. 2024-03-24T10:45:07.864 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-24T10:45:07.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.967+0000 7f33777c7640 1 Processor -- start 2024-03-24T10:45:07.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.968+0000 7f33777c7640 1 -- start start 2024-03-24T10:45:07.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.968+0000 7f33777c7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3370072c00 0x7f33700c4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.968+0000 7f33777c7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f33700c6230 0x7f3370072fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.968+0000 7f33777c7640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33700c3b30 con 0x7f3370156fd0 2024-03-24T10:45:07.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.968+0000 7f33777c7640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f3370073520 con 0x7f33700c6230 2024-03-24T10:45:07.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.968+0000 7f33777c7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f33700c6770 con 0x7f3370072c00 2024-03-24T10:45:07.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.969+0000 7f3375563640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3370072c00 0x7f33700c4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.969+0000 7f3374d62640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f33700c6230 0x7f3370072fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.969+0000 7f3375d64640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f3370156fd0 0x7f33700c7640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:34280/0 (socket says 172.21.15.5:34280) 2024-03-24T10:45:07.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.969+0000 7f3375d64640 1 -- 172.21.15.5:0/240204252 learned_addr learned my addr 172.21.15.5:0/240204252 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:07.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.971+0000 7f3367fff640 1 -- 172.21.15.5:0/240204252 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 124212166 0 0) 0x7f33700c3b30 con 0x7f3370156fd0 2024-03-24T10:45:07.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.971+0000 7f3367fff640 1 -- 172.21.15.5:0/240204252 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f335c003520 con 0x7f3370156fd0 2024-03-24T10:45:07.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.971+0000 7f3375563640 1 -- 172.21.15.5:0/240204252 >> v2:172.21.15.5:3300/0 conn(0x7f33700c6230 msgr2=0x7f3370072fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:07.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.971+0000 7f3375563640 1 --2- 172.21.15.5:0/240204252 >> v2:172.21.15.5:3300/0 conn(0x7f33700c6230 0x7f3370072fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.971+0000 7f3375563640 1 -- 172.21.15.5:0/240204252 >> v1:172.21.15.57:6789/0 conn(0x7f3370156fd0 legacy=0x7f33700c7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:07.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.971+0000 7f3375563640 1 -- 172.21.15.5:0/240204252 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33700c6a50 con 0x7f3370072c00 2024-03-24T10:45:07.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.973+0000 7f3375563640 1 --2- 172.21.15.5:0/240204252 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3370072c00 0x7f33700c4930 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f336c0097a0 tx=0x7f336c02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b63564807ddf96bc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:07.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.977+0000 7f3367fff640 1 -- 172.21.15.5:0/240204252 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c030610 con 0x7f3370072c00 2024-03-24T10:45:07.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.977+0000 7f3367fff640 1 -- 172.21.15.5:0/240204252 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f336c030770 con 0x7f3370072c00 2024-03-24T10:45:07.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.977+0000 7f3367fff640 1 -- 172.21.15.5:0/240204252 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f336c030b00 con 0x7f3370072c00 2024-03-24T10:45:07.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.978+0000 7f33777c7640 1 -- 172.21.15.5:0/240204252 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3370072c00 msgr2=0x7f33700c4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:07.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.978+0000 7f33777c7640 1 --2- 172.21.15.5:0/240204252 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3370072c00 0x7f33700c4930 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f336c0097a0 tx=0x7f336c02b960 comp rx=0 tx=0).stop 2024-03-24T10:45:07.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.978+0000 7f33777c7640 1 -- 172.21.15.5:0/240204252 shutdown_connections 2024-03-24T10:45:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.978+0000 7f33777c7640 1 --2- 172.21.15.5:0/240204252 >> v2:172.21.15.5:3300/0 conn(0x7f33700c6230 0x7f3370072fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.978+0000 7f33777c7640 1 --2- 172.21.15.5:0/240204252 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3370072c00 0x7f33700c4930 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.978+0000 7f33777c7640 1 -- 172.21.15.5:0/240204252 >> 172.21.15.5:0/240204252 conn(0x7f3370070130 msgr2=0x7f3370070510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.978+0000 7f33777c7640 1 -- 172.21.15.5:0/240204252 shutdown_connections 2024-03-24T10:45:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.978+0000 7f33777c7640 1 -- 172.21.15.5:0/240204252 wait complete. 2024-03-24T10:45:07.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.979+0000 7f33777c7640 1 Processor -- start 2024-03-24T10:45:07.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.979+0000 7f33777c7640 1 -- start start 2024-03-24T10:45:07.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.980+0000 7f33777c7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3370072c00 0x7f337014a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.980+0000 7f33777c7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33700c6230 0x7f3370153cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.980+0000 7f33777c7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3370156fd0 0x7f33701541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:07.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.980+0000 7f33777c7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f3370153820 con 0x7f33700c6230 2024-03-24T10:45:07.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.980+0000 7f33777c7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3370152800 con 0x7f3370156fd0 2024-03-24T10:45:07.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.980+0000 7f33777c7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f33701f1060 con 0x7f3370072c00 2024-03-24T10:45:07.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.980+0000 7f3374d62640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33700c6230 0x7f3370153cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.980+0000 7f3375d64640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3370156fd0 0x7f33701541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.980+0000 7f3375d64640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3370156fd0 0x7f33701541f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:34686/0 (socket says 172.21.15.5:34686) 2024-03-24T10:45:07.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.980+0000 7f3375d64640 1 -- 172.21.15.5:0/3680499886 learned_addr learned my addr 172.21.15.5:0/3680499886 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:07.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.981+0000 7f3375563640 1 --2- 172.21.15.5:0/3680499886 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3370072c00 0x7f337014a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:07.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.984+0000 7f3375d64640 1 -- 172.21.15.5:0/3680499886 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3370072c00 msgr2=0x7f337014a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:07.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.984+0000 7f3375d64640 1 --2- 172.21.15.5:0/3680499886 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3370072c00 0x7f337014a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.984+0000 7f3375d64640 1 -- 172.21.15.5:0/3680499886 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33700c6230 msgr2=0x7f3370153cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:07.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.984+0000 7f3375d64640 1 --2- 172.21.15.5:0/3680499886 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33700c6230 0x7f3370153cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:07.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.984+0000 7f3375d64640 1 -- 172.21.15.5:0/3680499886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f336c009400 con 0x7f3370156fd0 2024-03-24T10:45:07.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.987+0000 7f3375d64640 1 --2- 172.21.15.5:0/3680499886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3370156fd0 0x7f33701541f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f336800edd0 tx=0x7f336800c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:07.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.992+0000 7f3365ffb640 1 -- 172.21.15.5:0/3680499886 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3368011670 con 0x7f3370156fd0 2024-03-24T10:45:07.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.992+0000 7f3365ffb640 1 -- 172.21.15.5:0/3680499886 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33680117d0 con 0x7f3370156fd0 2024-03-24T10:45:07.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.992+0000 7f3365ffb640 1 -- 172.21.15.5:0/3680499886 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3368011ba0 con 0x7f3370156fd0 2024-03-24T10:45:07.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.992+0000 7f33777c7640 1 -- 172.21.15.5:0/3680499886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3370152c10 con 0x7f3370156fd0 2024-03-24T10:45:07.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.992+0000 7f33777c7640 1 -- 172.21.15.5:0/3680499886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33701531d0 con 0x7f3370156fd0 2024-03-24T10:45:07.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:07.993+0000 7f33777c7640 1 -- 172.21.15.5:0/3680499886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33700688e0 con 0x7f3370156fd0 2024-03-24T10:45:08.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.012+0000 7f3365ffb640 1 -- 172.21.15.5:0/3680499886 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3368011d00 con 0x7f3370156fd0 2024-03-24T10:45:08.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.012+0000 7f3365ffb640 1 --2- 172.21.15.5:0/3680499886 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f335406b890 0x7f335406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:08.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.012+0000 7f3375563640 1 --2- 172.21.15.5:0/3680499886 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f335406b890 0x7f335406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:08.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.013+0000 7f3375563640 1 --2- 172.21.15.5:0/3680499886 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f335406b890 0x7f335406dd50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f336c036000 tx=0x7f336c008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:08.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.014+0000 7f3365ffb640 1 -- 172.21.15.5:0/3680499886 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f336805bc60 con 0x7f3370156fd0 2024-03-24T10:45:08.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.276+0000 7f3365ffb640 1 -- 172.21.15.5:0/3680499886 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3368057f00 con 0x7f3370156fd0 2024-03-24T10:45:08.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.497+0000 7f33777c7640 1 -- 172.21.15.5:0/3680499886 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f3370157d60 con 0x7f3370156fd0 2024-03-24T10:45:08.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.841+0000 7f3365ffb640 1 -- 172.21.15.5:0/3680499886 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v11) ==== 115+0+0 (secure 0 0 0) 0x7f336805b5c0 con 0x7f3370156fd0 2024-03-24T10:45:08.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.844+0000 7f33777c7640 1 -- 172.21.15.5:0/3680499886 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f335406b890 msgr2=0x7f335406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:08.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.844+0000 7f33777c7640 1 --2- 172.21.15.5:0/3680499886 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f335406b890 0x7f335406dd50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f336c036000 tx=0x7f336c008030 comp rx=0 tx=0).stop 2024-03-24T10:45:08.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.844+0000 7f33777c7640 1 -- 172.21.15.5:0/3680499886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3370156fd0 msgr2=0x7f33701541f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.844+0000 7f33777c7640 1 --2- 172.21.15.5:0/3680499886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3370156fd0 0x7f33701541f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f336800edd0 tx=0x7f336800c640 comp rx=0 tx=0).stop 2024-03-24T10:45:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.844+0000 7f33777c7640 1 -- 172.21.15.5:0/3680499886 shutdown_connections 2024-03-24T10:45:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.844+0000 7f33777c7640 1 --2- 172.21.15.5:0/3680499886 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f335406b890 0x7f335406dd50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.844+0000 7f33777c7640 1 --2- 172.21.15.5:0/3680499886 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33700c6230 0x7f3370153cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.844+0000 7f33777c7640 1 --2- 172.21.15.5:0/3680499886 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3370156fd0 0x7f33701541f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.844+0000 7f33777c7640 1 --2- 172.21.15.5:0/3680499886 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3370072c00 0x7f337014a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.844+0000 7f33777c7640 1 -- 172.21.15.5:0/3680499886 >> 172.21.15.5:0/3680499886 conn(0x7f3370070130 msgr2=0x7f33700c8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.844+0000 7f33777c7640 1 -- 172.21.15.5:0/3680499886 shutdown_connections 2024-03-24T10:45:08.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.844+0000 7f33777c7640 1 -- 172.21.15.5:0/3680499886 wait complete. 2024-03-24T10:45:08.855 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T10:45:08.855 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:45:08.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.960+0000 7f228158a640 1 Processor -- start 2024-03-24T10:45:08.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.960+0000 7f228158a640 1 -- start start 2024-03-24T10:45:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.960+0000 7f228158a640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f227c15aea0 0x7f227c0cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.960+0000 7f228158a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f227c0cf100 0x7f227c0ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.960+0000 7f228158a640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f227c06cd50 con 0x7f227c15aac0 2024-03-24T10:45:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.960+0000 7f228158a640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f227c0ce790 con 0x7f227c15aea0 2024-03-24T10:45:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.960+0000 7f228158a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f227c0cdb10 con 0x7f227c0cf100 2024-03-24T10:45:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.961+0000 7f227affd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f227c15aac0 0x7f227c06c230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59618/0 (socket says 172.21.15.5:59618) 2024-03-24T10:45:08.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.961+0000 7f227affd640 1 -- 172.21.15.5:0/209237081 learned_addr learned my addr 172.21.15.5:0/209237081 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:08.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.961+0000 7f227a7fc640 1 --2- 172.21.15.5:0/209237081 >> v2:172.21.15.57:3300/0 conn(0x7f227c15aea0 0x7f227c0cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:08.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.964+0000 7f227a7fc640 1 -- 172.21.15.5:0/209237081 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f227c0cf100 msgr2=0x7f227c0ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:08.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.964+0000 7f227a7fc640 1 --2- 172.21.15.5:0/209237081 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f227c0cf100 0x7f227c0ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:08.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.964+0000 7f227a7fc640 1 -- 172.21.15.5:0/209237081 >> v1:172.21.15.5:6789/0 conn(0x7f227c15aac0 legacy=0x7f227c06c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:08.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.964+0000 7f227a7fc640 1 -- 172.21.15.5:0/209237081 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f227c0cddf0 con 0x7f227c15aea0 2024-03-24T10:45:08.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.966+0000 7f227a7fc640 1 --2- 172.21.15.5:0/209237081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f227c15aea0 0x7f227c0cebc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f22700097a0 tx=0x7f227002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1f15d9bd2ae54c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:08.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.970+0000 7f2279ffb640 1 -- 172.21.15.5:0/209237081 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2270030610 con 0x7f227c15aea0 2024-03-24T10:45:08.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.970+0000 7f2279ffb640 1 -- 172.21.15.5:0/209237081 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2270030770 con 0x7f227c15aea0 2024-03-24T10:45:08.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.970+0000 7f2279ffb640 1 -- 172.21.15.5:0/209237081 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2270030b00 con 0x7f227c15aea0 2024-03-24T10:45:08.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.971+0000 7f228158a640 1 -- 172.21.15.5:0/209237081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f227c15aea0 msgr2=0x7f227c0cebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:08.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.971+0000 7f228158a640 1 --2- 172.21.15.5:0/209237081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f227c15aea0 0x7f227c0cebc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f22700097a0 tx=0x7f227002b960 comp rx=0 tx=0).stop 2024-03-24T10:45:08.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.971+0000 7f228158a640 1 -- 172.21.15.5:0/209237081 shutdown_connections 2024-03-24T10:45:08.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.971+0000 7f228158a640 1 --2- 172.21.15.5:0/209237081 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f227c0cf100 0x7f227c0ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:08.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.971+0000 7f228158a640 1 --2- 172.21.15.5:0/209237081 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f227c15aea0 0x7f227c0cebc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:08.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.971+0000 7f228158a640 1 -- 172.21.15.5:0/209237081 >> 172.21.15.5:0/209237081 conn(0x7f227c0cfb90 msgr2=0x7f227c0cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:08.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.971+0000 7f228158a640 1 -- 172.21.15.5:0/209237081 shutdown_connections 2024-03-24T10:45:08.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.971+0000 7f228158a640 1 -- 172.21.15.5:0/209237081 wait complete. 2024-03-24T10:45:08.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.972+0000 7f228158a640 1 Processor -- start 2024-03-24T10:45:08.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.972+0000 7f228158a640 1 -- start start 2024-03-24T10:45:08.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.973+0000 7f228158a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f227c0cf100 0x7f227c0c6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:08.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.973+0000 7f228158a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f227c15aac0 0x7f227c0b4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:08.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.973+0000 7f228158a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f227c15aea0 0x7f227c0b5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:08.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.973+0000 7f228158a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f227c0b31c0 con 0x7f227c15aac0 2024-03-24T10:45:08.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.973+0000 7f228158a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f227c1f3650 con 0x7f227c0cf100 2024-03-24T10:45:08.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.973+0000 7f228158a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f227c0b5880 con 0x7f227c15aea0 2024-03-24T10:45:08.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.974+0000 7f227a7fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f227c15aac0 0x7f227c0b4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:08.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.974+0000 7f227affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f227c0cf100 0x7f227c0c6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:08.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.974+0000 7f227affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f227c0cf100 0x7f227c0c6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:34706/0 (socket says 172.21.15.5:34706) 2024-03-24T10:45:08.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.974+0000 7f227affd640 1 -- 172.21.15.5:0/1583419806 learned_addr learned my addr 172.21.15.5:0/1583419806 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:08.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.978+0000 7f227a7fc640 1 -- 172.21.15.5:0/1583419806 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f227c15aea0 msgr2=0x7f227c0b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:08.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.978+0000 7f227a7fc640 1 --2- 172.21.15.5:0/1583419806 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f227c15aea0 0x7f227c0b5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:08.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.978+0000 7f227a7fc640 1 -- 172.21.15.5:0/1583419806 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f227c0cf100 msgr2=0x7f227c0c6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:08.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.978+0000 7f227a7fc640 1 --2- 172.21.15.5:0/1583419806 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f227c0cf100 0x7f227c0c6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:08.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.978+0000 7f227a7fc640 1 -- 172.21.15.5:0/1583419806 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2270009400 con 0x7f227c15aac0 2024-03-24T10:45:08.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.981+0000 7f227a7fc640 1 --2- 172.21.15.5:0/1583419806 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f227c15aac0 0x7f227c0b4e00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2270033000 tx=0x7f22700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:08.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.983+0000 7f22637fe640 1 -- 172.21.15.5:0/1583419806 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227003e070 con 0x7f227c15aac0 2024-03-24T10:45:08.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.983+0000 7f228158a640 1 -- 172.21.15.5:0/1583419806 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f227c0b3680 con 0x7f227c15aac0 2024-03-24T10:45:08.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.983+0000 7f228158a640 1 -- 172.21.15.5:0/1583419806 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f227c0b33a0 con 0x7f227c15aac0 2024-03-24T10:45:08.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:08.984+0000 7f228158a640 1 -- 172.21.15.5:0/1583419806 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f227c0681d0 con 0x7f227c15aac0 2024-03-24T10:45:09.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.194+0000 7f22637fe640 1 -- 172.21.15.5:0/1583419806 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22700399d0 con 0x7f227c15aac0 2024-03-24T10:45:09.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.251+0000 7f22637fe640 1 -- 172.21.15.5:0/1583419806 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2270039d60 con 0x7f227c15aac0 2024-03-24T10:45:09.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.252+0000 7f22637fe640 1 -- 172.21.15.5:0/1583419806 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2270055430 con 0x7f227c15aac0 2024-03-24T10:45:09.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.252+0000 7f22637fe640 1 --2- 172.21.15.5:0/1583419806 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f226806b810 0x7f226806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:09.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.252+0000 7f227affd640 1 --2- 172.21.15.5:0/1583419806 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f226806b810 0x7f226806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:09.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.253+0000 7f227affd640 1 --2- 172.21.15.5:0/1583419806 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f226806b810 0x7f226806dcd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f226c004e40 tx=0x7f226c007f10 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:09.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.265+0000 7f22637fe640 1 -- 172.21.15.5:0/1583419806 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f227007c020 con 0x7f227c15aac0 2024-03-24T10:45:09.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.265+0000 7f22637fe640 1 -- 172.21.15.5:0/1583419806 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2270077bc0 con 0x7f227c15aac0 2024-03-24T10:45:09.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.502+0000 7f228158a640 1 -- 172.21.15.5:0/1583419806 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f227c0cf770 con 0x7f227c15aac0 2024-03-24T10:45:09.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.524+0000 7f22637fe640 1 -- 172.21.15.5:0/1583419806 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6847 (secure 0 0 0) 0x7f2270034020 con 0x7f227c15aac0 2024-03-24T10:45:09.524 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:45:09.525 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4251,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4252,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:45:08.776275+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2],"up":{"mds_0":4258,"mds_1":4257,"mds_2":4254},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:creating","state_seq":1,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4257":{"gid":4257,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.57:6837/1426147484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":1426147484},{"type":"v1","addr":"172.21.15.57:6837","nonce":1426147484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4257]},"id":1}]} 2024-03-24T10:45:09.525 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 12 2024-03-24T10:45:09.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.526+0000 7f228158a640 1 -- 172.21.15.5:0/1583419806 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f226806b810 msgr2=0x7f226806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:09.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.526+0000 7f228158a640 1 --2- 172.21.15.5:0/1583419806 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f226806b810 0x7f226806dcd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f226c004e40 tx=0x7f226c007f10 comp rx=0 tx=0).stop 2024-03-24T10:45:09.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.526+0000 7f228158a640 1 -- 172.21.15.5:0/1583419806 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f227c15aac0 msgr2=0x7f227c0b4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:09.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.526+0000 7f228158a640 1 --2- 172.21.15.5:0/1583419806 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f227c15aac0 0x7f227c0b4e00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2270033000 tx=0x7f22700046c0 comp rx=0 tx=0).stop 2024-03-24T10:45:09.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.526+0000 7f228158a640 1 -- 172.21.15.5:0/1583419806 shutdown_connections 2024-03-24T10:45:09.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.526+0000 7f228158a640 1 --2- 172.21.15.5:0/1583419806 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f226806b810 0x7f226806dcd0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:09.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.526+0000 7f228158a640 1 --2- 172.21.15.5:0/1583419806 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f227c15aac0 0x7f227c0b4e00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:09.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.526+0000 7f228158a640 1 --2- 172.21.15.5:0/1583419806 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f227c15aea0 0x7f227c0b5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:09.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.526+0000 7f228158a640 1 --2- 172.21.15.5:0/1583419806 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f227c0cf100 0x7f227c0c6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:09.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.526+0000 7f228158a640 1 -- 172.21.15.5:0/1583419806 >> 172.21.15.5:0/1583419806 conn(0x7f227c0cfb90 msgr2=0x7f227c153ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:09.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.526+0000 7f228158a640 1 -- 172.21.15.5:0/1583419806 shutdown_connections 2024-03-24T10:45:09.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.526+0000 7f228158a640 1 -- 172.21.15.5:0/1583419806 wait complete. 2024-03-24T10:45:09.538 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:45:09.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.639+0000 7fcd0342a640 1 Processor -- start 2024-03-24T10:45:09.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.639+0000 7fcd0342a640 1 -- start start 2024-03-24T10:45:09.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.639+0000 7fcd0342a640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fccfc15ac00 0x7fccfc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:09.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.640+0000 7fcd0342a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fccfc0ccaa0 0x7fccfc06bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:09.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.640+0000 7fcd0342a640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccfc0cb2f0 con 0x7fccfc1543a0 2024-03-24T10:45:09.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.640+0000 7fcd0342a640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fccfc0c6750 con 0x7fccfc0ccaa0 2024-03-24T10:45:09.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.640+0000 7fcd0342a640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fccfc0cc380 con 0x7fccfc15ac00 2024-03-24T10:45:09.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.640+0000 7fcd019c7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fccfc0ccaa0 0x7fccfc06bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:09.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.640+0000 7fcd009c5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fccfc15ac00 0x7fccfc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:09.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.640+0000 7fcd009c5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fccfc15ac00 0x7fccfc0cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:34712/0 (socket says 172.21.15.5:34712) 2024-03-24T10:45:09.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.640+0000 7fcd009c5640 1 -- 172.21.15.5:0/154534148 learned_addr learned my addr 172.21.15.5:0/154534148 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:09.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.643+0000 7fcd009c5640 1 -- 172.21.15.5:0/154534148 >> v2:172.21.15.5:3300/0 conn(0x7fccfc0ccaa0 msgr2=0x7fccfc06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:09.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.643+0000 7fcd009c5640 1 --2- 172.21.15.5:0/154534148 >> v2:172.21.15.5:3300/0 conn(0x7fccfc0ccaa0 0x7fccfc06bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:09.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.643+0000 7fcd009c5640 1 -- 172.21.15.5:0/154534148 >> v1:172.21.15.5:6789/0 conn(0x7fccfc1543a0 legacy=0x7fccfc06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:09.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.643+0000 7fcd009c5640 1 -- 172.21.15.5:0/154534148 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccfc15b070 con 0x7fccfc15ac00 2024-03-24T10:45:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.645+0000 7fcd009c5640 1 --2- 172.21.15.5:0/154534148 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fccfc15ac00 0x7fccfc0cc560 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fccfc0c6990 tx=0x7fccec02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4ab6a941119e501 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:09.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.649+0000 7fccf3fff640 1 -- 172.21.15.5:0/154534148 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec030860 con 0x7fccfc15ac00 2024-03-24T10:45:09.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.649+0000 7fccf3fff640 1 -- 172.21.15.5:0/154534148 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccec0309c0 con 0x7fccfc15ac00 2024-03-24T10:45:09.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.649+0000 7fccf3fff640 1 -- 172.21.15.5:0/154534148 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec030d50 con 0x7fccfc15ac00 2024-03-24T10:45:09.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.650+0000 7fcd0342a640 1 -- 172.21.15.5:0/154534148 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fccfc15ac00 msgr2=0x7fccfc0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:09.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.650+0000 7fcd0342a640 1 --2- 172.21.15.5:0/154534148 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fccfc15ac00 0x7fccfc0cc560 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fccfc0c6990 tx=0x7fccec02b9a0 comp rx=0 tx=0).stop 2024-03-24T10:45:09.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.650+0000 7fcd0342a640 1 -- 172.21.15.5:0/154534148 shutdown_connections 2024-03-24T10:45:09.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.650+0000 7fcd0342a640 1 --2- 172.21.15.5:0/154534148 >> v2:172.21.15.5:3300/0 conn(0x7fccfc0ccaa0 0x7fccfc06bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:09.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.650+0000 7fcd0342a640 1 --2- 172.21.15.5:0/154534148 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fccfc15ac00 0x7fccfc0cc560 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:09.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.650+0000 7fcd0342a640 1 -- 172.21.15.5:0/154534148 >> 172.21.15.5:0/154534148 conn(0x7fccfc072ef0 msgr2=0x7fccfc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:09.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.650+0000 7fcd0342a640 1 -- 172.21.15.5:0/154534148 shutdown_connections 2024-03-24T10:45:09.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.650+0000 7fcd0342a640 1 -- 172.21.15.5:0/154534148 wait complete. 2024-03-24T10:45:09.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.652+0000 7fcd0342a640 1 Processor -- start 2024-03-24T10:45:09.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.652+0000 7fcd0342a640 1 -- start start 2024-03-24T10:45:09.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.653+0000 7fcd0342a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fccfc0ccaa0 0x7fccfc0b3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:09.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.653+0000 7fcd0342a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fccfc1543a0 0x7fccfc1efbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:09.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.653+0000 7fcd0342a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fccfc15ac00 0x7fccfc1466c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:09.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.653+0000 7fcd0342a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fccfc0b5130 con 0x7fccfc1543a0 2024-03-24T10:45:09.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.653+0000 7fcd0342a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fccfc1e7590 con 0x7fccfc0ccaa0 2024-03-24T10:45:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.653+0000 7fcd0342a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fccfc1589d0 con 0x7fccfc15ac00 2024-03-24T10:45:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.654+0000 7fcd009c5640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fccfc1543a0 0x7fccfc1efbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.654+0000 7fcd011c6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fccfc0ccaa0 0x7fccfc0b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.654+0000 7fcd019c7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fccfc15ac00 0x7fccfc1466c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.654+0000 7fcd011c6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fccfc0ccaa0 0x7fccfc0b3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:34714/0 (socket says 172.21.15.5:34714) 2024-03-24T10:45:09.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.654+0000 7fcd011c6640 1 -- 172.21.15.5:0/383249187 learned_addr learned my addr 172.21.15.5:0/383249187 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.658+0000 7fcd009c5640 1 -- 172.21.15.5:0/383249187 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fccfc15ac00 msgr2=0x7fccfc1466c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.658+0000 7fcd009c5640 1 --2- 172.21.15.5:0/383249187 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fccfc15ac00 0x7fccfc1466c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.658+0000 7fcd009c5640 1 -- 172.21.15.5:0/383249187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fccfc0ccaa0 msgr2=0x7fccfc0b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.658+0000 7fcd009c5640 1 --2- 172.21.15.5:0/383249187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fccfc0ccaa0 0x7fccfc0b3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.658+0000 7fcd009c5640 1 -- 172.21.15.5:0/383249187 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccec0094d0 con 0x7fccfc1543a0 2024-03-24T10:45:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.660+0000 7fcd009c5640 1 --2- 172.21.15.5:0/383249187 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fccfc1543a0 0x7fccfc1efbc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fccec033000 tx=0x7fccec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:09.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.662+0000 7fccf1ffb640 1 -- 172.21.15.5:0/383249187 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec039a90 con 0x7fccfc1543a0 2024-03-24T10:45:09.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.662+0000 7fcd0342a640 1 -- 172.21.15.5:0/383249187 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccfc1e72b0 con 0x7fccfc1543a0 2024-03-24T10:45:09.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.663+0000 7fcd0342a640 1 -- 172.21.15.5:0/383249187 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccfc1f35b0 con 0x7fccfc1543a0 2024-03-24T10:45:09.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.664+0000 7fcd0342a640 1 -- 172.21.15.5:0/383249187 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccfc1f3130 con 0x7fccfc1543a0 2024-03-24T10:45:09.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.664+0000 7fccf1ffb640 1 -- 172.21.15.5:0/383249187 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccec039bf0 con 0x7fccfc1543a0 2024-03-24T10:45:09.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.664+0000 7fccf1ffb640 1 -- 172.21.15.5:0/383249187 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec04d440 con 0x7fccfc1543a0 2024-03-24T10:45:09.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.684+0000 7fccf1ffb640 1 -- 172.21.15.5:0/383249187 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fccec04c3b0 con 0x7fccfc1543a0 2024-03-24T10:45:09.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.685+0000 7fccf1ffb640 1 --2- 172.21.15.5:0/383249187 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcce006b810 0x7fcce006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:09.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.685+0000 7fcd011c6640 1 --2- 172.21.15.5:0/383249187 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcce006b810 0x7fcce006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:09.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.685+0000 7fcd011c6640 1 --2- 172.21.15.5:0/383249187 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcce006b810 0x7fcce006dcd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fccf8001840 tx=0x7fccf8009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:09.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.947+0000 7fccf1ffb640 1 -- 172.21.15.5:0/383249187 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fccec07b1b0 con 0x7fccfc1543a0 2024-03-24T10:45:09.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:09.947+0000 7fccf1ffb640 1 -- 172.21.15.5:0/383249187 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fccec077450 con 0x7fccfc1543a0 2024-03-24T10:45:10.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.167+0000 7fcd0342a640 1 -- 172.21.15.5:0/383249187 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fccfc1f3130 con 0x7fccfc1543a0 2024-03-24T10:45:10.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.215+0000 7fccf1ffb640 1 -- 172.21.15.5:0/383249187 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16485 (secure 0 0 0) 0x7fccfc1f3130 con 0x7fccfc1543a0 2024-03-24T10:45:10.215 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:45:10.215 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":21,"fsid":"7d9848db-2fe7-4d95-a6e2-b04f4f639ebc","created":"2024-03-24T10:44:30.435176+0000","modified":"2024-03-24T10:45:02.781257+0000","last_up_change":"2024-03-24T10:44:44.634047+0000","last_in_change":"2024-03-24T10:44:35.986391+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:44:45.829320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:44:58.435543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:45:00.129313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50f590b3-f5ec-4e1b-ac46-25cab307a27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6825","nonce":1991336051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6827","nonce":1991336051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6831","nonce":1991336051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6829","nonce":1991336051}]},"public_addr":"172.21.15.5:6825/1991336051","cluster_addr":"172.21.15.5:6827/1991336051","heartbeat_back_addr":"172.21.15.5:6831/1991336051","heartbeat_front_addr":"172.21.15.5:6829/1991336051","state":["exists","up"]},{"osd":1,"uuid":"ccba9e93-c704-472b-a832-8d4eeb456c47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6801","nonce":2140667501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6803","nonce":2140667501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6807","nonce":2140667501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6805","nonce":2140667501}]},"public_addr":"172.21.15.5:6801/2140667501","cluster_addr":"172.21.15.5:6803/2140667501","heartbeat_back_addr":"172.21.15.5:6807/2140667501","heartbeat_front_addr":"172.21.15.5:6805/2140667501","state":["exists","up"]},{"osd":2,"uuid":"e614c5b1-6504-452e-82e0-95c7ca271737","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6817","nonce":90414679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6819","nonce":90414679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6823","nonce":90414679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6821","nonce":90414679}]},"public_addr":"172.21.15.5:6817/90414679","cluster_addr":"172.21.15.5:6819/90414679","heartbeat_back_addr":"172.21.15.5:6823/90414679","heartbeat_front_addr":"172.21.15.5:6821/90414679","state":["exists","up"]},{"osd":3,"uuid":"c60154d0-85e0-46a1-8273-4863bd8ad2dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6809","nonce":4212473119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6811","nonce":4212473119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6815","nonce":4212473119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6813","nonce":4212473119}]},"public_addr":"172.21.15.5:6809/4212473119","cluster_addr":"172.21.15.5:6811/4212473119","heartbeat_back_addr":"172.21.15.5:6815/4212473119","heartbeat_front_addr":"172.21.15.5:6813/4212473119","state":["exists","up"]},{"osd":4,"uuid":"21d74109-da3a-4fc0-b2d8-93e9ae3ed30c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6817","nonce":2193986505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6819","nonce":2193986505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6823","nonce":2193986505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6821","nonce":2193986505}]},"public_addr":"172.21.15.57:6817/2193986505","cluster_addr":"172.21.15.57:6819/2193986505","heartbeat_back_addr":"172.21.15.57:6823/2193986505","heartbeat_front_addr":"172.21.15.57:6821/2193986505","state":["exists","up"]},{"osd":5,"uuid":"5cf19cc7-215f-4d69-a30a-80543c0293e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6809","nonce":3044987466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6811","nonce":3044987466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6815","nonce":3044987466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6813","nonce":3044987466}]},"public_addr":"172.21.15.57:6809/3044987466","cluster_addr":"172.21.15.57:6811/3044987466","heartbeat_back_addr":"172.21.15.57:6815/3044987466","heartbeat_front_addr":"172.21.15.57:6813/3044987466","state":["exists","up"]},{"osd":6,"uuid":"a5e76c69-c3fc-47b2-8f92-e560cf503e45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6801","nonce":4290897445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6803","nonce":4290897445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6807","nonce":4290897445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6805","nonce":4290897445}]},"public_addr":"172.21.15.57:6801/4290897445","cluster_addr":"172.21.15.57:6803/4290897445","heartbeat_back_addr":"172.21.15.57:6807/4290897445","heartbeat_front_addr":"172.21.15.57:6805/4290897445","state":["exists","up"]},{"osd":7,"uuid":"47d0b873-e3ac-4f40-a372-681436e45d0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6825","nonce":212605698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6827","nonce":212605698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6831","nonce":212605698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6829","nonce":212605698}]},"public_addr":"172.21.15.57:6825/212605698","cluster_addr":"172.21.15.57:6827/212605698","heartbeat_back_addr":"172.21.15.57:6831/212605698","heartbeat_front_addr":"172.21.15.57:6829/212605698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.211237+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.192822+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.273429+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.199212+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.277082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.218768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.190915+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.225294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:45:10.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.217+0000 7fcd0342a640 1 -- 172.21.15.5:0/383249187 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcce006b810 msgr2=0x7fcce006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:10.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.217+0000 7fcd0342a640 1 --2- 172.21.15.5:0/383249187 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcce006b810 0x7fcce006dcd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fccf8001840 tx=0x7fccf8009040 comp rx=0 tx=0).stop 2024-03-24T10:45:10.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.217+0000 7fcd0342a640 1 -- 172.21.15.5:0/383249187 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fccfc1543a0 msgr2=0x7fccfc1efbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:10.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.217+0000 7fcd0342a640 1 --2- 172.21.15.5:0/383249187 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fccfc1543a0 0x7fccfc1efbc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fccec033000 tx=0x7fccec004730 comp rx=0 tx=0).stop 2024-03-24T10:45:10.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.217+0000 7fcd0342a640 1 -- 172.21.15.5:0/383249187 shutdown_connections 2024-03-24T10:45:10.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.217+0000 7fcd0342a640 1 --2- 172.21.15.5:0/383249187 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcce006b810 0x7fcce006dcd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:10.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.217+0000 7fcd0342a640 1 --2- 172.21.15.5:0/383249187 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fccfc1543a0 0x7fccfc1efbc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:10.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.217+0000 7fcd0342a640 1 --2- 172.21.15.5:0/383249187 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fccfc15ac00 0x7fccfc1466c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:10.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.217+0000 7fcd0342a640 1 --2- 172.21.15.5:0/383249187 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fccfc0ccaa0 0x7fccfc0b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:10.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.217+0000 7fcd0342a640 1 -- 172.21.15.5:0/383249187 >> 172.21.15.5:0/383249187 conn(0x7fccfc072ef0 msgr2=0x7fccfc06c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:10.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.217+0000 7fcd0342a640 1 -- 172.21.15.5:0/383249187 shutdown_connections 2024-03-24T10:45:10.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.218+0000 7fcd0342a640 1 -- 172.21.15.5:0/383249187 wait complete. 2024-03-24T10:45:10.229 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T10:45:10.229 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:45:10.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.331+0000 7f0093fd0640 1 Processor -- start 2024-03-24T10:45:10.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.331+0000 7f0093fd0640 1 -- start start 2024-03-24T10:45:10.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.332+0000 7f0093fd0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f008c15be90 0x7f008c0c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:10.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.332+0000 7f0093fd0640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f008c0c8850 con 0x7f008c0c92f0 2024-03-24T10:45:10.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.332+0000 7f0093fd0640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f008c152560 con 0x7f008c0c8c30 2024-03-24T10:45:10.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.332+0000 7f0093fd0640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f008c0c9c10 con 0x7f008c15be90 2024-03-24T10:45:10.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.332+0000 7f009156b640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f008c0c8c30 0x7f008c160a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:34294/0 (socket says 172.21.15.5:34294) 2024-03-24T10:45:10.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.332+0000 7f009156b640 1 -- 172.21.15.5:0/342379276 learned_addr learned my addr 172.21.15.5:0/342379276 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:10.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.333+0000 7f009256d640 1 --2- 172.21.15.5:0/342379276 >> v2:172.21.15.5:3300/0 conn(0x7f008c15be90 0x7f008c0c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:10.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.336+0000 7f0090d6a640 1 -- 172.21.15.5:0/342379276 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2120325690 0 0) 0x7f008c152560 con 0x7f008c0c8c30 2024-03-24T10:45:10.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.336+0000 7f0090d6a640 1 -- 172.21.15.5:0/342379276 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0078003520 con 0x7f008c0c8c30 2024-03-24T10:45:10.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.336+0000 7f009256d640 1 -- 172.21.15.5:0/342379276 >> v1:172.21.15.57:6789/0 conn(0x7f008c0c8c30 legacy=0x7f008c160a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:10.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.336+0000 7f009256d640 1 -- 172.21.15.5:0/342379276 >> v1:172.21.15.5:6789/0 conn(0x7f008c0c92f0 legacy=0x7f008c15b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:10.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.336+0000 7f009256d640 1 -- 172.21.15.5:0/342379276 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f008c0b3b80 con 0x7f008c15be90 2024-03-24T10:45:10.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.340+0000 7f009256d640 1 --2- 172.21.15.5:0/342379276 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f008c15be90 0x7f008c0c96d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0084009d90 tx=0x7f008402bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42842b3ddc56431a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:10.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.344+0000 7f0090d6a640 1 -- 172.21.15.5:0/342379276 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008402cd80 con 0x7f008c15be90 2024-03-24T10:45:10.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.344+0000 7f0090d6a640 1 -- 172.21.15.5:0/342379276 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0084034440 con 0x7f008c15be90 2024-03-24T10:45:10.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.344+0000 7f0090d6a640 1 -- 172.21.15.5:0/342379276 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00840347d0 con 0x7f008c15be90 2024-03-24T10:45:10.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.345+0000 7f0093fd0640 1 -- 172.21.15.5:0/342379276 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f008c15be90 msgr2=0x7f008c0c96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:10.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.345+0000 7f0093fd0640 1 --2- 172.21.15.5:0/342379276 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f008c15be90 0x7f008c0c96d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0084009d90 tx=0x7f008402bfd0 comp rx=0 tx=0).stop 2024-03-24T10:45:10.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.345+0000 7f0093fd0640 1 -- 172.21.15.5:0/342379276 shutdown_connections 2024-03-24T10:45:10.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.345+0000 7f0093fd0640 1 --2- 172.21.15.5:0/342379276 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f008c15be90 0x7f008c0c96d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:10.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.345+0000 7f0093fd0640 1 -- 172.21.15.5:0/342379276 >> 172.21.15.5:0/342379276 conn(0x7f008c06f820 msgr2=0x7f008c061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:10.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.345+0000 7f0093fd0640 1 -- 172.21.15.5:0/342379276 shutdown_connections 2024-03-24T10:45:10.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.345+0000 7f0093fd0640 1 -- 172.21.15.5:0/342379276 wait complete. 2024-03-24T10:45:10.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.346+0000 7f0093fd0640 1 Processor -- start 2024-03-24T10:45:10.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.346+0000 7f0093fd0640 1 -- start start 2024-03-24T10:45:10.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.347+0000 7f0093fd0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f008c0c8c30 0x7f008c1f3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:10.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.347+0000 7f0093fd0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f008c0c92f0 0x7f008c0c3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:10.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.347+0000 7f0093fd0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f008c15be90 0x7f008c0c3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:10.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.347+0000 7f0093fd0640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f008c06daf0 con 0x7f008c0c92f0 2024-03-24T10:45:10.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.347+0000 7f0093fd0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f008c1f43b0 con 0x7f008c15be90 2024-03-24T10:45:10.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.347+0000 7f0093fd0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f008c06dd40 con 0x7f008c0c8c30 2024-03-24T10:45:10.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.347+0000 7f009156b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f008c0c92f0 0x7f008c0c3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:10.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.348+0000 7f009256d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f008c15be90 0x7f008c0c3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:10.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.348+0000 7f0091d6c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f008c0c8c30 0x7f008c1f3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:10.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.348+0000 7f009256d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f008c15be90 0x7f008c0c3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:34722/0 (socket says 172.21.15.5:34722) 2024-03-24T10:45:10.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.348+0000 7f009156b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f008c0c92f0 0x7f008c0c3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:34468/0 (socket says 172.21.15.5:34468) 2024-03-24T10:45:10.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.348+0000 7f009256d640 1 -- 172.21.15.5:0/3529757813 learned_addr learned my addr 172.21.15.5:0/3529757813 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:10.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.352+0000 7f009256d640 1 -- 172.21.15.5:0/3529757813 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f008c0c8c30 msgr2=0x7f008c1f3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:10.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.352+0000 7f009256d640 1 --2- 172.21.15.5:0/3529757813 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f008c0c8c30 0x7f008c1f3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:10.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.352+0000 7f009256d640 1 -- 172.21.15.5:0/3529757813 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f008c0c92f0 msgr2=0x7f008c0c3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:10.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.352+0000 7f009256d640 1 --2- 172.21.15.5:0/3529757813 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f008c0c92f0 0x7f008c0c3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:10.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.352+0000 7f009256d640 1 -- 172.21.15.5:0/3529757813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0084009a70 con 0x7f008c15be90 2024-03-24T10:45:10.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.355+0000 7f009256d640 1 --2- 172.21.15.5:0/3529757813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f008c15be90 0x7f008c0c3a70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0084009d90 tx=0x7f0084004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:10.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.360+0000 7f00827fc640 1 -- 172.21.15.5:0/3529757813 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084040650 con 0x7f008c15be90 2024-03-24T10:45:10.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.360+0000 7f00827fc640 1 -- 172.21.15.5:0/3529757813 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00840407b0 con 0x7f008c15be90 2024-03-24T10:45:10.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.360+0000 7f00827fc640 1 -- 172.21.15.5:0/3529757813 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0084040b60 con 0x7f008c15be90 2024-03-24T10:45:10.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.360+0000 7f0093fd0640 1 -- 172.21.15.5:0/3529757813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f008c06e020 con 0x7f008c15be90 2024-03-24T10:45:10.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.360+0000 7f0093fd0640 1 -- 172.21.15.5:0/3529757813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f008c1f3950 con 0x7f008c15be90 2024-03-24T10:45:10.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.362+0000 7f0093fd0640 1 -- 172.21.15.5:0/3529757813 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f008c06e210 con 0x7f008c15be90 2024-03-24T10:45:10.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.381+0000 7f00827fc640 1 -- 172.21.15.5:0/3529757813 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0084040cc0 con 0x7f008c15be90 2024-03-24T10:45:10.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.382+0000 7f00827fc640 1 --2- 172.21.15.5:0/3529757813 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f007006b860 0x7f007006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:10.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.382+0000 7f0091d6c640 1 --2- 172.21.15.5:0/3529757813 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f007006b860 0x7f007006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:10.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.382+0000 7f0091d6c640 1 --2- 172.21.15.5:0/3529757813 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f007006b860 0x7f007006dd20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0088001780 tx=0x7f0088009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:10.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.385+0000 7f00827fc640 1 -- 172.21.15.5:0/3529757813 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f008407bb70 con 0x7f008c15be90 2024-03-24T10:45:10.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.669+0000 7f00827fc640 1 -- 172.21.15.5:0/3529757813 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0084077e10 con 0x7f008c15be90 2024-03-24T10:45:10.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.846+0000 7f0093fd0640 1 -- 172.21.15.5:0/3529757813 --> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f008c1587f0 con 0x7f007006b860 2024-03-24T10:45:10.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.853+0000 7f00827fc640 1 -- 172.21.15.5:0/3529757813 <== mgr.4120 v2:172.21.15.5:6832/1102439181 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251988 (secure 0 0 0) 0x7f0088001420 con 0x7f007006b860 2024-03-24T10:45:10.856 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:45:10.858 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-03-24T10:45:10.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.856+0000 7f0093fd0640 1 -- 172.21.15.5:0/3529757813 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f007006b860 msgr2=0x7f007006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:10.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.856+0000 7f0093fd0640 1 --2- 172.21.15.5:0/3529757813 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f007006b860 0x7f007006dd20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0088001780 tx=0x7f0088009040 comp rx=0 tx=0).stop 2024-03-24T10:45:10.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.856+0000 7f0093fd0640 1 -- 172.21.15.5:0/3529757813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f008c15be90 msgr2=0x7f008c0c3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:10.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.856+0000 7f0093fd0640 1 --2- 172.21.15.5:0/3529757813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f008c15be90 0x7f008c0c3a70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0084009d90 tx=0x7f0084004a40 comp rx=0 tx=0).stop 2024-03-24T10:45:10.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.856+0000 7f0093fd0640 1 -- 172.21.15.5:0/3529757813 shutdown_connections 2024-03-24T10:45:10.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.856+0000 7f0093fd0640 1 --2- 172.21.15.5:0/3529757813 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f007006b860 0x7f007006dd20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:10.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.856+0000 7f0093fd0640 1 --2- 172.21.15.5:0/3529757813 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f008c0c92f0 0x7f008c0c3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:10.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.856+0000 7f0093fd0640 1 --2- 172.21.15.5:0/3529757813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f008c15be90 0x7f008c0c3a70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:10.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.856+0000 7f0093fd0640 1 --2- 172.21.15.5:0/3529757813 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f008c0c8c30 0x7f008c1f3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:10.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.856+0000 7f0093fd0640 1 -- 172.21.15.5:0/3529757813 >> 172.21.15.5:0/3529757813 conn(0x7f008c06f820 msgr2=0x7f008c0711d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:10.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.856+0000 7f0093fd0640 1 -- 172.21.15.5:0/3529757813 shutdown_connections 2024-03-24T10:45:10.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:10.856+0000 7f0093fd0640 1 -- 172.21.15.5:0/3529757813 wait complete. 2024-03-24T10:45:10.867 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-03-24T10:45:09.371678+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592868,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":217,"num_write_kb":2431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165,"ondisk_log_size":165,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":234676,"kb_used_data":1864,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754740044,"statfs":{"total":773094113280,"available":772853805056,"internally_reserved":0,"allocated":1908736,"data_stored":1779308,"data_compressed":65430,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2500,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001555"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956091+0000","last_change":"2024-03-24T10:45:01.894006+0000","last_active":"2024-03-24T10:45:02.956091+0000","last_peered":"2024-03-24T10:45:02.956091+0000","last_clean":"2024-03-24T10:45:02.956091+0000","last_became_active":"2024-03-24T10:45:01.893734+0000","last_became_peered":"2024-03-24T10:45:01.893734+0000","last_unstale":"2024-03-24T10:45:02.956091+0000","last_undegraded":"2024-03-24T10:45:02.956091+0000","last_fullsized":"2024-03-24T10:45:02.956091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:05:51.579388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.932998+0000","last_change":"2024-03-24T10:45:01.931707+0000","last_active":"2024-03-24T10:45:02.932998+0000","last_peered":"2024-03-24T10:45:02.932998+0000","last_clean":"2024-03-24T10:45:02.932998+0000","last_became_active":"2024-03-24T10:45:01.931449+0000","last_became_peered":"2024-03-24T10:45:01.931449+0000","last_unstale":"2024-03-24T10:45:02.932998+0000","last_undegraded":"2024-03-24T10:45:02.932998+0000","last_fullsized":"2024-03-24T10:45:02.932998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953238+0000","last_change":"2024-03-24T10:45:01.895929+0000","last_active":"2024-03-24T10:45:02.953238+0000","last_peered":"2024-03-24T10:45:02.953238+0000","last_clean":"2024-03-24T10:45:02.953238+0000","last_became_active":"2024-03-24T10:45:01.895654+0000","last_became_peered":"2024-03-24T10:45:01.895654+0000","last_unstale":"2024-03-24T10:45:02.953238+0000","last_undegraded":"2024-03-24T10:45:02.953238+0000","last_fullsized":"2024-03-24T10:45:02.953238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954093+0000","last_change":"2024-03-24T10:45:01.952468+0000","last_active":"2024-03-24T10:45:02.954093+0000","last_peered":"2024-03-24T10:45:02.954093+0000","last_clean":"2024-03-24T10:45:02.954093+0000","last_became_active":"2024-03-24T10:45:01.952219+0000","last_became_peered":"2024-03-24T10:45:01.952219+0000","last_unstale":"2024-03-24T10:45:02.954093+0000","last_undegraded":"2024-03-24T10:45:02.954093+0000","last_fullsized":"2024-03-24T10:45:02.954093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:27:12.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955739+0000","last_change":"2024-03-24T10:45:01.895885+0000","last_active":"2024-03-24T10:45:02.955739+0000","last_peered":"2024-03-24T10:45:02.955739+0000","last_clean":"2024-03-24T10:45:02.955739+0000","last_became_active":"2024-03-24T10:45:01.895622+0000","last_became_peered":"2024-03-24T10:45:01.895622+0000","last_unstale":"2024-03-24T10:45:02.955739+0000","last_undegraded":"2024-03-24T10:45:02.955739+0000","last_fullsized":"2024-03-24T10:45:02.955739+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:16.523177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954340+0000","last_change":"2024-03-24T10:45:01.958421+0000","last_active":"2024-03-24T10:45:02.954340+0000","last_peered":"2024-03-24T10:45:02.954340+0000","last_clean":"2024-03-24T10:45:02.954340+0000","last_became_active":"2024-03-24T10:45:01.957972+0000","last_became_peered":"2024-03-24T10:45:01.957972+0000","last_unstale":"2024-03-24T10:45:02.954340+0000","last_undegraded":"2024-03-24T10:45:02.954340+0000","last_fullsized":"2024-03-24T10:45:02.954340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:16.523177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955433+0000","last_change":"2024-03-24T10:45:01.892745+0000","last_active":"2024-03-24T10:45:02.955433+0000","last_peered":"2024-03-24T10:45:02.955433+0000","last_clean":"2024-03-24T10:45:02.955433+0000","last_became_active":"2024-03-24T10:45:01.892452+0000","last_became_peered":"2024-03-24T10:45:01.892452+0000","last_unstale":"2024-03-24T10:45:02.955433+0000","last_undegraded":"2024-03-24T10:45:02.955433+0000","last_fullsized":"2024-03-24T10:45:02.955433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954102+0000","last_change":"2024-03-24T10:45:01.953432+0000","last_active":"2024-03-24T10:45:02.954102+0000","last_peered":"2024-03-24T10:45:02.954102+0000","last_clean":"2024-03-24T10:45:02.954102+0000","last_became_active":"2024-03-24T10:45:01.953149+0000","last_became_peered":"2024-03-24T10:45:01.953149+0000","last_unstale":"2024-03-24T10:45:02.954102+0000","last_undegraded":"2024-03-24T10:45:02.954102+0000","last_fullsized":"2024-03-24T10:45:02.954102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956172+0000","last_change":"2024-03-24T10:45:01.956488+0000","last_active":"2024-03-24T10:45:02.956172+0000","last_peered":"2024-03-24T10:45:02.956172+0000","last_clean":"2024-03-24T10:45:02.956172+0000","last_became_active":"2024-03-24T10:45:01.956207+0000","last_became_peered":"2024-03-24T10:45:01.956207+0000","last_unstale":"2024-03-24T10:45:02.956172+0000","last_undegraded":"2024-03-24T10:45:02.956172+0000","last_fullsized":"2024-03-24T10:45:02.956172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:23:26.523809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953720+0000","last_change":"2024-03-24T10:45:01.956484+0000","last_active":"2024-03-24T10:45:02.953720+0000","last_peered":"2024-03-24T10:45:02.953720+0000","last_clean":"2024-03-24T10:45:02.953720+0000","last_became_active":"2024-03-24T10:45:01.956209+0000","last_became_peered":"2024-03-24T10:45:01.956209+0000","last_unstale":"2024-03-24T10:45:02.953720+0000","last_undegraded":"2024-03-24T10:45:02.953720+0000","last_fullsized":"2024-03-24T10:45:02.953720+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953838+0000","last_change":"2024-03-24T10:45:01.957103+0000","last_active":"2024-03-24T10:45:02.953838+0000","last_peered":"2024-03-24T10:45:02.953838+0000","last_clean":"2024-03-24T10:45:02.953838+0000","last_became_active":"2024-03-24T10:45:01.956849+0000","last_became_peered":"2024-03-24T10:45:01.956849+0000","last_unstale":"2024-03-24T10:45:02.953838+0000","last_undegraded":"2024-03-24T10:45:02.953838+0000","last_fullsized":"2024-03-24T10:45:02.953838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954691+0000","last_change":"2024-03-24T10:45:01.896701+0000","last_active":"2024-03-24T10:45:02.954691+0000","last_peered":"2024-03-24T10:45:02.954691+0000","last_clean":"2024-03-24T10:45:02.954691+0000","last_became_active":"2024-03-24T10:45:01.896400+0000","last_became_peered":"2024-03-24T10:45:01.896400+0000","last_unstale":"2024-03-24T10:45:02.954691+0000","last_undegraded":"2024-03-24T10:45:02.954691+0000","last_fullsized":"2024-03-24T10:45:02.954691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:44.380352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953999+0000","last_change":"2024-03-24T10:45:01.894954+0000","last_active":"2024-03-24T10:45:02.953999+0000","last_peered":"2024-03-24T10:45:02.953999+0000","last_clean":"2024-03-24T10:45:02.953999+0000","last_became_active":"2024-03-24T10:45:01.894632+0000","last_became_peered":"2024-03-24T10:45:01.894632+0000","last_unstale":"2024-03-24T10:45:02.953999+0000","last_undegraded":"2024-03-24T10:45:02.953999+0000","last_fullsized":"2024-03-24T10:45:02.953999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.952924+0000","last_change":"2024-03-24T10:45:01.929588+0000","last_active":"2024-03-24T10:45:02.952924+0000","last_peered":"2024-03-24T10:45:02.952924+0000","last_clean":"2024-03-24T10:45:02.952924+0000","last_became_active":"2024-03-24T10:45:01.928460+0000","last_became_peered":"2024-03-24T10:45:01.928460+0000","last_unstale":"2024-03-24T10:45:02.952924+0000","last_undegraded":"2024-03-24T10:45:02.952924+0000","last_fullsized":"2024-03-24T10:45:02.952924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:47:48.896914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933000+0000","last_change":"2024-03-24T10:45:01.928721+0000","last_active":"2024-03-24T10:45:02.933000+0000","last_peered":"2024-03-24T10:45:02.933000+0000","last_clean":"2024-03-24T10:45:02.933000+0000","last_became_active":"2024-03-24T10:45:01.928467+0000","last_became_peered":"2024-03-24T10:45:01.928467+0000","last_unstale":"2024-03-24T10:45:02.933000+0000","last_undegraded":"2024-03-24T10:45:02.933000+0000","last_fullsized":"2024-03-24T10:45:02.933000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953333+0000","last_change":"2024-03-24T10:45:01.956030+0000","last_active":"2024-03-24T10:45:02.953333+0000","last_peered":"2024-03-24T10:45:02.953333+0000","last_clean":"2024-03-24T10:45:02.953333+0000","last_became_active":"2024-03-24T10:45:01.955768+0000","last_became_peered":"2024-03-24T10:45:01.955768+0000","last_unstale":"2024-03-24T10:45:02.953333+0000","last_undegraded":"2024-03-24T10:45:02.953333+0000","last_fullsized":"2024-03-24T10:45:02.953333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954695+0000","last_change":"2024-03-24T10:45:01.956181+0000","last_active":"2024-03-24T10:45:02.954695+0000","last_peered":"2024-03-24T10:45:02.954695+0000","last_clean":"2024-03-24T10:45:02.954695+0000","last_became_active":"2024-03-24T10:45:01.955890+0000","last_became_peered":"2024-03-24T10:45:01.955890+0000","last_unstale":"2024-03-24T10:45:02.954695+0000","last_undegraded":"2024-03-24T10:45:02.954695+0000","last_fullsized":"2024-03-24T10:45:02.954695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:15:11.107915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.936626+0000","last_change":"2024-03-24T10:45:01.896663+0000","last_active":"2024-03-24T10:45:02.936626+0000","last_peered":"2024-03-24T10:45:02.936626+0000","last_clean":"2024-03-24T10:45:02.936626+0000","last_became_active":"2024-03-24T10:45:01.896391+0000","last_became_peered":"2024-03-24T10:45:01.896391+0000","last_unstale":"2024-03-24T10:45:02.936626+0000","last_undegraded":"2024-03-24T10:45:02.936626+0000","last_fullsized":"2024-03-24T10:45:02.936626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:30.942758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956053+0000","last_change":"2024-03-24T10:45:01.895951+0000","last_active":"2024-03-24T10:45:02.956053+0000","last_peered":"2024-03-24T10:45:02.956053+0000","last_clean":"2024-03-24T10:45:02.956053+0000","last_became_active":"2024-03-24T10:45:01.895668+0000","last_became_peered":"2024-03-24T10:45:01.895668+0000","last_unstale":"2024-03-24T10:45:02.956053+0000","last_undegraded":"2024-03-24T10:45:02.956053+0000","last_fullsized":"2024-03-24T10:45:02.956053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:56:43.462458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953964+0000","last_change":"2024-03-24T10:45:01.898126+0000","last_active":"2024-03-24T10:45:02.953964+0000","last_peered":"2024-03-24T10:45:02.953964+0000","last_clean":"2024-03-24T10:45:02.953964+0000","last_became_active":"2024-03-24T10:45:01.897878+0000","last_became_peered":"2024-03-24T10:45:01.897878+0000","last_unstale":"2024-03-24T10:45:02.953964+0000","last_undegraded":"2024-03-24T10:45:02.953964+0000","last_fullsized":"2024-03-24T10:45:02.953964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953247+0000","last_change":"2024-03-24T10:45:01.896109+0000","last_active":"2024-03-24T10:45:02.953247+0000","last_peered":"2024-03-24T10:45:02.953247+0000","last_clean":"2024-03-24T10:45:02.953247+0000","last_became_active":"2024-03-24T10:45:01.895696+0000","last_became_peered":"2024-03-24T10:45:01.895696+0000","last_unstale":"2024-03-24T10:45:02.953247+0000","last_undegraded":"2024-03-24T10:45:02.953247+0000","last_fullsized":"2024-03-24T10:45:02.953247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954483+0000","last_change":"2024-03-24T10:45:01.958119+0000","last_active":"2024-03-24T10:45:02.954483+0000","last_peered":"2024-03-24T10:45:02.954483+0000","last_clean":"2024-03-24T10:45:02.954483+0000","last_became_active":"2024-03-24T10:45:01.957832+0000","last_became_peered":"2024-03-24T10:45:01.957832+0000","last_unstale":"2024-03-24T10:45:02.954483+0000","last_undegraded":"2024-03-24T10:45:02.954483+0000","last_fullsized":"2024-03-24T10:45:02.954483+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:12.476490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953638+0000","last_change":"2024-03-24T10:45:01.842442+0000","last_active":"2024-03-24T10:45:02.953638+0000","last_peered":"2024-03-24T10:45:02.953638+0000","last_clean":"2024-03-24T10:45:02.953638+0000","last_became_active":"2024-03-24T10:45:01.842199+0000","last_became_peered":"2024-03-24T10:45:01.842199+0000","last_unstale":"2024-03-24T10:45:02.953638+0000","last_undegraded":"2024-03-24T10:45:02.953638+0000","last_fullsized":"2024-03-24T10:45:02.953638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953764+0000","last_change":"2024-03-24T10:45:01.952378+0000","last_active":"2024-03-24T10:45:02.953764+0000","last_peered":"2024-03-24T10:45:02.953764+0000","last_clean":"2024-03-24T10:45:02.953764+0000","last_became_active":"2024-03-24T10:45:01.952123+0000","last_became_peered":"2024-03-24T10:45:01.952123+0000","last_unstale":"2024-03-24T10:45:02.953764+0000","last_undegraded":"2024-03-24T10:45:02.953764+0000","last_fullsized":"2024-03-24T10:45:02.953764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954628+0000","last_change":"2024-03-24T10:45:01.842299+0000","last_active":"2024-03-24T10:45:02.954628+0000","last_peered":"2024-03-24T10:45:02.954628+0000","last_clean":"2024-03-24T10:45:02.954628+0000","last_became_active":"2024-03-24T10:45:01.842097+0000","last_became_peered":"2024-03-24T10:45:01.842097+0000","last_unstale":"2024-03-24T10:45:02.954628+0000","last_undegraded":"2024-03-24T10:45:02.954628+0000","last_fullsized":"2024-03-24T10:45:02.954628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:16.341360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953886+0000","last_change":"2024-03-24T10:45:01.893579+0000","last_active":"2024-03-24T10:45:02.953886+0000","last_peered":"2024-03-24T10:45:02.953886+0000","last_clean":"2024-03-24T10:45:02.953886+0000","last_became_active":"2024-03-24T10:45:01.893273+0000","last_became_peered":"2024-03-24T10:45:01.893273+0000","last_unstale":"2024-03-24T10:45:02.953886+0000","last_undegraded":"2024-03-24T10:45:02.953886+0000","last_fullsized":"2024-03-24T10:45:02.953886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933308+0000","last_change":"2024-03-24T10:45:01.930032+0000","last_active":"2024-03-24T10:45:02.933308+0000","last_peered":"2024-03-24T10:45:02.933308+0000","last_clean":"2024-03-24T10:45:02.933308+0000","last_became_active":"2024-03-24T10:45:01.929770+0000","last_became_peered":"2024-03-24T10:45:01.929770+0000","last_unstale":"2024-03-24T10:45:02.933308+0000","last_undegraded":"2024-03-24T10:45:02.933308+0000","last_fullsized":"2024-03-24T10:45:02.933308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954962+0000","last_change":"2024-03-24T10:45:01.842460+0000","last_active":"2024-03-24T10:45:02.954962+0000","last_peered":"2024-03-24T10:45:02.954962+0000","last_clean":"2024-03-24T10:45:02.954962+0000","last_became_active":"2024-03-24T10:45:01.842165+0000","last_became_peered":"2024-03-24T10:45:01.842165+0000","last_unstale":"2024-03-24T10:45:02.954962+0000","last_undegraded":"2024-03-24T10:45:02.954962+0000","last_fullsized":"2024-03-24T10:45:02.954962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953401+0000","last_change":"2024-03-24T10:45:01.895868+0000","last_active":"2024-03-24T10:45:02.953401+0000","last_peered":"2024-03-24T10:45:02.953401+0000","last_clean":"2024-03-24T10:45:02.953401+0000","last_became_active":"2024-03-24T10:45:01.895290+0000","last_became_peered":"2024-03-24T10:45:01.895290+0000","last_unstale":"2024-03-24T10:45:02.953401+0000","last_undegraded":"2024-03-24T10:45:02.953401+0000","last_fullsized":"2024-03-24T10:45:02.953401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954615+0000","last_change":"2024-03-24T10:45:01.957724+0000","last_active":"2024-03-24T10:45:02.954615+0000","last_peered":"2024-03-24T10:45:02.954615+0000","last_clean":"2024-03-24T10:45:02.954615+0000","last_became_active":"2024-03-24T10:45:01.957429+0000","last_became_peered":"2024-03-24T10:45:01.957429+0000","last_unstale":"2024-03-24T10:45:02.954615+0000","last_undegraded":"2024-03-24T10:45:02.954615+0000","last_fullsized":"2024-03-24T10:45:02.954615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:22.089070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.936465+0000","last_change":"2024-03-24T10:45:01.952159+0000","last_active":"2024-03-24T10:45:02.936465+0000","last_peered":"2024-03-24T10:45:02.936465+0000","last_clean":"2024-03-24T10:45:02.936465+0000","last_became_active":"2024-03-24T10:45:01.951961+0000","last_became_peered":"2024-03-24T10:45:01.951961+0000","last_unstale":"2024-03-24T10:45:02.936465+0000","last_undegraded":"2024-03-24T10:45:02.936465+0000","last_fullsized":"2024-03-24T10:45:02.936465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:44.380352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953610+0000","last_change":"2024-03-24T10:45:01.893452+0000","last_active":"2024-03-24T10:45:02.953610+0000","last_peered":"2024-03-24T10:45:02.953610+0000","last_clean":"2024-03-24T10:45:02.953610+0000","last_became_active":"2024-03-24T10:45:01.893139+0000","last_became_peered":"2024-03-24T10:45:01.893139+0000","last_unstale":"2024-03-24T10:45:02.953610+0000","last_undegraded":"2024-03-24T10:45:02.953610+0000","last_fullsized":"2024-03-24T10:45:02.953610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953542+0000","last_change":"2024-03-24T10:45:01.928827+0000","last_active":"2024-03-24T10:45:02.953542+0000","last_peered":"2024-03-24T10:45:02.953542+0000","last_clean":"2024-03-24T10:45:02.953542+0000","last_became_active":"2024-03-24T10:45:01.928521+0000","last_became_peered":"2024-03-24T10:45:01.928521+0000","last_unstale":"2024-03-24T10:45:02.953542+0000","last_undegraded":"2024-03-24T10:45:02.953542+0000","last_fullsized":"2024-03-24T10:45:02.953542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:59.643246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954243+0000","last_change":"2024-03-24T10:45:00.637993+0000","last_active":"2024-03-24T10:45:02.954243+0000","last_peered":"2024-03-24T10:45:02.954243+0000","last_clean":"2024-03-24T10:45:02.954243+0000","last_became_active":"2024-03-24T10:45:00.637803+0000","last_became_peered":"2024-03-24T10:45:00.637803+0000","last_unstale":"2024-03-24T10:45:02.954243+0000","last_undegraded":"2024-03-24T10:45:02.954243+0000","last_fullsized":"2024-03-24T10:45:02.954243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:28:28.854541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953550+0000","last_change":"2024-03-24T10:45:01.897686+0000","last_active":"2024-03-24T10:45:02.953550+0000","last_peered":"2024-03-24T10:45:02.953550+0000","last_clean":"2024-03-24T10:45:02.953550+0000","last_became_active":"2024-03-24T10:45:01.897320+0000","last_became_peered":"2024-03-24T10:45:01.897320+0000","last_unstale":"2024-03-24T10:45:02.953550+0000","last_undegraded":"2024-03-24T10:45:02.953550+0000","last_fullsized":"2024-03-24T10:45:02.953550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:03.132133+0000","last_change":"2024-03-24T10:45:00.638329+0000","last_active":"2024-03-24T10:45:03.132133+0000","last_peered":"2024-03-24T10:45:03.132133+0000","last_clean":"2024-03-24T10:45:03.132133+0000","last_became_active":"2024-03-24T10:45:00.638006+0000","last_became_peered":"2024-03-24T10:45:00.638006+0000","last_unstale":"2024-03-24T10:45:03.132133+0000","last_undegraded":"2024-03-24T10:45:03.132133+0000","last_fullsized":"2024-03-24T10:45:03.132133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953780+0000","last_change":"2024-03-24T10:45:00.633703+0000","last_active":"2024-03-24T10:45:02.953780+0000","last_peered":"2024-03-24T10:45:02.953780+0000","last_clean":"2024-03-24T10:45:02.953780+0000","last_became_active":"2024-03-24T10:45:00.633427+0000","last_became_peered":"2024-03-24T10:45:00.633427+0000","last_unstale":"2024-03-24T10:45:02.953780+0000","last_undegraded":"2024-03-24T10:45:02.953780+0000","last_fullsized":"2024-03-24T10:45:02.953780+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:30.350191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954240+0000","last_change":"2024-03-24T10:45:01.893902+0000","last_active":"2024-03-24T10:45:02.954240+0000","last_peered":"2024-03-24T10:45:02.954240+0000","last_clean":"2024-03-24T10:45:02.954240+0000","last_became_active":"2024-03-24T10:45:01.893617+0000","last_became_peered":"2024-03-24T10:45:01.893617+0000","last_unstale":"2024-03-24T10:45:02.954240+0000","last_undegraded":"2024-03-24T10:45:02.954240+0000","last_fullsized":"2024-03-24T10:45:02.954240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953934+0000","last_change":"2024-03-24T10:45:00.638778+0000","last_active":"2024-03-24T10:45:02.953934+0000","last_peered":"2024-03-24T10:45:02.953934+0000","last_clean":"2024-03-24T10:45:02.953934+0000","last_became_active":"2024-03-24T10:45:00.638089+0000","last_became_peered":"2024-03-24T10:45:00.638089+0000","last_unstale":"2024-03-24T10:45:02.953934+0000","last_undegraded":"2024-03-24T10:45:02.953934+0000","last_fullsized":"2024-03-24T10:45:02.953934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.932839+0000","last_change":"2024-03-24T10:45:01.930063+0000","last_active":"2024-03-24T10:45:02.932839+0000","last_peered":"2024-03-24T10:45:02.932839+0000","last_clean":"2024-03-24T10:45:02.932839+0000","last_became_active":"2024-03-24T10:45:01.929786+0000","last_became_peered":"2024-03-24T10:45:01.929786+0000","last_unstale":"2024-03-24T10:45:02.932839+0000","last_undegraded":"2024-03-24T10:45:02.932839+0000","last_fullsized":"2024-03-24T10:45:02.932839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.773089+0000","last_change":"2024-03-24T10:45:00.639822+0000","last_active":"2024-03-24T10:45:07.773089+0000","last_peered":"2024-03-24T10:45:07.773089+0000","last_clean":"2024-03-24T10:45:07.773089+0000","last_became_active":"2024-03-24T10:45:00.639600+0000","last_became_peered":"2024-03-24T10:45:00.639600+0000","last_unstale":"2024-03-24T10:45:07.773089+0000","last_undegraded":"2024-03-24T10:45:07.773089+0000","last_fullsized":"2024-03-24T10:45:07.773089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953735+0000","last_change":"2024-03-24T10:45:01.954015+0000","last_active":"2024-03-24T10:45:02.953735+0000","last_peered":"2024-03-24T10:45:02.953735+0000","last_clean":"2024-03-24T10:45:02.953735+0000","last_became_active":"2024-03-24T10:45:01.953767+0000","last_became_peered":"2024-03-24T10:45:01.953767+0000","last_unstale":"2024-03-24T10:45:02.953735+0000","last_undegraded":"2024-03-24T10:45:02.953735+0000","last_fullsized":"2024-03-24T10:45:02.953735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954410+0000","last_change":"2024-03-24T10:45:00.636553+0000","last_active":"2024-03-24T10:45:02.954410+0000","last_peered":"2024-03-24T10:45:02.954410+0000","last_clean":"2024-03-24T10:45:02.954410+0000","last_became_active":"2024-03-24T10:45:00.636282+0000","last_became_peered":"2024-03-24T10:45:00.636282+0000","last_unstale":"2024-03-24T10:45:02.954410+0000","last_undegraded":"2024-03-24T10:45:02.954410+0000","last_fullsized":"2024-03-24T10:45:02.954410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953458+0000","last_change":"2024-03-24T10:45:01.957905+0000","last_active":"2024-03-24T10:45:02.953458+0000","last_peered":"2024-03-24T10:45:02.953458+0000","last_clean":"2024-03-24T10:45:02.953458+0000","last_became_active":"2024-03-24T10:45:01.957676+0000","last_became_peered":"2024-03-24T10:45:01.957676+0000","last_unstale":"2024-03-24T10:45:02.953458+0000","last_undegraded":"2024-03-24T10:45:02.953458+0000","last_fullsized":"2024-03-24T10:45:02.953458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:03.132459+0000","last_change":"2024-03-24T10:45:00.635197+0000","last_active":"2024-03-24T10:45:03.132459+0000","last_peered":"2024-03-24T10:45:03.132459+0000","last_clean":"2024-03-24T10:45:03.132459+0000","last_became_active":"2024-03-24T10:45:00.634989+0000","last_became_peered":"2024-03-24T10:45:00.634989+0000","last_unstale":"2024-03-24T10:45:03.132459+0000","last_undegraded":"2024-03-24T10:45:03.132459+0000","last_fullsized":"2024-03-24T10:45:03.132459+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954821+0000","last_change":"2024-03-24T10:45:01.853785+0000","last_active":"2024-03-24T10:45:02.954821+0000","last_peered":"2024-03-24T10:45:02.954821+0000","last_clean":"2024-03-24T10:45:02.954821+0000","last_became_active":"2024-03-24T10:45:01.853504+0000","last_became_peered":"2024-03-24T10:45:01.853504+0000","last_unstale":"2024-03-24T10:45:02.954821+0000","last_undegraded":"2024-03-24T10:45:02.954821+0000","last_fullsized":"2024-03-24T10:45:02.954821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:30.942758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.773017+0000","last_change":"2024-03-24T10:45:00.632381+0000","last_active":"2024-03-24T10:45:07.773017+0000","last_peered":"2024-03-24T10:45:07.773017+0000","last_clean":"2024-03-24T10:45:07.773017+0000","last_became_active":"2024-03-24T10:45:00.632108+0000","last_became_peered":"2024-03-24T10:45:00.632108+0000","last_unstale":"2024-03-24T10:45:07.773017+0000","last_undegraded":"2024-03-24T10:45:07.773017+0000","last_fullsized":"2024-03-24T10:45:07.773017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:06.142451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955746+0000","last_change":"2024-03-24T10:45:01.853591+0000","last_active":"2024-03-24T10:45:02.955746+0000","last_peered":"2024-03-24T10:45:02.955746+0000","last_clean":"2024-03-24T10:45:02.955746+0000","last_became_active":"2024-03-24T10:45:01.853279+0000","last_became_peered":"2024-03-24T10:45:01.853279+0000","last_unstale":"2024-03-24T10:45:02.955746+0000","last_undegraded":"2024-03-24T10:45:02.955746+0000","last_fullsized":"2024-03-24T10:45:02.955746+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:07:24.149961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.788598+0000","last_change":"2024-03-24T10:45:00.634192+0000","last_active":"2024-03-24T10:45:07.788598+0000","last_peered":"2024-03-24T10:45:07.788598+0000","last_clean":"2024-03-24T10:45:07.788598+0000","last_became_active":"2024-03-24T10:45:00.633893+0000","last_became_peered":"2024-03-24T10:45:00.633893+0000","last_unstale":"2024-03-24T10:45:07.788598+0000","last_undegraded":"2024-03-24T10:45:07.788598+0000","last_fullsized":"2024-03-24T10:45:07.788598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:30.350191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953075+0000","last_change":"2024-03-24T10:45:01.893426+0000","last_active":"2024-03-24T10:45:02.953075+0000","last_peered":"2024-03-24T10:45:02.953075+0000","last_clean":"2024-03-24T10:45:02.953075+0000","last_became_active":"2024-03-24T10:45:01.893178+0000","last_became_peered":"2024-03-24T10:45:01.893178+0000","last_unstale":"2024-03-24T10:45:02.953075+0000","last_undegraded":"2024-03-24T10:45:02.953075+0000","last_fullsized":"2024-03-24T10:45:02.953075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:13.887947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954766+0000","last_change":"2024-03-24T10:45:00.638214+0000","last_active":"2024-03-24T10:45:02.954766+0000","last_peered":"2024-03-24T10:45:02.954766+0000","last_clean":"2024-03-24T10:45:02.954766+0000","last_became_active":"2024-03-24T10:45:00.637973+0000","last_became_peered":"2024-03-24T10:45:00.637973+0000","last_unstale":"2024-03-24T10:45:02.954766+0000","last_undegraded":"2024-03-24T10:45:02.954766+0000","last_fullsized":"2024-03-24T10:45:02.954766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:15.749451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955767+0000","last_change":"2024-03-24T10:45:01.856756+0000","last_active":"2024-03-24T10:45:02.955767+0000","last_peered":"2024-03-24T10:45:02.955767+0000","last_clean":"2024-03-24T10:45:02.955767+0000","last_became_active":"2024-03-24T10:45:01.856482+0000","last_became_peered":"2024-03-24T10:45:01.856482+0000","last_unstale":"2024-03-24T10:45:02.955767+0000","last_undegraded":"2024-03-24T10:45:02.955767+0000","last_fullsized":"2024-03-24T10:45:02.955767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:12.476490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953326+0000","last_change":"2024-03-24T10:45:00.636481+0000","last_active":"2024-03-24T10:45:02.953326+0000","last_peered":"2024-03-24T10:45:02.953326+0000","last_clean":"2024-03-24T10:45:02.953326+0000","last_became_active":"2024-03-24T10:45:00.636206+0000","last_became_peered":"2024-03-24T10:45:00.636206+0000","last_unstale":"2024-03-24T10:45:02.953326+0000","last_undegraded":"2024-03-24T10:45:02.953326+0000","last_fullsized":"2024-03-24T10:45:02.953326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954153+0000","last_change":"2024-03-24T10:45:01.897279+0000","last_active":"2024-03-24T10:45:02.954153+0000","last_peered":"2024-03-24T10:45:02.954153+0000","last_clean":"2024-03-24T10:45:02.954153+0000","last_became_active":"2024-03-24T10:45:01.896736+0000","last_became_peered":"2024-03-24T10:45:01.896736+0000","last_unstale":"2024-03-24T10:45:02.954153+0000","last_undegraded":"2024-03-24T10:45:02.954153+0000","last_fullsized":"2024-03-24T10:45:02.954153+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:03.136552+0000","last_change":"2024-03-24T10:45:00.564066+0000","last_active":"2024-03-24T10:45:03.136552+0000","last_peered":"2024-03-24T10:45:03.136552+0000","last_clean":"2024-03-24T10:45:03.136552+0000","last_became_active":"2024-03-24T10:45:00.563812+0000","last_became_peered":"2024-03-24T10:45:00.563812+0000","last_unstale":"2024-03-24T10:45:03.136552+0000","last_undegraded":"2024-03-24T10:45:03.136552+0000","last_fullsized":"2024-03-24T10:45:03.136552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:28:28.854541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955401+0000","last_change":"2024-03-24T10:45:01.854429+0000","last_active":"2024-03-24T10:45:02.955401+0000","last_peered":"2024-03-24T10:45:02.955401+0000","last_clean":"2024-03-24T10:45:02.955401+0000","last_became_active":"2024-03-24T10:45:01.854179+0000","last_became_peered":"2024-03-24T10:45:01.854179+0000","last_unstale":"2024-03-24T10:45:02.955401+0000","last_undegraded":"2024-03-24T10:45:02.955401+0000","last_fullsized":"2024-03-24T10:45:02.955401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:03.132419+0000","last_change":"2024-03-24T10:45:00.634223+0000","last_active":"2024-03-24T10:45:03.132419+0000","last_peered":"2024-03-24T10:45:03.132419+0000","last_clean":"2024-03-24T10:45:03.132419+0000","last_became_active":"2024-03-24T10:45:00.633355+0000","last_became_peered":"2024-03-24T10:45:00.633355+0000","last_unstale":"2024-03-24T10:45:03.132419+0000","last_undegraded":"2024-03-24T10:45:03.132419+0000","last_fullsized":"2024-03-24T10:45:03.132419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:04.930115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956031+0000","last_change":"2024-03-24T10:45:01.856935+0000","last_active":"2024-03-24T10:45:02.956031+0000","last_peered":"2024-03-24T10:45:02.956031+0000","last_clean":"2024-03-24T10:45:02.956031+0000","last_became_active":"2024-03-24T10:45:01.856653+0000","last_became_peered":"2024-03-24T10:45:01.856653+0000","last_unstale":"2024-03-24T10:45:02.956031+0000","last_undegraded":"2024-03-24T10:45:02.956031+0000","last_fullsized":"2024-03-24T10:45:02.956031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:15:11.107915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:03.133907+0000","last_change":"2024-03-24T10:45:00.626484+0000","last_active":"2024-03-24T10:45:03.133907+0000","last_peered":"2024-03-24T10:45:03.133907+0000","last_clean":"2024-03-24T10:45:03.133907+0000","last_became_active":"2024-03-24T10:45:00.626195+0000","last_became_peered":"2024-03-24T10:45:00.626195+0000","last_unstale":"2024-03-24T10:45:03.133907+0000","last_undegraded":"2024-03-24T10:45:03.133907+0000","last_fullsized":"2024-03-24T10:45:03.133907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956227+0000","last_change":"2024-03-24T10:45:01.856995+0000","last_active":"2024-03-24T10:45:02.956227+0000","last_peered":"2024-03-24T10:45:02.956227+0000","last_clean":"2024-03-24T10:45:02.956227+0000","last_became_active":"2024-03-24T10:45:01.856531+0000","last_became_peered":"2024-03-24T10:45:01.856531+0000","last_unstale":"2024-03-24T10:45:02.956227+0000","last_undegraded":"2024-03-24T10:45:02.956227+0000","last_fullsized":"2024-03-24T10:45:02.956227+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:29:31.508911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933424+0000","last_change":"2024-03-24T10:45:00.628438+0000","last_active":"2024-03-24T10:45:02.933424+0000","last_peered":"2024-03-24T10:45:02.933424+0000","last_clean":"2024-03-24T10:45:02.933424+0000","last_became_active":"2024-03-24T10:45:00.628158+0000","last_became_peered":"2024-03-24T10:45:00.628158+0000","last_unstale":"2024-03-24T10:45:02.933424+0000","last_undegraded":"2024-03-24T10:45:02.933424+0000","last_fullsized":"2024-03-24T10:45:02.933424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:29.826202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954370+0000","last_change":"2024-03-24T10:45:01.951927+0000","last_active":"2024-03-24T10:45:02.954370+0000","last_peered":"2024-03-24T10:45:02.954370+0000","last_clean":"2024-03-24T10:45:02.954370+0000","last_became_active":"2024-03-24T10:45:01.951165+0000","last_became_peered":"2024-03-24T10:45:01.951165+0000","last_unstale":"2024-03-24T10:45:02.954370+0000","last_undegraded":"2024-03-24T10:45:02.954370+0000","last_fullsized":"2024-03-24T10:45:02.954370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:07:24.149961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.936632+0000","last_change":"2024-03-24T10:45:00.639093+0000","last_active":"2024-03-24T10:45:02.936632+0000","last_peered":"2024-03-24T10:45:02.936632+0000","last_clean":"2024-03-24T10:45:02.936632+0000","last_became_active":"2024-03-24T10:45:00.638785+0000","last_became_peered":"2024-03-24T10:45:00.638785+0000","last_unstale":"2024-03-24T10:45:02.936632+0000","last_undegraded":"2024-03-24T10:45:02.936632+0000","last_fullsized":"2024-03-24T10:45:02.936632+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955920+0000","last_change":"2024-03-24T10:45:01.953772+0000","last_active":"2024-03-24T10:45:02.955920+0000","last_peered":"2024-03-24T10:45:02.955920+0000","last_clean":"2024-03-24T10:45:02.955920+0000","last_became_active":"2024-03-24T10:45:01.953402+0000","last_became_peered":"2024-03-24T10:45:01.953402+0000","last_unstale":"2024-03-24T10:45:02.955920+0000","last_undegraded":"2024-03-24T10:45:02.955920+0000","last_fullsized":"2024-03-24T10:45:02.955920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:22.089070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955402+0000","last_change":"2024-03-24T10:44:48.241074+0000","last_active":"2024-03-24T10:45:02.955402+0000","last_peered":"2024-03-24T10:45:02.955402+0000","last_clean":"2024-03-24T10:45:02.955402+0000","last_became_active":"2024-03-24T10:44:48.239986+0000","last_became_peered":"2024-03-24T10:44:48.239986+0000","last_unstale":"2024-03-24T10:45:02.955402+0000","last_undegraded":"2024-03-24T10:45:02.955402+0000","last_fullsized":"2024-03-24T10:45:02.955402+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:46.142247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:46.142247+0000","last_clean_scrub_stamp":"2024-03-24T10:44:46.142247+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:05:50.110100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953599+0000","last_change":"2024-03-24T10:45:00.637276+0000","last_active":"2024-03-24T10:45:02.953599+0000","last_peered":"2024-03-24T10:45:02.953599+0000","last_clean":"2024-03-24T10:45:02.953599+0000","last_became_active":"2024-03-24T10:45:00.636734+0000","last_became_peered":"2024-03-24T10:45:00.636734+0000","last_unstale":"2024-03-24T10:45:02.953599+0000","last_undegraded":"2024-03-24T10:45:02.953599+0000","last_fullsized":"2024-03-24T10:45:02.953599+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954655+0000","last_change":"2024-03-24T10:45:01.958505+0000","last_active":"2024-03-24T10:45:02.954655+0000","last_peered":"2024-03-24T10:45:02.954655+0000","last_clean":"2024-03-24T10:45:02.954655+0000","last_became_active":"2024-03-24T10:45:01.958268+0000","last_became_peered":"2024-03-24T10:45:01.958268+0000","last_unstale":"2024-03-24T10:45:02.954655+0000","last_undegraded":"2024-03-24T10:45:02.954655+0000","last_fullsized":"2024-03-24T10:45:02.954655+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:27:12.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955286+0000","last_change":"2024-03-24T10:45:00.597856+0000","last_active":"2024-03-24T10:45:02.955286+0000","last_peered":"2024-03-24T10:45:02.955286+0000","last_clean":"2024-03-24T10:45:02.955286+0000","last_became_active":"2024-03-24T10:45:00.597640+0000","last_became_peered":"2024-03-24T10:45:00.597640+0000","last_unstale":"2024-03-24T10:45:02.955286+0000","last_undegraded":"2024-03-24T10:45:02.955286+0000","last_fullsized":"2024-03-24T10:45:02.955286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:29.826202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.952842+0000","last_change":"2024-03-24T10:45:01.897502+0000","last_active":"2024-03-24T10:45:02.952842+0000","last_peered":"2024-03-24T10:45:02.952842+0000","last_clean":"2024-03-24T10:45:02.952842+0000","last_became_active":"2024-03-24T10:45:01.897250+0000","last_became_peered":"2024-03-24T10:45:01.897250+0000","last_unstale":"2024-03-24T10:45:02.952842+0000","last_undegraded":"2024-03-24T10:45:02.952842+0000","last_fullsized":"2024-03-24T10:45:02.952842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:09:46.341672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.936461+0000","last_change":"2024-03-24T10:45:00.627166+0000","last_active":"2024-03-24T10:45:02.936461+0000","last_peered":"2024-03-24T10:45:02.936461+0000","last_clean":"2024-03-24T10:45:02.936461+0000","last_became_active":"2024-03-24T10:45:00.626881+0000","last_became_peered":"2024-03-24T10:45:00.626881+0000","last_unstale":"2024-03-24T10:45:02.936461+0000","last_undegraded":"2024-03-24T10:45:02.936461+0000","last_fullsized":"2024-03-24T10:45:02.936461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.932815+0000","last_change":"2024-03-24T10:45:01.930271+0000","last_active":"2024-03-24T10:45:02.932815+0000","last_peered":"2024-03-24T10:45:02.932815+0000","last_clean":"2024-03-24T10:45:02.932815+0000","last_became_active":"2024-03-24T10:45:01.929998+0000","last_became_peered":"2024-03-24T10:45:01.929998+0000","last_unstale":"2024-03-24T10:45:02.932815+0000","last_undegraded":"2024-03-24T10:45:02.932815+0000","last_fullsized":"2024-03-24T10:45:02.932815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954850+0000","last_change":"2024-03-24T10:45:00.638309+0000","last_active":"2024-03-24T10:45:02.954850+0000","last_peered":"2024-03-24T10:45:02.954850+0000","last_clean":"2024-03-24T10:45:02.954850+0000","last_became_active":"2024-03-24T10:45:00.638041+0000","last_became_peered":"2024-03-24T10:45:00.638041+0000","last_unstale":"2024-03-24T10:45:02.954850+0000","last_undegraded":"2024-03-24T10:45:02.954850+0000","last_fullsized":"2024-03-24T10:45:02.954850+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:14.049099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953823+0000","last_change":"2024-03-24T10:45:01.897722+0000","last_active":"2024-03-24T10:45:02.953823+0000","last_peered":"2024-03-24T10:45:02.953823+0000","last_clean":"2024-03-24T10:45:02.953823+0000","last_became_active":"2024-03-24T10:45:01.897159+0000","last_became_peered":"2024-03-24T10:45:01.897159+0000","last_unstale":"2024-03-24T10:45:02.953823+0000","last_undegraded":"2024-03-24T10:45:02.953823+0000","last_fullsized":"2024-03-24T10:45:02.953823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:42.293342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933199+0000","last_change":"2024-03-24T10:45:00.629267+0000","last_active":"2024-03-24T10:45:02.933199+0000","last_peered":"2024-03-24T10:45:02.933199+0000","last_clean":"2024-03-24T10:45:02.933199+0000","last_became_active":"2024-03-24T10:45:00.628971+0000","last_became_peered":"2024-03-24T10:45:00.628971+0000","last_unstale":"2024-03-24T10:45:02.933199+0000","last_undegraded":"2024-03-24T10:45:02.933199+0000","last_fullsized":"2024-03-24T10:45:02.933199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:41.176786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954115+0000","last_change":"2024-03-24T10:45:01.952527+0000","last_active":"2024-03-24T10:45:02.954115+0000","last_peered":"2024-03-24T10:45:02.954115+0000","last_clean":"2024-03-24T10:45:02.954115+0000","last_became_active":"2024-03-24T10:45:01.952192+0000","last_became_peered":"2024-03-24T10:45:01.952192+0000","last_unstale":"2024-03-24T10:45:02.954115+0000","last_undegraded":"2024-03-24T10:45:02.954115+0000","last_fullsized":"2024-03-24T10:45:02.954115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956004+0000","last_change":"2024-03-24T10:45:00.631941+0000","last_active":"2024-03-24T10:45:02.956004+0000","last_peered":"2024-03-24T10:45:02.956004+0000","last_clean":"2024-03-24T10:45:02.956004+0000","last_became_active":"2024-03-24T10:45:00.631670+0000","last_became_peered":"2024-03-24T10:45:00.631670+0000","last_unstale":"2024-03-24T10:45:02.956004+0000","last_undegraded":"2024-03-24T10:45:02.956004+0000","last_fullsized":"2024-03-24T10:45:02.956004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:09:45.225115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933116+0000","last_change":"2024-03-24T10:45:01.930483+0000","last_active":"2024-03-24T10:45:02.933116+0000","last_peered":"2024-03-24T10:45:02.933116+0000","last_clean":"2024-03-24T10:45:02.933116+0000","last_became_active":"2024-03-24T10:45:01.930220+0000","last_became_peered":"2024-03-24T10:45:01.930220+0000","last_unstale":"2024-03-24T10:45:02.933116+0000","last_undegraded":"2024-03-24T10:45:02.933116+0000","last_fullsized":"2024-03-24T10:45:02.933116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933234+0000","last_change":"2024-03-24T10:45:00.633650+0000","last_active":"2024-03-24T10:45:02.933234+0000","last_peered":"2024-03-24T10:45:02.933234+0000","last_clean":"2024-03-24T10:45:02.933234+0000","last_became_active":"2024-03-24T10:45:00.633329+0000","last_became_peered":"2024-03-24T10:45:00.633329+0000","last_unstale":"2024-03-24T10:45:02.933234+0000","last_undegraded":"2024-03-24T10:45:02.933234+0000","last_fullsized":"2024-03-24T10:45:02.933234+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954759+0000","last_change":"2024-03-24T10:45:01.853487+0000","last_active":"2024-03-24T10:45:02.954759+0000","last_peered":"2024-03-24T10:45:02.954759+0000","last_clean":"2024-03-24T10:45:02.954759+0000","last_became_active":"2024-03-24T10:45:01.853257+0000","last_became_peered":"2024-03-24T10:45:01.853257+0000","last_unstale":"2024-03-24T10:45:02.954759+0000","last_undegraded":"2024-03-24T10:45:02.954759+0000","last_fullsized":"2024-03-24T10:45:02.954759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.771497+0000","last_change":"2024-03-24T10:45:00.639282+0000","last_active":"2024-03-24T10:45:07.771497+0000","last_peered":"2024-03-24T10:45:07.771497+0000","last_clean":"2024-03-24T10:45:07.771497+0000","last_became_active":"2024-03-24T10:45:00.638999+0000","last_became_peered":"2024-03-24T10:45:00.638999+0000","last_unstale":"2024-03-24T10:45:07.771497+0000","last_undegraded":"2024-03-24T10:45:07.771497+0000","last_fullsized":"2024-03-24T10:45:07.771497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:41.176786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953535+0000","last_change":"2024-03-24T10:45:01.928900+0000","last_active":"2024-03-24T10:45:02.953535+0000","last_peered":"2024-03-24T10:45:02.953535+0000","last_clean":"2024-03-24T10:45:02.953535+0000","last_became_active":"2024-03-24T10:45:01.928610+0000","last_became_peered":"2024-03-24T10:45:01.928610+0000","last_unstale":"2024-03-24T10:45:02.953535+0000","last_undegraded":"2024-03-24T10:45:02.953535+0000","last_fullsized":"2024-03-24T10:45:02.953535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955286+0000","last_change":"2024-03-24T10:45:00.631327+0000","last_active":"2024-03-24T10:45:02.955286+0000","last_peered":"2024-03-24T10:45:02.955286+0000","last_clean":"2024-03-24T10:45:02.955286+0000","last_became_active":"2024-03-24T10:45:00.631078+0000","last_became_peered":"2024-03-24T10:45:00.631078+0000","last_unstale":"2024-03-24T10:45:02.955286+0000","last_undegraded":"2024-03-24T10:45:02.955286+0000","last_fullsized":"2024-03-24T10:45:02.955286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:43.263796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953696+0000","last_change":"2024-03-24T10:45:01.897329+0000","last_active":"2024-03-24T10:45:02.953696+0000","last_peered":"2024-03-24T10:45:02.953696+0000","last_clean":"2024-03-24T10:45:02.953696+0000","last_became_active":"2024-03-24T10:45:01.897063+0000","last_became_peered":"2024-03-24T10:45:01.897063+0000","last_unstale":"2024-03-24T10:45:02.953696+0000","last_undegraded":"2024-03-24T10:45:02.953696+0000","last_fullsized":"2024-03-24T10:45:02.953696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:07.259007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933122+0000","last_change":"2024-03-24T10:45:00.597948+0000","last_active":"2024-03-24T10:45:02.933122+0000","last_peered":"2024-03-24T10:45:02.933122+0000","last_clean":"2024-03-24T10:45:02.933122+0000","last_became_active":"2024-03-24T10:45:00.597671+0000","last_became_peered":"2024-03-24T10:45:00.597671+0000","last_unstale":"2024-03-24T10:45:02.933122+0000","last_undegraded":"2024-03-24T10:45:02.933122+0000","last_fullsized":"2024-03-24T10:45:02.933122+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:06.142451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954368+0000","last_change":"2024-03-24T10:45:01.927822+0000","last_active":"2024-03-24T10:45:02.954368+0000","last_peered":"2024-03-24T10:45:02.954368+0000","last_clean":"2024-03-24T10:45:02.954368+0000","last_became_active":"2024-03-24T10:45:01.927538+0000","last_became_peered":"2024-03-24T10:45:01.927538+0000","last_unstale":"2024-03-24T10:45:02.954368+0000","last_undegraded":"2024-03-24T10:45:02.954368+0000","last_fullsized":"2024-03-24T10:45:02.954368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933096+0000","last_change":"2024-03-24T10:45:00.626818+0000","last_active":"2024-03-24T10:45:02.933096+0000","last_peered":"2024-03-24T10:45:02.933096+0000","last_clean":"2024-03-24T10:45:02.933096+0000","last_became_active":"2024-03-24T10:45:00.626557+0000","last_became_peered":"2024-03-24T10:45:00.626557+0000","last_unstale":"2024-03-24T10:45:02.933096+0000","last_undegraded":"2024-03-24T10:45:02.933096+0000","last_fullsized":"2024-03-24T10:45:02.933096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:43.263796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954131+0000","last_change":"2024-03-24T10:45:01.893192+0000","last_active":"2024-03-24T10:45:02.954131+0000","last_peered":"2024-03-24T10:45:02.954131+0000","last_clean":"2024-03-24T10:45:02.954131+0000","last_became_active":"2024-03-24T10:45:01.892934+0000","last_became_peered":"2024-03-24T10:45:01.892934+0000","last_unstale":"2024-03-24T10:45:02.954131+0000","last_undegraded":"2024-03-24T10:45:02.954131+0000","last_fullsized":"2024-03-24T10:45:02.954131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:03.267523+0000","last_change":"2024-03-24T10:45:00.630147+0000","last_active":"2024-03-24T10:45:03.267523+0000","last_peered":"2024-03-24T10:45:03.267523+0000","last_clean":"2024-03-24T10:45:03.267523+0000","last_became_active":"2024-03-24T10:45:00.629850+0000","last_became_peered":"2024-03-24T10:45:00.629850+0000","last_unstale":"2024-03-24T10:45:03.267523+0000","last_undegraded":"2024-03-24T10:45:03.267523+0000","last_fullsized":"2024-03-24T10:45:03.267523+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:01:12.843222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953404+0000","last_change":"2024-03-24T10:45:01.897333+0000","last_active":"2024-03-24T10:45:02.953404+0000","last_peered":"2024-03-24T10:45:02.953404+0000","last_clean":"2024-03-24T10:45:02.953404+0000","last_became_active":"2024-03-24T10:45:01.897080+0000","last_became_peered":"2024-03-24T10:45:01.897080+0000","last_unstale":"2024-03-24T10:45:02.953404+0000","last_undegraded":"2024-03-24T10:45:02.953404+0000","last_fullsized":"2024-03-24T10:45:02.953404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:13.887947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953968+0000","last_change":"2024-03-24T10:45:01.956041+0000","last_active":"2024-03-24T10:45:02.953968+0000","last_peered":"2024-03-24T10:45:02.953968+0000","last_clean":"2024-03-24T10:45:02.953968+0000","last_became_active":"2024-03-24T10:45:01.955766+0000","last_became_peered":"2024-03-24T10:45:01.955766+0000","last_unstale":"2024-03-24T10:45:02.953968+0000","last_undegraded":"2024-03-24T10:45:02.953968+0000","last_fullsized":"2024-03-24T10:45:02.953968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:16.341360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:03.262738+0000","last_change":"2024-03-24T10:45:00.639913+0000","last_active":"2024-03-24T10:45:03.262738+0000","last_peered":"2024-03-24T10:45:03.262738+0000","last_clean":"2024-03-24T10:45:03.262738+0000","last_became_active":"2024-03-24T10:45:00.639653+0000","last_became_peered":"2024-03-24T10:45:00.639653+0000","last_unstale":"2024-03-24T10:45:03.262738+0000","last_undegraded":"2024-03-24T10:45:03.262738+0000","last_fullsized":"2024-03-24T10:45:03.262738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:15.749451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954233+0000","last_change":"2024-03-24T10:45:01.952340+0000","last_active":"2024-03-24T10:45:02.954233+0000","last_peered":"2024-03-24T10:45:02.954233+0000","last_clean":"2024-03-24T10:45:02.954233+0000","last_became_active":"2024-03-24T10:45:01.952074+0000","last_became_peered":"2024-03-24T10:45:01.952074+0000","last_unstale":"2024-03-24T10:45:02.954233+0000","last_undegraded":"2024-03-24T10:45:02.954233+0000","last_fullsized":"2024-03-24T10:45:02.954233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:03.262782+0000","last_change":"2024-03-24T10:45:00.633790+0000","last_active":"2024-03-24T10:45:03.262782+0000","last_peered":"2024-03-24T10:45:03.262782+0000","last_clean":"2024-03-24T10:45:03.262782+0000","last_became_active":"2024-03-24T10:45:00.633494+0000","last_became_peered":"2024-03-24T10:45:00.633494+0000","last_unstale":"2024-03-24T10:45:03.262782+0000","last_undegraded":"2024-03-24T10:45:03.262782+0000","last_fullsized":"2024-03-24T10:45:03.262782+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:14.049099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953320+0000","last_change":"2024-03-24T10:45:01.957958+0000","last_active":"2024-03-24T10:45:02.953320+0000","last_peered":"2024-03-24T10:45:02.953320+0000","last_clean":"2024-03-24T10:45:02.953320+0000","last_became_active":"2024-03-24T10:45:01.957676+0000","last_became_peered":"2024-03-24T10:45:01.957676+0000","last_unstale":"2024-03-24T10:45:02.953320+0000","last_undegraded":"2024-03-24T10:45:02.953320+0000","last_fullsized":"2024-03-24T10:45:02.953320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954519+0000","last_change":"2024-03-24T10:45:00.629120+0000","last_active":"2024-03-24T10:45:02.954519+0000","last_peered":"2024-03-24T10:45:02.954519+0000","last_clean":"2024-03-24T10:45:02.954519+0000","last_became_active":"2024-03-24T10:45:00.628712+0000","last_became_peered":"2024-03-24T10:45:00.628712+0000","last_unstale":"2024-03-24T10:45:02.954519+0000","last_undegraded":"2024-03-24T10:45:02.954519+0000","last_fullsized":"2024-03-24T10:45:02.954519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954346+0000","last_change":"2024-03-24T10:45:01.894602+0000","last_active":"2024-03-24T10:45:02.954346+0000","last_peered":"2024-03-24T10:45:02.954346+0000","last_clean":"2024-03-24T10:45:02.954346+0000","last_became_active":"2024-03-24T10:45:01.894362+0000","last_became_peered":"2024-03-24T10:45:01.894362+0000","last_unstale":"2024-03-24T10:45:02.954346+0000","last_undegraded":"2024-03-24T10:45:02.954346+0000","last_fullsized":"2024-03-24T10:45:02.954346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:03.223274+0000","last_change":"2024-03-24T10:45:00.639171+0000","last_active":"2024-03-24T10:45:03.223274+0000","last_peered":"2024-03-24T10:45:03.223274+0000","last_clean":"2024-03-24T10:45:03.223274+0000","last_became_active":"2024-03-24T10:45:00.638871+0000","last_became_peered":"2024-03-24T10:45:00.638871+0000","last_unstale":"2024-03-24T10:45:03.223274+0000","last_undegraded":"2024-03-24T10:45:03.223274+0000","last_fullsized":"2024-03-24T10:45:03.223274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:52:46.322196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2500,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48,"ondisk_log_size":48,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65430,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542152,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":282624,"data_stored":665155,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542152,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":74295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542152,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":74205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":221184,"data_stored":74365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":229376,"data_stored":75907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":74331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":294912,"data_stored":665233,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:45:10.871 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:45:11.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.008+0000 7fdd17cb5640 1 Processor -- start 2024-03-24T10:45:11.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.008+0000 7fdd17cb5640 1 -- start start 2024-03-24T10:45:11.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.009+0000 7fdd17cb5640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdd100c86e0 0x7fdd100c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:11.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.009+0000 7fdd17cb5640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd1015a6f0 con 0x7fdd100ca500 2024-03-24T10:45:11.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.009+0000 7fdd17cb5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd1015b290 con 0x7fdd10154680 2024-03-24T10:45:11.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.009+0000 7fdd17cb5640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fdd10154a60 con 0x7fdd100c86e0 2024-03-24T10:45:11.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.010+0000 7fdd15a51640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fdd10154680 0x7fdd1015ab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:34308/0 (socket says 172.21.15.5:34308) 2024-03-24T10:45:11.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.010+0000 7fdd15a51640 1 -- 172.21.15.5:0/1781803046 learned_addr learned my addr 172.21.15.5:0/1781803046 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:11.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.010+0000 7fdd16252640 1 --2- 172.21.15.5:0/1781803046 >> v2:172.21.15.5:3300/0 conn(0x7fdd100c86e0 0x7fdd100c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:11.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.012+0000 7fdd14a4f640 1 -- 172.21.15.5:0/1781803046 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3773312764 0 0) 0x7fdd1015b290 con 0x7fdd10154680 2024-03-24T10:45:11.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.012+0000 7fdd14a4f640 1 -- 172.21.15.5:0/1781803046 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdcfc003520 con 0x7fdd10154680 2024-03-24T10:45:11.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.013+0000 7fdd16252640 1 -- 172.21.15.5:0/1781803046 >> v1:172.21.15.57:6789/0 conn(0x7fdd10154680 legacy=0x7fdd1015ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:11.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.013+0000 7fdd16252640 1 -- 172.21.15.5:0/1781803046 >> v1:172.21.15.5:6789/0 conn(0x7fdd100ca500 legacy=0x7fdd100c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:11.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.013+0000 7fdd16252640 1 -- 172.21.15.5:0/1781803046 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd1015a3c0 con 0x7fdd100c86e0 2024-03-24T10:45:11.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.018+0000 7fdd16252640 1 --2- 172.21.15.5:0/1781803046 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd100c86e0 0x7fdd100c8b30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdd08009d90 tx=0x7fdd0802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de6a93aad462a510 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:11.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.022+0000 7fdd14a4f640 1 -- 172.21.15.5:0/1781803046 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0802cd00 con 0x7fdd100c86e0 2024-03-24T10:45:11.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.022+0000 7fdd14a4f640 1 -- 172.21.15.5:0/1781803046 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd0802ce60 con 0x7fdd100c86e0 2024-03-24T10:45:11.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.022+0000 7fdd14a4f640 1 -- 172.21.15.5:0/1781803046 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd08034660 con 0x7fdd100c86e0 2024-03-24T10:45:11.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.023+0000 7fdd17cb5640 1 -- 172.21.15.5:0/1781803046 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd100c86e0 msgr2=0x7fdd100c8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:11.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.023+0000 7fdd17cb5640 1 --2- 172.21.15.5:0/1781803046 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd100c86e0 0x7fdd100c8b30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdd08009d90 tx=0x7fdd0802bf50 comp rx=0 tx=0).stop 2024-03-24T10:45:11.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.023+0000 7fdd17cb5640 1 -- 172.21.15.5:0/1781803046 shutdown_connections 2024-03-24T10:45:11.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.023+0000 7fdd17cb5640 1 --2- 172.21.15.5:0/1781803046 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd100c86e0 0x7fdd100c8b30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:11.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.023+0000 7fdd17cb5640 1 -- 172.21.15.5:0/1781803046 >> 172.21.15.5:0/1781803046 conn(0x7fdd1006f880 msgr2=0x7fdd100712e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:11.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.023+0000 7fdd17cb5640 1 -- 172.21.15.5:0/1781803046 shutdown_connections 2024-03-24T10:45:11.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.023+0000 7fdd17cb5640 1 -- 172.21.15.5:0/1781803046 wait complete. 2024-03-24T10:45:11.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.024+0000 7fdd17cb5640 1 Processor -- start 2024-03-24T10:45:11.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.024+0000 7fdd17cb5640 1 -- start start 2024-03-24T10:45:11.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.025+0000 7fdd17cb5640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd100c86e0 0x7fdd100cc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:11.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.025+0000 7fdd17cb5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd100ca500 0x7fdd100cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:11.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.025+0000 7fdd17cb5640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd10154680 0x7fdd1015c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:11.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.025+0000 7fdd17cb5640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1015d930 con 0x7fdd100c86e0 2024-03-24T10:45:11.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.025+0000 7fdd17cb5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fdd100c4680 con 0x7fdd100ca500 2024-03-24T10:45:11.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.025+0000 7fdd17cb5640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fdd101f00b0 con 0x7fdd10154680 2024-03-24T10:45:11.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.026+0000 7fdd15a51640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd100c86e0 0x7fdd100cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:11.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.026+0000 7fdd15250640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd100ca500 0x7fdd100cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:11.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.026+0000 7fdd16252640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd10154680 0x7fdd1015c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:11.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.026+0000 7fdd15250640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd100ca500 0x7fdd100cc7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:34736/0 (socket says 172.21.15.5:34736) 2024-03-24T10:45:11.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.026+0000 7fdd15250640 1 -- 172.21.15.5:0/2420836154 learned_addr learned my addr 172.21.15.5:0/2420836154 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:11.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.029+0000 7fdd15a51640 1 -- 172.21.15.5:0/2420836154 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd10154680 msgr2=0x7fdd1015c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:11.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.029+0000 7fdd15a51640 1 --2- 172.21.15.5:0/2420836154 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd10154680 0x7fdd1015c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:11.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.030+0000 7fdd15a51640 1 -- 172.21.15.5:0/2420836154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd100ca500 msgr2=0x7fdd100cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:11.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.030+0000 7fdd15a51640 1 --2- 172.21.15.5:0/2420836154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd100ca500 0x7fdd100cc7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:11.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.030+0000 7fdd15a51640 1 -- 172.21.15.5:0/2420836154 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd08009a70 con 0x7fdd100c86e0 2024-03-24T10:45:11.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.032+0000 7fdd15a51640 1 --2- 172.21.15.5:0/2420836154 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd100c86e0 0x7fdd100cc270 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00c390 tx=0x7fdd0c001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:11.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.035+0000 7fdd067fc640 1 -- 172.21.15.5:0/2420836154 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c005ce0 con 0x7fdd100c86e0 2024-03-24T10:45:11.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.035+0000 7fdd067fc640 1 -- 172.21.15.5:0/2420836154 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd0c005e40 con 0x7fdd100c86e0 2024-03-24T10:45:11.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.035+0000 7fdd17cb5640 1 -- 172.21.15.5:0/2420836154 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd101f0390 con 0x7fdd100c86e0 2024-03-24T10:45:11.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.035+0000 7fdd067fc640 1 -- 172.21.15.5:0/2420836154 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c0072f0 con 0x7fdd100c86e0 2024-03-24T10:45:11.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.035+0000 7fdd17cb5640 1 -- 172.21.15.5:0/2420836154 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd101f0950 con 0x7fdd100c86e0 2024-03-24T10:45:11.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.036+0000 7fdd17cb5640 1 -- 172.21.15.5:0/2420836154 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd1015c3c0 con 0x7fdd100c86e0 2024-03-24T10:45:11.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.304+0000 7fdd067fc640 1 -- 172.21.15.5:0/2420836154 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdd0c007510 con 0x7fdd100c86e0 2024-03-24T10:45:11.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.304+0000 7fdd067fc640 1 --2- 172.21.15.5:0/2420836154 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdcf406b860 0x7fdcf406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:11.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.305+0000 7fdd15250640 1 --2- 172.21.15.5:0/2420836154 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdcf406b860 0x7fdcf406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:11.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.305+0000 7fdd15250640 1 --2- 172.21.15.5:0/2420836154 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdcf406b860 0x7fdcf406dd20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdd1015c290 tx=0x7fdd00007ef0 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:11.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.319+0000 7fdd067fc640 1 -- 172.21.15.5:0/2420836154 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdd0c007e20 con 0x7fdd100c86e0 2024-03-24T10:45:11.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.319+0000 7fdd067fc640 1 -- 172.21.15.5:0/2420836154 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdd0c054420 con 0x7fdd100c86e0 2024-03-24T10:45:11.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.504+0000 7fdd17cb5640 1 -- 172.21.15.5:0/2420836154 --> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdd100c9070 con 0x7fdcf406b860 2024-03-24T10:45:11.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.510+0000 7fdd067fc640 1 -- 172.21.15.5:0/2420836154 <== mgr.4120 v2:172.21.15.5:6832/1102439181 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251997 (secure 0 0 0) 0x7fdd00003a00 con 0x7fdcf406b860 2024-03-24T10:45:11.512 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:45:11.513 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-03-24T10:45:11.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.512+0000 7fdd17cb5640 1 -- 172.21.15.5:0/2420836154 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdcf406b860 msgr2=0x7fdcf406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:11.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.512+0000 7fdd17cb5640 1 --2- 172.21.15.5:0/2420836154 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdcf406b860 0x7fdcf406dd20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdd1015c290 tx=0x7fdd00007ef0 comp rx=0 tx=0).stop 2024-03-24T10:45:11.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.512+0000 7fdd17cb5640 1 -- 172.21.15.5:0/2420836154 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd100c86e0 msgr2=0x7fdd100cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:11.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.512+0000 7fdd17cb5640 1 --2- 172.21.15.5:0/2420836154 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd100c86e0 0x7fdd100cc270 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00c390 tx=0x7fdd0c001e70 comp rx=0 tx=0).stop 2024-03-24T10:45:11.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.513+0000 7fdd17cb5640 1 -- 172.21.15.5:0/2420836154 shutdown_connections 2024-03-24T10:45:11.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.513+0000 7fdd17cb5640 1 --2- 172.21.15.5:0/2420836154 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdcf406b860 0x7fdcf406dd20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:11.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.513+0000 7fdd17cb5640 1 --2- 172.21.15.5:0/2420836154 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd10154680 0x7fdd1015c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:11.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.513+0000 7fdd17cb5640 1 --2- 172.21.15.5:0/2420836154 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd100ca500 0x7fdd100cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:11.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.513+0000 7fdd17cb5640 1 --2- 172.21.15.5:0/2420836154 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd100c86e0 0x7fdd100cc270 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:11.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.513+0000 7fdd17cb5640 1 -- 172.21.15.5:0/2420836154 >> 172.21.15.5:0/2420836154 conn(0x7fdd1006f880 msgr2=0x7fdd101528b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:11.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.513+0000 7fdd17cb5640 1 -- 172.21.15.5:0/2420836154 shutdown_connections 2024-03-24T10:45:11.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.513+0000 7fdd17cb5640 1 -- 172.21.15.5:0/2420836154 wait complete. 2024-03-24T10:45:11.524 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-03-24T10:45:11.372055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595002,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":268,"num_write_kb":2453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":212,"ondisk_log_size":212,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":234700,"kb_used_data":1932,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754740020,"statfs":{"total":773094113280,"available":772853780480,"internally_reserved":0,"allocated":1978368,"data_stored":1782938,"data_compressed":65430,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4634,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.431239"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956091+0000","last_change":"2024-03-24T10:45:01.894006+0000","last_active":"2024-03-24T10:45:02.956091+0000","last_peered":"2024-03-24T10:45:02.956091+0000","last_clean":"2024-03-24T10:45:02.956091+0000","last_became_active":"2024-03-24T10:45:01.893734+0000","last_became_peered":"2024-03-24T10:45:01.893734+0000","last_unstale":"2024-03-24T10:45:02.956091+0000","last_undegraded":"2024-03-24T10:45:02.956091+0000","last_fullsized":"2024-03-24T10:45:02.956091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:05:51.579388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.932998+0000","last_change":"2024-03-24T10:45:01.931707+0000","last_active":"2024-03-24T10:45:02.932998+0000","last_peered":"2024-03-24T10:45:02.932998+0000","last_clean":"2024-03-24T10:45:02.932998+0000","last_became_active":"2024-03-24T10:45:01.931449+0000","last_became_peered":"2024-03-24T10:45:01.931449+0000","last_unstale":"2024-03-24T10:45:02.932998+0000","last_undegraded":"2024-03-24T10:45:02.932998+0000","last_fullsized":"2024-03-24T10:45:02.932998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953238+0000","last_change":"2024-03-24T10:45:01.895929+0000","last_active":"2024-03-24T10:45:02.953238+0000","last_peered":"2024-03-24T10:45:02.953238+0000","last_clean":"2024-03-24T10:45:02.953238+0000","last_became_active":"2024-03-24T10:45:01.895654+0000","last_became_peered":"2024-03-24T10:45:01.895654+0000","last_unstale":"2024-03-24T10:45:02.953238+0000","last_undegraded":"2024-03-24T10:45:02.953238+0000","last_fullsized":"2024-03-24T10:45:02.953238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954093+0000","last_change":"2024-03-24T10:45:01.952468+0000","last_active":"2024-03-24T10:45:02.954093+0000","last_peered":"2024-03-24T10:45:02.954093+0000","last_clean":"2024-03-24T10:45:02.954093+0000","last_became_active":"2024-03-24T10:45:01.952219+0000","last_became_peered":"2024-03-24T10:45:01.952219+0000","last_unstale":"2024-03-24T10:45:02.954093+0000","last_undegraded":"2024-03-24T10:45:02.954093+0000","last_fullsized":"2024-03-24T10:45:02.954093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:27:12.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955739+0000","last_change":"2024-03-24T10:45:01.895885+0000","last_active":"2024-03-24T10:45:02.955739+0000","last_peered":"2024-03-24T10:45:02.955739+0000","last_clean":"2024-03-24T10:45:02.955739+0000","last_became_active":"2024-03-24T10:45:01.895622+0000","last_became_peered":"2024-03-24T10:45:01.895622+0000","last_unstale":"2024-03-24T10:45:02.955739+0000","last_undegraded":"2024-03-24T10:45:02.955739+0000","last_fullsized":"2024-03-24T10:45:02.955739+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:16.523177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954340+0000","last_change":"2024-03-24T10:45:01.958421+0000","last_active":"2024-03-24T10:45:02.954340+0000","last_peered":"2024-03-24T10:45:02.954340+0000","last_clean":"2024-03-24T10:45:02.954340+0000","last_became_active":"2024-03-24T10:45:01.957972+0000","last_became_peered":"2024-03-24T10:45:01.957972+0000","last_unstale":"2024-03-24T10:45:02.954340+0000","last_undegraded":"2024-03-24T10:45:02.954340+0000","last_fullsized":"2024-03-24T10:45:02.954340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:16.523177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955433+0000","last_change":"2024-03-24T10:45:01.892745+0000","last_active":"2024-03-24T10:45:02.955433+0000","last_peered":"2024-03-24T10:45:02.955433+0000","last_clean":"2024-03-24T10:45:02.955433+0000","last_became_active":"2024-03-24T10:45:01.892452+0000","last_became_peered":"2024-03-24T10:45:01.892452+0000","last_unstale":"2024-03-24T10:45:02.955433+0000","last_undegraded":"2024-03-24T10:45:02.955433+0000","last_fullsized":"2024-03-24T10:45:02.955433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954102+0000","last_change":"2024-03-24T10:45:01.953432+0000","last_active":"2024-03-24T10:45:02.954102+0000","last_peered":"2024-03-24T10:45:02.954102+0000","last_clean":"2024-03-24T10:45:02.954102+0000","last_became_active":"2024-03-24T10:45:01.953149+0000","last_became_peered":"2024-03-24T10:45:01.953149+0000","last_unstale":"2024-03-24T10:45:02.954102+0000","last_undegraded":"2024-03-24T10:45:02.954102+0000","last_fullsized":"2024-03-24T10:45:02.954102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956172+0000","last_change":"2024-03-24T10:45:01.956488+0000","last_active":"2024-03-24T10:45:02.956172+0000","last_peered":"2024-03-24T10:45:02.956172+0000","last_clean":"2024-03-24T10:45:02.956172+0000","last_became_active":"2024-03-24T10:45:01.956207+0000","last_became_peered":"2024-03-24T10:45:01.956207+0000","last_unstale":"2024-03-24T10:45:02.956172+0000","last_undegraded":"2024-03-24T10:45:02.956172+0000","last_fullsized":"2024-03-24T10:45:02.956172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:23:26.523809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953720+0000","last_change":"2024-03-24T10:45:01.956484+0000","last_active":"2024-03-24T10:45:02.953720+0000","last_peered":"2024-03-24T10:45:02.953720+0000","last_clean":"2024-03-24T10:45:02.953720+0000","last_became_active":"2024-03-24T10:45:01.956209+0000","last_became_peered":"2024-03-24T10:45:01.956209+0000","last_unstale":"2024-03-24T10:45:02.953720+0000","last_undegraded":"2024-03-24T10:45:02.953720+0000","last_fullsized":"2024-03-24T10:45:02.953720+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953838+0000","last_change":"2024-03-24T10:45:01.957103+0000","last_active":"2024-03-24T10:45:02.953838+0000","last_peered":"2024-03-24T10:45:02.953838+0000","last_clean":"2024-03-24T10:45:02.953838+0000","last_became_active":"2024-03-24T10:45:01.956849+0000","last_became_peered":"2024-03-24T10:45:01.956849+0000","last_unstale":"2024-03-24T10:45:02.953838+0000","last_undegraded":"2024-03-24T10:45:02.953838+0000","last_fullsized":"2024-03-24T10:45:02.953838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954691+0000","last_change":"2024-03-24T10:45:01.896701+0000","last_active":"2024-03-24T10:45:02.954691+0000","last_peered":"2024-03-24T10:45:02.954691+0000","last_clean":"2024-03-24T10:45:02.954691+0000","last_became_active":"2024-03-24T10:45:01.896400+0000","last_became_peered":"2024-03-24T10:45:01.896400+0000","last_unstale":"2024-03-24T10:45:02.954691+0000","last_undegraded":"2024-03-24T10:45:02.954691+0000","last_fullsized":"2024-03-24T10:45:02.954691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:44.380352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953999+0000","last_change":"2024-03-24T10:45:01.894954+0000","last_active":"2024-03-24T10:45:02.953999+0000","last_peered":"2024-03-24T10:45:02.953999+0000","last_clean":"2024-03-24T10:45:02.953999+0000","last_became_active":"2024-03-24T10:45:01.894632+0000","last_became_peered":"2024-03-24T10:45:01.894632+0000","last_unstale":"2024-03-24T10:45:02.953999+0000","last_undegraded":"2024-03-24T10:45:02.953999+0000","last_fullsized":"2024-03-24T10:45:02.953999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.952924+0000","last_change":"2024-03-24T10:45:01.929588+0000","last_active":"2024-03-24T10:45:02.952924+0000","last_peered":"2024-03-24T10:45:02.952924+0000","last_clean":"2024-03-24T10:45:02.952924+0000","last_became_active":"2024-03-24T10:45:01.928460+0000","last_became_peered":"2024-03-24T10:45:01.928460+0000","last_unstale":"2024-03-24T10:45:02.952924+0000","last_undegraded":"2024-03-24T10:45:02.952924+0000","last_fullsized":"2024-03-24T10:45:02.952924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:47:48.896914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933000+0000","last_change":"2024-03-24T10:45:01.928721+0000","last_active":"2024-03-24T10:45:02.933000+0000","last_peered":"2024-03-24T10:45:02.933000+0000","last_clean":"2024-03-24T10:45:02.933000+0000","last_became_active":"2024-03-24T10:45:01.928467+0000","last_became_peered":"2024-03-24T10:45:01.928467+0000","last_unstale":"2024-03-24T10:45:02.933000+0000","last_undegraded":"2024-03-24T10:45:02.933000+0000","last_fullsized":"2024-03-24T10:45:02.933000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953333+0000","last_change":"2024-03-24T10:45:01.956030+0000","last_active":"2024-03-24T10:45:02.953333+0000","last_peered":"2024-03-24T10:45:02.953333+0000","last_clean":"2024-03-24T10:45:02.953333+0000","last_became_active":"2024-03-24T10:45:01.955768+0000","last_became_peered":"2024-03-24T10:45:01.955768+0000","last_unstale":"2024-03-24T10:45:02.953333+0000","last_undegraded":"2024-03-24T10:45:02.953333+0000","last_fullsized":"2024-03-24T10:45:02.953333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954695+0000","last_change":"2024-03-24T10:45:01.956181+0000","last_active":"2024-03-24T10:45:02.954695+0000","last_peered":"2024-03-24T10:45:02.954695+0000","last_clean":"2024-03-24T10:45:02.954695+0000","last_became_active":"2024-03-24T10:45:01.955890+0000","last_became_peered":"2024-03-24T10:45:01.955890+0000","last_unstale":"2024-03-24T10:45:02.954695+0000","last_undegraded":"2024-03-24T10:45:02.954695+0000","last_fullsized":"2024-03-24T10:45:02.954695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:15:11.107915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.936626+0000","last_change":"2024-03-24T10:45:01.896663+0000","last_active":"2024-03-24T10:45:02.936626+0000","last_peered":"2024-03-24T10:45:02.936626+0000","last_clean":"2024-03-24T10:45:02.936626+0000","last_became_active":"2024-03-24T10:45:01.896391+0000","last_became_peered":"2024-03-24T10:45:01.896391+0000","last_unstale":"2024-03-24T10:45:02.936626+0000","last_undegraded":"2024-03-24T10:45:02.936626+0000","last_fullsized":"2024-03-24T10:45:02.936626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:30.942758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956053+0000","last_change":"2024-03-24T10:45:01.895951+0000","last_active":"2024-03-24T10:45:02.956053+0000","last_peered":"2024-03-24T10:45:02.956053+0000","last_clean":"2024-03-24T10:45:02.956053+0000","last_became_active":"2024-03-24T10:45:01.895668+0000","last_became_peered":"2024-03-24T10:45:01.895668+0000","last_unstale":"2024-03-24T10:45:02.956053+0000","last_undegraded":"2024-03-24T10:45:02.956053+0000","last_fullsized":"2024-03-24T10:45:02.956053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:56:43.462458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953964+0000","last_change":"2024-03-24T10:45:01.898126+0000","last_active":"2024-03-24T10:45:02.953964+0000","last_peered":"2024-03-24T10:45:02.953964+0000","last_clean":"2024-03-24T10:45:02.953964+0000","last_became_active":"2024-03-24T10:45:01.897878+0000","last_became_peered":"2024-03-24T10:45:01.897878+0000","last_unstale":"2024-03-24T10:45:02.953964+0000","last_undegraded":"2024-03-24T10:45:02.953964+0000","last_fullsized":"2024-03-24T10:45:02.953964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953247+0000","last_change":"2024-03-24T10:45:01.896109+0000","last_active":"2024-03-24T10:45:02.953247+0000","last_peered":"2024-03-24T10:45:02.953247+0000","last_clean":"2024-03-24T10:45:02.953247+0000","last_became_active":"2024-03-24T10:45:01.895696+0000","last_became_peered":"2024-03-24T10:45:01.895696+0000","last_unstale":"2024-03-24T10:45:02.953247+0000","last_undegraded":"2024-03-24T10:45:02.953247+0000","last_fullsized":"2024-03-24T10:45:02.953247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954483+0000","last_change":"2024-03-24T10:45:01.958119+0000","last_active":"2024-03-24T10:45:02.954483+0000","last_peered":"2024-03-24T10:45:02.954483+0000","last_clean":"2024-03-24T10:45:02.954483+0000","last_became_active":"2024-03-24T10:45:01.957832+0000","last_became_peered":"2024-03-24T10:45:01.957832+0000","last_unstale":"2024-03-24T10:45:02.954483+0000","last_undegraded":"2024-03-24T10:45:02.954483+0000","last_fullsized":"2024-03-24T10:45:02.954483+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:12.476490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953638+0000","last_change":"2024-03-24T10:45:01.842442+0000","last_active":"2024-03-24T10:45:02.953638+0000","last_peered":"2024-03-24T10:45:02.953638+0000","last_clean":"2024-03-24T10:45:02.953638+0000","last_became_active":"2024-03-24T10:45:01.842199+0000","last_became_peered":"2024-03-24T10:45:01.842199+0000","last_unstale":"2024-03-24T10:45:02.953638+0000","last_undegraded":"2024-03-24T10:45:02.953638+0000","last_fullsized":"2024-03-24T10:45:02.953638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953764+0000","last_change":"2024-03-24T10:45:01.952378+0000","last_active":"2024-03-24T10:45:02.953764+0000","last_peered":"2024-03-24T10:45:02.953764+0000","last_clean":"2024-03-24T10:45:02.953764+0000","last_became_active":"2024-03-24T10:45:01.952123+0000","last_became_peered":"2024-03-24T10:45:01.952123+0000","last_unstale":"2024-03-24T10:45:02.953764+0000","last_undegraded":"2024-03-24T10:45:02.953764+0000","last_fullsized":"2024-03-24T10:45:02.953764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954628+0000","last_change":"2024-03-24T10:45:01.842299+0000","last_active":"2024-03-24T10:45:02.954628+0000","last_peered":"2024-03-24T10:45:02.954628+0000","last_clean":"2024-03-24T10:45:02.954628+0000","last_became_active":"2024-03-24T10:45:01.842097+0000","last_became_peered":"2024-03-24T10:45:01.842097+0000","last_unstale":"2024-03-24T10:45:02.954628+0000","last_undegraded":"2024-03-24T10:45:02.954628+0000","last_fullsized":"2024-03-24T10:45:02.954628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:16.341360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953886+0000","last_change":"2024-03-24T10:45:01.893579+0000","last_active":"2024-03-24T10:45:02.953886+0000","last_peered":"2024-03-24T10:45:02.953886+0000","last_clean":"2024-03-24T10:45:02.953886+0000","last_became_active":"2024-03-24T10:45:01.893273+0000","last_became_peered":"2024-03-24T10:45:01.893273+0000","last_unstale":"2024-03-24T10:45:02.953886+0000","last_undegraded":"2024-03-24T10:45:02.953886+0000","last_fullsized":"2024-03-24T10:45:02.953886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933308+0000","last_change":"2024-03-24T10:45:01.930032+0000","last_active":"2024-03-24T10:45:02.933308+0000","last_peered":"2024-03-24T10:45:02.933308+0000","last_clean":"2024-03-24T10:45:02.933308+0000","last_became_active":"2024-03-24T10:45:01.929770+0000","last_became_peered":"2024-03-24T10:45:01.929770+0000","last_unstale":"2024-03-24T10:45:02.933308+0000","last_undegraded":"2024-03-24T10:45:02.933308+0000","last_fullsized":"2024-03-24T10:45:02.933308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954962+0000","last_change":"2024-03-24T10:45:01.842460+0000","last_active":"2024-03-24T10:45:02.954962+0000","last_peered":"2024-03-24T10:45:02.954962+0000","last_clean":"2024-03-24T10:45:02.954962+0000","last_became_active":"2024-03-24T10:45:01.842165+0000","last_became_peered":"2024-03-24T10:45:01.842165+0000","last_unstale":"2024-03-24T10:45:02.954962+0000","last_undegraded":"2024-03-24T10:45:02.954962+0000","last_fullsized":"2024-03-24T10:45:02.954962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953401+0000","last_change":"2024-03-24T10:45:01.895868+0000","last_active":"2024-03-24T10:45:02.953401+0000","last_peered":"2024-03-24T10:45:02.953401+0000","last_clean":"2024-03-24T10:45:02.953401+0000","last_became_active":"2024-03-24T10:45:01.895290+0000","last_became_peered":"2024-03-24T10:45:01.895290+0000","last_unstale":"2024-03-24T10:45:02.953401+0000","last_undegraded":"2024-03-24T10:45:02.953401+0000","last_fullsized":"2024-03-24T10:45:02.953401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954615+0000","last_change":"2024-03-24T10:45:01.957724+0000","last_active":"2024-03-24T10:45:02.954615+0000","last_peered":"2024-03-24T10:45:02.954615+0000","last_clean":"2024-03-24T10:45:02.954615+0000","last_became_active":"2024-03-24T10:45:01.957429+0000","last_became_peered":"2024-03-24T10:45:01.957429+0000","last_unstale":"2024-03-24T10:45:02.954615+0000","last_undegraded":"2024-03-24T10:45:02.954615+0000","last_fullsized":"2024-03-24T10:45:02.954615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:22.089070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.936465+0000","last_change":"2024-03-24T10:45:01.952159+0000","last_active":"2024-03-24T10:45:02.936465+0000","last_peered":"2024-03-24T10:45:02.936465+0000","last_clean":"2024-03-24T10:45:02.936465+0000","last_became_active":"2024-03-24T10:45:01.951961+0000","last_became_peered":"2024-03-24T10:45:01.951961+0000","last_unstale":"2024-03-24T10:45:02.936465+0000","last_undegraded":"2024-03-24T10:45:02.936465+0000","last_fullsized":"2024-03-24T10:45:02.936465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:44.380352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953610+0000","last_change":"2024-03-24T10:45:01.893452+0000","last_active":"2024-03-24T10:45:02.953610+0000","last_peered":"2024-03-24T10:45:02.953610+0000","last_clean":"2024-03-24T10:45:02.953610+0000","last_became_active":"2024-03-24T10:45:01.893139+0000","last_became_peered":"2024-03-24T10:45:01.893139+0000","last_unstale":"2024-03-24T10:45:02.953610+0000","last_undegraded":"2024-03-24T10:45:02.953610+0000","last_fullsized":"2024-03-24T10:45:02.953610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953542+0000","last_change":"2024-03-24T10:45:01.928827+0000","last_active":"2024-03-24T10:45:02.953542+0000","last_peered":"2024-03-24T10:45:02.953542+0000","last_clean":"2024-03-24T10:45:02.953542+0000","last_became_active":"2024-03-24T10:45:01.928521+0000","last_became_peered":"2024-03-24T10:45:01.928521+0000","last_unstale":"2024-03-24T10:45:02.953542+0000","last_undegraded":"2024-03-24T10:45:02.953542+0000","last_fullsized":"2024-03-24T10:45:02.953542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:59.643246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.699015+0000","last_change":"2024-03-24T10:45:00.637993+0000","last_active":"2024-03-24T10:45:07.699015+0000","last_peered":"2024-03-24T10:45:07.699015+0000","last_clean":"2024-03-24T10:45:07.699015+0000","last_became_active":"2024-03-24T10:45:00.637803+0000","last_became_peered":"2024-03-24T10:45:00.637803+0000","last_unstale":"2024-03-24T10:45:07.699015+0000","last_undegraded":"2024-03-24T10:45:07.699015+0000","last_fullsized":"2024-03-24T10:45:07.699015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:28:28.854541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953550+0000","last_change":"2024-03-24T10:45:01.897686+0000","last_active":"2024-03-24T10:45:02.953550+0000","last_peered":"2024-03-24T10:45:02.953550+0000","last_clean":"2024-03-24T10:45:02.953550+0000","last_became_active":"2024-03-24T10:45:01.897320+0000","last_became_peered":"2024-03-24T10:45:01.897320+0000","last_unstale":"2024-03-24T10:45:02.953550+0000","last_undegraded":"2024-03-24T10:45:02.953550+0000","last_fullsized":"2024-03-24T10:45:02.953550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:03.132133+0000","last_change":"2024-03-24T10:45:00.638329+0000","last_active":"2024-03-24T10:45:03.132133+0000","last_peered":"2024-03-24T10:45:03.132133+0000","last_clean":"2024-03-24T10:45:03.132133+0000","last_became_active":"2024-03-24T10:45:00.638006+0000","last_became_peered":"2024-03-24T10:45:00.638006+0000","last_unstale":"2024-03-24T10:45:03.132133+0000","last_undegraded":"2024-03-24T10:45:03.132133+0000","last_fullsized":"2024-03-24T10:45:03.132133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:10.355252+0000","last_change":"2024-03-24T10:45:00.633703+0000","last_active":"2024-03-24T10:45:10.355252+0000","last_peered":"2024-03-24T10:45:10.355252+0000","last_clean":"2024-03-24T10:45:10.355252+0000","last_became_active":"2024-03-24T10:45:00.633427+0000","last_became_peered":"2024-03-24T10:45:00.633427+0000","last_unstale":"2024-03-24T10:45:10.355252+0000","last_undegraded":"2024-03-24T10:45:10.355252+0000","last_fullsized":"2024-03-24T10:45:10.355252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:30.350191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954240+0000","last_change":"2024-03-24T10:45:01.893902+0000","last_active":"2024-03-24T10:45:02.954240+0000","last_peered":"2024-03-24T10:45:02.954240+0000","last_clean":"2024-03-24T10:45:02.954240+0000","last_became_active":"2024-03-24T10:45:01.893617+0000","last_became_peered":"2024-03-24T10:45:01.893617+0000","last_unstale":"2024-03-24T10:45:02.954240+0000","last_undegraded":"2024-03-24T10:45:02.954240+0000","last_fullsized":"2024-03-24T10:45:02.954240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953975+0000","last_change":"2024-03-24T10:45:00.638778+0000","last_active":"2024-03-24T10:45:02.953975+0000","last_peered":"2024-03-24T10:45:02.953975+0000","last_clean":"2024-03-24T10:45:02.953975+0000","last_became_active":"2024-03-24T10:45:00.638089+0000","last_became_peered":"2024-03-24T10:45:00.638089+0000","last_unstale":"2024-03-24T10:45:02.953975+0000","last_undegraded":"2024-03-24T10:45:02.953975+0000","last_fullsized":"2024-03-24T10:45:02.953975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.932839+0000","last_change":"2024-03-24T10:45:01.930063+0000","last_active":"2024-03-24T10:45:02.932839+0000","last_peered":"2024-03-24T10:45:02.932839+0000","last_clean":"2024-03-24T10:45:02.932839+0000","last_became_active":"2024-03-24T10:45:01.929786+0000","last_became_peered":"2024-03-24T10:45:01.929786+0000","last_unstale":"2024-03-24T10:45:02.932839+0000","last_undegraded":"2024-03-24T10:45:02.932839+0000","last_fullsized":"2024-03-24T10:45:02.932839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.773089+0000","last_change":"2024-03-24T10:45:00.639822+0000","last_active":"2024-03-24T10:45:07.773089+0000","last_peered":"2024-03-24T10:45:07.773089+0000","last_clean":"2024-03-24T10:45:07.773089+0000","last_became_active":"2024-03-24T10:45:00.639600+0000","last_became_peered":"2024-03-24T10:45:00.639600+0000","last_unstale":"2024-03-24T10:45:07.773089+0000","last_undegraded":"2024-03-24T10:45:07.773089+0000","last_fullsized":"2024-03-24T10:45:07.773089+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953735+0000","last_change":"2024-03-24T10:45:01.954015+0000","last_active":"2024-03-24T10:45:02.953735+0000","last_peered":"2024-03-24T10:45:02.953735+0000","last_clean":"2024-03-24T10:45:02.953735+0000","last_became_active":"2024-03-24T10:45:01.953767+0000","last_became_peered":"2024-03-24T10:45:01.953767+0000","last_unstale":"2024-03-24T10:45:02.953735+0000","last_undegraded":"2024-03-24T10:45:02.953735+0000","last_fullsized":"2024-03-24T10:45:02.953735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.696215+0000","last_change":"2024-03-24T10:45:00.636553+0000","last_active":"2024-03-24T10:45:07.696215+0000","last_peered":"2024-03-24T10:45:07.696215+0000","last_clean":"2024-03-24T10:45:07.696215+0000","last_became_active":"2024-03-24T10:45:00.636282+0000","last_became_peered":"2024-03-24T10:45:00.636282+0000","last_unstale":"2024-03-24T10:45:07.696215+0000","last_undegraded":"2024-03-24T10:45:07.696215+0000","last_fullsized":"2024-03-24T10:45:07.696215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953458+0000","last_change":"2024-03-24T10:45:01.957905+0000","last_active":"2024-03-24T10:45:02.953458+0000","last_peered":"2024-03-24T10:45:02.953458+0000","last_clean":"2024-03-24T10:45:02.953458+0000","last_became_active":"2024-03-24T10:45:01.957676+0000","last_became_peered":"2024-03-24T10:45:01.957676+0000","last_unstale":"2024-03-24T10:45:02.953458+0000","last_undegraded":"2024-03-24T10:45:02.953458+0000","last_fullsized":"2024-03-24T10:45:02.953458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:03.133117+0000","last_change":"2024-03-24T10:45:00.635197+0000","last_active":"2024-03-24T10:45:03.133117+0000","last_peered":"2024-03-24T10:45:03.133117+0000","last_clean":"2024-03-24T10:45:03.133117+0000","last_became_active":"2024-03-24T10:45:00.634989+0000","last_became_peered":"2024-03-24T10:45:00.634989+0000","last_unstale":"2024-03-24T10:45:03.133117+0000","last_undegraded":"2024-03-24T10:45:03.133117+0000","last_fullsized":"2024-03-24T10:45:03.133117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954821+0000","last_change":"2024-03-24T10:45:01.853785+0000","last_active":"2024-03-24T10:45:02.954821+0000","last_peered":"2024-03-24T10:45:02.954821+0000","last_clean":"2024-03-24T10:45:02.954821+0000","last_became_active":"2024-03-24T10:45:01.853504+0000","last_became_peered":"2024-03-24T10:45:01.853504+0000","last_unstale":"2024-03-24T10:45:02.954821+0000","last_undegraded":"2024-03-24T10:45:02.954821+0000","last_fullsized":"2024-03-24T10:45:02.954821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:30.942758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.773017+0000","last_change":"2024-03-24T10:45:00.632381+0000","last_active":"2024-03-24T10:45:07.773017+0000","last_peered":"2024-03-24T10:45:07.773017+0000","last_clean":"2024-03-24T10:45:07.773017+0000","last_became_active":"2024-03-24T10:45:00.632108+0000","last_became_peered":"2024-03-24T10:45:00.632108+0000","last_unstale":"2024-03-24T10:45:07.773017+0000","last_undegraded":"2024-03-24T10:45:07.773017+0000","last_fullsized":"2024-03-24T10:45:07.773017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:06.142451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955746+0000","last_change":"2024-03-24T10:45:01.853591+0000","last_active":"2024-03-24T10:45:02.955746+0000","last_peered":"2024-03-24T10:45:02.955746+0000","last_clean":"2024-03-24T10:45:02.955746+0000","last_became_active":"2024-03-24T10:45:01.853279+0000","last_became_peered":"2024-03-24T10:45:01.853279+0000","last_unstale":"2024-03-24T10:45:02.955746+0000","last_undegraded":"2024-03-24T10:45:02.955746+0000","last_fullsized":"2024-03-24T10:45:02.955746+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:07:24.149961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.788598+0000","last_change":"2024-03-24T10:45:00.634192+0000","last_active":"2024-03-24T10:45:07.788598+0000","last_peered":"2024-03-24T10:45:07.788598+0000","last_clean":"2024-03-24T10:45:07.788598+0000","last_became_active":"2024-03-24T10:45:00.633893+0000","last_became_peered":"2024-03-24T10:45:00.633893+0000","last_unstale":"2024-03-24T10:45:07.788598+0000","last_undegraded":"2024-03-24T10:45:07.788598+0000","last_fullsized":"2024-03-24T10:45:07.788598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:30.350191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953075+0000","last_change":"2024-03-24T10:45:01.893426+0000","last_active":"2024-03-24T10:45:02.953075+0000","last_peered":"2024-03-24T10:45:02.953075+0000","last_clean":"2024-03-24T10:45:02.953075+0000","last_became_active":"2024-03-24T10:45:01.893178+0000","last_became_peered":"2024-03-24T10:45:01.893178+0000","last_unstale":"2024-03-24T10:45:02.953075+0000","last_undegraded":"2024-03-24T10:45:02.953075+0000","last_fullsized":"2024-03-24T10:45:02.953075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:13.887947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.697184+0000","last_change":"2024-03-24T10:45:00.638214+0000","last_active":"2024-03-24T10:45:07.697184+0000","last_peered":"2024-03-24T10:45:07.697184+0000","last_clean":"2024-03-24T10:45:07.697184+0000","last_became_active":"2024-03-24T10:45:00.637973+0000","last_became_peered":"2024-03-24T10:45:00.637973+0000","last_unstale":"2024-03-24T10:45:07.697184+0000","last_undegraded":"2024-03-24T10:45:07.697184+0000","last_fullsized":"2024-03-24T10:45:07.697184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:15.749451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955767+0000","last_change":"2024-03-24T10:45:01.856756+0000","last_active":"2024-03-24T10:45:02.955767+0000","last_peered":"2024-03-24T10:45:02.955767+0000","last_clean":"2024-03-24T10:45:02.955767+0000","last_became_active":"2024-03-24T10:45:01.856482+0000","last_became_peered":"2024-03-24T10:45:01.856482+0000","last_unstale":"2024-03-24T10:45:02.955767+0000","last_undegraded":"2024-03-24T10:45:02.955767+0000","last_fullsized":"2024-03-24T10:45:02.955767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:12.476490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953326+0000","last_change":"2024-03-24T10:45:00.636481+0000","last_active":"2024-03-24T10:45:02.953326+0000","last_peered":"2024-03-24T10:45:02.953326+0000","last_clean":"2024-03-24T10:45:02.953326+0000","last_became_active":"2024-03-24T10:45:00.636206+0000","last_became_peered":"2024-03-24T10:45:00.636206+0000","last_unstale":"2024-03-24T10:45:02.953326+0000","last_undegraded":"2024-03-24T10:45:02.953326+0000","last_fullsized":"2024-03-24T10:45:02.953326+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954153+0000","last_change":"2024-03-24T10:45:01.897279+0000","last_active":"2024-03-24T10:45:02.954153+0000","last_peered":"2024-03-24T10:45:02.954153+0000","last_clean":"2024-03-24T10:45:02.954153+0000","last_became_active":"2024-03-24T10:45:01.896736+0000","last_became_peered":"2024-03-24T10:45:01.896736+0000","last_unstale":"2024-03-24T10:45:02.954153+0000","last_undegraded":"2024-03-24T10:45:02.954153+0000","last_fullsized":"2024-03-24T10:45:02.954153+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'15","reported_seq":29,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:10.391603+0000","last_change":"2024-03-24T10:45:00.564066+0000","last_active":"2024-03-24T10:45:10.391603+0000","last_peered":"2024-03-24T10:45:10.391603+0000","last_clean":"2024-03-24T10:45:10.391603+0000","last_became_active":"2024-03-24T10:45:00.563812+0000","last_became_peered":"2024-03-24T10:45:00.563812+0000","last_unstale":"2024-03-24T10:45:10.391603+0000","last_undegraded":"2024-03-24T10:45:10.391603+0000","last_fullsized":"2024-03-24T10:45:10.391603+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:28:28.854541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955401+0000","last_change":"2024-03-24T10:45:01.854429+0000","last_active":"2024-03-24T10:45:02.955401+0000","last_peered":"2024-03-24T10:45:02.955401+0000","last_clean":"2024-03-24T10:45:02.955401+0000","last_became_active":"2024-03-24T10:45:01.854179+0000","last_became_peered":"2024-03-24T10:45:01.854179+0000","last_unstale":"2024-03-24T10:45:02.955401+0000","last_undegraded":"2024-03-24T10:45:02.955401+0000","last_fullsized":"2024-03-24T10:45:02.955401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:03.132419+0000","last_change":"2024-03-24T10:45:00.634223+0000","last_active":"2024-03-24T10:45:03.132419+0000","last_peered":"2024-03-24T10:45:03.132419+0000","last_clean":"2024-03-24T10:45:03.132419+0000","last_became_active":"2024-03-24T10:45:00.633355+0000","last_became_peered":"2024-03-24T10:45:00.633355+0000","last_unstale":"2024-03-24T10:45:03.132419+0000","last_undegraded":"2024-03-24T10:45:03.132419+0000","last_fullsized":"2024-03-24T10:45:03.132419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:04.930115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956031+0000","last_change":"2024-03-24T10:45:01.856935+0000","last_active":"2024-03-24T10:45:02.956031+0000","last_peered":"2024-03-24T10:45:02.956031+0000","last_clean":"2024-03-24T10:45:02.956031+0000","last_became_active":"2024-03-24T10:45:01.856653+0000","last_became_peered":"2024-03-24T10:45:01.856653+0000","last_unstale":"2024-03-24T10:45:02.956031+0000","last_undegraded":"2024-03-24T10:45:02.956031+0000","last_fullsized":"2024-03-24T10:45:02.956031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:15:11.107915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:03.133907+0000","last_change":"2024-03-24T10:45:00.626484+0000","last_active":"2024-03-24T10:45:03.133907+0000","last_peered":"2024-03-24T10:45:03.133907+0000","last_clean":"2024-03-24T10:45:03.133907+0000","last_became_active":"2024-03-24T10:45:00.626195+0000","last_became_peered":"2024-03-24T10:45:00.626195+0000","last_unstale":"2024-03-24T10:45:03.133907+0000","last_undegraded":"2024-03-24T10:45:03.133907+0000","last_fullsized":"2024-03-24T10:45:03.133907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956227+0000","last_change":"2024-03-24T10:45:01.856995+0000","last_active":"2024-03-24T10:45:02.956227+0000","last_peered":"2024-03-24T10:45:02.956227+0000","last_clean":"2024-03-24T10:45:02.956227+0000","last_became_active":"2024-03-24T10:45:01.856531+0000","last_became_peered":"2024-03-24T10:45:01.856531+0000","last_unstale":"2024-03-24T10:45:02.956227+0000","last_undegraded":"2024-03-24T10:45:02.956227+0000","last_fullsized":"2024-03-24T10:45:02.956227+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:29:31.508911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933424+0000","last_change":"2024-03-24T10:45:00.628438+0000","last_active":"2024-03-24T10:45:02.933424+0000","last_peered":"2024-03-24T10:45:02.933424+0000","last_clean":"2024-03-24T10:45:02.933424+0000","last_became_active":"2024-03-24T10:45:00.628158+0000","last_became_peered":"2024-03-24T10:45:00.628158+0000","last_unstale":"2024-03-24T10:45:02.933424+0000","last_undegraded":"2024-03-24T10:45:02.933424+0000","last_fullsized":"2024-03-24T10:45:02.933424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:29.826202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954370+0000","last_change":"2024-03-24T10:45:01.951927+0000","last_active":"2024-03-24T10:45:02.954370+0000","last_peered":"2024-03-24T10:45:02.954370+0000","last_clean":"2024-03-24T10:45:02.954370+0000","last_became_active":"2024-03-24T10:45:01.951165+0000","last_became_peered":"2024-03-24T10:45:01.951165+0000","last_unstale":"2024-03-24T10:45:02.954370+0000","last_undegraded":"2024-03-24T10:45:02.954370+0000","last_fullsized":"2024-03-24T10:45:02.954370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:07:24.149961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:10.385450+0000","last_change":"2024-03-24T10:45:00.639093+0000","last_active":"2024-03-24T10:45:10.385450+0000","last_peered":"2024-03-24T10:45:10.385450+0000","last_clean":"2024-03-24T10:45:10.385450+0000","last_became_active":"2024-03-24T10:45:00.638785+0000","last_became_peered":"2024-03-24T10:45:00.638785+0000","last_unstale":"2024-03-24T10:45:10.385450+0000","last_undegraded":"2024-03-24T10:45:10.385450+0000","last_fullsized":"2024-03-24T10:45:10.385450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955920+0000","last_change":"2024-03-24T10:45:01.953772+0000","last_active":"2024-03-24T10:45:02.955920+0000","last_peered":"2024-03-24T10:45:02.955920+0000","last_clean":"2024-03-24T10:45:02.955920+0000","last_became_active":"2024-03-24T10:45:01.953402+0000","last_became_peered":"2024-03-24T10:45:01.953402+0000","last_unstale":"2024-03-24T10:45:02.955920+0000","last_undegraded":"2024-03-24T10:45:02.955920+0000","last_fullsized":"2024-03-24T10:45:02.955920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:22.089070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955402+0000","last_change":"2024-03-24T10:44:48.241074+0000","last_active":"2024-03-24T10:45:02.955402+0000","last_peered":"2024-03-24T10:45:02.955402+0000","last_clean":"2024-03-24T10:45:02.955402+0000","last_became_active":"2024-03-24T10:44:48.239986+0000","last_became_peered":"2024-03-24T10:44:48.239986+0000","last_unstale":"2024-03-24T10:45:02.955402+0000","last_undegraded":"2024-03-24T10:45:02.955402+0000","last_fullsized":"2024-03-24T10:45:02.955402+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:46.142247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:46.142247+0000","last_clean_scrub_stamp":"2024-03-24T10:44:46.142247+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:05:50.110100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.697116+0000","last_change":"2024-03-24T10:45:00.637276+0000","last_active":"2024-03-24T10:45:07.697116+0000","last_peered":"2024-03-24T10:45:07.697116+0000","last_clean":"2024-03-24T10:45:07.697116+0000","last_became_active":"2024-03-24T10:45:00.636734+0000","last_became_peered":"2024-03-24T10:45:00.636734+0000","last_unstale":"2024-03-24T10:45:07.697116+0000","last_undegraded":"2024-03-24T10:45:07.697116+0000","last_fullsized":"2024-03-24T10:45:07.697116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954655+0000","last_change":"2024-03-24T10:45:01.958505+0000","last_active":"2024-03-24T10:45:02.954655+0000","last_peered":"2024-03-24T10:45:02.954655+0000","last_clean":"2024-03-24T10:45:02.954655+0000","last_became_active":"2024-03-24T10:45:01.958268+0000","last_became_peered":"2024-03-24T10:45:01.958268+0000","last_unstale":"2024-03-24T10:45:02.954655+0000","last_undegraded":"2024-03-24T10:45:02.954655+0000","last_fullsized":"2024-03-24T10:45:02.954655+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:27:12.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:10.260890+0000","last_change":"2024-03-24T10:45:00.597856+0000","last_active":"2024-03-24T10:45:10.260890+0000","last_peered":"2024-03-24T10:45:10.260890+0000","last_clean":"2024-03-24T10:45:10.260890+0000","last_became_active":"2024-03-24T10:45:00.597640+0000","last_became_peered":"2024-03-24T10:45:00.597640+0000","last_unstale":"2024-03-24T10:45:10.260890+0000","last_undegraded":"2024-03-24T10:45:10.260890+0000","last_fullsized":"2024-03-24T10:45:10.260890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:29.826202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.952842+0000","last_change":"2024-03-24T10:45:01.897502+0000","last_active":"2024-03-24T10:45:02.952842+0000","last_peered":"2024-03-24T10:45:02.952842+0000","last_clean":"2024-03-24T10:45:02.952842+0000","last_became_active":"2024-03-24T10:45:01.897250+0000","last_became_peered":"2024-03-24T10:45:01.897250+0000","last_unstale":"2024-03-24T10:45:02.952842+0000","last_undegraded":"2024-03-24T10:45:02.952842+0000","last_fullsized":"2024-03-24T10:45:02.952842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:09:46.341672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:10.385435+0000","last_change":"2024-03-24T10:45:00.627166+0000","last_active":"2024-03-24T10:45:10.385435+0000","last_peered":"2024-03-24T10:45:10.385435+0000","last_clean":"2024-03-24T10:45:10.385435+0000","last_became_active":"2024-03-24T10:45:00.626881+0000","last_became_peered":"2024-03-24T10:45:00.626881+0000","last_unstale":"2024-03-24T10:45:10.385435+0000","last_undegraded":"2024-03-24T10:45:10.385435+0000","last_fullsized":"2024-03-24T10:45:10.385435+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.932815+0000","last_change":"2024-03-24T10:45:01.930271+0000","last_active":"2024-03-24T10:45:02.932815+0000","last_peered":"2024-03-24T10:45:02.932815+0000","last_clean":"2024-03-24T10:45:02.932815+0000","last_became_active":"2024-03-24T10:45:01.929998+0000","last_became_peered":"2024-03-24T10:45:01.929998+0000","last_unstale":"2024-03-24T10:45:02.932815+0000","last_undegraded":"2024-03-24T10:45:02.932815+0000","last_fullsized":"2024-03-24T10:45:02.932815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954888+0000","last_change":"2024-03-24T10:45:00.638309+0000","last_active":"2024-03-24T10:45:02.954888+0000","last_peered":"2024-03-24T10:45:02.954888+0000","last_clean":"2024-03-24T10:45:02.954888+0000","last_became_active":"2024-03-24T10:45:00.638041+0000","last_became_peered":"2024-03-24T10:45:00.638041+0000","last_unstale":"2024-03-24T10:45:02.954888+0000","last_undegraded":"2024-03-24T10:45:02.954888+0000","last_fullsized":"2024-03-24T10:45:02.954888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:14.049099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953823+0000","last_change":"2024-03-24T10:45:01.897722+0000","last_active":"2024-03-24T10:45:02.953823+0000","last_peered":"2024-03-24T10:45:02.953823+0000","last_clean":"2024-03-24T10:45:02.953823+0000","last_became_active":"2024-03-24T10:45:01.897159+0000","last_became_peered":"2024-03-24T10:45:01.897159+0000","last_unstale":"2024-03-24T10:45:02.953823+0000","last_undegraded":"2024-03-24T10:45:02.953823+0000","last_fullsized":"2024-03-24T10:45:02.953823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:42.293342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933199+0000","last_change":"2024-03-24T10:45:00.629267+0000","last_active":"2024-03-24T10:45:02.933199+0000","last_peered":"2024-03-24T10:45:02.933199+0000","last_clean":"2024-03-24T10:45:02.933199+0000","last_became_active":"2024-03-24T10:45:00.628971+0000","last_became_peered":"2024-03-24T10:45:00.628971+0000","last_unstale":"2024-03-24T10:45:02.933199+0000","last_undegraded":"2024-03-24T10:45:02.933199+0000","last_fullsized":"2024-03-24T10:45:02.933199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:41.176786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954115+0000","last_change":"2024-03-24T10:45:01.952527+0000","last_active":"2024-03-24T10:45:02.954115+0000","last_peered":"2024-03-24T10:45:02.954115+0000","last_clean":"2024-03-24T10:45:02.954115+0000","last_became_active":"2024-03-24T10:45:01.952192+0000","last_became_peered":"2024-03-24T10:45:01.952192+0000","last_unstale":"2024-03-24T10:45:02.954115+0000","last_undegraded":"2024-03-24T10:45:02.954115+0000","last_fullsized":"2024-03-24T10:45:02.954115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956049+0000","last_change":"2024-03-24T10:45:00.631941+0000","last_active":"2024-03-24T10:45:02.956049+0000","last_peered":"2024-03-24T10:45:02.956049+0000","last_clean":"2024-03-24T10:45:02.956049+0000","last_became_active":"2024-03-24T10:45:00.631670+0000","last_became_peered":"2024-03-24T10:45:00.631670+0000","last_unstale":"2024-03-24T10:45:02.956049+0000","last_undegraded":"2024-03-24T10:45:02.956049+0000","last_fullsized":"2024-03-24T10:45:02.956049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:09:45.225115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933116+0000","last_change":"2024-03-24T10:45:01.930483+0000","last_active":"2024-03-24T10:45:02.933116+0000","last_peered":"2024-03-24T10:45:02.933116+0000","last_clean":"2024-03-24T10:45:02.933116+0000","last_became_active":"2024-03-24T10:45:01.930220+0000","last_became_peered":"2024-03-24T10:45:01.930220+0000","last_unstale":"2024-03-24T10:45:02.933116+0000","last_undegraded":"2024-03-24T10:45:02.933116+0000","last_fullsized":"2024-03-24T10:45:02.933116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:10.391033+0000","last_change":"2024-03-24T10:45:00.633650+0000","last_active":"2024-03-24T10:45:10.391033+0000","last_peered":"2024-03-24T10:45:10.391033+0000","last_clean":"2024-03-24T10:45:10.391033+0000","last_became_active":"2024-03-24T10:45:00.633329+0000","last_became_peered":"2024-03-24T10:45:00.633329+0000","last_unstale":"2024-03-24T10:45:10.391033+0000","last_undegraded":"2024-03-24T10:45:10.391033+0000","last_fullsized":"2024-03-24T10:45:10.391033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954759+0000","last_change":"2024-03-24T10:45:01.853487+0000","last_active":"2024-03-24T10:45:02.954759+0000","last_peered":"2024-03-24T10:45:02.954759+0000","last_clean":"2024-03-24T10:45:02.954759+0000","last_became_active":"2024-03-24T10:45:01.853257+0000","last_became_peered":"2024-03-24T10:45:01.853257+0000","last_unstale":"2024-03-24T10:45:02.954759+0000","last_undegraded":"2024-03-24T10:45:02.954759+0000","last_fullsized":"2024-03-24T10:45:02.954759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.771497+0000","last_change":"2024-03-24T10:45:00.639282+0000","last_active":"2024-03-24T10:45:07.771497+0000","last_peered":"2024-03-24T10:45:07.771497+0000","last_clean":"2024-03-24T10:45:07.771497+0000","last_became_active":"2024-03-24T10:45:00.638999+0000","last_became_peered":"2024-03-24T10:45:00.638999+0000","last_unstale":"2024-03-24T10:45:07.771497+0000","last_undegraded":"2024-03-24T10:45:07.771497+0000","last_fullsized":"2024-03-24T10:45:07.771497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:41.176786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953535+0000","last_change":"2024-03-24T10:45:01.928900+0000","last_active":"2024-03-24T10:45:02.953535+0000","last_peered":"2024-03-24T10:45:02.953535+0000","last_clean":"2024-03-24T10:45:02.953535+0000","last_became_active":"2024-03-24T10:45:01.928610+0000","last_became_peered":"2024-03-24T10:45:01.928610+0000","last_unstale":"2024-03-24T10:45:02.953535+0000","last_undegraded":"2024-03-24T10:45:02.953535+0000","last_fullsized":"2024-03-24T10:45:02.953535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.772043+0000","last_change":"2024-03-24T10:45:00.631327+0000","last_active":"2024-03-24T10:45:07.772043+0000","last_peered":"2024-03-24T10:45:07.772043+0000","last_clean":"2024-03-24T10:45:07.772043+0000","last_became_active":"2024-03-24T10:45:00.631078+0000","last_became_peered":"2024-03-24T10:45:00.631078+0000","last_unstale":"2024-03-24T10:45:07.772043+0000","last_undegraded":"2024-03-24T10:45:07.772043+0000","last_fullsized":"2024-03-24T10:45:07.772043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:43.263796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953696+0000","last_change":"2024-03-24T10:45:01.897329+0000","last_active":"2024-03-24T10:45:02.953696+0000","last_peered":"2024-03-24T10:45:02.953696+0000","last_clean":"2024-03-24T10:45:02.953696+0000","last_became_active":"2024-03-24T10:45:01.897063+0000","last_became_peered":"2024-03-24T10:45:01.897063+0000","last_unstale":"2024-03-24T10:45:02.953696+0000","last_undegraded":"2024-03-24T10:45:02.953696+0000","last_fullsized":"2024-03-24T10:45:02.953696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:07.259007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.797764+0000","last_change":"2024-03-24T10:45:00.597948+0000","last_active":"2024-03-24T10:45:07.797764+0000","last_peered":"2024-03-24T10:45:07.797764+0000","last_clean":"2024-03-24T10:45:07.797764+0000","last_became_active":"2024-03-24T10:45:00.597671+0000","last_became_peered":"2024-03-24T10:45:00.597671+0000","last_unstale":"2024-03-24T10:45:07.797764+0000","last_undegraded":"2024-03-24T10:45:07.797764+0000","last_fullsized":"2024-03-24T10:45:07.797764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:06.142451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954368+0000","last_change":"2024-03-24T10:45:01.927822+0000","last_active":"2024-03-24T10:45:02.954368+0000","last_peered":"2024-03-24T10:45:02.954368+0000","last_clean":"2024-03-24T10:45:02.954368+0000","last_became_active":"2024-03-24T10:45:01.927538+0000","last_became_peered":"2024-03-24T10:45:01.927538+0000","last_unstale":"2024-03-24T10:45:02.954368+0000","last_undegraded":"2024-03-24T10:45:02.954368+0000","last_fullsized":"2024-03-24T10:45:02.954368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933096+0000","last_change":"2024-03-24T10:45:00.626818+0000","last_active":"2024-03-24T10:45:02.933096+0000","last_peered":"2024-03-24T10:45:02.933096+0000","last_clean":"2024-03-24T10:45:02.933096+0000","last_became_active":"2024-03-24T10:45:00.626557+0000","last_became_peered":"2024-03-24T10:45:00.626557+0000","last_unstale":"2024-03-24T10:45:02.933096+0000","last_undegraded":"2024-03-24T10:45:02.933096+0000","last_fullsized":"2024-03-24T10:45:02.933096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:43.263796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954131+0000","last_change":"2024-03-24T10:45:01.893192+0000","last_active":"2024-03-24T10:45:02.954131+0000","last_peered":"2024-03-24T10:45:02.954131+0000","last_clean":"2024-03-24T10:45:02.954131+0000","last_became_active":"2024-03-24T10:45:01.892934+0000","last_became_peered":"2024-03-24T10:45:01.892934+0000","last_unstale":"2024-03-24T10:45:02.954131+0000","last_undegraded":"2024-03-24T10:45:02.954131+0000","last_fullsized":"2024-03-24T10:45:02.954131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.772069+0000","last_change":"2024-03-24T10:45:00.630147+0000","last_active":"2024-03-24T10:45:07.772069+0000","last_peered":"2024-03-24T10:45:07.772069+0000","last_clean":"2024-03-24T10:45:07.772069+0000","last_became_active":"2024-03-24T10:45:00.629850+0000","last_became_peered":"2024-03-24T10:45:00.629850+0000","last_unstale":"2024-03-24T10:45:07.772069+0000","last_undegraded":"2024-03-24T10:45:07.772069+0000","last_fullsized":"2024-03-24T10:45:07.772069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:01:12.843222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953404+0000","last_change":"2024-03-24T10:45:01.897333+0000","last_active":"2024-03-24T10:45:02.953404+0000","last_peered":"2024-03-24T10:45:02.953404+0000","last_clean":"2024-03-24T10:45:02.953404+0000","last_became_active":"2024-03-24T10:45:01.897080+0000","last_became_peered":"2024-03-24T10:45:01.897080+0000","last_unstale":"2024-03-24T10:45:02.953404+0000","last_undegraded":"2024-03-24T10:45:02.953404+0000","last_fullsized":"2024-03-24T10:45:02.953404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:13.887947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953968+0000","last_change":"2024-03-24T10:45:01.956041+0000","last_active":"2024-03-24T10:45:02.953968+0000","last_peered":"2024-03-24T10:45:02.953968+0000","last_clean":"2024-03-24T10:45:02.953968+0000","last_became_active":"2024-03-24T10:45:01.955766+0000","last_became_peered":"2024-03-24T10:45:01.955766+0000","last_unstale":"2024-03-24T10:45:02.953968+0000","last_undegraded":"2024-03-24T10:45:02.953968+0000","last_fullsized":"2024-03-24T10:45:02.953968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:16.341360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:03.262738+0000","last_change":"2024-03-24T10:45:00.639913+0000","last_active":"2024-03-24T10:45:03.262738+0000","last_peered":"2024-03-24T10:45:03.262738+0000","last_clean":"2024-03-24T10:45:03.262738+0000","last_became_active":"2024-03-24T10:45:00.639653+0000","last_became_peered":"2024-03-24T10:45:00.639653+0000","last_unstale":"2024-03-24T10:45:03.262738+0000","last_undegraded":"2024-03-24T10:45:03.262738+0000","last_fullsized":"2024-03-24T10:45:03.262738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:15.749451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954233+0000","last_change":"2024-03-24T10:45:01.952340+0000","last_active":"2024-03-24T10:45:02.954233+0000","last_peered":"2024-03-24T10:45:02.954233+0000","last_clean":"2024-03-24T10:45:02.954233+0000","last_became_active":"2024-03-24T10:45:01.952074+0000","last_became_peered":"2024-03-24T10:45:01.952074+0000","last_unstale":"2024-03-24T10:45:02.954233+0000","last_undegraded":"2024-03-24T10:45:02.954233+0000","last_fullsized":"2024-03-24T10:45:02.954233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:10.260946+0000","last_change":"2024-03-24T10:45:00.633790+0000","last_active":"2024-03-24T10:45:10.260946+0000","last_peered":"2024-03-24T10:45:10.260946+0000","last_clean":"2024-03-24T10:45:10.260946+0000","last_became_active":"2024-03-24T10:45:00.633494+0000","last_became_peered":"2024-03-24T10:45:00.633494+0000","last_unstale":"2024-03-24T10:45:10.260946+0000","last_undegraded":"2024-03-24T10:45:10.260946+0000","last_fullsized":"2024-03-24T10:45:10.260946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:14.049099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953320+0000","last_change":"2024-03-24T10:45:01.957958+0000","last_active":"2024-03-24T10:45:02.953320+0000","last_peered":"2024-03-24T10:45:02.953320+0000","last_clean":"2024-03-24T10:45:02.953320+0000","last_became_active":"2024-03-24T10:45:01.957676+0000","last_became_peered":"2024-03-24T10:45:01.957676+0000","last_unstale":"2024-03-24T10:45:02.953320+0000","last_undegraded":"2024-03-24T10:45:02.953320+0000","last_fullsized":"2024-03-24T10:45:02.953320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:03.159944+0000","last_change":"2024-03-24T10:45:00.629120+0000","last_active":"2024-03-24T10:45:03.159944+0000","last_peered":"2024-03-24T10:45:03.159944+0000","last_clean":"2024-03-24T10:45:03.159944+0000","last_became_active":"2024-03-24T10:45:00.628712+0000","last_became_peered":"2024-03-24T10:45:00.628712+0000","last_unstale":"2024-03-24T10:45:03.159944+0000","last_undegraded":"2024-03-24T10:45:03.159944+0000","last_fullsized":"2024-03-24T10:45:03.159944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954346+0000","last_change":"2024-03-24T10:45:01.894602+0000","last_active":"2024-03-24T10:45:02.954346+0000","last_peered":"2024-03-24T10:45:02.954346+0000","last_clean":"2024-03-24T10:45:02.954346+0000","last_became_active":"2024-03-24T10:45:01.894362+0000","last_became_peered":"2024-03-24T10:45:01.894362+0000","last_unstale":"2024-03-24T10:45:02.954346+0000","last_undegraded":"2024-03-24T10:45:02.954346+0000","last_fullsized":"2024-03-24T10:45:02.954346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:03.223274+0000","last_change":"2024-03-24T10:45:00.639171+0000","last_active":"2024-03-24T10:45:03.223274+0000","last_peered":"2024-03-24T10:45:03.223274+0000","last_clean":"2024-03-24T10:45:03.223274+0000","last_became_active":"2024-03-24T10:45:00.638871+0000","last_became_peered":"2024-03-24T10:45:00.638871+0000","last_unstale":"2024-03-24T10:45:03.223274+0000","last_undegraded":"2024-03-24T10:45:03.223274+0000","last_fullsized":"2024-03-24T10:45:03.223274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:52:46.322196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4634,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":9268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65430,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542153,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31468,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340372,"statfs":{"total":96636764160,"available":96604540928,"internally_reserved":0,"allocated":299008,"data_stored":665889,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542153,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":74967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542152,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":76439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542153,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":75023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542153,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":237568,"data_stored":75149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":229376,"data_stored":75907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":74331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31464,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340376,"statfs":{"total":96636764160,"available":96604545024,"internally_reserved":0,"allocated":294912,"data_stored":665233,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:45:11.527 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T10:45:11.527 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T10:45:11.528 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-24T10:45:11.528 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-24T10:45:11.528 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-24T10:45:11.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.669+0000 7f269296c640 1 Processor -- start 2024-03-24T10:45:11.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.669+0000 7f269296c640 1 -- start start 2024-03-24T10:45:11.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.669+0000 7f269296c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f268c1543a0 0x7f268c15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:11.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.669+0000 7f269296c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f268c06d4f0 0x7f268c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:11.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.669+0000 7f269296c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f268c0cb2f0 con 0x7f268c0ccaa0 2024-03-24T10:45:11.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.669+0000 7f269296c640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f268c0c6750 con 0x7f268c1543a0 2024-03-24T10:45:11.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.669+0000 7f269296c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f268c0cc380 con 0x7f268c06d4f0 2024-03-24T10:45:11.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.670+0000 7f268bfff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f268c1543a0 0x7f268c15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:11.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.670+0000 7f268bfff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f268c1543a0 0x7f268c15ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:34500/0 (socket says 172.21.15.5:34500) 2024-03-24T10:45:11.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.670+0000 7f268bfff640 1 -- 172.21.15.5:0/3883103077 learned_addr learned my addr 172.21.15.5:0/3883103077 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:11.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.671+0000 7f268b7fe640 1 --2- 172.21.15.5:0/3883103077 >> v2:172.21.15.57:3300/0 conn(0x7f268c06d4f0 0x7f268c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:11.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.673+0000 7f268bfff640 1 -- 172.21.15.5:0/3883103077 >> v2:172.21.15.57:3300/0 conn(0x7f268c06d4f0 msgr2=0x7f268c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:11.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.673+0000 7f268bfff640 1 --2- 172.21.15.5:0/3883103077 >> v2:172.21.15.57:3300/0 conn(0x7f268c06d4f0 0x7f268c0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:11.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.673+0000 7f268bfff640 1 -- 172.21.15.5:0/3883103077 >> v1:172.21.15.57:6789/0 conn(0x7f268c0ccaa0 legacy=0x7f268c06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:11.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.673+0000 7f268bfff640 1 -- 172.21.15.5:0/3883103077 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268c0ca7e0 con 0x7f268c1543a0 2024-03-24T10:45:11.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.674+0000 7f268bfff640 1 --2- 172.21.15.5:0/3883103077 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f268c1543a0 0x7f268c15ac00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f268c0c6990 tx=0x7f267c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=689adfca7d4b0394 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:11.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.678+0000 7f268affd640 1 -- 172.21.15.5:0/3883103077 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c02f850 con 0x7f268c1543a0 2024-03-24T10:45:11.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.678+0000 7f268affd640 1 -- 172.21.15.5:0/3883103077 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f267c02f9b0 con 0x7f268c1543a0 2024-03-24T10:45:11.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.678+0000 7f268affd640 1 -- 172.21.15.5:0/3883103077 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c02fd40 con 0x7f268c1543a0 2024-03-24T10:45:11.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.679+0000 7f269296c640 1 -- 172.21.15.5:0/3883103077 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f268c1543a0 msgr2=0x7f268c15ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:11.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.679+0000 7f269296c640 1 --2- 172.21.15.5:0/3883103077 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f268c1543a0 0x7f268c15ac00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f268c0c6990 tx=0x7f267c02ba30 comp rx=0 tx=0).stop 2024-03-24T10:45:11.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.679+0000 7f269296c640 1 -- 172.21.15.5:0/3883103077 shutdown_connections 2024-03-24T10:45:11.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.679+0000 7f269296c640 1 --2- 172.21.15.5:0/3883103077 >> v2:172.21.15.57:3300/0 conn(0x7f268c06d4f0 0x7f268c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:11.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.679+0000 7f269296c640 1 --2- 172.21.15.5:0/3883103077 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f268c1543a0 0x7f268c15ac00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:11.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.679+0000 7f269296c640 1 -- 172.21.15.5:0/3883103077 >> 172.21.15.5:0/3883103077 conn(0x7f268c072ef0 msgr2=0x7f268c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:11.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.679+0000 7f269296c640 1 -- 172.21.15.5:0/3883103077 shutdown_connections 2024-03-24T10:45:11.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.679+0000 7f269296c640 1 -- 172.21.15.5:0/3883103077 wait complete. 2024-03-24T10:45:11.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.680+0000 7f269296c640 1 Processor -- start 2024-03-24T10:45:11.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.681+0000 7f269296c640 1 -- start start 2024-03-24T10:45:11.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.681+0000 7f269296c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f268c06d4f0 0x7f268c0b4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:11.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.681+0000 7f269296c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f268c0ccaa0 0x7f268c15c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:11.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.681+0000 7f269296c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f268c1543a0 0x7f268c157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:11.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.681+0000 7f269296c640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f268c1f3dd0 con 0x7f268c06d4f0 2024-03-24T10:45:11.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.681+0000 7f269296c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f268c154a60 con 0x7f268c0ccaa0 2024-03-24T10:45:11.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.681+0000 7f269296c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f268c0b47b0 con 0x7f268c1543a0 2024-03-24T10:45:11.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.682+0000 7f268bfff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f268c06d4f0 0x7f268c0b4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:11.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.682+0000 7f268b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f268c0ccaa0 0x7f268c15c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:11.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.682+0000 7f268b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f268c0ccaa0 0x7f268c15c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:34754/0 (socket says 172.21.15.5:34754) 2024-03-24T10:45:11.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.682+0000 7f268b7fe640 1 -- 172.21.15.5:0/2221862626 learned_addr learned my addr 172.21.15.5:0/2221862626 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:11.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.682+0000 7f2690f09640 1 --2- 172.21.15.5:0/2221862626 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f268c1543a0 0x7f268c157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:11.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.686+0000 7f268bfff640 1 -- 172.21.15.5:0/2221862626 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f268c1543a0 msgr2=0x7f268c157480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:11.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.686+0000 7f268bfff640 1 --2- 172.21.15.5:0/2221862626 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f268c1543a0 0x7f268c157480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:11.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.686+0000 7f268bfff640 1 -- 172.21.15.5:0/2221862626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f268c0ccaa0 msgr2=0x7f268c15c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:11.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.686+0000 7f268bfff640 1 --2- 172.21.15.5:0/2221862626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f268c0ccaa0 0x7f268c15c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:11.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.686+0000 7f268bfff640 1 -- 172.21.15.5:0/2221862626 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f267c0094d0 con 0x7f268c06d4f0 2024-03-24T10:45:11.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.688+0000 7f268bfff640 1 --2- 172.21.15.5:0/2221862626 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f268c06d4f0 0x7f268c0b4c40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f267c033000 tx=0x7f267c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:11.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.692+0000 7f2688ff9640 1 -- 172.21.15.5:0/2221862626 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c037c40 con 0x7f268c06d4f0 2024-03-24T10:45:11.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.692+0000 7f2688ff9640 1 -- 172.21.15.5:0/2221862626 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f267c037dc0 con 0x7f268c06d4f0 2024-03-24T10:45:11.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.692+0000 7f269296c640 1 -- 172.21.15.5:0/2221862626 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f268c0b30f0 con 0x7f268c06d4f0 2024-03-24T10:45:11.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.692+0000 7f2688ff9640 1 -- 172.21.15.5:0/2221862626 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c041640 con 0x7f268c06d4f0 2024-03-24T10:45:11.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.692+0000 7f269296c640 1 -- 172.21.15.5:0/2221862626 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f268c1f3af0 con 0x7f268c06d4f0 2024-03-24T10:45:11.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.693+0000 7f269296c640 1 -- 172.21.15.5:0/2221862626 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f268c0681d0 con 0x7f268c06d4f0 2024-03-24T10:45:11.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.960+0000 7f2688ff9640 1 -- 172.21.15.5:0/2221862626 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f267c0417a0 con 0x7f268c06d4f0 2024-03-24T10:45:11.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.960+0000 7f2688ff9640 1 --2- 172.21.15.5:0/2221862626 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f267806b890 0x7f267806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:11.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.960+0000 7f268b7fe640 1 --2- 172.21.15.5:0/2221862626 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f267806b890 0x7f267806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:11.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.961+0000 7f268b7fe640 1 --2- 172.21.15.5:0/2221862626 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f267806b890 0x7f267806dd50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f268c0ca1c0 tx=0x7f2680008040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:11.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.974+0000 7f2688ff9640 1 -- 172.21.15.5:0/2221862626 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f267c03c070 con 0x7f268c06d4f0 2024-03-24T10:45:11.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:11.974+0000 7f2688ff9640 1 -- 172.21.15.5:0/2221862626 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f267c0783a0 con 0x7f268c06d4f0 2024-03-24T10:45:12.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.204+0000 7f269296c640 1 -- 172.21.15.5:0/2221862626 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f268c0cae80 con 0x7f268c06d4f0 2024-03-24T10:45:12.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.352+0000 7f2688ff9640 1 -- 172.21.15.5:0/2221862626 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+171776 (secure 0 0 0) 0x7f267c07ba60 con 0x7f268c06d4f0 2024-03-24T10:45:12.355 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:45:12.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.355+0000 7f269296c640 1 -- 172.21.15.5:0/2221862626 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f267806b890 msgr2=0x7f267806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:12.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.355+0000 7f269296c640 1 --2- 172.21.15.5:0/2221862626 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f267806b890 0x7f267806dd50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f268c0ca1c0 tx=0x7f2680008040 comp rx=0 tx=0).stop 2024-03-24T10:45:12.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.355+0000 7f269296c640 1 -- 172.21.15.5:0/2221862626 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f268c06d4f0 msgr2=0x7f268c0b4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:12.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.355+0000 7f269296c640 1 --2- 172.21.15.5:0/2221862626 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f268c06d4f0 0x7f268c0b4c40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f267c033000 tx=0x7f267c004730 comp rx=0 tx=0).stop 2024-03-24T10:45:12.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.355+0000 7f269296c640 1 -- 172.21.15.5:0/2221862626 shutdown_connections 2024-03-24T10:45:12.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.355+0000 7f269296c640 1 --2- 172.21.15.5:0/2221862626 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f267806b890 0x7f267806dd50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:12.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.355+0000 7f269296c640 1 --2- 172.21.15.5:0/2221862626 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f268c1543a0 0x7f268c157480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:12.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.355+0000 7f269296c640 1 --2- 172.21.15.5:0/2221862626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f268c0ccaa0 0x7f268c15c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:12.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.355+0000 7f269296c640 1 --2- 172.21.15.5:0/2221862626 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f268c06d4f0 0x7f268c0b4c40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:12.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.355+0000 7f269296c640 1 -- 172.21.15.5:0/2221862626 >> 172.21.15.5:0/2221862626 conn(0x7f268c072ef0 msgr2=0x7f268c153480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:12.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.355+0000 7f269296c640 1 -- 172.21.15.5:0/2221862626 shutdown_connections 2024-03-24T10:45:12.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.355+0000 7f269296c640 1 -- 172.21.15.5:0/2221862626 wait complete. 2024-03-24T10:45:12.367 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":5,"flags":0,"active_gid":4120,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":1102439181},{"type":"v1","addr":"172.21.15.5:6833","nonce":1102439181}]},"active_addr":"172.21.15.5:6833/1102439181","active_change":"2024-03-24T10:44:37.089074+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4107,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":510072697}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":3457551642}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":3826288372}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":185556538}]}]} 2024-03-24T10:45:12.371 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-24T10:45:12.371 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-24T10:45:12.371 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:45:12.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.477+0000 7fabb5c4e640 1 Processor -- start 2024-03-24T10:45:12.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.477+0000 7fabb5c4e640 1 -- start start 2024-03-24T10:45:12.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.477+0000 7fabb5c4e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fabb015ac00 0x7fabb00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:12.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.477+0000 7fabb5c4e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fabb00ccaa0 0x7fabb006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:12.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.477+0000 7fabb5c4e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabb00cb2f0 con 0x7fabb01543a0 2024-03-24T10:45:12.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.477+0000 7fabb5c4e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fabb00c6750 con 0x7fabb00ccaa0 2024-03-24T10:45:12.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.477+0000 7fabb5c4e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fabb00cc380 con 0x7fabb015ac00 2024-03-24T10:45:12.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.478+0000 7fabaf7fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fabb01543a0 0x7fabb006d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59660/0 (socket says 172.21.15.5:59660) 2024-03-24T10:45:12.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.478+0000 7fabaf7fe640 1 -- 172.21.15.5:0/2630357610 learned_addr learned my addr 172.21.15.5:0/2630357610 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:12.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.481+0000 7fabae7fc640 1 -- 172.21.15.5:0/2630357610 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3382647457 0 0) 0x7fabb00cb2f0 con 0x7fabb01543a0 2024-03-24T10:45:12.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.481+0000 7fabae7fc640 1 -- 172.21.15.5:0/2630357610 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab98003520 con 0x7fabb01543a0 2024-03-24T10:45:12.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.486+0000 7fabae7fc640 1 -- 172.21.15.5:0/2630357610 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2689742410 0 0) 0x7fab98003520 con 0x7fabb01543a0 2024-03-24T10:45:12.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.486+0000 7fabae7fc640 1 -- 172.21.15.5:0/2630357610 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fabb015ac00 msgr2=0x7fabb00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:12.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.486+0000 7fabae7fc640 1 --2- 172.21.15.5:0/2630357610 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fabb015ac00 0x7fabb00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:12.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.486+0000 7fabae7fc640 1 -- 172.21.15.5:0/2630357610 >> v2:172.21.15.57:3300/0 conn(0x7fabb00ccaa0 msgr2=0x7fabb006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:12.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.486+0000 7fabae7fc640 1 --2- 172.21.15.5:0/2630357610 >> v2:172.21.15.57:3300/0 conn(0x7fabb00ccaa0 0x7fabb006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:12.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.486+0000 7fabae7fc640 1 -- 172.21.15.5:0/2630357610 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb015b070 con 0x7fabb01543a0 2024-03-24T10:45:12.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.486+0000 7fabae7fc640 1 -- 172.21.15.5:0/2630357610 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 100681612 0 0) 0x7faba0001670 con 0x7fabb01543a0 2024-03-24T10:45:12.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.486+0000 7fabae7fc640 1 -- 172.21.15.5:0/2630357610 >> v1:172.21.15.5:6789/0 conn(0x7fabb01543a0 legacy=0x7fabb006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:12.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.486+0000 7fabae7fc640 1 --2- 172.21.15.5:0/2630357610 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab98003970 0x7fab98003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:12.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.486+0000 7fabae7fc640 1 --2- 172.21.15.5:0/2630357610 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fabb015ac00 0x7fab980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:12.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.486+0000 7fabae7fc640 1 --2- 172.21.15.5:0/2630357610 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab98004e20 0x7fab9800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:12.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.486+0000 7fabae7fc640 1 -- 172.21.15.5:0/2630357610 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fabb00c6750 con 0x7fab98003970 2024-03-24T10:45:12.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.486+0000 7fabae7fc640 1 -- 172.21.15.5:0/2630357610 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fabb00cc380 con 0x7fab98004e20 2024-03-24T10:45:12.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.486+0000 7fabae7fc640 1 -- 172.21.15.5:0/2630357610 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fab98005580 con 0x7fabb015ac00 2024-03-24T10:45:12.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.490+0000 7fabaf7fe640 1 --2- 172.21.15.5:0/2630357610 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab98003970 0x7fab98003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:12.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.493+0000 7fabaf7fe640 1 -- 172.21.15.5:0/2630357610 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fabb015ac00 msgr2=0x7fab980047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:12.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.493+0000 7fabaf7fe640 1 --2- 172.21.15.5:0/2630357610 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fabb015ac00 0x7fab980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:12.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.493+0000 7fabaf7fe640 1 -- 172.21.15.5:0/2630357610 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab98004e20 msgr2=0x7fab9800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:12.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.493+0000 7fabaf7fe640 1 --2- 172.21.15.5:0/2630357610 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab98004e20 0x7fab9800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:12.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.493+0000 7fabaf7fe640 1 -- 172.21.15.5:0/2630357610 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab98003520 con 0x7fab98003970 2024-03-24T10:45:12.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.495+0000 7fabaf7fe640 1 --2- 172.21.15.5:0/2630357610 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab98003970 0x7fab98003d90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faba000c0e0 tx=0x7fab980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9436f1a0f5b36bd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:12.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.497+0000 7fabae7fc640 1 -- 172.21.15.5:0/2630357610 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba0005ad0 con 0x7fab98003970 2024-03-24T10:45:12.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.500+0000 7fabae7fc640 1 -- 172.21.15.5:0/2630357610 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba0005c30 con 0x7fab98003970 2024-03-24T10:45:12.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.500+0000 7fabae7fc640 1 -- 172.21.15.5:0/2630357610 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba002c810 con 0x7fab98003970 2024-03-24T10:45:12.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.501+0000 7fabb5c4e640 1 -- 172.21.15.5:0/2630357610 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab98003970 msgr2=0x7fab98003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:12.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.501+0000 7fabb5c4e640 1 --2- 172.21.15.5:0/2630357610 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab98003970 0x7fab98003d90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faba000c0e0 tx=0x7fab980057f0 comp rx=0 tx=0).stop 2024-03-24T10:45:12.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.501+0000 7fabaffff640 1 -- 172.21.15.5:0/2630357610 reap_dead start 2024-03-24T10:45:12.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.501+0000 7fabb5c4e640 1 -- 172.21.15.5:0/2630357610 shutdown_connections 2024-03-24T10:45:12.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.501+0000 7fabb5c4e640 1 -- 172.21.15.5:0/2630357610 >> 172.21.15.5:0/2630357610 conn(0x7fabb0072ef0 msgr2=0x7fabb0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:12.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.501+0000 7fabb5c4e640 1 -- 172.21.15.5:0/2630357610 shutdown_connections 2024-03-24T10:45:12.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.501+0000 7fabb5c4e640 1 -- 172.21.15.5:0/2630357610 wait complete. 2024-03-24T10:45:12.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.503+0000 7fabb5c4e640 1 Processor -- start 2024-03-24T10:45:12.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.503+0000 7fabb5c4e640 1 -- start start 2024-03-24T10:45:12.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.504+0000 7fabb5c4e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fab98003970 0x7fabb0155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:12.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.504+0000 7fabb5c4e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabb0152a30 0x7fabb0154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:12.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.504+0000 7fabb5c4e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabb015ce20 0x7fabb01f2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:12.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.504+0000 7fabb5c4e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fabb015dff0 con 0x7fabb0152a30 2024-03-24T10:45:12.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.504+0000 7fabb5c4e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fabb00c5530 con 0x7fabb015ce20 2024-03-24T10:45:12.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.504+0000 7fabb5c4e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fabb00c4c90 con 0x7fab98003970 2024-03-24T10:45:12.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.504+0000 7fabaeffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabb0152a30 0x7fabb0154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:12.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.504+0000 7fabaf7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fab98003970 0x7fabb0155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:12.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.505+0000 7fabaf7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fab98003970 0x7fabb0155f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:51322/0 (socket says 172.21.15.5:51322) 2024-03-24T10:45:12.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.505+0000 7fabaf7fe640 1 -- 172.21.15.5:0/2416760043 learned_addr learned my addr 172.21.15.5:0/2416760043 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:12.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.508+0000 7fabaf7fe640 1 -- 172.21.15.5:0/2416760043 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabb015ce20 msgr2=0x7fabb01f2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:12.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.508+0000 7fabaf7fe640 1 --2- 172.21.15.5:0/2416760043 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabb015ce20 0x7fabb01f2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:12.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.508+0000 7fabaf7fe640 1 -- 172.21.15.5:0/2416760043 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabb0152a30 msgr2=0x7fabb0154cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:12.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.508+0000 7fabaf7fe640 1 --2- 172.21.15.5:0/2416760043 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabb0152a30 0x7fabb0154cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:12.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.508+0000 7fabaf7fe640 1 -- 172.21.15.5:0/2416760043 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb00c4f70 con 0x7fab98003970 2024-03-24T10:45:12.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.511+0000 7fabaf7fe640 1 --2- 172.21.15.5:0/2416760043 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fab98003970 0x7fabb0155f90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7faba000c200 tx=0x7fabb00b3010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:12.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.515+0000 7fab97fff640 1 -- 172.21.15.5:0/2416760043 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba002cc20 con 0x7fab98003970 2024-03-24T10:45:12.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.515+0000 7fab97fff640 1 -- 172.21.15.5:0/2416760043 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba002cda0 con 0x7fab98003970 2024-03-24T10:45:12.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.515+0000 7fab97fff640 1 -- 172.21.15.5:0/2416760043 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba00072b0 con 0x7fab98003970 2024-03-24T10:45:12.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.515+0000 7fabb5c4e640 1 -- 172.21.15.5:0/2416760043 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabb00c5250 con 0x7fab98003970 2024-03-24T10:45:12.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.515+0000 7fabb5c4e640 1 -- 172.21.15.5:0/2416760043 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fabb0155210 con 0x7fab98003970 2024-03-24T10:45:12.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.517+0000 7fabb5c4e640 1 -- 172.21.15.5:0/2416760043 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabb00681d0 con 0x7fab98003970 2024-03-24T10:45:12.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.792+0000 7fab97fff640 1 -- 172.21.15.5:0/2416760043 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7faba0032070 con 0x7fab98003970 2024-03-24T10:45:12.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.793+0000 7fab97fff640 1 --2- 172.21.15.5:0/2416760043 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fab9c06b860 0x7fab9c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:12.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.793+0000 7fabaeffd640 1 --2- 172.21.15.5:0/2416760043 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fab9c06b860 0x7fab9c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:12.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.793+0000 7fabaeffd640 1 --2- 172.21.15.5:0/2416760043 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fab9c06b860 0x7fab9c06dd20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fabb00b2bd0 tx=0x7faba4006f90 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:12.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.796+0000 7fab97fff640 1 -- 172.21.15.5:0/2416760043 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7faba003c040 con 0x7fab98003970 2024-03-24T10:45:12.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.808+0000 7fab97fff640 1 -- 172.21.15.5:0/2416760043 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7faba0076450 con 0x7fab98003970 2024-03-24T10:45:12.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:12.999+0000 7fabb5c4e640 1 -- 172.21.15.5:0/2416760043 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fabb00c4dd0 con 0x7fab98003970 2024-03-24T10:45:13.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.040+0000 7fab97fff640 1 -- 172.21.15.5:0/2416760043 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16485 (secure 0 0 0) 0x7fabb00c4dd0 con 0x7fab98003970 2024-03-24T10:45:13.041 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:45:13.041 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":21,"fsid":"7d9848db-2fe7-4d95-a6e2-b04f4f639ebc","created":"2024-03-24T10:44:30.435176+0000","modified":"2024-03-24T10:45:02.781257+0000","last_up_change":"2024-03-24T10:44:44.634047+0000","last_in_change":"2024-03-24T10:44:35.986391+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:44:45.829320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:44:58.435543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:45:00.129313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50f590b3-f5ec-4e1b-ac46-25cab307a27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6825","nonce":1991336051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6827","nonce":1991336051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6831","nonce":1991336051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6829","nonce":1991336051}]},"public_addr":"172.21.15.5:6825/1991336051","cluster_addr":"172.21.15.5:6827/1991336051","heartbeat_back_addr":"172.21.15.5:6831/1991336051","heartbeat_front_addr":"172.21.15.5:6829/1991336051","state":["exists","up"]},{"osd":1,"uuid":"ccba9e93-c704-472b-a832-8d4eeb456c47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6801","nonce":2140667501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6803","nonce":2140667501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6807","nonce":2140667501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6805","nonce":2140667501}]},"public_addr":"172.21.15.5:6801/2140667501","cluster_addr":"172.21.15.5:6803/2140667501","heartbeat_back_addr":"172.21.15.5:6807/2140667501","heartbeat_front_addr":"172.21.15.5:6805/2140667501","state":["exists","up"]},{"osd":2,"uuid":"e614c5b1-6504-452e-82e0-95c7ca271737","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6817","nonce":90414679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6819","nonce":90414679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6823","nonce":90414679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6821","nonce":90414679}]},"public_addr":"172.21.15.5:6817/90414679","cluster_addr":"172.21.15.5:6819/90414679","heartbeat_back_addr":"172.21.15.5:6823/90414679","heartbeat_front_addr":"172.21.15.5:6821/90414679","state":["exists","up"]},{"osd":3,"uuid":"c60154d0-85e0-46a1-8273-4863bd8ad2dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6809","nonce":4212473119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6811","nonce":4212473119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6815","nonce":4212473119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6813","nonce":4212473119}]},"public_addr":"172.21.15.5:6809/4212473119","cluster_addr":"172.21.15.5:6811/4212473119","heartbeat_back_addr":"172.21.15.5:6815/4212473119","heartbeat_front_addr":"172.21.15.5:6813/4212473119","state":["exists","up"]},{"osd":4,"uuid":"21d74109-da3a-4fc0-b2d8-93e9ae3ed30c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6817","nonce":2193986505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6819","nonce":2193986505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6823","nonce":2193986505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6821","nonce":2193986505}]},"public_addr":"172.21.15.57:6817/2193986505","cluster_addr":"172.21.15.57:6819/2193986505","heartbeat_back_addr":"172.21.15.57:6823/2193986505","heartbeat_front_addr":"172.21.15.57:6821/2193986505","state":["exists","up"]},{"osd":5,"uuid":"5cf19cc7-215f-4d69-a30a-80543c0293e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6809","nonce":3044987466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6811","nonce":3044987466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6815","nonce":3044987466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6813","nonce":3044987466}]},"public_addr":"172.21.15.57:6809/3044987466","cluster_addr":"172.21.15.57:6811/3044987466","heartbeat_back_addr":"172.21.15.57:6815/3044987466","heartbeat_front_addr":"172.21.15.57:6813/3044987466","state":["exists","up"]},{"osd":6,"uuid":"a5e76c69-c3fc-47b2-8f92-e560cf503e45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6801","nonce":4290897445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6803","nonce":4290897445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6807","nonce":4290897445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6805","nonce":4290897445}]},"public_addr":"172.21.15.57:6801/4290897445","cluster_addr":"172.21.15.57:6803/4290897445","heartbeat_back_addr":"172.21.15.57:6807/4290897445","heartbeat_front_addr":"172.21.15.57:6805/4290897445","state":["exists","up"]},{"osd":7,"uuid":"47d0b873-e3ac-4f40-a372-681436e45d0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6825","nonce":212605698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6827","nonce":212605698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6831","nonce":212605698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6829","nonce":212605698}]},"public_addr":"172.21.15.57:6825/212605698","cluster_addr":"172.21.15.57:6827/212605698","heartbeat_back_addr":"172.21.15.57:6831/212605698","heartbeat_front_addr":"172.21.15.57:6829/212605698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.211237+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.192822+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.273429+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.199212+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.277082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.218768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.190915+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.225294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:45:13.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.043+0000 7fabb5c4e640 1 -- 172.21.15.5:0/2416760043 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fab9c06b860 msgr2=0x7fab9c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.043+0000 7fabb5c4e640 1 --2- 172.21.15.5:0/2416760043 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fab9c06b860 0x7fab9c06dd20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fabb00b2bd0 tx=0x7faba4006f90 comp rx=0 tx=0).stop 2024-03-24T10:45:13.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.043+0000 7fabb5c4e640 1 -- 172.21.15.5:0/2416760043 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fab98003970 msgr2=0x7fabb0155f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.043+0000 7fabb5c4e640 1 --2- 172.21.15.5:0/2416760043 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fab98003970 0x7fabb0155f90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7faba000c200 tx=0x7fabb00b3010 comp rx=0 tx=0).stop 2024-03-24T10:45:13.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.043+0000 7fabb5c4e640 1 -- 172.21.15.5:0/2416760043 shutdown_connections 2024-03-24T10:45:13.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.043+0000 7fabb5c4e640 1 --2- 172.21.15.5:0/2416760043 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fab9c06b860 0x7fab9c06dd20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.043+0000 7fabb5c4e640 1 --2- 172.21.15.5:0/2416760043 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabb0152a30 0x7fabb0154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.043+0000 7fabb5c4e640 1 --2- 172.21.15.5:0/2416760043 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabb015ce20 0x7fabb01f2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.043+0000 7fabb5c4e640 1 --2- 172.21.15.5:0/2416760043 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fab98003970 0x7fabb0155f90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.043+0000 7fabb5c4e640 1 -- 172.21.15.5:0/2416760043 >> 172.21.15.5:0/2416760043 conn(0x7fabb0072ef0 msgr2=0x7fabb00b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:13.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.043+0000 7fabb5c4e640 1 -- 172.21.15.5:0/2416760043 shutdown_connections 2024-03-24T10:45:13.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.043+0000 7fabb5c4e640 1 -- 172.21.15.5:0/2416760043 wait complete. 2024-03-24T10:45:13.054 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-24T10:45:13.054 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:45:13.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.156+0000 7f5000055640 1 Processor -- start 2024-03-24T10:45:13.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.156+0000 7f5000055640 1 -- start start 2024-03-24T10:45:13.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.157+0000 7f5000055640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4ff8152430 0x7f4ff8158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.157+0000 7f5000055640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ff8152810 con 0x7f4ff80733c0 2024-03-24T10:45:13.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.157+0000 7f5000055640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ff8072390 con 0x7f4ff806d2f0 2024-03-24T10:45:13.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.157+0000 7f5000055640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f4ff806d780 con 0x7f4ff8152430 2024-03-24T10:45:13.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.158+0000 7f4ffddf1640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4ff8152430 0x7f4ff8158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.158+0000 7f4ffe5f2640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f4ff806d2f0 0x7f4ff80c5760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:47000/0 (socket says 172.21.15.5:47000) 2024-03-24T10:45:13.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.158+0000 7f4ffe5f2640 1 -- 172.21.15.5:0/3329419481 learned_addr learned my addr 172.21.15.5:0/3329419481 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:13.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.161+0000 7f4ffddf1640 1 -- 172.21.15.5:0/3329419481 >> v1:172.21.15.57:6789/0 conn(0x7f4ff806d2f0 legacy=0x7f4ff80c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.161+0000 7f4ffddf1640 1 -- 172.21.15.5:0/3329419481 >> v1:172.21.15.5:6789/0 conn(0x7f4ff80733c0 legacy=0x7f4ff8071c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:13.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.161+0000 7f4ffddf1640 1 -- 172.21.15.5:0/3329419481 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff80c8320 con 0x7f4ff8152430 2024-03-24T10:45:13.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.163+0000 7f4ffddf1640 1 --2- 172.21.15.5:0/3329419481 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ff8152430 0x7f4ff8158ca0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0097a0 tx=0x7f4fec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=877b81825cfb4c17 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:13.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.168+0000 7f4ffcdef640 1 -- 172.21.15.5:0/3329419481 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fec031830 con 0x7f4ff8152430 2024-03-24T10:45:13.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.168+0000 7f4ffcdef640 1 -- 172.21.15.5:0/3329419481 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fec031990 con 0x7f4ff8152430 2024-03-24T10:45:13.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.168+0000 7f4ffcdef640 1 -- 172.21.15.5:0/3329419481 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fec031d20 con 0x7f4ff8152430 2024-03-24T10:45:13.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.169+0000 7f5000055640 1 -- 172.21.15.5:0/3329419481 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ff8152430 msgr2=0x7f4ff8158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.169+0000 7f5000055640 1 --2- 172.21.15.5:0/3329419481 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ff8152430 0x7f4ff8158ca0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0097a0 tx=0x7f4fec02b8d0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.169+0000 7f5000055640 1 -- 172.21.15.5:0/3329419481 shutdown_connections 2024-03-24T10:45:13.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.169+0000 7f5000055640 1 --2- 172.21.15.5:0/3329419481 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ff8152430 0x7f4ff8158ca0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.169+0000 7f5000055640 1 -- 172.21.15.5:0/3329419481 >> 172.21.15.5:0/3329419481 conn(0x7f4ff80cc050 msgr2=0x7f4ff80cc460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:13.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.169+0000 7f5000055640 1 -- 172.21.15.5:0/3329419481 shutdown_connections 2024-03-24T10:45:13.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.169+0000 7f5000055640 1 -- 172.21.15.5:0/3329419481 wait complete. 2024-03-24T10:45:13.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.170+0000 7f5000055640 1 Processor -- start 2024-03-24T10:45:13.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.170+0000 7f5000055640 1 -- start start 2024-03-24T10:45:13.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.171+0000 7f5000055640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ff806d2f0 0x7f4ff814d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.171+0000 7f5000055640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ff80733c0 0x7f4ff81e4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.171+0000 7f5000055640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ff8152430 0x7f4ff814ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.171+0000 7f5000055640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff8151820 con 0x7f4ff80733c0 2024-03-24T10:45:13.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.171+0000 7f5000055640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff80c5340 con 0x7f4ff8152430 2024-03-24T10:45:13.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.171+0000 7f5000055640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4ff814bdd0 con 0x7f4ff806d2f0 2024-03-24T10:45:13.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.172+0000 7f4ffd5f0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ff80733c0 0x7f4ff81e4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.172+0000 7f4ffe5f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ff8152430 0x7f4ff814ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.172+0000 7f4ffe5f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ff8152430 0x7f4ff814ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36420/0 (socket says 172.21.15.5:36420) 2024-03-24T10:45:13.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.172+0000 7f4ffe5f2640 1 -- 172.21.15.5:0/1232097606 learned_addr learned my addr 172.21.15.5:0/1232097606 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:13.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.172+0000 7f4ffddf1640 1 --2- 172.21.15.5:0/1232097606 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ff806d2f0 0x7f4ff814d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.175+0000 7f4ffd5f0640 1 -- 172.21.15.5:0/1232097606 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ff806d2f0 msgr2=0x7f4ff814d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.175+0000 7f4ffd5f0640 1 --2- 172.21.15.5:0/1232097606 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ff806d2f0 0x7f4ff814d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.175+0000 7f4ffd5f0640 1 -- 172.21.15.5:0/1232097606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ff8152430 msgr2=0x7f4ff814ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.175+0000 7f4ffd5f0640 1 --2- 172.21.15.5:0/1232097606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ff8152430 0x7f4ff814ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.175+0000 7f4ffd5f0640 1 -- 172.21.15.5:0/1232097606 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fec009400 con 0x7f4ff80733c0 2024-03-24T10:45:13.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.177+0000 7f4ffd5f0640 1 --2- 172.21.15.5:0/1232097606 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ff80733c0 0x7f4ff81e4e00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4ff400c410 tx=0x7f4ff400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:13.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.181+0000 7f4fea7fc640 1 -- 172.21.15.5:0/1232097606 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4005230 con 0x7f4ff80733c0 2024-03-24T10:45:13.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.181+0000 7f4fea7fc640 1 -- 172.21.15.5:0/1232097606 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ff4005390 con 0x7f4ff80733c0 2024-03-24T10:45:13.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.181+0000 7f4fea7fc640 1 -- 172.21.15.5:0/1232097606 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4004820 con 0x7f4ff80733c0 2024-03-24T10:45:13.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.181+0000 7f5000055640 1 -- 172.21.15.5:0/1232097606 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ff80c5060 con 0x7f4ff80733c0 2024-03-24T10:45:13.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.181+0000 7f5000055640 1 -- 172.21.15.5:0/1232097606 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ff814f180 con 0x7f4ff80733c0 2024-03-24T10:45:13.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.182+0000 7f5000055640 1 -- 172.21.15.5:0/1232097606 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ff80c88c0 con 0x7f4ff80733c0 2024-03-24T10:45:13.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.450+0000 7f4fea7fc640 1 -- 172.21.15.5:0/1232097606 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4ff4004980 con 0x7f4ff80733c0 2024-03-24T10:45:13.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.450+0000 7f4fea7fc640 1 --2- 172.21.15.5:0/1232097606 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4fdc06b860 0x7f4fdc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.450+0000 7f4ffddf1640 1 --2- 172.21.15.5:0/1232097606 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4fdc06b860 0x7f4fdc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.451+0000 7f4ffddf1640 1 --2- 172.21.15.5:0/1232097606 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4fdc06b860 0x7f4fdc06dd20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4fec036000 tx=0x7f4fec03c040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:13.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.479+0000 7f4fea7fc640 1 -- 172.21.15.5:0/1232097606 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4ff4057e90 con 0x7f4ff80733c0 2024-03-24T10:45:13.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.479+0000 7f4fea7fc640 1 -- 172.21.15.5:0/1232097606 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4ff4054130 con 0x7f4ff80733c0 2024-03-24T10:45:13.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.667+0000 7f5000055640 1 -- 172.21.15.5:0/1232097606 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4ff80c88c0 con 0x7f4ff80733c0 2024-03-24T10:45:13.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.706+0000 7f4fea7fc640 1 -- 172.21.15.5:0/1232097606 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16485 (secure 0 0 0) 0x7f4ff80c88c0 con 0x7f4ff80733c0 2024-03-24T10:45:13.706 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:45:13.706 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":21,"fsid":"7d9848db-2fe7-4d95-a6e2-b04f4f639ebc","created":"2024-03-24T10:44:30.435176+0000","modified":"2024-03-24T10:45:02.781257+0000","last_up_change":"2024-03-24T10:44:44.634047+0000","last_in_change":"2024-03-24T10:44:35.986391+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:44:45.829320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:44:58.435543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:45:00.129313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50f590b3-f5ec-4e1b-ac46-25cab307a27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6825","nonce":1991336051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6827","nonce":1991336051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6831","nonce":1991336051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6829","nonce":1991336051}]},"public_addr":"172.21.15.5:6825/1991336051","cluster_addr":"172.21.15.5:6827/1991336051","heartbeat_back_addr":"172.21.15.5:6831/1991336051","heartbeat_front_addr":"172.21.15.5:6829/1991336051","state":["exists","up"]},{"osd":1,"uuid":"ccba9e93-c704-472b-a832-8d4eeb456c47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6801","nonce":2140667501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6803","nonce":2140667501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6807","nonce":2140667501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6805","nonce":2140667501}]},"public_addr":"172.21.15.5:6801/2140667501","cluster_addr":"172.21.15.5:6803/2140667501","heartbeat_back_addr":"172.21.15.5:6807/2140667501","heartbeat_front_addr":"172.21.15.5:6805/2140667501","state":["exists","up"]},{"osd":2,"uuid":"e614c5b1-6504-452e-82e0-95c7ca271737","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6817","nonce":90414679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6819","nonce":90414679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6823","nonce":90414679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6821","nonce":90414679}]},"public_addr":"172.21.15.5:6817/90414679","cluster_addr":"172.21.15.5:6819/90414679","heartbeat_back_addr":"172.21.15.5:6823/90414679","heartbeat_front_addr":"172.21.15.5:6821/90414679","state":["exists","up"]},{"osd":3,"uuid":"c60154d0-85e0-46a1-8273-4863bd8ad2dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6809","nonce":4212473119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6811","nonce":4212473119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6815","nonce":4212473119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6813","nonce":4212473119}]},"public_addr":"172.21.15.5:6809/4212473119","cluster_addr":"172.21.15.5:6811/4212473119","heartbeat_back_addr":"172.21.15.5:6815/4212473119","heartbeat_front_addr":"172.21.15.5:6813/4212473119","state":["exists","up"]},{"osd":4,"uuid":"21d74109-da3a-4fc0-b2d8-93e9ae3ed30c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6817","nonce":2193986505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6819","nonce":2193986505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6823","nonce":2193986505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6821","nonce":2193986505}]},"public_addr":"172.21.15.57:6817/2193986505","cluster_addr":"172.21.15.57:6819/2193986505","heartbeat_back_addr":"172.21.15.57:6823/2193986505","heartbeat_front_addr":"172.21.15.57:6821/2193986505","state":["exists","up"]},{"osd":5,"uuid":"5cf19cc7-215f-4d69-a30a-80543c0293e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6809","nonce":3044987466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6811","nonce":3044987466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6815","nonce":3044987466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6813","nonce":3044987466}]},"public_addr":"172.21.15.57:6809/3044987466","cluster_addr":"172.21.15.57:6811/3044987466","heartbeat_back_addr":"172.21.15.57:6815/3044987466","heartbeat_front_addr":"172.21.15.57:6813/3044987466","state":["exists","up"]},{"osd":6,"uuid":"a5e76c69-c3fc-47b2-8f92-e560cf503e45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6801","nonce":4290897445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6803","nonce":4290897445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6807","nonce":4290897445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6805","nonce":4290897445}]},"public_addr":"172.21.15.57:6801/4290897445","cluster_addr":"172.21.15.57:6803/4290897445","heartbeat_back_addr":"172.21.15.57:6807/4290897445","heartbeat_front_addr":"172.21.15.57:6805/4290897445","state":["exists","up"]},{"osd":7,"uuid":"47d0b873-e3ac-4f40-a372-681436e45d0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6825","nonce":212605698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6827","nonce":212605698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6831","nonce":212605698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6829","nonce":212605698}]},"public_addr":"172.21.15.57:6825/212605698","cluster_addr":"172.21.15.57:6827/212605698","heartbeat_back_addr":"172.21.15.57:6831/212605698","heartbeat_front_addr":"172.21.15.57:6829/212605698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.211237+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.192822+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.273429+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.199212+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.277082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.218768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.190915+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.225294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:45:13.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.708+0000 7f5000055640 1 -- 172.21.15.5:0/1232097606 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4fdc06b860 msgr2=0x7f4fdc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.708+0000 7f5000055640 1 --2- 172.21.15.5:0/1232097606 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4fdc06b860 0x7f4fdc06dd20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4fec036000 tx=0x7f4fec03c040 comp rx=0 tx=0).stop 2024-03-24T10:45:13.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.708+0000 7f5000055640 1 -- 172.21.15.5:0/1232097606 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ff80733c0 msgr2=0x7f4ff81e4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.708+0000 7f5000055640 1 --2- 172.21.15.5:0/1232097606 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ff80733c0 0x7f4ff81e4e00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4ff400c410 tx=0x7f4ff400c8d0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.708+0000 7f5000055640 1 -- 172.21.15.5:0/1232097606 shutdown_connections 2024-03-24T10:45:13.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.708+0000 7f5000055640 1 --2- 172.21.15.5:0/1232097606 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4fdc06b860 0x7f4fdc06dd20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.708+0000 7f5000055640 1 --2- 172.21.15.5:0/1232097606 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ff80733c0 0x7f4ff81e4e00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.708+0000 7f5000055640 1 --2- 172.21.15.5:0/1232097606 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ff8152430 0x7f4ff814ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.708+0000 7f5000055640 1 --2- 172.21.15.5:0/1232097606 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ff806d2f0 0x7f4ff814d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.708+0000 7f5000055640 1 -- 172.21.15.5:0/1232097606 >> 172.21.15.5:0/1232097606 conn(0x7f4ff80cc050 msgr2=0x7f4ff80c9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:13.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.708+0000 7f5000055640 1 -- 172.21.15.5:0/1232097606 shutdown_connections 2024-03-24T10:45:13.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.708+0000 7f5000055640 1 -- 172.21.15.5:0/1232097606 wait complete. 2024-03-24T10:45:13.719 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-24T10:45:13.720 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-24T10:45:13.720 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-24T10:45:13.721 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-24T10:45:13.721 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-24T10:45:13.721 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-24T10:45:13.722 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-24T10:45:13.722 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-24T10:45:13.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.917+0000 7f8efa72e640 1 Processor -- start 2024-03-24T10:45:13.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.918+0000 7f8efa72e640 1 -- start start 2024-03-24T10:45:13.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.919+0000 7f8efa72e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8ef4071ff0 0x7f8ef406d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.919+0000 7f8efa72e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ef406f6b0 0x7f8ef406fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.919+0000 7f8efa72e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ef4156e10 con 0x7f8ef4150740 2024-03-24T10:45:13.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.919+0000 7f8efa72e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f8ef4156500 con 0x7f8ef4071ff0 2024-03-24T10:45:13.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.919+0000 7f8efa72e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f8ef40c2c30 con 0x7f8ef406f6b0 2024-03-24T10:45:13.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.922+0000 7f8ef37fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8ef4071ff0 0x7f8ef406d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.922+0000 7f8ef37fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8ef4071ff0 0x7f8ef406d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36436/0 (socket says 172.21.15.5:36436) 2024-03-24T10:45:13.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.922+0000 7f8ef37fe640 1 -- 172.21.15.5:0/4228272042 learned_addr learned my addr 172.21.15.5:0/4228272042 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:13.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.922+0000 7f8ef8ccb640 1 --2- 172.21.15.5:0/4228272042 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ef406f6b0 0x7f8ef406fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.924+0000 7f37c76f4640 1 Processor -- start 2024-03-24T10:45:13.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.925+0000 7fd7a7642640 1 Processor -- start 2024-03-24T10:45:13.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.925+0000 7f37c76f4640 1 -- start start 2024-03-24T10:45:13.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.926+0000 7f37c76f4640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f37c00c7d60 0x7f37c00c5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.926+0000 7f37c76f4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37c00c4950 con 0x7f37c00c58a0 2024-03-24T10:45:13.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.926+0000 7f37c76f4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37c00c4c30 con 0x7f37c01506d0 2024-03-24T10:45:13.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.926+0000 7f37c76f4640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f37c00c3700 con 0x7f37c00c7d60 2024-03-24T10:45:13.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.924+0000 7f7f2e93a640 1 Processor -- start 2024-03-24T10:45:13.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.924+0000 7f7f2e93a640 1 -- start start 2024-03-24T10:45:13.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.925+0000 7f7f2e93a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f280c5c40 0x7f7f281670c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.925+0000 7f7f2e93a640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f280c57b0 con 0x7f7f280cbf20 2024-03-24T10:45:13.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.925+0000 7f7f2e93a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f280c48e0 con 0x7f7f280cbb40 2024-03-24T10:45:13.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.925+0000 7f7f2e93a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7f280c3290 con 0x7f7f280c5c40 2024-03-24T10:45:13.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.926+0000 7fd7a7642640 1 -- start start 2024-03-24T10:45:13.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.926+0000 7f8ef37fe640 1 -- 172.21.15.5:0/4228272042 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ef406f6b0 msgr2=0x7f8ef406fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:13.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.926+0000 7f8ef37fe640 1 --2- 172.21.15.5:0/4228272042 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ef406f6b0 0x7f8ef406fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.926+0000 7f8ef37fe640 1 -- 172.21.15.5:0/4228272042 >> v1:172.21.15.57:6789/0 conn(0x7f8ef4150740 legacy=0x7f8ef40c23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:13.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.926+0000 7f8ef37fe640 1 -- 172.21.15.5:0/4228272042 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ef40c2f10 con 0x7f8ef4071ff0 2024-03-24T10:45:13.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.927+0000 7f37c5490640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f37c01506d0 0x7f37c015a320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:47024/0 (socket says 172.21.15.5:47024) 2024-03-24T10:45:13.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.927+0000 7f37c5490640 1 -- 172.21.15.5:0/1470222649 learned_addr learned my addr 172.21.15.5:0/1470222649 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:13.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.927+0000 7fd7a7642640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd7a01506d0 0x7fd7a015a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.927+0000 7fd7a7642640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd7a00c7d60 0x7fd7a00cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.927+0000 7fd7a7642640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7a015a860 con 0x7fd7a006fc70 2024-03-24T10:45:13.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.927+0000 7fd7a7642640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fd7a00c3700 con 0x7fd7a01506d0 2024-03-24T10:45:13.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.927+0000 7fd7a7642640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fd7a00c4950 con 0x7fd7a00c7d60 2024-03-24T10:45:13.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.927+0000 7f7f27fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f7f280cbb40 0x7f7f280c4110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:47014/0 (socket says 172.21.15.5:47014) 2024-03-24T10:45:13.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.927+0000 7f7f27fff640 1 -- 172.21.15.5:0/220284835 learned_addr learned my addr 172.21.15.5:0/220284835 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:13.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.930+0000 7f7f277fe640 1 --2- 172.21.15.5:0/220284835 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f280c5c40 0x7f7f281670c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.932+0000 7f0c9a42b640 1 Processor -- start 2024-03-24T10:45:13.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.932+0000 7f0c9a42b640 1 -- start start 2024-03-24T10:45:13.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.933+0000 7fd7a5bdf640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd7a00c7d60 0x7fd7a00cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.933+0000 7f0c9a42b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0c8c046160 0x7f0c8c009140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.933+0000 7f0c9a42b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0c8c036500 0x7f0c8c036930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.933+0000 7f0c9a42b640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c8c006db0 con 0x7f0c8c0d6ee0 2024-03-24T10:45:13.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.933+0000 7f0c9a42b640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f0c8c0d72c0 con 0x7f0c8c036500 2024-03-24T10:45:13.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.933+0000 7f0c9a42b640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f0c8c04c310 con 0x7f0c8c046160 2024-03-24T10:45:13.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.934+0000 7f7f277fe640 1 -- 172.21.15.5:0/220284835 >> v1:172.21.15.57:6789/0 conn(0x7f7f280cbb40 legacy=0x7f7f280c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.934+0000 7f7f277fe640 1 -- 172.21.15.5:0/220284835 >> v1:172.21.15.5:6789/0 conn(0x7f7f280cbf20 legacy=0x7f7f28168530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:13.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.934+0000 7f7f277fe640 1 -- 172.21.15.5:0/220284835 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f280c4ba0 con 0x7f7f280c5c40 2024-03-24T10:45:13.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.934+0000 7fd7a5bdf640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd7a00c7d60 0x7fd7a00cbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36440/0 (socket says 172.21.15.5:36440) 2024-03-24T10:45:13.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.934+0000 7fd7a5bdf640 1 -- 172.21.15.5:0/1523006898 learned_addr learned my addr 172.21.15.5:0/1523006898 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:13.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.934+0000 7f8ef37fe640 1 --2- 172.21.15.5:0/4228272042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ef4071ff0 0x7f8ef406d6d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8eec00d370 tx=0x7f8eec02e2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11f2898bcc7eba88 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:13.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.935+0000 7f2582f1f640 1 Processor -- start 2024-03-24T10:45:13.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.935+0000 7f2582f1f640 1 -- start start 2024-03-24T10:45:13.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.936+0000 7f2582f1f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f25740d5cc0 0x7f257404c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.936+0000 7fb1d71d7640 1 Processor -- start 2024-03-24T10:45:13.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.936+0000 7fb1d71d7640 1 -- start start 2024-03-24T10:45:13.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.936+0000 7f2582f1f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2574046a40 0x7f25740df030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.936+0000 7f2582f1f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f257404cb50 con 0x7f257404d7a0 2024-03-24T10:45:13.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.936+0000 7f2582f1f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f257404c950 con 0x7f2574046a40 2024-03-24T10:45:13.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.936+0000 7f2582f1f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f2574008640 con 0x7f25740d5cc0 2024-03-24T10:45:13.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.937+0000 7f2581f1d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f25740d5cc0 0x7f257404c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.937+0000 7f2581f1d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f25740d5cc0 0x7f257404c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:51364/0 (socket says 172.21.15.5:51364) 2024-03-24T10:45:13.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.937+0000 7f2581f1d640 1 -- 172.21.15.5:0/4018945540 learned_addr learned my addr 172.21.15.5:0/4018945540 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:13.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.937+0000 7fb1d71d7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb1d00c7680 0x7fb1d00c7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.937+0000 7fd7a5bdf640 1 -- 172.21.15.5:0/1523006898 >> v2:172.21.15.5:3300/0 conn(0x7fd7a01506d0 msgr2=0x7fd7a015a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:13.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.938+0000 7fd7a5bdf640 1 --2- 172.21.15.5:0/1523006898 >> v2:172.21.15.5:3300/0 conn(0x7fd7a01506d0 0x7fd7a015a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.938+0000 7fd7a5bdf640 1 -- 172.21.15.5:0/1523006898 >> v1:172.21.15.5:6789/0 conn(0x7fd7a006fc70 legacy=0x7fd7a00c5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:13.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.938+0000 7fd7a5bdf640 1 -- 172.21.15.5:0/1523006898 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a00c4c30 con 0x7fd7a00c7d60 2024-03-24T10:45:13.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.938+0000 7fb1d71d7640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1d00c1d80 con 0x7fb1d0151290 2024-03-24T10:45:13.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.938+0000 7fb1d71d7640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1d00c18f0 con 0x7fb1d0157030 2024-03-24T10:45:13.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.938+0000 7fb1d71d7640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fb1d00c6c70 con 0x7fb1d00c7680 2024-03-24T10:45:13.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.938+0000 7f7f277fe640 1 --2- 172.21.15.5:0/220284835 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f280c5c40 0x7f7f281670c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7f2000aed0 tx=0x7f7f2002cfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a2b046493b11c3b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:13.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.941+0000 7f87da49d640 1 Processor -- start 2024-03-24T10:45:13.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.942+0000 7f87da49d640 1 -- start start 2024-03-24T10:45:13.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.942+0000 7f2581f1d640 1 -- 172.21.15.5:0/4018945540 >> v2:172.21.15.57:3300/0 conn(0x7f2574046a40 msgr2=0x7f25740df030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:13.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.942+0000 7f2581f1d640 1 --2- 172.21.15.5:0/4018945540 >> v2:172.21.15.57:3300/0 conn(0x7f2574046a40 0x7f25740df030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.942+0000 7f0c98c28640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0c8c046160 0x7f0c8c009140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.942+0000 7f2581f1d640 1 -- 172.21.15.5:0/4018945540 >> v1:172.21.15.57:6789/0 conn(0x7f257404d7a0 legacy=0x7f2574046330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:13.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.942+0000 7f2581f1d640 1 -- 172.21.15.5:0/4018945540 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2574008920 con 0x7f25740d5cc0 2024-03-24T10:45:13.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.943+0000 7f0c98c28640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0c8c046160 0x7f0c8c009140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36446/0 (socket says 172.21.15.5:36446) 2024-03-24T10:45:13.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.943+0000 7f0c98c28640 1 -- 172.21.15.5:0/3105784729 learned_addr learned my addr 172.21.15.5:0/3105784729 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:13.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.943+0000 7f87da49d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f87d40cf340 0x7f87d40cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.943+0000 7f87da49d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f87d40ccb80 0x7f87d406bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.943+0000 7fd7a5bdf640 1 --2- 172.21.15.5:0/1523006898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7a00c7d60 0x7fd7a00cbcf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd7a006e6e0 tx=0x7fd79802bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c3de908d8265cd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:13.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.943+0000 7f87da49d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87d40cc3e0 con 0x7f87d40cc000 2024-03-24T10:45:13.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.943+0000 7f87da49d640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f87d415a400 con 0x7f87d40ccb80 2024-03-24T10:45:13.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.943+0000 7f87da49d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f87d4154bd0 con 0x7f87d40cf340 2024-03-24T10:45:13.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.943+0000 7f37c4c8f640 1 --2- 172.21.15.5:0/1470222649 >> v2:172.21.15.5:3300/0 conn(0x7f37c00c7d60 0x7f37c00c5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.946+0000 7f37c4c8f640 1 -- 172.21.15.5:0/1470222649 >> v1:172.21.15.57:6789/0 conn(0x7f37c01506d0 legacy=0x7f37c015a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.946+0000 7f37c4c8f640 1 -- 172.21.15.5:0/1470222649 >> v1:172.21.15.5:6789/0 conn(0x7f37c00c58a0 legacy=0x7f37c00cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:13.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.946+0000 7f37c4c8f640 1 -- 172.21.15.5:0/1470222649 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37c00c4f10 con 0x7f37c00c7d60 2024-03-24T10:45:13.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.947+0000 7f2581f1d640 1 --2- 172.21.15.5:0/4018945540 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f25740d5cc0 0x7f257404c410 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2578009870 tx=0x7f257802b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0cde8c6d9a6959b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:13.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.948+0000 7f0c98c28640 1 -- 172.21.15.5:0/3105784729 >> v2:172.21.15.5:3300/0 conn(0x7f0c8c036500 msgr2=0x7f0c8c036930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:13.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.948+0000 7f0c98c28640 1 --2- 172.21.15.5:0/3105784729 >> v2:172.21.15.5:3300/0 conn(0x7f0c8c036500 0x7f0c8c036930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.948+0000 7f0c98c28640 1 -- 172.21.15.5:0/3105784729 >> v1:172.21.15.5:6789/0 conn(0x7f0c8c0d6ee0 legacy=0x7f0c8c04bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:13.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.948+0000 7f0c98c28640 1 -- 172.21.15.5:0/3105784729 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c8c0dcc60 con 0x7f0c8c046160 2024-03-24T10:45:13.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.948+0000 7f7f26ffd640 1 -- 172.21.15.5:0/220284835 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2002db40 con 0x7f7f280c5c40 2024-03-24T10:45:13.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.948+0000 7f7f26ffd640 1 -- 172.21.15.5:0/220284835 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f2002dca0 con 0x7f7f280c5c40 2024-03-24T10:45:13.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.948+0000 7f7f26ffd640 1 -- 172.21.15.5:0/220284835 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f20008660 con 0x7f7f280c5c40 2024-03-24T10:45:13.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.949+0000 7f7f2e93a640 1 -- 172.21.15.5:0/220284835 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f280c5c40 msgr2=0x7f7f281670c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.949+0000 7f7f2e93a640 1 --2- 172.21.15.5:0/220284835 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f280c5c40 0x7f7f281670c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7f2000aed0 tx=0x7f7f2002cfa0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.949+0000 7f7f2e93a640 1 -- 172.21.15.5:0/220284835 shutdown_connections 2024-03-24T10:45:13.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.949+0000 7f7f2e93a640 1 --2- 172.21.15.5:0/220284835 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f280c5c40 0x7f7f281670c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.949+0000 7f7f2e93a640 1 -- 172.21.15.5:0/220284835 >> 172.21.15.5:0/220284835 conn(0x7f7f2806f180 msgr2=0x7f7f2806f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:13.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.949+0000 7f7f2e93a640 1 -- 172.21.15.5:0/220284835 shutdown_connections 2024-03-24T10:45:13.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.949+0000 7f7f2e93a640 1 -- 172.21.15.5:0/220284835 wait complete. 2024-03-24T10:45:13.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.950+0000 7f7f2e93a640 1 Processor -- start 2024-03-24T10:45:13.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.951+0000 7f7f2e93a640 1 -- start start 2024-03-24T10:45:13.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.951+0000 7f7f2e93a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f280cbb40 0x7f7f28203e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.951+0000 7f7f2e93a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f280cbf20 0x7f7f280c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.951+0000 7f7f2e93a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f282015d0 0x7f7f2815f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.951+0000 7f7f2e93a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f7f282008e0 con 0x7f7f280cbf20 2024-03-24T10:45:13.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.951+0000 7f7f2e93a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7f280cace0 con 0x7f7f280cbb40 2024-03-24T10:45:13.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.951+0000 7f7f2e93a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7f280cafc0 con 0x7f7f282015d0 2024-03-24T10:45:13.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.952+0000 7f0c98c28640 1 --2- 172.21.15.5:0/3105784729 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c8c046160 0x7f0c8c009140 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c0466f0 tx=0x7f0c8402b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1032388d617513ab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:13.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.953+0000 7fd793fff640 1 -- 172.21.15.5:0/1523006898 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79802cb20 con 0x7fd7a00c7d60 2024-03-24T10:45:13.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.953+0000 7fd793fff640 1 -- 172.21.15.5:0/1523006898 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd79802cc80 con 0x7fd7a00c7d60 2024-03-24T10:45:13.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.953+0000 7fd793fff640 1 -- 172.21.15.5:0/1523006898 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7980343f0 con 0x7fd7a00c7d60 2024-03-24T10:45:13.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.953+0000 7fd7a7642640 1 -- 172.21.15.5:0/1523006898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7a00c7d60 msgr2=0x7fd7a00cbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.953+0000 7fd7a7642640 1 --2- 172.21.15.5:0/1523006898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7a00c7d60 0x7fd7a00cbcf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd7a006e6e0 tx=0x7fd79802bf80 comp rx=0 tx=0).stop 2024-03-24T10:45:13.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.953+0000 7fd7a7642640 1 -- 172.21.15.5:0/1523006898 shutdown_connections 2024-03-24T10:45:13.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.953+0000 7fd7a7642640 1 --2- 172.21.15.5:0/1523006898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7a00c7d60 0x7fd7a00cbcf0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.953+0000 7fd7a7642640 1 --2- 172.21.15.5:0/1523006898 >> v2:172.21.15.5:3300/0 conn(0x7fd7a01506d0 0x7fd7a015a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.953+0000 7fd7a7642640 1 -- 172.21.15.5:0/1523006898 >> 172.21.15.5:0/1523006898 conn(0x7fd7a006f180 msgr2=0x7fd7a006f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:13.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.954+0000 7fd7a7642640 1 -- 172.21.15.5:0/1523006898 shutdown_connections 2024-03-24T10:45:13.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.954+0000 7fd7a7642640 1 -- 172.21.15.5:0/1523006898 wait complete. 2024-03-24T10:45:13.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.954+0000 7f8ef2ffd640 1 -- 172.21.15.5:0/4228272042 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eec038600 con 0x7f8ef4071ff0 2024-03-24T10:45:13.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.954+0000 7f8ef2ffd640 1 -- 172.21.15.5:0/4228272042 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8eec032070 con 0x7f8ef4071ff0 2024-03-24T10:45:13.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.954+0000 7f87d3fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f87d40cc000 0x7f87d415e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:47056/0 (socket says 172.21.15.5:47056) 2024-03-24T10:45:13.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.954+0000 7f37c4c8f640 1 --2- 172.21.15.5:0/1470222649 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f37c00c7d60 0x7f37c00c5360 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f37b800c6a0 tx=0x7f37b802c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63050c49b4c0d78f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:13.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.954+0000 7f87d3fff640 1 -- 172.21.15.5:0/1126236484 learned_addr learned my addr 172.21.15.5:0/1126236484 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:13.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.954+0000 7f8ef2ffd640 1 -- 172.21.15.5:0/4228272042 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eec0389c0 con 0x7f8ef4071ff0 2024-03-24T10:45:13.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.954+0000 7f8efa72e640 1 -- 172.21.15.5:0/4228272042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ef4071ff0 msgr2=0x7f8ef406d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.954+0000 7f8efa72e640 1 --2- 172.21.15.5:0/4228272042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ef4071ff0 0x7f8ef406d6d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8eec00d370 tx=0x7f8eec02e2a0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.955+0000 7fb1d5774640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb1d0157030 0x7fb1d01619b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:47048/0 (socket says 172.21.15.5:47048) 2024-03-24T10:45:13.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.955+0000 7fb1d5774640 1 -- 172.21.15.5:0/713318047 learned_addr learned my addr 172.21.15.5:0/713318047 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:13.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.955+0000 7f8efa72e640 1 -- 172.21.15.5:0/4228272042 shutdown_connections 2024-03-24T10:45:13.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.955+0000 7f8efa72e640 1 --2- 172.21.15.5:0/4228272042 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ef406f6b0 0x7f8ef406fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.955+0000 7f8efa72e640 1 --2- 172.21.15.5:0/4228272042 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ef4071ff0 0x7f8ef406d6d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.955+0000 7f8efa72e640 1 -- 172.21.15.5:0/4228272042 >> 172.21.15.5:0/4228272042 conn(0x7f8ef40c8b70 msgr2=0x7f8ef40c8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:13.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.955+0000 7f8efa72e640 1 -- 172.21.15.5:0/4228272042 shutdown_connections 2024-03-24T10:45:13.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.955+0000 7f8efa72e640 1 -- 172.21.15.5:0/4228272042 wait complete. 2024-03-24T10:45:13.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.955+0000 7fd7a7642640 1 Processor -- start 2024-03-24T10:45:13.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.955+0000 7fd7a7642640 1 -- start start 2024-03-24T10:45:13.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.955+0000 7f7f2ced7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f282015d0 0x7f7f2815f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.955+0000 7f2580f1b640 1 -- 172.21.15.5:0/4018945540 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2578031870 con 0x7f25740d5cc0 2024-03-24T10:45:13.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.955+0000 7f2580f1b640 1 -- 172.21.15.5:0/4018945540 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25780319d0 con 0x7f25740d5cc0 2024-03-24T10:45:13.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.955+0000 7f2580f1b640 1 -- 172.21.15.5:0/4018945540 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2578031d60 con 0x7f25740d5cc0 2024-03-24T10:45:13.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.956+0000 7f7f2ced7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f282015d0 0x7f7f2815f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:51374/0 (socket says 172.21.15.5:51374) 2024-03-24T10:45:13.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.956+0000 7f7f2ced7640 1 -- 172.21.15.5:0/2606280843 learned_addr learned my addr 172.21.15.5:0/2606280843 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:13.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.956+0000 7fd7a7642640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd7a006fc70 0x7fd7a01f8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.956+0000 7fd7a7642640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7a00c7d60 0x7fd7a014abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.956+0000 7f8efa72e640 1 Processor -- start 2024-03-24T10:45:13.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.956+0000 7fd7a7642640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7a01506d0 0x7fd7a00cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.956+0000 7f8efa72e640 1 -- start start 2024-03-24T10:45:13.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.956+0000 7fd7a7642640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a014a3e0 con 0x7fd7a00c7d60 2024-03-24T10:45:13.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.956+0000 7f2582f1f640 1 -- 172.21.15.5:0/4018945540 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f25740d5cc0 msgr2=0x7f257404c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.956+0000 7fd7a7642640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a01f8550 con 0x7fd7a01506d0 2024-03-24T10:45:13.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.956+0000 7f2582f1f640 1 --2- 172.21.15.5:0/4018945540 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f25740d5cc0 0x7f257404c410 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2578009870 tx=0x7f257802b910 comp rx=0 tx=0).stop 2024-03-24T10:45:13.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.957+0000 7fd7a7642640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd7a0149b80 con 0x7fd7a006fc70 2024-03-24T10:45:13.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.957+0000 7f2582f1f640 1 -- 172.21.15.5:0/4018945540 shutdown_connections 2024-03-24T10:45:13.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.957+0000 7f2582f1f640 1 --2- 172.21.15.5:0/4018945540 >> v2:172.21.15.57:3300/0 conn(0x7f2574046a40 0x7f25740df030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.957+0000 7f2582f1f640 1 --2- 172.21.15.5:0/4018945540 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f25740d5cc0 0x7f257404c410 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.957+0000 7f2582f1f640 1 -- 172.21.15.5:0/4018945540 >> 172.21.15.5:0/4018945540 conn(0x7f2574051290 msgr2=0x7f25740516a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:13.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.957+0000 7f2582f1f640 1 -- 172.21.15.5:0/4018945540 shutdown_connections 2024-03-24T10:45:13.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.957+0000 7f2582f1f640 1 -- 172.21.15.5:0/4018945540 wait complete. 2024-03-24T10:45:13.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.957+0000 7f8efa72e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ef406f6b0 0x7f8ef40c3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.957+0000 7f8efa72e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ef4071ff0 0x7f8ef414f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.957+0000 7f8efa72e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ef4150740 0x7f8ef41f4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.957+0000 7f8efa72e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef40cb410 con 0x7f8ef406f6b0 2024-03-24T10:45:13.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.957+0000 7f8efa72e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef40c8050 con 0x7f8ef4071ff0 2024-03-24T10:45:13.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.957+0000 7f8efa72e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f8ef40cbc00 con 0x7f8ef4150740 2024-03-24T10:45:13.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.959+0000 7f2582f1f640 1 Processor -- start 2024-03-24T10:45:13.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.959+0000 7f2582f1f640 1 -- start start 2024-03-24T10:45:13.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.960+0000 7f7f2ced7640 1 -- 172.21.15.5:0/2606280843 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f280cbb40 msgr2=0x7f7f28203e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.960+0000 7f7f2ced7640 1 --2- 172.21.15.5:0/2606280843 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f280cbb40 0x7f7f28203e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.960+0000 7f7f2ced7640 1 -- 172.21.15.5:0/2606280843 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f280cbf20 msgr2=0x7f7f280c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.960+0000 7f7f2ced7640 1 --2- 172.21.15.5:0/2606280843 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f280cbf20 0x7f7f280c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.960+0000 7f7f2ced7640 1 -- 172.21.15.5:0/2606280843 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f2000ab80 con 0x7f7f282015d0 2024-03-24T10:45:13.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.961+0000 7f2582f1f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2574046a40 0x7f2574036bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.961+0000 7f2582f1f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f257404d7a0 0x7f25741710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.961+0000 7f2582f1f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25740d5cc0 0x7f25741673a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.961+0000 7f2582f1f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f2574037130 con 0x7f25740d5cc0 2024-03-24T10:45:13.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.961+0000 7f87d8a3a640 1 --2- 172.21.15.5:0/1126236484 >> v2:172.21.15.5:3300/0 conn(0x7f87d40ccb80 0x7f87d406bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.961+0000 7f2582f1f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2574035fd0 con 0x7f2574046a40 2024-03-24T10:45:13.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.961+0000 7f2582f1f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f2574171600 con 0x7f257404d7a0 2024-03-24T10:45:13.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.962+0000 7fb1cffff640 1 --2- 172.21.15.5:0/713318047 >> v2:172.21.15.5:3300/0 conn(0x7fb1d00c7680 0x7fb1d00c7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.964+0000 7f7f2ced7640 1 --2- 172.21.15.5:0/2606280843 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f282015d0 0x7f7f2815f2d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c00c3e0 tx=0x7f7f1c00c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:13.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.966+0000 7fb1cffff640 1 -- 172.21.15.5:0/713318047 >> v1:172.21.15.57:6789/0 conn(0x7fb1d0157030 legacy=0x7fb1d01619b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.966+0000 7fb1cffff640 1 -- 172.21.15.5:0/713318047 >> v1:172.21.15.5:6789/0 conn(0x7fb1d0151290 legacy=0x7fb1d006c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:13.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.966+0000 7fb1cffff640 1 -- 172.21.15.5:0/713318047 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d00c8a90 con 0x7fb1d00c7680 2024-03-24T10:45:13.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.967+0000 7f0c8bfff640 1 -- 172.21.15.5:0/3105784729 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c84031800 con 0x7f0c8c046160 2024-03-24T10:45:13.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.967+0000 7f0c8bfff640 1 -- 172.21.15.5:0/3105784729 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c84031960 con 0x7f0c8c046160 2024-03-24T10:45:13.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.967+0000 7f0c8bfff640 1 -- 172.21.15.5:0/3105784729 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c84031cf0 con 0x7f0c8c046160 2024-03-24T10:45:13.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.968+0000 7f0c9a42b640 1 -- 172.21.15.5:0/3105784729 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c8c046160 msgr2=0x7f0c8c009140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.968+0000 7f0c9a42b640 1 --2- 172.21.15.5:0/3105784729 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c8c046160 0x7f0c8c009140 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c0466f0 tx=0x7f0c8402b8a0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.968+0000 7f0c9a42b640 1 -- 172.21.15.5:0/3105784729 shutdown_connections 2024-03-24T10:45:13.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.968+0000 7f0c9a42b640 1 --2- 172.21.15.5:0/3105784729 >> v2:172.21.15.5:3300/0 conn(0x7f0c8c036500 0x7f0c8c036930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.968+0000 7f0c9a42b640 1 --2- 172.21.15.5:0/3105784729 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c8c046160 0x7f0c8c009140 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.968+0000 7f0c9a42b640 1 -- 172.21.15.5:0/3105784729 >> 172.21.15.5:0/3105784729 conn(0x7f0c8c051280 msgr2=0x7f0c8c051690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:13.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.968+0000 7f87d8a3a640 1 -- 172.21.15.5:0/1126236484 >> v2:172.21.15.57:3300/0 conn(0x7f87d40cf340 msgr2=0x7f87d40cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:13.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.968+0000 7f87d8a3a640 1 --2- 172.21.15.5:0/1126236484 >> v2:172.21.15.57:3300/0 conn(0x7f87d40cf340 0x7f87d40cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.968+0000 7f87d8a3a640 1 -- 172.21.15.5:0/1126236484 >> v1:172.21.15.57:6789/0 conn(0x7f87d40cc000 legacy=0x7f87d415e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.968+0000 7f87d8a3a640 1 -- 172.21.15.5:0/1126236484 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d415a5f0 con 0x7f87d40ccb80 2024-03-24T10:45:13.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.968+0000 7f0c9a42b640 1 -- 172.21.15.5:0/3105784729 shutdown_connections 2024-03-24T10:45:13.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.968+0000 7f0c9a42b640 1 -- 172.21.15.5:0/3105784729 wait complete. 2024-03-24T10:45:13.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.970+0000 7f0c9a42b640 1 Processor -- start 2024-03-24T10:45:13.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.970+0000 7f0c9a42b640 1 -- start start 2024-03-24T10:45:13.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.971+0000 7f0c9a42b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c8c036500 0x7f0c8c0d8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.971+0000 7f0c9a42b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c8c046160 0x7f0c8c0d9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.971+0000 7f0c9a42b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c8c0d6ee0 0x7f0c8c035bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.971+0000 7f0c9a42b640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f0c8c0d9120 con 0x7f0c8c0d6ee0 2024-03-24T10:45:13.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.971+0000 7f0c9a42b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0c8c0da040 con 0x7f0c8c036500 2024-03-24T10:45:13.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.971+0000 7f0c9a42b640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0c8c0d7ba0 con 0x7f0c8c046160 2024-03-24T10:45:13.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.971+0000 7f8ef37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ef4071ff0 0x7f8ef414f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.972+0000 7f8ef37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ef4071ff0 0x7f8ef414f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36494/0 (socket says 172.21.15.5:36494) 2024-03-24T10:45:13.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.972+0000 7f8ef37fe640 1 -- 172.21.15.5:0/766319016 learned_addr learned my addr 172.21.15.5:0/766319016 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:13.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.972+0000 7fb1cffff640 1 --2- 172.21.15.5:0/713318047 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d00c7680 0x7fb1d00c7ac0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc0097a0 tx=0x7fb1bc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be06258092faaa66 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:13.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.973+0000 7fd7a53de640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd7a006fc70 0x7fd7a01f8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.973+0000 7f87d8a3a640 1 --2- 172.21.15.5:0/1126236484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87d40ccb80 0x7f87d406bf50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f87d40cd790 tx=0x7f87cc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a0f401f563a0a95 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:13.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.973+0000 7f8ef8ccb640 1 --2- 172.21.15.5:0/766319016 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ef4150740 0x7f8ef41f4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.973+0000 7fd7a53de640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd7a006fc70 0x7fd7a01f8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:51378/0 (socket says 172.21.15.5:51378) 2024-03-24T10:45:13.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.973+0000 7fd7a53de640 1 -- 172.21.15.5:0/719227015 learned_addr learned my addr 172.21.15.5:0/719227015 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:13.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.979+0000 7f8ef8ccb640 1 -- 172.21.15.5:0/766319016 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ef4071ff0 msgr2=0x7f8ef414f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.979+0000 7f8ef8ccb640 1 --2- 172.21.15.5:0/766319016 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ef4071ff0 0x7f8ef414f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.979+0000 7f8ef8ccb640 1 -- 172.21.15.5:0/766319016 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ef406f6b0 msgr2=0x7f8ef40c3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.979+0000 7f8ef8ccb640 1 --2- 172.21.15.5:0/766319016 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ef406f6b0 0x7f8ef40c3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.979+0000 7f8ef8ccb640 1 -- 172.21.15.5:0/766319016 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eec00d050 con 0x7f8ef4150740 2024-03-24T10:45:13.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.979+0000 7fd7a5bdf640 1 --2- 172.21.15.5:0/719227015 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7a01506d0 0x7fd7a00cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.981+0000 7f37b7fff640 1 -- 172.21.15.5:0/1470222649 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b8031680 con 0x7f37c00c7d60 2024-03-24T10:45:13.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.981+0000 7f37b7fff640 1 -- 172.21.15.5:0/1470222649 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37b80317e0 con 0x7f37c00c7d60 2024-03-24T10:45:13.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.981+0000 7f37b7fff640 1 -- 172.21.15.5:0/1470222649 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b8031b70 con 0x7f37c00c7d60 2024-03-24T10:45:13.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.982+0000 7f37c76f4640 1 -- 172.21.15.5:0/1470222649 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f37c00c7d60 msgr2=0x7f37c00c5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.982+0000 7f37c76f4640 1 --2- 172.21.15.5:0/1470222649 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f37c00c7d60 0x7f37c00c5360 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f37b800c6a0 tx=0x7f37b802c9d0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.982+0000 7f37c76f4640 1 -- 172.21.15.5:0/1470222649 shutdown_connections 2024-03-24T10:45:13.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.982+0000 7f37c76f4640 1 --2- 172.21.15.5:0/1470222649 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f37c00c7d60 0x7f37c00c5360 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.982+0000 7f37c76f4640 1 -- 172.21.15.5:0/1470222649 >> 172.21.15.5:0/1470222649 conn(0x7f37c006f180 msgr2=0x7f37c006f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:13.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.982+0000 7f37c76f4640 1 -- 172.21.15.5:0/1470222649 shutdown_connections 2024-03-24T10:45:13.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.982+0000 7f37c76f4640 1 -- 172.21.15.5:0/1470222649 wait complete. 2024-03-24T10:45:13.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.982+0000 7fb1cf7fe640 1 -- 172.21.15.5:0/713318047 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc031830 con 0x7fb1d00c7680 2024-03-24T10:45:13.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.982+0000 7fb1cf7fe640 1 -- 172.21.15.5:0/713318047 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1bc031990 con 0x7fb1d00c7680 2024-03-24T10:45:13.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.982+0000 7fb1cf7fe640 1 -- 172.21.15.5:0/713318047 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc031d20 con 0x7fb1d00c7680 2024-03-24T10:45:13.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.983+0000 7f37c76f4640 1 Processor -- start 2024-03-24T10:45:13.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.983+0000 7f37c76f4640 1 -- start start 2024-03-24T10:45:13.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.983+0000 7fb1d71d7640 1 -- 172.21.15.5:0/713318047 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d00c7680 msgr2=0x7fb1d00c7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.983+0000 7fb1d71d7640 1 --2- 172.21.15.5:0/713318047 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d00c7680 0x7fb1d00c7ac0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc0097a0 tx=0x7fb1bc02b8d0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.983+0000 7f2581f1d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2574046a40 0x7f2574036bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7fb1d71d7640 1 -- 172.21.15.5:0/713318047 shutdown_connections 2024-03-24T10:45:13.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7fb1d71d7640 1 --2- 172.21.15.5:0/713318047 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d00c7680 0x7fb1d00c7ac0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7fb1d71d7640 1 -- 172.21.15.5:0/713318047 >> 172.21.15.5:0/713318047 conn(0x7fb1d00c8c50 msgr2=0x7fb1d00c9030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:13.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7fd7a53de640 1 -- 172.21.15.5:0/719227015 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7a01506d0 msgr2=0x7fd7a00cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7fd7a53de640 1 --2- 172.21.15.5:0/719227015 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7a01506d0 0x7fd7a00cab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7fd7a53de640 1 -- 172.21.15.5:0/719227015 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7a00c7d60 msgr2=0x7fd7a014abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7fd7a53de640 1 --2- 172.21.15.5:0/719227015 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7a00c7d60 0x7fd7a014abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7fd7a53de640 1 -- 172.21.15.5:0/719227015 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd798009b30 con 0x7fd7a006fc70 2024-03-24T10:45:13.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7fb1d71d7640 1 -- 172.21.15.5:0/713318047 shutdown_connections 2024-03-24T10:45:13.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7fb1d71d7640 1 -- 172.21.15.5:0/713318047 wait complete. 2024-03-24T10:45:13.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7f7f24ff9640 1 -- 172.21.15.5:0/2606280843 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c0035f0 con 0x7f7f282015d0 2024-03-24T10:45:13.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7f7f24ff9640 1 -- 172.21.15.5:0/2606280843 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f1c003770 con 0x7f7f282015d0 2024-03-24T10:45:13.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7f7f2e93a640 1 -- 172.21.15.5:0/2606280843 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f28200af0 con 0x7f7f282015d0 2024-03-24T10:45:13.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7f7f24ff9640 1 -- 172.21.15.5:0/2606280843 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c003b20 con 0x7f7f282015d0 2024-03-24T10:45:13.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7f7f2e93a640 1 -- 172.21.15.5:0/2606280843 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f282010b0 con 0x7f7f282015d0 2024-03-24T10:45:13.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7f37c76f4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37c00c58a0 0x7f37c0159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7f37c76f4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f37c00c7d60 0x7f37c01eaf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7f37c76f4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f37c01506d0 0x7f37c014b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7f37c76f4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f37c0158c80 con 0x7f37c00c7d60 2024-03-24T10:45:13.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7f37c76f4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f37c01597c0 con 0x7f37c00c58a0 2024-03-24T10:45:13.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.984+0000 7f37c76f4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f37c01589a0 con 0x7f37c01506d0 2024-03-24T10:45:13.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.985+0000 7f2581f1d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2574046a40 0x7f2574036bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36508/0 (socket says 172.21.15.5:36508) 2024-03-24T10:45:13.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.985+0000 7f2581f1d640 1 -- 172.21.15.5:0/3274246168 learned_addr learned my addr 172.21.15.5:0/3274246168 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:13.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.985+0000 7f7f2e93a640 1 -- 172.21.15.5:0/2606280843 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7f28071cb0 con 0x7f7f282015d0 2024-03-24T10:45:13.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.986+0000 7f258271e640 1 --2- 172.21.15.5:0/3274246168 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25740d5cc0 0x7f25741673a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.986+0000 7fb1d71d7640 1 Processor -- start 2024-03-24T10:45:13.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.986+0000 7f258171c640 1 --2- 172.21.15.5:0/3274246168 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f257404d7a0 0x7f25741710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.986+0000 7fb1d71d7640 1 -- start start 2024-03-24T10:45:13.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.988+0000 7fb1d71d7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d00c7680 0x7fb1d0148130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.988+0000 7fb1d71d7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb1d0151290 0x7fb1d00af200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.988+0000 7fb1d71d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1d0157030 0x7fb1d01ed1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.988+0000 7fb1d71d7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d00af740 con 0x7fb1d00c7680 2024-03-24T10:45:13.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.988+0000 7fb1d71d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d01ed730 con 0x7fb1d0157030 2024-03-24T10:45:13.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.988+0000 7fb1d71d7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fb1d0147400 con 0x7fb1d0151290 2024-03-24T10:45:13.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.988+0000 7f37c5490640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37c00c58a0 0x7f37c0159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.989+0000 7f0c99429640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c8c036500 0x7f0c8c0d8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.990+0000 7fd7a53de640 1 --2- 172.21.15.5:0/719227015 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd7a006fc70 0x7fd7a01f8010 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00b4a0 tx=0x7fd79c00b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:13.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.991+0000 7f8ef8ccb640 1 --2- 172.21.15.5:0/766319016 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ef4150740 0x7f8ef41f4020 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8ee800e7f0 tx=0x7f8ee800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:13.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.992+0000 7f0c98c28640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c8c046160 0x7f0c8c0d9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:13.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.992+0000 7f0c99429640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c8c036500 0x7f0c8c0d8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36510/0 (socket says 172.21.15.5:36510) 2024-03-24T10:45:13.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.992+0000 7f0c99429640 1 -- 172.21.15.5:0/1197529209 learned_addr learned my addr 172.21.15.5:0/1197529209 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:13.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.992+0000 7f87d2ffd640 1 -- 172.21.15.5:0/1126236484 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc02cc80 con 0x7f87d40ccb80 2024-03-24T10:45:13.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.992+0000 7f87d2ffd640 1 -- 172.21.15.5:0/1126236484 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87cc02cde0 con 0x7f87d40ccb80 2024-03-24T10:45:13.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.992+0000 7f87d2ffd640 1 -- 172.21.15.5:0/1126236484 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc034660 con 0x7f87d40ccb80 2024-03-24T10:45:13.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.992+0000 7f37c5490640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37c00c58a0 0x7f37c0159280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36520/0 (socket says 172.21.15.5:36520) 2024-03-24T10:45:13.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.992+0000 7f37c5490640 1 -- 172.21.15.5:0/1755794961 learned_addr learned my addr 172.21.15.5:0/1755794961 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:13.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.993+0000 7f87da49d640 1 -- 172.21.15.5:0/1126236484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87d40ccb80 msgr2=0x7f87d406bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.993+0000 7f87da49d640 1 --2- 172.21.15.5:0/1126236484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87d40ccb80 0x7f87d406bf50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f87d40cd790 tx=0x7f87cc02bed0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.993+0000 7f87da49d640 1 -- 172.21.15.5:0/1126236484 shutdown_connections 2024-03-24T10:45:13.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.993+0000 7f87da49d640 1 --2- 172.21.15.5:0/1126236484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87d40ccb80 0x7f87d406bf50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.993+0000 7f87da49d640 1 --2- 172.21.15.5:0/1126236484 >> v2:172.21.15.57:3300/0 conn(0x7f87d40cf340 0x7f87d40cc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.993+0000 7f87da49d640 1 -- 172.21.15.5:0/1126236484 >> 172.21.15.5:0/1126236484 conn(0x7f87d40cfc30 msgr2=0x7f87d40d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:13.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.993+0000 7f87da49d640 1 -- 172.21.15.5:0/1126236484 shutdown_connections 2024-03-24T10:45:13.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.993+0000 7f87da49d640 1 -- 172.21.15.5:0/1126236484 wait complete. 2024-03-24T10:45:13.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.995+0000 7f87da49d640 1 Processor -- start 2024-03-24T10:45:13.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.995+0000 7f87da49d640 1 -- start start 2024-03-24T10:45:13.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.996+0000 7f258171c640 1 -- 172.21.15.5:0/3274246168 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2574046a40 msgr2=0x7f2574036bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.996+0000 7f258171c640 1 --2- 172.21.15.5:0/3274246168 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2574046a40 0x7f2574036bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.996+0000 7f258171c640 1 -- 172.21.15.5:0/3274246168 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25740d5cc0 msgr2=0x7f25741673a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:13.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.996+0000 7f258171c640 1 --2- 172.21.15.5:0/3274246168 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25740d5cc0 0x7f25741673a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:13.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.996+0000 7f258171c640 1 -- 172.21.15.5:0/3274246168 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25780094d0 con 0x7f257404d7a0 2024-03-24T10:45:13.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.996+0000 7f87da49d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87d40cc000 0x7f87d4151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.996+0000 7f87da49d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f87d40ccb80 0x7f87d415d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.997+0000 7f87da49d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87d40cf340 0x7f87d41f1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:13.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.997+0000 7f87da49d640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f87d40b1e80 con 0x7f87d40cc000 2024-03-24T10:45:13.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.997+0000 7f87da49d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f87d4151d40 con 0x7f87d40cf340 2024-03-24T10:45:13.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.997+0000 7f87da49d640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f87d40b0ff0 con 0x7f87d40ccb80 2024-03-24T10:45:13.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:13.999+0000 7f37c4c8f640 1 --2- 172.21.15.5:0/1755794961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f37c00c7d60 0x7f37c01eaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.000+0000 7f258171c640 1 --2- 172.21.15.5:0/3274246168 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f257404d7a0 0x7f25741710c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f256c00c410 tx=0x7f256c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.002+0000 7f0c99429640 1 -- 172.21.15.5:0/1197529209 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c8c046160 msgr2=0x7f0c8c0d9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.002+0000 7f0c99429640 1 --2- 172.21.15.5:0/1197529209 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c8c046160 0x7f0c8c0d9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.002+0000 7f0c99429640 1 -- 172.21.15.5:0/1197529209 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c8c0d6ee0 msgr2=0x7f0c8c035bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.002+0000 7f0c99429640 1 --2- 172.21.15.5:0/1197529209 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c8c0d6ee0 0x7f0c8c035bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.002+0000 7f0c99429640 1 -- 172.21.15.5:0/1197529209 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c840094d0 con 0x7f0c8c036500 2024-03-24T10:45:14.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.006+0000 7f37c5490640 1 -- 172.21.15.5:0/1755794961 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f37c01506d0 msgr2=0x7f37c014b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.006+0000 7f37c5490640 1 --2- 172.21.15.5:0/1755794961 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f37c01506d0 0x7f37c014b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.006+0000 7f37c5490640 1 -- 172.21.15.5:0/1755794961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f37c00c7d60 msgr2=0x7f37c01eaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.006+0000 7f37c5490640 1 --2- 172.21.15.5:0/1755794961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f37c00c7d60 0x7f37c01eaf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.006+0000 7f37c5490640 1 -- 172.21.15.5:0/1755794961 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37b800c300 con 0x7f37c00c58a0 2024-03-24T10:45:14.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.010+0000 7f37c5490640 1 --2- 172.21.15.5:0/1755794961 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37c00c58a0 0x7f37c0159280 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f37b000c480 tx=0x7f37b000c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.010+0000 7f0c99429640 1 --2- 172.21.15.5:0/1197529209 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c8c036500 0x7f0c8c0d8a90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0c9000c410 tx=0x7f0c9000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.017+0000 7fb1d4f73640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d00c7680 0x7fb1d0148130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.017+0000 7f87d3fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87d40cc000 0x7f87d4151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.017+0000 7fb1d4f73640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d00c7680 0x7fb1d0148130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60032/0 (socket says 172.21.15.5:60032) 2024-03-24T10:45:14.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.017+0000 7fb1d4f73640 1 -- 172.21.15.5:0/423773519 learned_addr learned my addr 172.21.15.5:0/423773519 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:14.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.019+0000 7fb1d5774640 1 --2- 172.21.15.5:0/423773519 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1d0157030 0x7fb1d01ed1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.021+0000 7f37b5ffb640 1 -- 172.21.15.5:0/1755794961 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b0005ce0 con 0x7f37c00c58a0 2024-03-24T10:45:14.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.021+0000 7f37b5ffb640 1 -- 172.21.15.5:0/1755794961 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37b0005e40 con 0x7f37c00c58a0 2024-03-24T10:45:14.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.021+0000 7f37b5ffb640 1 -- 172.21.15.5:0/1755794961 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b00072f0 con 0x7f37c00c58a0 2024-03-24T10:45:14.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.021+0000 7f37c76f4640 1 -- 172.21.15.5:0/1755794961 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37c0072070 con 0x7f37c00c58a0 2024-03-24T10:45:14.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.021+0000 7f37c76f4640 1 -- 172.21.15.5:0/1755794961 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37c0072630 con 0x7f37c00c58a0 2024-03-24T10:45:14.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.022+0000 7f37c76f4640 1 -- 172.21.15.5:0/1755794961 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f37c00c2ed0 con 0x7f37c00c58a0 2024-03-24T10:45:14.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.023+0000 7f8ef0ff9640 1 -- 172.21.15.5:0/766319016 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee800c730 con 0x7f8ef4150740 2024-03-24T10:45:14.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.023+0000 7f8ef0ff9640 1 -- 172.21.15.5:0/766319016 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ee800c890 con 0x7f8ef4150740 2024-03-24T10:45:14.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.023+0000 7f8efa72e640 1 -- 172.21.15.5:0/766319016 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ef40c3690 con 0x7f8ef4150740 2024-03-24T10:45:14.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.023+0000 7fd791ffb640 1 -- 172.21.15.5:0/719227015 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c010610 con 0x7fd7a006fc70 2024-03-24T10:45:14.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.023+0000 7fd791ffb640 1 -- 172.21.15.5:0/719227015 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd79c010770 con 0x7fd7a006fc70 2024-03-24T10:45:14.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.023+0000 7f8ef0ff9640 1 -- 172.21.15.5:0/766319016 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee800cc40 con 0x7f8ef4150740 2024-03-24T10:45:14.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.023+0000 7f8efa72e640 1 -- 172.21.15.5:0/766319016 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ef40cb920 con 0x7f8ef4150740 2024-03-24T10:45:14.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.023+0000 7fd7a7642640 1 -- 172.21.15.5:0/719227015 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7a0149e20 con 0x7fd7a006fc70 2024-03-24T10:45:14.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.024+0000 7fd791ffb640 1 -- 172.21.15.5:0/719227015 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c010b20 con 0x7fd7a006fc70 2024-03-24T10:45:14.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.024+0000 7fd7a7642640 1 -- 172.21.15.5:0/719227015 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7a013cf80 con 0x7fd7a006fc70 2024-03-24T10:45:14.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.024+0000 7f8efa72e640 1 -- 172.21.15.5:0/766319016 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8ef406dc10 con 0x7f8ef4150740 2024-03-24T10:45:14.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.024+0000 7fd7a7642640 1 -- 172.21.15.5:0/719227015 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd7a0150ab0 con 0x7fd7a006fc70 2024-03-24T10:45:14.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.025+0000 7f0c89ffb640 1 -- 172.21.15.5:0/1197529209 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c900035f0 con 0x7f0c8c036500 2024-03-24T10:45:14.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.025+0000 7f0c89ffb640 1 -- 172.21.15.5:0/1197529209 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c90003750 con 0x7f0c8c036500 2024-03-24T10:45:14.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.025+0000 7f87d37fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f87d40ccb80 0x7f87d415d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.025+0000 7f0c9a42b640 1 -- 172.21.15.5:0/1197529209 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c8c0d75e0 con 0x7f0c8c036500 2024-03-24T10:45:14.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.025+0000 7f0c89ffb640 1 -- 172.21.15.5:0/1197529209 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c90003b00 con 0x7f0c8c036500 2024-03-24T10:45:14.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.025+0000 7f87d8a3a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87d40cf340 0x7f87d41f1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.025+0000 7f0c9a42b640 1 -- 172.21.15.5:0/1197529209 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c8c0cbd90 con 0x7f0c8c036500 2024-03-24T10:45:14.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.026+0000 7f0c9a42b640 1 -- 172.21.15.5:0/1197529209 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0c8c0dd500 con 0x7f0c8c036500 2024-03-24T10:45:14.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.028+0000 7fb1d5774640 1 -- 172.21.15.5:0/423773519 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb1d0151290 msgr2=0x7fb1d00af200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.028+0000 7fb1d5774640 1 --2- 172.21.15.5:0/423773519 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb1d0151290 0x7fb1d00af200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.028+0000 7fb1d5774640 1 -- 172.21.15.5:0/423773519 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d00c7680 msgr2=0x7fb1d0148130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.028+0000 7fb1d5774640 1 --2- 172.21.15.5:0/423773519 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb1d00c7680 0x7fb1d0148130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.028+0000 7fb1d5774640 1 -- 172.21.15.5:0/423773519 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1bc009400 con 0x7fb1d0157030 2024-03-24T10:45:14.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.031+0000 7f87d3fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87d40cc000 0x7f87d4151800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60048/0 (socket says 172.21.15.5:60048) 2024-03-24T10:45:14.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.031+0000 7f87d3fff640 1 -- 172.21.15.5:0/153296255 learned_addr learned my addr 172.21.15.5:0/153296255 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:14.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.036+0000 7f25727fc640 1 -- 172.21.15.5:0/3274246168 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c0035f0 con 0x7f257404d7a0 2024-03-24T10:45:14.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.036+0000 7f25727fc640 1 -- 172.21.15.5:0/3274246168 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f256c003750 con 0x7f257404d7a0 2024-03-24T10:45:14.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.036+0000 7f25727fc640 1 -- 172.21.15.5:0/3274246168 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c003ae0 con 0x7f257404d7a0 2024-03-24T10:45:14.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.036+0000 7f2582f1f640 1 -- 172.21.15.5:0/3274246168 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2574037310 con 0x7f257404d7a0 2024-03-24T10:45:14.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.036+0000 7f2582f1f640 1 -- 172.21.15.5:0/3274246168 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2574035980 con 0x7f257404d7a0 2024-03-24T10:45:14.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.037+0000 7f2582f1f640 1 -- 172.21.15.5:0/3274246168 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f25740091c0 con 0x7f257404d7a0 2024-03-24T10:45:14.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.042+0000 7f87d37fe640 1 -- 172.21.15.5:0/153296255 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87d40cf340 msgr2=0x7f87d41f1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.042+0000 7f87d37fe640 1 --2- 172.21.15.5:0/153296255 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87d40cf340 0x7f87d41f1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.042+0000 7f87d37fe640 1 -- 172.21.15.5:0/153296255 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87d40cc000 msgr2=0x7f87d4151800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.042+0000 7f87d37fe640 1 --2- 172.21.15.5:0/153296255 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87d40cc000 0x7f87d4151800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.042+0000 7f87d37fe640 1 -- 172.21.15.5:0/153296255 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87cc009a70 con 0x7f87d40ccb80 2024-03-24T10:45:14.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.056+0000 7f37b5ffb640 1 -- 172.21.15.5:0/1755794961 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f37b0007450 con 0x7f37c00c58a0 2024-03-24T10:45:14.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.056+0000 7f37b5ffb640 1 --2- 172.21.15.5:0/1755794961 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f37a406b840 0x7f37a406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.057+0000 7f37c4c8f640 1 --2- 172.21.15.5:0/1755794961 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f37a406b840 0x7f37a406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.057+0000 7f37c4c8f640 1 --2- 172.21.15.5:0/1755794961 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f37a406b840 0x7f37a406dd00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f37b8037000 tx=0x7f37b8032000 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.062+0000 7f37b5ffb640 1 -- 172.21.15.5:0/1755794961 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f37b00586e0 con 0x7f37c00c58a0 2024-03-24T10:45:14.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.062+0000 7f37b5ffb640 1 --2- 172.21.15.5:0/1755794961 >> [v2:172.21.15.5:6824/1991336051,v1:172.21.15.5:6825/1991336051] conn(0x7f37a4073a40 0x7f37a4075e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.062+0000 7f37b5ffb640 1 -- 172.21.15.5:0/1755794961 --> [v2:172.21.15.5:6824/1991336051,v1:172.21.15.5:6825/1991336051] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f37a4076510 con 0x7f37a4073a40 2024-03-24T10:45:14.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.062+0000 7f37c5c91640 1 --2- 172.21.15.5:0/1755794961 >> [v2:172.21.15.5:6824/1991336051,v1:172.21.15.5:6825/1991336051] conn(0x7f37a4073a40 0x7f37a4075e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.063+0000 7fb1d5774640 1 --2- 172.21.15.5:0/423773519 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1d0157030 0x7fb1d01ed1f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb1c800ca70 tx=0x7fb1c800cf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.063+0000 7f37c5c91640 1 --2- 172.21.15.5:0/1755794961 >> [v2:172.21.15.5:6824/1991336051,v1:172.21.15.5:6825/1991336051] conn(0x7f37a4073a40 0x7f37a4075e60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.064+0000 7f37b5ffb640 1 -- 172.21.15.5:0/1755794961 <== osd.0 v2:172.21.15.5:6824/1991336051 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f37a4076510 con 0x7f37a4073a40 2024-03-24T10:45:14.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.064+0000 7f7f24ff9640 1 -- 172.21.15.5:0/2606280843 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7f1c003c80 con 0x7f7f282015d0 2024-03-24T10:45:14.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.064+0000 7f7f24ff9640 1 --2- 172.21.15.5:0/2606280843 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7f1406b860 0x7f7f1406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.064+0000 7f7f27fff640 1 --2- 172.21.15.5:0/2606280843 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7f1406b860 0x7f7f1406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.065+0000 7f7f27fff640 1 --2- 172.21.15.5:0/2606280843 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7f1406b860 0x7f7f1406dd20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7f180047c0 tx=0x7f7f18009120 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.078+0000 7f37c76f4640 1 -- 172.21.15.5:0/1755794961 --> [v2:172.21.15.5:6824/1991336051,v1:172.21.15.5:6825/1991336051] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f37c0159e00 con 0x7f37a4073a40 2024-03-24T10:45:14.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.078+0000 7f37b5ffb640 1 -- 172.21.15.5:0/1755794961 <== osd.0 v2:172.21.15.5:6824/1991336051 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f37bc0050b0 con 0x7f37a4073a40 2024-03-24T10:45:14.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.079+0000 7f37c76f4640 1 -- 172.21.15.5:0/1755794961 >> [v2:172.21.15.5:6824/1991336051,v1:172.21.15.5:6825/1991336051] conn(0x7f37a4073a40 msgr2=0x7f37a4075e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.079+0000 7f37c76f4640 1 --2- 172.21.15.5:0/1755794961 >> [v2:172.21.15.5:6824/1991336051,v1:172.21.15.5:6825/1991336051] conn(0x7f37a4073a40 0x7f37a4075e60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.079+0000 7f37c76f4640 1 -- 172.21.15.5:0/1755794961 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f37a406b840 msgr2=0x7f37a406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.079+0000 7f37c76f4640 1 --2- 172.21.15.5:0/1755794961 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f37a406b840 0x7f37a406dd00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f37b8037000 tx=0x7f37b8032000 comp rx=0 tx=0).stop 2024-03-24T10:45:14.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.079+0000 7f37c76f4640 1 -- 172.21.15.5:0/1755794961 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37c00c58a0 msgr2=0x7f37c0159280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.079+0000 7f37c76f4640 1 --2- 172.21.15.5:0/1755794961 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37c00c58a0 0x7f37c0159280 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f37b000c480 tx=0x7f37b000c940 comp rx=0 tx=0).stop 2024-03-24T10:45:14.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.079+0000 7f37c5c91640 1 -- 172.21.15.5:0/1755794961 reap_dead start 2024-03-24T10:45:14.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.079+0000 7f37c76f4640 1 -- 172.21.15.5:0/1755794961 shutdown_connections 2024-03-24T10:45:14.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.079+0000 7f37c76f4640 1 -- 172.21.15.5:0/1755794961 >> 172.21.15.5:0/1755794961 conn(0x7f37c006f180 msgr2=0x7f37c00c8140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:14.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.079+0000 7f37c76f4640 1 -- 172.21.15.5:0/1755794961 shutdown_connections 2024-03-24T10:45:14.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.079+0000 7f37c76f4640 1 -- 172.21.15.5:0/1755794961 wait complete. 2024-03-24T10:45:14.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.086+0000 7f0c89ffb640 1 -- 172.21.15.5:0/1197529209 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0c90003c60 con 0x7f0c8c036500 2024-03-24T10:45:14.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.086+0000 7f0c89ffb640 1 --2- 172.21.15.5:0/1197529209 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0c7806b840 0x7f0c7806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.087+0000 7f0c98c28640 1 --2- 172.21.15.5:0/1197529209 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0c7806b840 0x7f0c7806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.087+0000 7f0c98c28640 1 --2- 172.21.15.5:0/1197529209 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0c7806b840 0x7f0c7806dd00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0c84036000 tx=0x7f0c8403c040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.090 INFO:teuthology.orchestra.run.smithi005.stdout:55834574856 2024-03-24T10:45:14.090 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-24T10:45:14.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.090+0000 7f0c89ffb640 1 -- 172.21.15.5:0/1197529209 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0c900010c0 con 0x7f0c8c036500 2024-03-24T10:45:14.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.090+0000 7f0c89ffb640 1 --2- 172.21.15.5:0/1197529209 >> [v2:172.21.15.5:6816/90414679,v1:172.21.15.5:6817/90414679] conn(0x7f0c78073a40 0x7f0c78075e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.090+0000 7f0c89ffb640 1 -- 172.21.15.5:0/1197529209 --> [v2:172.21.15.5:6816/90414679,v1:172.21.15.5:6817/90414679] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c78076510 con 0x7f0c78073a40 2024-03-24T10:45:14.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.090+0000 7f0c99c2a640 1 --2- 172.21.15.5:0/1197529209 >> [v2:172.21.15.5:6816/90414679,v1:172.21.15.5:6817/90414679] conn(0x7f0c78073a40 0x7f0c78075e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.091+0000 7f0c99c2a640 1 --2- 172.21.15.5:0/1197529209 >> [v2:172.21.15.5:6816/90414679,v1:172.21.15.5:6817/90414679] conn(0x7f0c78073a40 0x7f0c78075e60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.091+0000 7f8ef0ff9640 1 -- 172.21.15.5:0/766319016 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8ee8013470 con 0x7f8ef4150740 2024-03-24T10:45:14.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.091+0000 7f8ef0ff9640 1 --2- 172.21.15.5:0/766319016 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8ee006b810 0x7f8ee006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.091+0000 7f8ef3fff640 1 --2- 172.21.15.5:0/766319016 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8ee006b810 0x7f8ee006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.092+0000 7f8ef3fff640 1 --2- 172.21.15.5:0/766319016 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8ee006b810 0x7f8ee006dcd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8ee4003cc0 tx=0x7f8ee40079b0 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.092+0000 7f0c89ffb640 1 -- 172.21.15.5:0/1197529209 <== osd.2 v2:172.21.15.5:6816/90414679 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0c78076510 con 0x7f0c78073a40 2024-03-24T10:45:14.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.096+0000 7f7f24ff9640 1 -- 172.21.15.5:0/2606280843 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7f1c058d20 con 0x7f7f282015d0 2024-03-24T10:45:14.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.096+0000 7f7f24ff9640 1 --2- 172.21.15.5:0/2606280843 >> [v2:172.21.15.5:6800/2140667501,v1:172.21.15.5:6801/2140667501] conn(0x7f7f14073a60 0x7f7f14075e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.096+0000 7f7f24ff9640 1 -- 172.21.15.5:0/2606280843 --> [v2:172.21.15.5:6800/2140667501,v1:172.21.15.5:6801/2140667501] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f14076530 con 0x7f7f14073a60 2024-03-24T10:45:14.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.096+0000 7f7f277fe640 1 --2- 172.21.15.5:0/2606280843 >> [v2:172.21.15.5:6800/2140667501,v1:172.21.15.5:6801/2140667501] conn(0x7f7f14073a60 0x7f7f14075e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.096+0000 7f7f277fe640 1 --2- 172.21.15.5:0/2606280843 >> [v2:172.21.15.5:6800/2140667501,v1:172.21.15.5:6801/2140667501] conn(0x7f7f14073a60 0x7f7f14075e80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.097+0000 7f7f24ff9640 1 -- 172.21.15.5:0/2606280843 <== osd.1 v2:172.21.15.5:6800/2140667501 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7f14076530 con 0x7f7f14073a60 2024-03-24T10:45:14.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.101+0000 7fb1cd7fa640 1 -- 172.21.15.5:0/423773519 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c800aa90 con 0x7fb1d0157030 2024-03-24T10:45:14.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.101+0000 7fb1cd7fa640 1 -- 172.21.15.5:0/423773519 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1c800abf0 con 0x7fb1d0157030 2024-03-24T10:45:14.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.101+0000 7fb1cd7fa640 1 -- 172.21.15.5:0/423773519 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8012400 con 0x7fb1d0157030 2024-03-24T10:45:14.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.101+0000 7fb1d71d7640 1 -- 172.21.15.5:0/423773519 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1d01464d0 con 0x7fb1d0157030 2024-03-24T10:45:14.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.101+0000 7fb1d71d7640 1 -- 172.21.15.5:0/423773519 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1d0146a10 con 0x7fb1d0157030 2024-03-24T10:45:14.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.102+0000 7f8ef0ff9640 1 -- 172.21.15.5:0/766319016 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8ee8059360 con 0x7f8ef4150740 2024-03-24T10:45:14.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.102+0000 7f8ef0ff9640 1 --2- 172.21.15.5:0/766319016 >> [v2:172.21.15.57:6808/3044987466,v1:172.21.15.57:6809/3044987466] conn(0x7f8ee0073a10 0x7f8ee0075e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.102+0000 7f8ef0ff9640 1 -- 172.21.15.5:0/766319016 --> [v2:172.21.15.57:6808/3044987466,v1:172.21.15.57:6809/3044987466] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ee00764e0 con 0x7f8ee0073a10 2024-03-24T10:45:14.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.102+0000 7fb1d71d7640 1 -- 172.21.15.5:0/423773519 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb1d006c950 con 0x7fb1d0157030 2024-03-24T10:45:14.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.102+0000 7f8ef37fe640 1 --2- 172.21.15.5:0/766319016 >> [v2:172.21.15.57:6808/3044987466,v1:172.21.15.57:6809/3044987466] conn(0x7f8ee0073a10 0x7f8ee0075e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.103+0000 7f8ef37fe640 1 --2- 172.21.15.5:0/766319016 >> [v2:172.21.15.57:6808/3044987466,v1:172.21.15.57:6809/3044987466] conn(0x7f8ee0073a10 0x7f8ee0075e30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.104+0000 7f8ef0ff9640 1 -- 172.21.15.5:0/766319016 <== osd.5 v2:172.21.15.57:6808/3044987466 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8ee00764e0 con 0x7f8ee0073a10 2024-03-24T10:45:14.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.105+0000 7f87d37fe640 1 --2- 172.21.15.5:0/153296255 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f87d40ccb80 0x7f87d415d380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f87c800b3f0 tx=0x7f87c800b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.120+0000 7f0c9a42b640 1 -- 172.21.15.5:0/1197529209 --> [v2:172.21.15.5:6816/90414679,v1:172.21.15.5:6817/90414679] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0c8c0dd220 con 0x7f0c78073a40 2024-03-24T10:45:14.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.120+0000 7f0c89ffb640 1 -- 172.21.15.5:0/1197529209 <== osd.2 v2:172.21.15.5:6816/90414679 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0c9406d040 con 0x7f0c78073a40 2024-03-24T10:45:14.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.121+0000 7f0c9a42b640 1 -- 172.21.15.5:0/1197529209 >> [v2:172.21.15.5:6816/90414679,v1:172.21.15.5:6817/90414679] conn(0x7f0c78073a40 msgr2=0x7f0c78075e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.121+0000 7f0c9a42b640 1 --2- 172.21.15.5:0/1197529209 >> [v2:172.21.15.5:6816/90414679,v1:172.21.15.5:6817/90414679] conn(0x7f0c78073a40 0x7f0c78075e60 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.121+0000 7f0c9a42b640 1 -- 172.21.15.5:0/1197529209 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0c7806b840 msgr2=0x7f0c7806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.121+0000 7f0c9a42b640 1 --2- 172.21.15.5:0/1197529209 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0c7806b840 0x7f0c7806dd00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0c84036000 tx=0x7f0c8403c040 comp rx=0 tx=0).stop 2024-03-24T10:45:14.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.121+0000 7f0c9a42b640 1 -- 172.21.15.5:0/1197529209 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c8c036500 msgr2=0x7f0c8c0d8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.121+0000 7f0c9a42b640 1 --2- 172.21.15.5:0/1197529209 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c8c036500 0x7f0c8c0d8a90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0c9000c410 tx=0x7f0c9000c8d0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.122+0000 7f0c99c2a640 1 -- 172.21.15.5:0/1197529209 reap_dead start 2024-03-24T10:45:14.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.122+0000 7f0c9a42b640 1 -- 172.21.15.5:0/1197529209 shutdown_connections 2024-03-24T10:45:14.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.122+0000 7f0c9a42b640 1 -- 172.21.15.5:0/1197529209 >> 172.21.15.5:0/1197529209 conn(0x7f0c8c051280 msgr2=0x7f0c8c004e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:14.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.122+0000 7f0c9a42b640 1 -- 172.21.15.5:0/1197529209 shutdown_connections 2024-03-24T10:45:14.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.122+0000 7f0c9a42b640 1 -- 172.21.15.5:0/1197529209 wait complete. 2024-03-24T10:45:14.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.125+0000 7f8efa72e640 1 -- 172.21.15.5:0/766319016 --> [v2:172.21.15.57:6808/3044987466,v1:172.21.15.57:6809/3044987466] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8ef4156aa0 con 0x7f8ee0073a10 2024-03-24T10:45:14.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.125+0000 7f8ef0ff9640 1 -- 172.21.15.5:0/766319016 <== osd.5 v2:172.21.15.57:6808/3044987466 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8eec007b20 con 0x7f8ee0073a10 2024-03-24T10:45:14.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.126+0000 7f8efa72e640 1 -- 172.21.15.5:0/766319016 >> [v2:172.21.15.57:6808/3044987466,v1:172.21.15.57:6809/3044987466] conn(0x7f8ee0073a10 msgr2=0x7f8ee0075e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.126+0000 7f8efa72e640 1 --2- 172.21.15.5:0/766319016 >> [v2:172.21.15.57:6808/3044987466,v1:172.21.15.57:6809/3044987466] conn(0x7f8ee0073a10 0x7f8ee0075e30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.126+0000 7f8efa72e640 1 -- 172.21.15.5:0/766319016 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8ee006b810 msgr2=0x7f8ee006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.126+0000 7f8efa72e640 1 --2- 172.21.15.5:0/766319016 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8ee006b810 0x7f8ee006dcd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8ee4003cc0 tx=0x7f8ee40079b0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.126+0000 7f8efa72e640 1 -- 172.21.15.5:0/766319016 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ef4150740 msgr2=0x7f8ef41f4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.126+0000 7f8efa72e640 1 --2- 172.21.15.5:0/766319016 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ef4150740 0x7f8ef41f4020 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8ee800e7f0 tx=0x7f8ee800ecb0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.126+0000 7f8ef8ccb640 1 -- 172.21.15.5:0/766319016 reap_dead start 2024-03-24T10:45:14.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.127+0000 7f8efa72e640 1 -- 172.21.15.5:0/766319016 shutdown_connections 2024-03-24T10:45:14.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.127+0000 7f8efa72e640 1 -- 172.21.15.5:0/766319016 >> 172.21.15.5:0/766319016 conn(0x7f8ef40c8b70 msgr2=0x7f8ef414d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:14.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.127+0000 7f8efa72e640 1 -- 172.21.15.5:0/766319016 shutdown_connections 2024-03-24T10:45:14.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.127+0000 7f8efa72e640 1 -- 172.21.15.5:0/766319016 wait complete. 2024-03-24T10:45:14.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.128+0000 7f7f2e93a640 1 -- 172.21.15.5:0/2606280843 --> [v2:172.21.15.5:6800/2140667501,v1:172.21.15.5:6801/2140667501] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7f280c5440 con 0x7f7f14073a60 2024-03-24T10:45:14.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.128+0000 7f7f24ff9640 1 -- 172.21.15.5:0/2606280843 <== osd.1 v2:172.21.15.5:6800/2140667501 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7f2002da40 con 0x7f7f14073a60 2024-03-24T10:45:14.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.129+0000 7f7f2e93a640 1 -- 172.21.15.5:0/2606280843 >> [v2:172.21.15.5:6800/2140667501,v1:172.21.15.5:6801/2140667501] conn(0x7f7f14073a60 msgr2=0x7f7f14075e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.129+0000 7f7f2e93a640 1 --2- 172.21.15.5:0/2606280843 >> [v2:172.21.15.5:6800/2140667501,v1:172.21.15.5:6801/2140667501] conn(0x7f7f14073a60 0x7f7f14075e80 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.129+0000 7f7f2e93a640 1 -- 172.21.15.5:0/2606280843 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7f1406b860 msgr2=0x7f7f1406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.129+0000 7f7f2e93a640 1 --2- 172.21.15.5:0/2606280843 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7f1406b860 0x7f7f1406dd20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7f180047c0 tx=0x7f7f18009120 comp rx=0 tx=0).stop 2024-03-24T10:45:14.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.129+0000 7f7f2e93a640 1 -- 172.21.15.5:0/2606280843 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f282015d0 msgr2=0x7f7f2815f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.129+0000 7f7f2e93a640 1 --2- 172.21.15.5:0/2606280843 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f282015d0 0x7f7f2815f2d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c00c3e0 tx=0x7f7f1c00c8a0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.129+0000 7f7f2ced7640 1 -- 172.21.15.5:0/2606280843 reap_dead start 2024-03-24T10:45:14.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.129+0000 7f7f2e93a640 1 -- 172.21.15.5:0/2606280843 shutdown_connections 2024-03-24T10:45:14.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.129+0000 7f7f2e93a640 1 -- 172.21.15.5:0/2606280843 >> 172.21.15.5:0/2606280843 conn(0x7f7f2806f180 msgr2=0x7f7f280715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:14.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.129+0000 7f7f2e93a640 1 -- 172.21.15.5:0/2606280843 shutdown_connections 2024-03-24T10:45:14.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.129+0000 7f7f2e93a640 1 -- 172.21.15.5:0/2606280843 wait complete. 2024-03-24T10:45:14.137 INFO:teuthology.orchestra.run.smithi005.stdout:60129542154 2024-03-24T10:45:14.138 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-24T10:45:14.143 INFO:teuthology.orchestra.run.smithi005.stdout:60129542154 2024-03-24T10:45:14.143 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-24T10:45:14.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.143+0000 7f25727fc640 1 -- 172.21.15.5:0/3274246168 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f256c003c40 con 0x7f257404d7a0 2024-03-24T10:45:14.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.144+0000 7f25727fc640 1 --2- 172.21.15.5:0/3274246168 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f256006b860 0x7f256006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.144+0000 7f2581f1d640 1 --2- 172.21.15.5:0/3274246168 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f256006b860 0x7f256006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.145+0000 7f2581f1d640 1 --2- 172.21.15.5:0/3274246168 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f256006b860 0x7f256006dd20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2578034040 tx=0x7f257803c040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.148 INFO:teuthology.orchestra.run.smithi005.stdout:60129542154 2024-03-24T10:45:14.148 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-24T10:45:14.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.150+0000 7f25727fc640 1 -- 172.21.15.5:0/3274246168 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f256c058aa0 con 0x7f257404d7a0 2024-03-24T10:45:14.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.150+0000 7f25727fc640 1 --2- 172.21.15.5:0/3274246168 >> [v2:172.21.15.57:6800/4290897445,v1:172.21.15.57:6801/4290897445] conn(0x7f2560073a60 0x7f2560075e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.150+0000 7f25727fc640 1 -- 172.21.15.5:0/3274246168 --> [v2:172.21.15.57:6800/4290897445,v1:172.21.15.57:6801/4290897445] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2560076530 con 0x7f2560073a60 2024-03-24T10:45:14.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.151+0000 7f258271e640 1 --2- 172.21.15.5:0/3274246168 >> [v2:172.21.15.57:6800/4290897445,v1:172.21.15.57:6801/4290897445] conn(0x7f2560073a60 0x7f2560075e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.151+0000 7f258271e640 1 --2- 172.21.15.5:0/3274246168 >> [v2:172.21.15.57:6800/4290897445,v1:172.21.15.57:6801/4290897445] conn(0x7f2560073a60 0x7f2560075e80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.152+0000 7f25727fc640 1 -- 172.21.15.5:0/3274246168 <== osd.6 v2:172.21.15.57:6800/4290897445 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2560076530 con 0x7f2560073a60 2024-03-24T10:45:14.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.176+0000 7f87d0ff9640 1 -- 172.21.15.5:0/153296255 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c8011830 con 0x7f87d40ccb80 2024-03-24T10:45:14.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.176+0000 7f87d0ff9640 1 -- 172.21.15.5:0/153296255 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87c8011990 con 0x7f87d40ccb80 2024-03-24T10:45:14.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.176+0000 7f87d0ff9640 1 -- 172.21.15.5:0/153296255 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c8011d40 con 0x7f87d40ccb80 2024-03-24T10:45:14.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.177+0000 7f87da49d640 1 -- 172.21.15.5:0/153296255 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87d40b1270 con 0x7f87d40ccb80 2024-03-24T10:45:14.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.177+0000 7f87da49d640 1 -- 172.21.15.5:0/153296255 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f87d40b1830 con 0x7f87d40ccb80 2024-03-24T10:45:14.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.178+0000 7f87ba7fc640 1 -- 172.21.15.5:0/153296255 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f87ac001030 con 0x7f87d40ccb80 2024-03-24T10:45:14.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.180+0000 7f2582f1f640 1 -- 172.21.15.5:0/3274246168 --> [v2:172.21.15.57:6800/4290897445,v1:172.21.15.57:6801/4290897445] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2574008ee0 con 0x7f2560073a60 2024-03-24T10:45:14.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.181+0000 7f25727fc640 1 -- 172.21.15.5:0/3274246168 <== osd.6 v2:172.21.15.57:6800/4290897445 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f257c06d040 con 0x7f2560073a60 2024-03-24T10:45:14.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.181+0000 7f2582f1f640 1 -- 172.21.15.5:0/3274246168 >> [v2:172.21.15.57:6800/4290897445,v1:172.21.15.57:6801/4290897445] conn(0x7f2560073a60 msgr2=0x7f2560075e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.181+0000 7f2582f1f640 1 --2- 172.21.15.5:0/3274246168 >> [v2:172.21.15.57:6800/4290897445,v1:172.21.15.57:6801/4290897445] conn(0x7f2560073a60 0x7f2560075e80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.181+0000 7f2582f1f640 1 -- 172.21.15.5:0/3274246168 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f256006b860 msgr2=0x7f256006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.181+0000 7f2582f1f640 1 --2- 172.21.15.5:0/3274246168 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f256006b860 0x7f256006dd20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2578034040 tx=0x7f257803c040 comp rx=0 tx=0).stop 2024-03-24T10:45:14.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.181+0000 7f2582f1f640 1 -- 172.21.15.5:0/3274246168 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f257404d7a0 msgr2=0x7f25741710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.181+0000 7f2582f1f640 1 --2- 172.21.15.5:0/3274246168 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f257404d7a0 0x7f25741710c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f256c00c410 tx=0x7f256c00c8d0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.182+0000 7f258271e640 1 -- 172.21.15.5:0/3274246168 reap_dead start 2024-03-24T10:45:14.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.182+0000 7f2582f1f640 1 -- 172.21.15.5:0/3274246168 shutdown_connections 2024-03-24T10:45:14.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.182+0000 7f2582f1f640 1 -- 172.21.15.5:0/3274246168 >> 172.21.15.5:0/3274246168 conn(0x7f2574051290 msgr2=0x7f25740d2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:14.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.182+0000 7f2582f1f640 1 -- 172.21.15.5:0/3274246168 shutdown_connections 2024-03-24T10:45:14.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.182+0000 7f2582f1f640 1 -- 172.21.15.5:0/3274246168 wait complete. 2024-03-24T10:45:14.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.190+0000 7fd791ffb640 1 -- 172.21.15.5:0/719227015 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd79c010c80 con 0x7fd7a006fc70 2024-03-24T10:45:14.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.191+0000 7fd791ffb640 1 --2- 172.21.15.5:0/719227015 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd78406b860 0x7fd78406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.191+0000 7fd7a4bdd640 1 --2- 172.21.15.5:0/719227015 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd78406b860 0x7fd78406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.192+0000 7fd7a4bdd640 1 --2- 172.21.15.5:0/719227015 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd78406b860 0x7fd78406dd20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd7a00c6bb0 tx=0x7fd794006ba0 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.196+0000 7fb1cd7fa640 1 -- 172.21.15.5:0/423773519 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb1c8007070 con 0x7fb1d0157030 2024-03-24T10:45:14.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.197+0000 7fb1cd7fa640 1 --2- 172.21.15.5:0/423773519 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb1b406b860 0x7fb1b406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.197+0000 7fb1d4f73640 1 --2- 172.21.15.5:0/423773519 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb1b406b860 0x7fb1b406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.198+0000 7fb1d4f73640 1 --2- 172.21.15.5:0/423773519 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb1b406b860 0x7fb1b406dd20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4001770 tx=0x7fb1c4009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.200 INFO:teuthology.orchestra.run.smithi005.stdout:55834574856 2024-03-24T10:45:14.201 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-24T10:45:14.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.200+0000 7fb1cd7fa640 1 -- 172.21.15.5:0/423773519 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb1c8059590 con 0x7fb1d0157030 2024-03-24T10:45:14.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.200+0000 7fb1cd7fa640 1 --2- 172.21.15.5:0/423773519 >> [v2:172.21.15.5:6808/4212473119,v1:172.21.15.5:6809/4212473119] conn(0x7fb1b4075c10 0x7fb1b4078030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.200+0000 7fb1cd7fa640 1 -- 172.21.15.5:0/423773519 --> [v2:172.21.15.5:6808/4212473119,v1:172.21.15.5:6809/4212473119] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb1b40786e0 con 0x7fb1b4075c10 2024-03-24T10:45:14.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.200+0000 7fb1cffff640 1 --2- 172.21.15.5:0/423773519 >> [v2:172.21.15.5:6808/4212473119,v1:172.21.15.5:6809/4212473119] conn(0x7fb1b4075c10 0x7fb1b4078030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.201+0000 7fd791ffb640 1 -- 172.21.15.5:0/719227015 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd79c059230 con 0x7fd7a006fc70 2024-03-24T10:45:14.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.201+0000 7fb1cffff640 1 --2- 172.21.15.5:0/423773519 >> [v2:172.21.15.5:6808/4212473119,v1:172.21.15.5:6809/4212473119] conn(0x7fb1b4075c10 0x7fb1b4078030 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.201+0000 7fd791ffb640 1 --2- 172.21.15.5:0/719227015 >> [v2:172.21.15.57:6816/2193986505,v1:172.21.15.57:6817/2193986505] conn(0x7fd784073a60 0x7fd784075e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.201+0000 7fd791ffb640 1 -- 172.21.15.5:0/719227015 --> [v2:172.21.15.57:6816/2193986505,v1:172.21.15.57:6817/2193986505] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd784076530 con 0x7fd784073a60 2024-03-24T10:45:14.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.201+0000 7fd7a5bdf640 1 --2- 172.21.15.5:0/719227015 >> [v2:172.21.15.57:6816/2193986505,v1:172.21.15.57:6817/2193986505] conn(0x7fd784073a60 0x7fd784075e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.202+0000 7fd7a5bdf640 1 --2- 172.21.15.5:0/719227015 >> [v2:172.21.15.57:6816/2193986505,v1:172.21.15.57:6817/2193986505] conn(0x7fd784073a60 0x7fd784075e80 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.202+0000 7fb1cd7fa640 1 -- 172.21.15.5:0/423773519 <== osd.3 v2:172.21.15.5:6808/4212473119 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb1b40786e0 con 0x7fb1b4075c10 2024-03-24T10:45:14.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.203+0000 7fd791ffb640 1 -- 172.21.15.5:0/719227015 <== osd.4 v2:172.21.15.57:6816/2193986505 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd784076530 con 0x7fd784073a60 2024-03-24T10:45:14.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.236+0000 7fd7a7642640 1 -- 172.21.15.5:0/719227015 --> [v2:172.21.15.57:6816/2193986505,v1:172.21.15.57:6817/2193986505] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd7a014ed80 con 0x7fd784073a60 2024-03-24T10:45:14.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.237+0000 7fd791ffb640 1 -- 172.21.15.5:0/719227015 <== osd.4 v2:172.21.15.57:6816/2193986505 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd7a014ed80 con 0x7fd784073a60 2024-03-24T10:45:14.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.237+0000 7fd7a7642640 1 -- 172.21.15.5:0/719227015 >> [v2:172.21.15.57:6816/2193986505,v1:172.21.15.57:6817/2193986505] conn(0x7fd784073a60 msgr2=0x7fd784075e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.237+0000 7fd7a7642640 1 --2- 172.21.15.5:0/719227015 >> [v2:172.21.15.57:6816/2193986505,v1:172.21.15.57:6817/2193986505] conn(0x7fd784073a60 0x7fd784075e80 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.238+0000 7fd7a7642640 1 -- 172.21.15.5:0/719227015 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd78406b860 msgr2=0x7fd78406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.238+0000 7fd7a7642640 1 --2- 172.21.15.5:0/719227015 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd78406b860 0x7fd78406dd20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd7a00c6bb0 tx=0x7fd794006ba0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.238+0000 7fd7a7642640 1 -- 172.21.15.5:0/719227015 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd7a006fc70 msgr2=0x7fd7a01f8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.238+0000 7fd7a7642640 1 --2- 172.21.15.5:0/719227015 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd7a006fc70 0x7fd7a01f8010 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd79c00b4a0 tx=0x7fd79c00b960 comp rx=0 tx=0).stop 2024-03-24T10:45:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.238+0000 7fd7a5bdf640 1 -- 172.21.15.5:0/719227015 reap_dead start 2024-03-24T10:45:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.238+0000 7fd7a7642640 1 -- 172.21.15.5:0/719227015 shutdown_connections 2024-03-24T10:45:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.238+0000 7fd7a7642640 1 -- 172.21.15.5:0/719227015 >> 172.21.15.5:0/719227015 conn(0x7fd7a006f180 msgr2=0x7fd7a006f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.239+0000 7fd7a7642640 1 -- 172.21.15.5:0/719227015 shutdown_connections 2024-03-24T10:45:14.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.239+0000 7fd7a7642640 1 -- 172.21.15.5:0/719227015 wait complete. 2024-03-24T10:45:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.241+0000 7fb1d71d7640 1 -- 172.21.15.5:0/423773519 --> [v2:172.21.15.5:6808/4212473119,v1:172.21.15.5:6809/4212473119] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb1d00c8250 con 0x7fb1b4075c10 2024-03-24T10:45:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.241+0000 7fb1cd7fa640 1 -- 172.21.15.5:0/423773519 <== osd.3 v2:172.21.15.5:6808/4212473119 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb1bc031b00 con 0x7fb1b4075c10 2024-03-24T10:45:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.242+0000 7fb1d71d7640 1 -- 172.21.15.5:0/423773519 >> [v2:172.21.15.5:6808/4212473119,v1:172.21.15.5:6809/4212473119] conn(0x7fb1b4075c10 msgr2=0x7fb1b4078030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.242+0000 7fb1d71d7640 1 --2- 172.21.15.5:0/423773519 >> [v2:172.21.15.5:6808/4212473119,v1:172.21.15.5:6809/4212473119] conn(0x7fb1b4075c10 0x7fb1b4078030 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.242+0000 7fb1d71d7640 1 -- 172.21.15.5:0/423773519 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb1b406b860 msgr2=0x7fb1b406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.242+0000 7fb1d71d7640 1 --2- 172.21.15.5:0/423773519 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb1b406b860 0x7fb1b406dd20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4001770 tx=0x7fb1c4009040 comp rx=0 tx=0).stop 2024-03-24T10:45:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.243+0000 7fb1d71d7640 1 -- 172.21.15.5:0/423773519 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1d0157030 msgr2=0x7fb1d01ed1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.243+0000 7fb1d71d7640 1 --2- 172.21.15.5:0/423773519 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb1d0157030 0x7fb1d01ed1f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb1c800ca70 tx=0x7fb1c800cf30 comp rx=0 tx=0).stop 2024-03-24T10:45:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.243+0000 7fb1d5774640 1 -- 172.21.15.5:0/423773519 reap_dead start 2024-03-24T10:45:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.243+0000 7fb1d71d7640 1 -- 172.21.15.5:0/423773519 shutdown_connections 2024-03-24T10:45:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.243+0000 7fb1d71d7640 1 -- 172.21.15.5:0/423773519 >> 172.21.15.5:0/423773519 conn(0x7fb1d00c8c50 msgr2=0x7fb1d0156000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.243+0000 7fb1d71d7640 1 -- 172.21.15.5:0/423773519 shutdown_connections 2024-03-24T10:45:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.243+0000 7fb1d71d7640 1 -- 172.21.15.5:0/423773519 wait complete. 2024-03-24T10:45:14.258 INFO:teuthology.orchestra.run.smithi005.stdout:55834574857 2024-03-24T10:45:14.258 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-24T10:45:14.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.258+0000 7fabfa6b1640 1 Processor -- start 2024-03-24T10:45:14.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.258+0000 7fabfa6b1640 1 -- start start 2024-03-24T10:45:14.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.259+0000 7fabfa6b1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fabec04b4f0 0x7fabec0dac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.259+0000 7fabfa6b1640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fabec04b8d0 0x7fabec005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.259+0000 7fabfa6b1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fabec006410 0x7fabec034a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.260+0000 7fabfa6b1640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fabec007750 con 0x7fabec04b8d0 2024-03-24T10:45:14.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.260+0000 7fabfa6b1640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fabec0068f0 con 0x7fabec006410 2024-03-24T10:45:14.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.260+0000 7fabfa6b1640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fabec006bd0 con 0x7fabec04b4f0 2024-03-24T10:45:14.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.260+0000 7fabf9eb0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fabec006410 0x7fabec034a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.262+0000 7fabf9eb0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fabec006410 0x7fabec034a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36544/0 (socket says 172.21.15.5:36544) 2024-03-24T10:45:14.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.262+0000 7fabf9eb0640 1 -- 172.21.15.5:0/2604692335 learned_addr learned my addr 172.21.15.5:0/2604692335 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:14.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.262+0000 7fabf8eae640 1 --2- 172.21.15.5:0/2604692335 >> v2:172.21.15.5:3300/0 conn(0x7fabec04b8d0 0x7fabec005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.266 INFO:teuthology.orchestra.run.smithi005.stdout:60129542153 2024-03-24T10:45:14.266 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-24T10:45:14.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.267+0000 7fabf8eae640 1 -- 172.21.15.5:0/2604692335 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fabec04b4f0 msgr2=0x7fabec0dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:14.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.267+0000 7fabf8eae640 1 --2- 172.21.15.5:0/2604692335 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fabec04b4f0 0x7fabec0dac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.268+0000 7fabf8eae640 1 -- 172.21.15.5:0/2604692335 >> v2:172.21.15.57:3300/0 conn(0x7fabec006410 msgr2=0x7fabec034a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:14.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.268+0000 7fabf8eae640 1 --2- 172.21.15.5:0/2604692335 >> v2:172.21.15.57:3300/0 conn(0x7fabec006410 0x7fabec034a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.268+0000 7fabf8eae640 1 -- 172.21.15.5:0/2604692335 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabec006eb0 con 0x7fabec04b8d0 2024-03-24T10:45:14.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.279+0000 7fabf8eae640 1 --2- 172.21.15.5:0/2604692335 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabec04b8d0 0x7fabec005ed0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fabec04aef0 tx=0x7fabe402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d212459292e57bd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.285+0000 7f87d0ff9640 1 -- 172.21.15.5:0/153296255 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f87c801a400 con 0x7f87d40ccb80 2024-03-24T10:45:14.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.285+0000 7f87d0ff9640 1 --2- 172.21.15.5:0/153296255 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f87c006b860 0x7f87c006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.286+0000 7f87d3fff640 1 --2- 172.21.15.5:0/153296255 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f87c006b860 0x7f87c006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.286+0000 7f87d3fff640 1 --2- 172.21.15.5:0/153296255 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f87c006b860 0x7f87c006dd20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f87c4005880 tx=0x7f87c4009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.288+0000 7f87d0ff9640 1 -- 172.21.15.5:0/153296255 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f87c80597d0 con 0x7f87d40ccb80 2024-03-24T10:45:14.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.289+0000 7f87d0ff9640 1 --2- 172.21.15.5:0/153296255 >> [v2:172.21.15.57:6824/212605698,v1:172.21.15.57:6825/212605698] conn(0x7f87c0073a60 0x7f87c0075e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.289+0000 7f87d0ff9640 1 -- 172.21.15.5:0/153296255 --> [v2:172.21.15.57:6824/212605698,v1:172.21.15.57:6825/212605698] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f87c0076530 con 0x7f87c0073a60 2024-03-24T10:45:14.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.289+0000 7f87d8a3a640 1 --2- 172.21.15.5:0/153296255 >> [v2:172.21.15.57:6824/212605698,v1:172.21.15.57:6825/212605698] conn(0x7f87c0073a60 0x7f87c0075e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.291+0000 7f87d8a3a640 1 --2- 172.21.15.5:0/153296255 >> [v2:172.21.15.57:6824/212605698,v1:172.21.15.57:6825/212605698] conn(0x7f87c0073a60 0x7f87c0075e80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.292+0000 7f87d0ff9640 1 -- 172.21.15.5:0/153296255 <== osd.7 v2:172.21.15.57:6824/212605698 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f87c0076530 con 0x7f87c0073a60 2024-03-24T10:45:14.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.319+0000 7f5be8c7d640 1 Processor -- start 2024-03-24T10:45:14.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.319+0000 7f5be8c7d640 1 -- start start 2024-03-24T10:45:14.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.328+0000 7f56fb042640 1 Processor -- start 2024-03-24T10:45:14.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.328+0000 7f56fb042640 1 -- start start 2024-03-24T10:45:14.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.329+0000 7f56fb042640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f56f406fc70 0x7f56f415a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.329+0000 7f56fb042640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f56f40c7d60 0x7f56f4150460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.329+0000 7f56fb042640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56f41509a0 0x7f56f40c5360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.329+0000 7f56fb042640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f56f40c3700 con 0x7f56f406fc70 2024-03-24T10:45:14.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.329+0000 7f56fb042640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f56f415a860 con 0x7f56f40c7d60 2024-03-24T10:45:14.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.329+0000 7f56fb042640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f56f40c4c30 con 0x7f56f41509a0 2024-03-24T10:45:14.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.321+0000 7f5be8c7d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5bdc0d9be0 0x7f5bdc0e3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.321+0000 7f5be8c7d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5bdc04bc00 0x7f5bdc007c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.321+0000 7f5be8c7d640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bdc0d9fc0 con 0x7f5bdc046160 2024-03-24T10:45:14.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.321+0000 7f5be8c7d640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f5bdc04db90 con 0x7f5bdc04bc00 2024-03-24T10:45:14.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.321+0000 7f5be8c7d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f5bdc0e4100 con 0x7f5bdc0d9be0 2024-03-24T10:45:14.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.337+0000 7f56f8dde640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f56f406fc70 0x7f56f415a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.337+0000 7f56f8dde640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f56f406fc70 0x7f56f415a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60074/0 (socket says 172.21.15.5:60074) 2024-03-24T10:45:14.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.337+0000 7f56f8dde640 1 -- 172.21.15.5:0/1976185934 learned_addr learned my addr 172.21.15.5:0/1976185934 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:14.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.338+0000 7f6395e3b640 1 Processor -- start 2024-03-24T10:45:14.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.338+0000 7f6395e3b640 1 -- start start 2024-03-24T10:45:14.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.339+0000 7f6395e3b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f63880fc970 0x7f638804b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.339+0000 7f6395e3b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f638804b5c0 0x7f6388045210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.339+0000 7f6395e3b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63880075e0 con 0x7f63880f6d40 2024-03-24T10:45:14.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.339+0000 7f6395e3b640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f6388047150 con 0x7f638804b5c0 2024-03-24T10:45:14.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.339+0000 7f6395e3b640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f63880f7120 con 0x7f63880fc970 2024-03-24T10:45:14.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.336+0000 7f5be2ffd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f5bdc046160 0x7f5bdc046e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:35372/0 (socket says 172.21.15.5:35372) 2024-03-24T10:45:14.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.336+0000 7f5be2ffd640 1 -- 172.21.15.5:0/494323021 learned_addr learned my addr 172.21.15.5:0/494323021 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:14.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.337+0000 7f5be3fff640 1 --2- 172.21.15.5:0/494323021 >> v2:172.21.15.5:3300/0 conn(0x7f5bdc04bc00 0x7f5bdc007c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.340+0000 7f87ba7fc640 1 -- 172.21.15.5:0/153296255 --> [v2:172.21.15.57:6824/212605698,v1:172.21.15.57:6825/212605698] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f87ac002d30 con 0x7f87c0073a60 2024-03-24T10:45:14.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.341+0000 7f87d0ff9640 1 -- 172.21.15.5:0/153296255 <== osd.7 v2:172.21.15.57:6824/212605698 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f87ac002d30 con 0x7f87c0073a60 2024-03-24T10:45:14.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.342+0000 7f87ba7fc640 1 -- 172.21.15.5:0/153296255 >> [v2:172.21.15.57:6824/212605698,v1:172.21.15.57:6825/212605698] conn(0x7f87c0073a60 msgr2=0x7f87c0075e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.342+0000 7f87ba7fc640 1 --2- 172.21.15.5:0/153296255 >> [v2:172.21.15.57:6824/212605698,v1:172.21.15.57:6825/212605698] conn(0x7f87c0073a60 0x7f87c0075e80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.342+0000 7f87ba7fc640 1 -- 172.21.15.5:0/153296255 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f87c006b860 msgr2=0x7f87c006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.342+0000 7f87ba7fc640 1 --2- 172.21.15.5:0/153296255 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f87c006b860 0x7f87c006dd20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f87c4005880 tx=0x7f87c4009040 comp rx=0 tx=0).stop 2024-03-24T10:45:14.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.342+0000 7f87ba7fc640 1 -- 172.21.15.5:0/153296255 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f87d40ccb80 msgr2=0x7f87d415d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.342+0000 7f87ba7fc640 1 --2- 172.21.15.5:0/153296255 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f87d40ccb80 0x7f87d415d380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f87c800b3f0 tx=0x7f87c800b8b0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.342+0000 7f87d8a3a640 1 -- 172.21.15.5:0/153296255 reap_dead start 2024-03-24T10:45:14.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.342+0000 7f87ba7fc640 1 -- 172.21.15.5:0/153296255 shutdown_connections 2024-03-24T10:45:14.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.342+0000 7f87ba7fc640 1 -- 172.21.15.5:0/153296255 >> 172.21.15.5:0/153296255 conn(0x7f87d40cfc30 msgr2=0x7f87d40c7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:14.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.342+0000 7f87ba7fc640 1 -- 172.21.15.5:0/153296255 shutdown_connections 2024-03-24T10:45:14.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.342+0000 7f87ba7fc640 1 -- 172.21.15.5:0/153296255 wait complete. 2024-03-24T10:45:14.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.346+0000 7f639563a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f638804b5c0 0x7f6388045210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.347+0000 7f639563a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f638804b5c0 0x7f6388045210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60088/0 (socket says 172.21.15.5:60088) 2024-03-24T10:45:14.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.347+0000 7f639563a640 1 -- 172.21.15.5:0/322964483 learned_addr learned my addr 172.21.15.5:0/322964483 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:14.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.350+0000 7f56f8dde640 1 -- 172.21.15.5:0/1976185934 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56f41509a0 msgr2=0x7f56f40c5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:14.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.350+0000 7f56f8dde640 1 --2- 172.21.15.5:0/1976185934 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56f41509a0 0x7f56f40c5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.350+0000 7f56f8dde640 1 -- 172.21.15.5:0/1976185934 >> v2:172.21.15.57:3300/0 conn(0x7f56f40c7d60 msgr2=0x7f56f4150460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:14.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.350+0000 7f56f8dde640 1 --2- 172.21.15.5:0/1976185934 >> v2:172.21.15.57:3300/0 conn(0x7f56f40c7d60 0x7f56f4150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.350+0000 7f56f8dde640 1 -- 172.21.15.5:0/1976185934 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f40c4f10 con 0x7f56f406fc70 2024-03-24T10:45:14.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.352+0000 7f5be3fff640 1 -- 172.21.15.5:0/494323021 >> v2:172.21.15.57:3300/0 conn(0x7f5bdc0d9be0 msgr2=0x7f5bdc0e3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:14.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.352+0000 7f5be3fff640 1 --2- 172.21.15.5:0/494323021 >> v2:172.21.15.57:3300/0 conn(0x7f5bdc0d9be0 0x7f5bdc0e3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.352+0000 7f5be3fff640 1 -- 172.21.15.5:0/494323021 >> v1:172.21.15.5:6789/0 conn(0x7f5bdc046160 legacy=0x7f5bdc046e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.353+0000 7f5be3fff640 1 -- 172.21.15.5:0/494323021 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bdc006c80 con 0x7f5bdc04bc00 2024-03-24T10:45:14.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.355+0000 7f638ffff640 1 --2- 172.21.15.5:0/322964483 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f63880fc970 0x7f638804b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.355+0000 7f56f8dde640 1 --2- 172.21.15.5:0/1976185934 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56f406fc70 0x7f56f415a320 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f56f40c3970 tx=0x7f56e402b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a41ed1e93a2ac1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.361 INFO:teuthology.orchestra.run.smithi005.stdout:60129542154 2024-03-24T10:45:14.361 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-24T10:45:14.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.360+0000 7f638ffff640 1 -- 172.21.15.5:0/322964483 >> v2:172.21.15.5:3300/0 conn(0x7f638804b5c0 msgr2=0x7f6388045210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:14.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.360+0000 7f638ffff640 1 --2- 172.21.15.5:0/322964483 >> v2:172.21.15.5:3300/0 conn(0x7f638804b5c0 0x7f6388045210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.361+0000 7f638ffff640 1 -- 172.21.15.5:0/322964483 >> v1:172.21.15.57:6789/0 conn(0x7f63880f6d40 legacy=0x7f6388005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:14.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.361+0000 7f638ffff640 1 -- 172.21.15.5:0/322964483 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63880066f0 con 0x7f63880fc970 2024-03-24T10:45:14.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.363+0000 7f638ffff640 1 --2- 172.21.15.5:0/322964483 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f63880fc970 0x7f638804b080 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f63840097a0 tx=0x7f638402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=958fc9c5cfd30a31 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.366+0000 7f638f7fe640 1 -- 172.21.15.5:0/322964483 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638402d040 con 0x7f63880fc970 2024-03-24T10:45:14.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.368+0000 7f638f7fe640 1 -- 172.21.15.5:0/322964483 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6384031810 con 0x7f63880fc970 2024-03-24T10:45:14.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.368+0000 7f638f7fe640 1 -- 172.21.15.5:0/322964483 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6384031ba0 con 0x7f63880fc970 2024-03-24T10:45:14.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.369+0000 7f6395e3b640 1 -- 172.21.15.5:0/322964483 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f63880fc970 msgr2=0x7f638804b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.369+0000 7f6395e3b640 1 --2- 172.21.15.5:0/322964483 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f63880fc970 0x7f638804b080 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f63840097a0 tx=0x7f638402b8b0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.369+0000 7f6395e3b640 1 -- 172.21.15.5:0/322964483 shutdown_connections 2024-03-24T10:45:14.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.369+0000 7f6395e3b640 1 --2- 172.21.15.5:0/322964483 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f63880fc970 0x7f638804b080 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.369+0000 7f6395e3b640 1 --2- 172.21.15.5:0/322964483 >> v2:172.21.15.5:3300/0 conn(0x7f638804b5c0 0x7f6388045210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.369+0000 7f6395e3b640 1 -- 172.21.15.5:0/322964483 >> 172.21.15.5:0/322964483 conn(0x7f6388050170 msgr2=0x7f6388050580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:14.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.369+0000 7f6395e3b640 1 -- 172.21.15.5:0/322964483 shutdown_connections 2024-03-24T10:45:14.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.369+0000 7f6395e3b640 1 -- 172.21.15.5:0/322964483 wait complete. 2024-03-24T10:45:14.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.371+0000 7f6395e3b640 1 Processor -- start 2024-03-24T10:45:14.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.371+0000 7f6395e3b640 1 -- start start 2024-03-24T10:45:14.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.372+0000 7f6395e3b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f638804b5c0 0x7f63880f26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.372+0000 7f6395e3b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f63880f6d40 0x7f63880ec2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.372+0000 7f6395e3b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63880fc970 0x7f63880f5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.372+0000 7f6395e3b640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f63880f5440 con 0x7f638804b5c0 2024-03-24T10:45:14.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.373+0000 7f6395e3b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f63880f4b30 con 0x7f63880fc970 2024-03-24T10:45:14.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.373+0000 7f6395e3b640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f63880f4df0 con 0x7f63880f6d40 2024-03-24T10:45:14.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.375+0000 7f638ffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f63880f6d40 0x7f63880ec2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.375+0000 7f638ffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f63880f6d40 0x7f63880ec2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:51452/0 (socket says 172.21.15.5:51452) 2024-03-24T10:45:14.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.375+0000 7f638ffff640 1 -- 172.21.15.5:0/3128326878 learned_addr learned my addr 172.21.15.5:0/3128326878 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:14.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.380+0000 7f638ffff640 1 -- 172.21.15.5:0/3128326878 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63880fc970 msgr2=0x7f63880f5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.380+0000 7f638ffff640 1 --2- 172.21.15.5:0/3128326878 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63880fc970 0x7f63880f5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.380+0000 7f638ffff640 1 -- 172.21.15.5:0/3128326878 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f638804b5c0 msgr2=0x7f63880f26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.380+0000 7f638ffff640 1 --2- 172.21.15.5:0/3128326878 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f638804b5c0 0x7f63880f26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.380+0000 7f638ffff640 1 -- 172.21.15.5:0/3128326878 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6384009400 con 0x7f63880f6d40 2024-03-24T10:45:14.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.383+0000 7f638ffff640 1 --2- 172.21.15.5:0/3128326878 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f63880f6d40 0x7f63880ec2c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f63840098c0 tx=0x7f63840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.388+0000 7f638d7fa640 1 -- 172.21.15.5:0/3128326878 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6384006e10 con 0x7f63880f6d40 2024-03-24T10:45:14.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.388+0000 7f638d7fa640 1 -- 172.21.15.5:0/3128326878 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f638403ad60 con 0x7f63880f6d40 2024-03-24T10:45:14.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.388+0000 7f638d7fa640 1 -- 172.21.15.5:0/3128326878 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6384044610 con 0x7f63880f6d40 2024-03-24T10:45:14.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.388+0000 7f6395e3b640 1 -- 172.21.15.5:0/3128326878 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f638818b580 con 0x7f63880f6d40 2024-03-24T10:45:14.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.388+0000 7f6395e3b640 1 -- 172.21.15.5:0/3128326878 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f638818bb20 con 0x7f63880f6d40 2024-03-24T10:45:14.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.390+0000 7f6395e3b640 1 -- 172.21.15.5:0/3128326878 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6370005290 con 0x7f63880f6d40 2024-03-24T10:45:14.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.415+0000 7f5be3fff640 1 --2- 172.21.15.5:0/494323021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bdc04bc00 0x7f5bdc007c90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc00ea10 tx=0x7f5bd002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ddc8a8ffcaa2b18 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.420+0000 7faa8cc6f640 1 Processor -- start 2024-03-24T10:45:14.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.421+0000 7faa8cc6f640 1 -- start start 2024-03-24T10:45:14.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.422+0000 7faa8cc6f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7faa880c8a70 0x7faa880ce090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.422+0000 7faa8cc6f640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faa880cafd0 0x7faa88151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.422+0000 7faa8cc6f640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa88159200 con 0x7faa880c8e50 2024-03-24T10:45:14.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.422+0000 7faa8cc6f640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7faa880cddf0 con 0x7faa880cafd0 2024-03-24T10:45:14.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.423+0000 7faa8659c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7faa880c8a70 0x7faa880ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.423+0000 7faa8cc6f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7faa880cb480 con 0x7faa880c8a70 2024-03-24T10:45:14.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.423+0000 7faa8659c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7faa880c8a70 0x7faa880ce090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36578/0 (socket says 172.21.15.5:36578) 2024-03-24T10:45:14.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.423+0000 7faa8659c640 1 -- 172.21.15.5:0/2361843353 learned_addr learned my addr 172.21.15.5:0/2361843353 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:14.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.424+0000 7f638d7fa640 1 -- 172.21.15.5:0/3128326878 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6384044770 con 0x7f63880f6d40 2024-03-24T10:45:14.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.425+0000 7f638d7fa640 1 --2- 172.21.15.5:0/3128326878 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f637406b860 0x7f637406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.425+0000 7f6394e39640 1 --2- 172.21.15.5:0/3128326878 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f637406b860 0x7f637406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.426+0000 7faa86d9d640 1 --2- 172.21.15.5:0/2361843353 >> v2:172.21.15.5:3300/0 conn(0x7faa880cafd0 0x7faa88151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.426+0000 7f6394e39640 1 --2- 172.21.15.5:0/3128326878 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f637406b860 0x7f637406dd20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f637c001560 tx=0x7f637c007990 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.426+0000 7faa8659c640 1 -- 172.21.15.5:0/2361843353 >> v2:172.21.15.5:3300/0 conn(0x7faa880cafd0 msgr2=0x7faa88151fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:14.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.426+0000 7faa8659c640 1 --2- 172.21.15.5:0/2361843353 >> v2:172.21.15.5:3300/0 conn(0x7faa880cafd0 0x7faa88151fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.426+0000 7faa8659c640 1 -- 172.21.15.5:0/2361843353 >> v1:172.21.15.5:6789/0 conn(0x7faa880c8e50 legacy=0x7faa8815bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:14.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.426+0000 7faa8659c640 1 -- 172.21.15.5:0/2361843353 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa88158680 con 0x7faa880c8a70 2024-03-24T10:45:14.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.429+0000 7faa8659c640 1 --2- 172.21.15.5:0/2361843353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa880c8a70 0x7faa880ce090 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7faa880ce950 tx=0x7faa7802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=173607775b70a881 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.437+0000 7faa85d9b640 1 -- 172.21.15.5:0/2361843353 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa78031870 con 0x7faa880c8a70 2024-03-24T10:45:14.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.437+0000 7faa85d9b640 1 -- 172.21.15.5:0/2361843353 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa780319d0 con 0x7faa880c8a70 2024-03-24T10:45:14.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.437+0000 7faa85d9b640 1 -- 172.21.15.5:0/2361843353 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa78031d60 con 0x7faa880c8a70 2024-03-24T10:45:14.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.438+0000 7f2ae39f4640 1 Processor -- start 2024-03-24T10:45:14.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.438+0000 7faa8cc6f640 1 -- 172.21.15.5:0/2361843353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa880c8a70 msgr2=0x7faa880ce090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.438+0000 7faa8cc6f640 1 --2- 172.21.15.5:0/2361843353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa880c8a70 0x7faa880ce090 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7faa880ce950 tx=0x7faa7802b910 comp rx=0 tx=0).stop 2024-03-24T10:45:14.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.438+0000 7f2ae39f4640 1 -- start start 2024-03-24T10:45:14.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.438+0000 7faa8cc6f640 1 -- 172.21.15.5:0/2361843353 shutdown_connections 2024-03-24T10:45:14.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.438+0000 7faa8cc6f640 1 --2- 172.21.15.5:0/2361843353 >> v2:172.21.15.5:3300/0 conn(0x7faa880cafd0 0x7faa88151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.438+0000 7faa8cc6f640 1 --2- 172.21.15.5:0/2361843353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa880c8a70 0x7faa880ce090 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.438+0000 7faa8cc6f640 1 -- 172.21.15.5:0/2361843353 >> 172.21.15.5:0/2361843353 conn(0x7faa8806fa80 msgr2=0x7faa88061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:14.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.438+0000 7faa8cc6f640 1 -- 172.21.15.5:0/2361843353 shutdown_connections 2024-03-24T10:45:14.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.438+0000 7faa8cc6f640 1 -- 172.21.15.5:0/2361843353 wait complete. 2024-03-24T10:45:14.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.439+0000 7f2ae39f4640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2adc0c66b0 0x7f2adc151e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.439+0000 7f2ae39f4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2adc158140 con 0x7f2adc0c62d0 2024-03-24T10:45:14.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.439+0000 7f2ae39f4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2adc06e810 con 0x7f2adc06e390 2024-03-24T10:45:14.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.439+0000 7f2ae39f4640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f2adc15b730 con 0x7f2adc0c66b0 2024-03-24T10:45:14.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.441+0000 7faa8cc6f640 1 Processor -- start 2024-03-24T10:45:14.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.441+0000 7faa8cc6f640 1 -- start start 2024-03-24T10:45:14.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.441+0000 7f2ae0f8f640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f2adc06e390 0x7f2adc15be30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:47074/0 (socket says 172.21.15.5:47074) 2024-03-24T10:45:14.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.441+0000 7f2ae0f8f640 1 -- 172.21.15.5:0/35776567 learned_addr learned my addr 172.21.15.5:0/35776567 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.442+0000 7faa8cc6f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faa880c8a70 0x7faa881534f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.442+0000 7faa8cc6f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa880c8e50 0x7faa8814e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.443+0000 7faa8cc6f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa880cafd0 0x7faa881e1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.443+0000 7faa8cc6f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7faa88155a10 con 0x7faa880c8e50 2024-03-24T10:45:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.443+0000 7faa8cc6f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7faa881e1e00 con 0x7faa880cafd0 2024-03-24T10:45:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.443+0000 7faa8cc6f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7faa881550e0 con 0x7faa880c8a70 2024-03-24T10:45:14.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.445+0000 7fa50195e640 1 Processor -- start 2024-03-24T10:45:14.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.445+0000 7fa50195e640 1 -- start start 2024-03-24T10:45:14.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.445+0000 7fabebfff640 1 -- 172.21.15.5:0/2604692335 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe4031870 con 0x7fabec04b8d0 2024-03-24T10:45:14.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.445+0000 7faa86d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa880cafd0 0x7faa881e1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.445+0000 7fabebfff640 1 -- 172.21.15.5:0/2604692335 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabe40319d0 con 0x7fabec04b8d0 2024-03-24T10:45:14.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.445+0000 7faa86d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa880cafd0 0x7faa881e1530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36584/0 (socket says 172.21.15.5:36584) 2024-03-24T10:45:14.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.445+0000 7faa86d9d640 1 -- 172.21.15.5:0/4121801674 learned_addr learned my addr 172.21.15.5:0/4121801674 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:14.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.445+0000 7f2ad3fff640 1 -- 172.21.15.5:0/35776567 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3516186344 0 0) 0x7f2adc06e810 con 0x7f2adc06e390 2024-03-24T10:45:14.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.446+0000 7f2ad3fff640 1 -- 172.21.15.5:0/35776567 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ac8003520 con 0x7f2adc06e390 2024-03-24T10:45:14.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.446+0000 7fabebfff640 1 -- 172.21.15.5:0/2604692335 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe4031da0 con 0x7fabec04b8d0 2024-03-24T10:45:14.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.446+0000 7fa50195e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa4fc0c7600 0x7fa4fc0c3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.446+0000 7fabfa6b1640 1 -- 172.21.15.5:0/2604692335 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabec04b8d0 msgr2=0x7fabec005ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.446+0000 7fabfa6b1640 1 --2- 172.21.15.5:0/2604692335 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabec04b8d0 0x7fabec005ed0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fabec04aef0 tx=0x7fabe402b910 comp rx=0 tx=0).stop 2024-03-24T10:45:14.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.446+0000 7fa50195e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4fc0726b0 con 0x7fa4fc0c1b60 2024-03-24T10:45:14.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.446+0000 7fa50195e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4fc1572b0 con 0x7fa4fc14d950 2024-03-24T10:45:14.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.446+0000 7fa50195e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fa4fc157e50 con 0x7fa4fc0c7600 2024-03-24T10:45:14.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.446+0000 7fabfa6b1640 1 -- 172.21.15.5:0/2604692335 shutdown_connections 2024-03-24T10:45:14.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.446+0000 7fabfa6b1640 1 --2- 172.21.15.5:0/2604692335 >> v2:172.21.15.57:3300/0 conn(0x7fabec006410 0x7fabec034a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.446+0000 7fabfa6b1640 1 --2- 172.21.15.5:0/2604692335 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabec04b8d0 0x7fabec005ed0 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fabec04aef0 tx=0x7fabe402b910 comp rx=0 tx=0).stop 2024-03-24T10:45:14.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.446+0000 7fabfa6b1640 1 --2- 172.21.15.5:0/2604692335 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fabec04b4f0 0x7fabec0dac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.446+0000 7fabfa6b1640 1 -- 172.21.15.5:0/2604692335 >> 172.21.15.5:0/2604692335 conn(0x7fabec050170 msgr2=0x7fabec050580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:14.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.446+0000 7f56f37fe640 1 -- 172.21.15.5:0/1976185934 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e4030890 con 0x7f56f406fc70 2024-03-24T10:45:14.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.446+0000 7f56f37fe640 1 -- 172.21.15.5:0/1976185934 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56e40309f0 con 0x7f56f406fc70 2024-03-24T10:45:14.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.446+0000 7fabfa6b1640 1 -- 172.21.15.5:0/2604692335 shutdown_connections 2024-03-24T10:45:14.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.447+0000 7f56f37fe640 1 -- 172.21.15.5:0/1976185934 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e4030dc0 con 0x7f56f406fc70 2024-03-24T10:45:14.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.446+0000 7fabfa6b1640 1 -- 172.21.15.5:0/2604692335 wait complete. 2024-03-24T10:45:14.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.447+0000 7f56fb042640 1 -- 172.21.15.5:0/1976185934 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56f406fc70 msgr2=0x7f56f415a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.447+0000 7f56fb042640 1 --2- 172.21.15.5:0/1976185934 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56f406fc70 0x7f56f415a320 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f56f40c3970 tx=0x7f56e402b9d0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.447+0000 7f56fb042640 1 -- 172.21.15.5:0/1976185934 shutdown_connections 2024-03-24T10:45:14.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.447+0000 7f56fb042640 1 --2- 172.21.15.5:0/1976185934 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56f41509a0 0x7f56f40c5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.447+0000 7f56fb042640 1 --2- 172.21.15.5:0/1976185934 >> v2:172.21.15.57:3300/0 conn(0x7f56f40c7d60 0x7f56f4150460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.447+0000 7f56fb042640 1 --2- 172.21.15.5:0/1976185934 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56f406fc70 0x7f56f415a320 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.447+0000 7f56fb042640 1 -- 172.21.15.5:0/1976185934 >> 172.21.15.5:0/1976185934 conn(0x7f56f406f180 msgr2=0x7f56f406f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:14.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.447+0000 7f56fb042640 1 -- 172.21.15.5:0/1976185934 shutdown_connections 2024-03-24T10:45:14.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.447+0000 7f56fb042640 1 -- 172.21.15.5:0/1976185934 wait complete. 2024-03-24T10:45:14.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.448+0000 7fabfa6b1640 1 Processor -- start 2024-03-24T10:45:14.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.448+0000 7fabfa6b1640 1 -- start start 2024-03-24T10:45:14.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.449+0000 7f56fb042640 1 Processor -- start 2024-03-24T10:45:14.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.449+0000 7fabfa6b1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabec006410 0x7fabec0cad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.449+0000 7fabfa6b1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fabec04b4f0 0x7fabec176cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.450+0000 7fabfa6b1640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabec0c2450 0x7fabec0cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.450+0000 7fabfa6b1640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fabec0cb620 con 0x7fabec0c2450 2024-03-24T10:45:14.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.450+0000 7fabfa6b1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fabec172d30 con 0x7fabec006410 2024-03-24T10:45:14.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.450+0000 7fabfa6b1640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fabec173d00 con 0x7fabec04b4f0 2024-03-24T10:45:14.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.449+0000 7f56fb042640 1 -- start start 2024-03-24T10:45:14.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.450+0000 7f56fb042640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56f406fc70 0x7f56f414f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.450+0000 7f56fb042640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56f40c7d60 0x7f56f412bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.450+0000 7f56fb042640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56f41509a0 0x7f56f414ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.450+0000 7f56fb042640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f56f412e170 con 0x7f56f406fc70 2024-03-24T10:45:14.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.450+0000 7f56fb042640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f56f4153db0 con 0x7f56f41509a0 2024-03-24T10:45:14.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.450+0000 7f56fb042640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f56f412e6e0 con 0x7f56f40c7d60 2024-03-24T10:45:14.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.451+0000 7f2ae1f91640 1 --2- 172.21.15.5:0/35776567 >> v2:172.21.15.5:3300/0 conn(0x7f2adc0c66b0 0x7f2adc151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.452+0000 7faa7ffff640 1 --2- 172.21.15.5:0/4121801674 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa880c8e50 0x7faa8814e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.454+0000 7faa86d9d640 1 -- 172.21.15.5:0/4121801674 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faa880c8a70 msgr2=0x7faa881534f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.454+0000 7faa86d9d640 1 --2- 172.21.15.5:0/4121801674 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faa880c8a70 0x7faa881534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.454+0000 7faa86d9d640 1 -- 172.21.15.5:0/4121801674 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa880c8e50 msgr2=0x7faa8814e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.454+0000 7faa86d9d640 1 --2- 172.21.15.5:0/4121801674 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa880c8e50 0x7faa8814e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.454+0000 7faa86d9d640 1 -- 172.21.15.5:0/4121801674 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa780094d0 con 0x7faa880cafd0 2024-03-24T10:45:14.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.458+0000 7fa4faffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa4fc14d950 0x7fa4fc157740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:47090/0 (socket says 172.21.15.5:47090) 2024-03-24T10:45:14.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.458+0000 7fa4faffd640 1 -- 172.21.15.5:0/3508667336 learned_addr learned my addr 172.21.15.5:0/3508667336 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:14.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.460+0000 7f2ad3fff640 1 -- 172.21.15.5:0/35776567 <== mon.1 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1313059776 0 0) 0x7f2ac8003520 con 0x7f2adc06e390 2024-03-24T10:45:14.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.460+0000 7f2ad3fff640 1 -- 172.21.15.5:0/35776567 >> v2:172.21.15.5:3300/0 conn(0x7f2adc0c66b0 msgr2=0x7f2adc151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:14.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.460+0000 7f2ad3fff640 1 --2- 172.21.15.5:0/35776567 >> v2:172.21.15.5:3300/0 conn(0x7f2adc0c66b0 0x7f2adc151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.460+0000 7f2ad3fff640 1 -- 172.21.15.5:0/35776567 >> v1:172.21.15.5:6789/0 conn(0x7f2adc0c62d0 legacy=0x7f2adc06b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:14.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.460+0000 7f2ad3fff640 1 -- 172.21.15.5:0/35776567 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2adc06eaf0 con 0x7f2adc06e390 2024-03-24T10:45:14.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.460+0000 7f2ad3fff640 1 -- 172.21.15.5:0/35776567 <== mon.1 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 100681612 0 0) 0x7f2acc001670 con 0x7f2adc06e390 2024-03-24T10:45:14.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.460+0000 7f2ad3fff640 1 -- 172.21.15.5:0/35776567 >> v1:172.21.15.57:6789/0 conn(0x7f2adc06e390 legacy=0x7f2adc15be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.460+0000 7f2ad3fff640 1 --2- 172.21.15.5:0/35776567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac8003970 0x7f2ac8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.460+0000 7f2ad3fff640 1 --2- 172.21.15.5:0/35776567 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2ac80047c0 0x7f2ac8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.460+0000 7f2ad3fff640 1 --2- 172.21.15.5:0/35776567 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ac8005180 0x7f2ac800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.460+0000 7f2ad3fff640 1 -- 172.21.15.5:0/35776567 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc15b730 con 0x7f2ac8005180 2024-03-24T10:45:14.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.460+0000 7f2ad3fff640 1 -- 172.21.15.5:0/35776567 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac800c030 con 0x7f2ac8003970 2024-03-24T10:45:14.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.460+0000 7f2ad3fff640 1 -- 172.21.15.5:0/35776567 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f2ac800c1f0 con 0x7f2ac80047c0 2024-03-24T10:45:14.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.461+0000 7faa86d9d640 1 --2- 172.21.15.5:0/4121801674 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa880cafd0 0x7faa881e1530 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faa8000ccb0 tx=0x7faa8000a5c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.464+0000 7f56f8dde640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56f406fc70 0x7f56f414f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.464+0000 7fa4fa7fc640 1 --2- 172.21.15.5:0/3508667336 >> v2:172.21.15.5:3300/0 conn(0x7fa4fc0c7600 0x7fa4fc0c3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.465+0000 7f5be27fc640 1 -- 172.21.15.5:0/494323021 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0030840 con 0x7f5bdc04bc00 2024-03-24T10:45:14.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.465+0000 7f5be27fc640 1 -- 172.21.15.5:0/494323021 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bd00309a0 con 0x7f5bdc04bc00 2024-03-24T10:45:14.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.465+0000 7fabf9eb0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabec0c2450 0x7fabec0cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.466+0000 7f5be27fc640 1 -- 172.21.15.5:0/494323021 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0030d70 con 0x7f5bdc04bc00 2024-03-24T10:45:14.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.466+0000 7f5be8c7d640 1 -- 172.21.15.5:0/494323021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bdc04bc00 msgr2=0x7f5bdc007c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.466+0000 7f5be8c7d640 1 --2- 172.21.15.5:0/494323021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bdc04bc00 0x7f5bdc007c90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc00ea10 tx=0x7f5bd002b980 comp rx=0 tx=0).stop 2024-03-24T10:45:14.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.466+0000 7f5be8c7d640 1 -- 172.21.15.5:0/494323021 shutdown_connections 2024-03-24T10:45:14.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.466+0000 7f5be8c7d640 1 --2- 172.21.15.5:0/494323021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bdc04bc00 0x7f5bdc007c90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.466+0000 7f5be8c7d640 1 --2- 172.21.15.5:0/494323021 >> v2:172.21.15.57:3300/0 conn(0x7f5bdc0d9be0 0x7f5bdc0e3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.466+0000 7f5be8c7d640 1 -- 172.21.15.5:0/494323021 >> 172.21.15.5:0/494323021 conn(0x7f5bdc051280 msgr2=0x7f5bdc051690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:14.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.466+0000 7f5be8c7d640 1 -- 172.21.15.5:0/494323021 shutdown_connections 2024-03-24T10:45:14.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.466+0000 7f5be8c7d640 1 -- 172.21.15.5:0/494323021 wait complete. 2024-03-24T10:45:14.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.467+0000 7fabf9eb0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabec0c2450 0x7fabec0cb8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60148/0 (socket says 172.21.15.5:60148) 2024-03-24T10:45:14.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.467+0000 7fabf9eb0640 1 -- 172.21.15.5:0/2507962584 learned_addr learned my addr 172.21.15.5:0/2507962584 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:14.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.467+0000 7faa7effd640 1 -- 172.21.15.5:0/4121801674 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8000ce60 con 0x7faa880cafd0 2024-03-24T10:45:14.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.467+0000 7faa8cc6f640 1 -- 172.21.15.5:0/4121801674 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa881553c0 con 0x7faa880cafd0 2024-03-24T10:45:14.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.467+0000 7faa8cc6f640 1 -- 172.21.15.5:0/4121801674 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa88154de0 con 0x7faa880cafd0 2024-03-24T10:45:14.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.467+0000 7f56f8dde640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56f406fc70 0x7f56f414f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60156/0 (socket says 172.21.15.5:60156) 2024-03-24T10:45:14.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.467+0000 7f56f8dde640 1 -- 172.21.15.5:0/1763748615 learned_addr learned my addr 172.21.15.5:0/1763748615 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:14.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.468+0000 7f5be8c7d640 1 Processor -- start 2024-03-24T10:45:14.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.468+0000 7f5be8c7d640 1 -- start start 2024-03-24T10:45:14.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.468+0000 7fa4f9ffb640 1 -- 172.21.15.5:0/3508667336 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3459807796 0 0) 0x7fa4fc1572b0 con 0x7fa4fc14d950 2024-03-24T10:45:14.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.468+0000 7fa4f9ffb640 1 -- 172.21.15.5:0/3508667336 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4e4003520 con 0x7fa4fc14d950 2024-03-24T10:45:14.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.469+0000 7faa8cc6f640 1 -- 172.21.15.5:0/4121801674 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa880688e0 con 0x7faa880cafd0 2024-03-24T10:45:14.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.469+0000 7f5be8c7d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5bdc046160 0x7f5bdc0494a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.469+0000 7f5be8c7d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bdc04bc00 0x7f5bdc0dbe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.469+0000 7f5be8c7d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5bdc0d9be0 0x7f5bdc0da840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.469+0000 7f5be8c7d640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc049080 con 0x7f5bdc04bc00 2024-03-24T10:45:14.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.469+0000 7f5be8c7d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc0d0fe0 con 0x7f5bdc046160 2024-03-24T10:45:14.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.469+0000 7f5be8c7d640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f5bdc048700 con 0x7f5bdc0d9be0 2024-03-24T10:45:14.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.472+0000 7f2ae1790640 1 --2- 172.21.15.5:0/35776567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac8003970 0x7f2ac8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.471+0000 7faa7effd640 1 -- 172.21.15.5:0/4121801674 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa80007820 con 0x7faa880cafd0 2024-03-24T10:45:14.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.471+0000 7faa7effd640 1 -- 172.21.15.5:0/4121801674 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa80003bb0 con 0x7faa880cafd0 2024-03-24T10:45:14.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.473+0000 7f56f8dde640 1 -- 172.21.15.5:0/1763748615 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56f40c7d60 msgr2=0x7f56f412bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.473+0000 7f56f8dde640 1 --2- 172.21.15.5:0/1763748615 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56f40c7d60 0x7f56f412bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.473+0000 7f56f8dde640 1 -- 172.21.15.5:0/1763748615 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56f41509a0 msgr2=0x7f56f414ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.473+0000 7f56f8dde640 1 --2- 172.21.15.5:0/1763748615 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56f41509a0 0x7f56f414ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.473+0000 7f56f8dde640 1 -- 172.21.15.5:0/1763748615 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56e40094d0 con 0x7f56f406fc70 2024-03-24T10:45:14.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.476+0000 7fa4fa7fc640 1 -- 172.21.15.5:0/3508667336 >> v1:172.21.15.57:6789/0 conn(0x7fa4fc14d950 legacy=0x7fa4fc157740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.476+0000 7fa4fa7fc640 1 -- 172.21.15.5:0/3508667336 >> v1:172.21.15.5:6789/0 conn(0x7fa4fc0c1b60 legacy=0x7fa4fc161920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:14.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.476+0000 7fa4fa7fc640 1 -- 172.21.15.5:0/3508667336 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4fc156fa0 con 0x7fa4fc0c7600 2024-03-24T10:45:14.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.478+0000 7fabf96af640 1 --2- 172.21.15.5:0/2507962584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabec006410 0x7fabec0cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.480+0000 7fabf9eb0640 1 -- 172.21.15.5:0/2507962584 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fabec04b4f0 msgr2=0x7fabec176cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.480+0000 7fabf9eb0640 1 --2- 172.21.15.5:0/2507962584 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fabec04b4f0 0x7fabec176cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.480+0000 7fabf9eb0640 1 -- 172.21.15.5:0/2507962584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabec006410 msgr2=0x7fabec0cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.480+0000 7fabf9eb0640 1 --2- 172.21.15.5:0/2507962584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabec006410 0x7fabec0cad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.480+0000 7fabf9eb0640 1 -- 172.21.15.5:0/2507962584 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe4009400 con 0x7fabec0c2450 2024-03-24T10:45:14.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.485+0000 7fa4fa7fc640 1 --2- 172.21.15.5:0/3508667336 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4fc0c7600 0x7fa4fc0c3470 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa4f00097a0 tx=0x7fa4f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72494809090d2b13 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.485+0000 7fabf9eb0640 1 --2- 172.21.15.5:0/2507962584 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabec0c2450 0x7fabec0cb8a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fabf000e840 tx=0x7fabf000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.486+0000 7f56f8dde640 1 --2- 172.21.15.5:0/1763748615 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56f406fc70 0x7f56f414f920 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f56e4009960 tx=0x7f56e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.497+0000 7f56f17fa640 1 -- 172.21.15.5:0/1763748615 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e4039b20 con 0x7f56f406fc70 2024-03-24T10:45:14.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.497+0000 7f56f17fa640 1 -- 172.21.15.5:0/1763748615 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56e4039ca0 con 0x7f56f406fc70 2024-03-24T10:45:14.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.497+0000 7f56f17fa640 1 -- 172.21.15.5:0/1763748615 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e4043630 con 0x7f56f406fc70 2024-03-24T10:45:14.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.497+0000 7f56fb042640 1 -- 172.21.15.5:0/1763748615 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56f412e9c0 con 0x7f56f406fc70 2024-03-24T10:45:14.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.497+0000 7f56fb042640 1 -- 172.21.15.5:0/1763748615 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56f4153a40 con 0x7f56f406fc70 2024-03-24T10:45:14.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.498+0000 7f56fb042640 1 -- 172.21.15.5:0/1763748615 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f41535c0 con 0x7f56f406fc70 2024-03-24T10:45:14.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.503+0000 7f2b68be2640 1 Processor -- start 2024-03-24T10:45:14.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.503+0000 7f2b68be2640 1 -- start start 2024-03-24T10:45:14.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.504+0000 7f2b68be2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2b640cc000 0x7f2b6415e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.504+0000 7f2b68be2640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b640cc3e0 con 0x7f2b6406bf50 2024-03-24T10:45:14.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.504+0000 7f2b68be2640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b6415a5f0 con 0x7f2b640cf340 2024-03-24T10:45:14.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.504+0000 7f2b68be2640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f2b6415a400 con 0x7f2b640cc000 2024-03-24T10:45:14.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.682+0000 7f638d7fa640 1 -- 172.21.15.5:0/3128326878 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f638407bc10 con 0x7f63880f6d40 2024-03-24T10:45:14.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.694+0000 7f638d7fa640 1 -- 172.21.15.5:0/3128326878 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f63840777b0 con 0x7f63880f6d40 2024-03-24T10:45:14.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.702+0000 7f2ae0f8f640 1 --2- 172.21.15.5:0/35776567 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2ac80047c0 0x7f2ac8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.705+0000 7f2b6259c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2b640cc000 0x7f2b6415e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.705+0000 7f2b6259c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2b640cc000 0x7f2b6415e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:51516/0 (socket says 172.21.15.5:51516) 2024-03-24T10:45:14.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.705+0000 7f2b6259c640 1 -- 172.21.15.5:0/3744045928 learned_addr learned my addr 172.21.15.5:0/3744045928 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:14.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.706+0000 7fa4f9ffb640 1 -- 172.21.15.5:0/3508667336 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0031830 con 0x7fa4fc0c7600 2024-03-24T10:45:14.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.708+0000 7f2b6259c640 1 -- 172.21.15.5:0/3744045928 >> v1:172.21.15.57:6789/0 conn(0x7f2b640cf340 legacy=0x7f2b640cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:14.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.708+0000 7f2b6259c640 1 -- 172.21.15.5:0/3744045928 >> v1:172.21.15.5:6789/0 conn(0x7f2b6406bf50 legacy=0x7f2b6406c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:14.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.708+0000 7f2b6259c640 1 -- 172.21.15.5:0/3744045928 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b6415a8b0 con 0x7f2b640cc000 2024-03-24T10:45:14.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.711+0000 7f2ae0f8f640 1 -- 172.21.15.5:0/35776567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac8003970 msgr2=0x7f2ac8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:14.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.711+0000 7f2ae0f8f640 1 --2- 172.21.15.5:0/35776567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2ac8003970 0x7f2ac8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.711+0000 7f2ae0f8f640 1 -- 172.21.15.5:0/35776567 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ac8005180 msgr2=0x7f2ac800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:14.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.711+0000 7f2ae0f8f640 1 --2- 172.21.15.5:0/35776567 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2ac8005180 0x7f2ac800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.711+0000 7f2ae0f8f640 1 -- 172.21.15.5:0/35776567 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2adc06eaf0 con 0x7f2ac80047c0 2024-03-24T10:45:14.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.711+0000 7f2ae1f91640 1 -- 172.21.15.5:0/35776567 reap_dead start 2024-03-24T10:45:14.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.717+0000 7f2ae0f8f640 1 --2- 172.21.15.5:0/35776567 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2ac80047c0 0x7f2ac8004c40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00c210 tx=0x7f2ac80057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2376c9fc09ebec8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.718+0000 7f2b6259c640 1 --2- 172.21.15.5:0/3744045928 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2b640cc000 0x7f2b6415e3b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2b540098a0 tx=0x7f2b5402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9ea57f528c2d95c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.719+0000 7f5be3fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5bdc0d9be0 0x7f5bdc0da840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.722+0000 7f5be3fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5bdc0d9be0 0x7f5bdc0da840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:51500/0 (socket says 172.21.15.5:51500) 2024-03-24T10:45:14.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.722+0000 7f5be3fff640 1 -- 172.21.15.5:0/2088755695 learned_addr learned my addr 172.21.15.5:0/2088755695 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:14.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.732+0000 7f5be3fff640 1 -- 172.21.15.5:0/2088755695 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5bdc046160 msgr2=0x7f5bdc0494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.732+0000 7f5be3fff640 1 --2- 172.21.15.5:0/2088755695 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5bdc046160 0x7f5bdc0494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.732+0000 7f5be3fff640 1 -- 172.21.15.5:0/2088755695 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bdc04bc00 msgr2=0x7f5bdc0dbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.732+0000 7f5be3fff640 1 --2- 172.21.15.5:0/2088755695 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bdc04bc00 0x7f5bdc0dbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.732+0000 7f5be3fff640 1 -- 172.21.15.5:0/2088755695 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bd0009400 con 0x7f5bdc0d9be0 2024-03-24T10:45:14.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.735+0000 7f2b6159a640 1 -- 172.21.15.5:0/3744045928 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b54030840 con 0x7f2b640cc000 2024-03-24T10:45:14.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.735+0000 7f2b6159a640 1 -- 172.21.15.5:0/3744045928 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b540309a0 con 0x7f2b640cc000 2024-03-24T10:45:14.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.735+0000 7f2b6159a640 1 -- 172.21.15.5:0/3744045928 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b54030d30 con 0x7f2b640cc000 2024-03-24T10:45:14.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.736+0000 7f5be3fff640 1 --2- 172.21.15.5:0/2088755695 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5bdc0d9be0 0x7f5bdc0da840 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0009860 tx=0x7f5bd0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.736+0000 7f2b68be2640 1 -- 172.21.15.5:0/3744045928 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2b640cc000 msgr2=0x7f2b6415e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.736+0000 7f2b68be2640 1 --2- 172.21.15.5:0/3744045928 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2b640cc000 0x7f2b6415e3b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2b540098a0 tx=0x7f2b5402b980 comp rx=0 tx=0).stop 2024-03-24T10:45:14.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.736+0000 7f2b68be2640 1 -- 172.21.15.5:0/3744045928 shutdown_connections 2024-03-24T10:45:14.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.736+0000 7f2b68be2640 1 --2- 172.21.15.5:0/3744045928 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2b640cc000 0x7f2b6415e3b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.736+0000 7f2b68be2640 1 -- 172.21.15.5:0/3744045928 >> 172.21.15.5:0/3744045928 conn(0x7f2b640cfc30 msgr2=0x7f2b640d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:14.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.736+0000 7f2b68be2640 1 -- 172.21.15.5:0/3744045928 shutdown_connections 2024-03-24T10:45:14.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.736+0000 7f2b68be2640 1 -- 172.21.15.5:0/3744045928 wait complete. 2024-03-24T10:45:14.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.737+0000 7f2b68be2640 1 Processor -- start 2024-03-24T10:45:14.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.737+0000 7f2b68be2640 1 -- start start 2024-03-24T10:45:14.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.738+0000 7f2b68be2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2b6406bf50 0x7f2b6415c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.738+0000 7f2b68be2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b640cc000 0x7f2b640b0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.738+0000 7f2b68be2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2b640cf340 0x7f2b641e6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.738+0000 7f2b68be2640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f2b64068a20 con 0x7f2b640cf340 2024-03-24T10:45:14.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.738+0000 7f2b68be2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6415d0b0 con 0x7f2b640cc000 2024-03-24T10:45:14.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.738+0000 7f2b68be2640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f2b641f0740 con 0x7f2b6406bf50 2024-03-24T10:45:14.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.741+0000 7f2ad3fff640 1 -- 172.21.15.5:0/35776567 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc02cb00 con 0x7f2ac80047c0 2024-03-24T10:45:14.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.741+0000 7f2ad3fff640 1 -- 172.21.15.5:0/35776567 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2acc02cc60 con 0x7f2ac80047c0 2024-03-24T10:45:14.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.741+0000 7f2ad3fff640 1 -- 172.21.15.5:0/35776567 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc0058d0 con 0x7f2ac80047c0 2024-03-24T10:45:14.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.742+0000 7f2ae39f4640 1 -- 172.21.15.5:0/35776567 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2ac80047c0 msgr2=0x7f2ac8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.742+0000 7f2ae39f4640 1 --2- 172.21.15.5:0/35776567 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2ac80047c0 0x7f2ac8004c40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00c210 tx=0x7f2ac80057f0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.742+0000 7f2ae39f4640 1 -- 172.21.15.5:0/35776567 shutdown_connections 2024-03-24T10:45:14.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.742+0000 7f2ae39f4640 1 --2- 172.21.15.5:0/35776567 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2ac80047c0 0x7f2ac8004c40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.742+0000 7f2ae39f4640 1 -- 172.21.15.5:0/35776567 >> 172.21.15.5:0/35776567 conn(0x7f2adc070130 msgr2=0x7f2adc070510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:14.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.742+0000 7f2ae39f4640 1 -- 172.21.15.5:0/35776567 shutdown_connections 2024-03-24T10:45:14.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.742+0000 7f2ae39f4640 1 -- 172.21.15.5:0/35776567 wait complete. 2024-03-24T10:45:14.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.743+0000 7f2ae39f4640 1 Processor -- start 2024-03-24T10:45:14.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.743+0000 7f5bcffff640 1 -- 172.21.15.5:0/2088755695 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd003e040 con 0x7f5bdc0d9be0 2024-03-24T10:45:14.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.743+0000 7f5bcffff640 1 -- 172.21.15.5:0/2088755695 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bd00443f0 con 0x7f5bdc0d9be0 2024-03-24T10:45:14.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.743+0000 7f5bcffff640 1 -- 172.21.15.5:0/2088755695 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0044780 con 0x7f5bdc0d9be0 2024-03-24T10:45:14.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.743+0000 7f2ae39f4640 1 -- start start 2024-03-24T10:45:14.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.743+0000 7f5be8c7d640 1 -- 172.21.15.5:0/2088755695 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bdc0da560 con 0x7f5bdc0d9be0 2024-03-24T10:45:14.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.743+0000 7f2b6259c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2b6406bf50 0x7f2b6415c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.743+0000 7f5be8c7d640 1 -- 172.21.15.5:0/2088755695 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bdc048ca0 con 0x7f5bdc0d9be0 2024-03-24T10:45:14.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.744+0000 7f2ae39f4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2ac80047c0 0x7f2adc150510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.744+0000 7f2ae39f4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2adc14a300 0x7f2adc1f13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.744+0000 7f2ae39f4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2adc1f1900 0x7f2adc1496b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.744+0000 7f2ae39f4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc15a1b0 con 0x7f2adc1f1900 2024-03-24T10:45:14.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.744+0000 7f2ae39f4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc0c9e20 con 0x7f2adc14a300 2024-03-24T10:45:14.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.744+0000 7f2ae39f4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f2adc1f1d80 con 0x7f2ac80047c0 2024-03-24T10:45:14.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.744+0000 7f2b6259c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2b6406bf50 0x7f2b6415c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:51524/0 (socket says 172.21.15.5:51524) 2024-03-24T10:45:14.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.744+0000 7f2b6259c640 1 -- 172.21.15.5:0/3031462177 learned_addr learned my addr 172.21.15.5:0/3031462177 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:14.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.745+0000 7f5be8c7d640 1 -- 172.21.15.5:0/2088755695 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bdc005700 con 0x7f5bdc0d9be0 2024-03-24T10:45:14.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.763+0000 7fa4f9ffb640 1 -- 172.21.15.5:0/3508667336 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4f0031990 con 0x7fa4fc0c7600 2024-03-24T10:45:14.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.763+0000 7fa4f9ffb640 1 -- 172.21.15.5:0/3508667336 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0031d20 con 0x7fa4fc0c7600 2024-03-24T10:45:14.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.763+0000 7fa50195e640 1 -- 172.21.15.5:0/3508667336 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4fc0c7600 msgr2=0x7fa4fc0c3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.763+0000 7fa50195e640 1 --2- 172.21.15.5:0/3508667336 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4fc0c7600 0x7fa4fc0c3470 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa4f00097a0 tx=0x7fa4f002b8d0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.763+0000 7fa50195e640 1 -- 172.21.15.5:0/3508667336 shutdown_connections 2024-03-24T10:45:14.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.763+0000 7fa50195e640 1 --2- 172.21.15.5:0/3508667336 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4fc0c7600 0x7fa4fc0c3470 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.763+0000 7fa50195e640 1 -- 172.21.15.5:0/3508667336 >> 172.21.15.5:0/3508667336 conn(0x7fa4fc06e480 msgr2=0x7fa4fc06e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:14.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.764+0000 7fa50195e640 1 -- 172.21.15.5:0/3508667336 shutdown_connections 2024-03-24T10:45:14.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.764+0000 7fa50195e640 1 -- 172.21.15.5:0/3508667336 wait complete. 2024-03-24T10:45:14.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.765+0000 7fa50195e640 1 Processor -- start 2024-03-24T10:45:14.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.766+0000 7fa50195e640 1 -- start start 2024-03-24T10:45:14.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.767+0000 7fa50195e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa4fc0c1b60 0x7fa4fc0c5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.767+0000 7fa50195e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4fc0c7600 0x7fa4fc145940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.767+0000 7fa50195e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4fc14d950 0x7fa4fc1edb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.767+0000 7fa50195e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa4fc159cf0 con 0x7fa4fc0c7600 2024-03-24T10:45:14.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.767+0000 7fa50195e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa4fc06bf60 con 0x7fa4fc14d950 2024-03-24T10:45:14.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.767+0000 7fa50195e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fa4fc1ed690 con 0x7fa4fc0c1b60 2024-03-24T10:45:14.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.779+0000 7f56f17fa640 1 -- 172.21.15.5:0/1763748615 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f56e402f3e0 con 0x7f56f406fc70 2024-03-24T10:45:14.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.779+0000 7f56f17fa640 1 --2- 172.21.15.5:0/1763748615 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f56e006b890 0x7f56e006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.779+0000 7f56f3fff640 1 --2- 172.21.15.5:0/1763748615 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f56e006b890 0x7f56e006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.780+0000 7f56f3fff640 1 --2- 172.21.15.5:0/1763748615 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f56e006b890 0x7f56e006dd50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f56f412edc0 tx=0x7f56ec009250 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.790+0000 7faa7effd640 1 -- 172.21.15.5:0/4121801674 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7faa80003d10 con 0x7faa880cafd0 2024-03-24T10:45:14.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.790+0000 7faa7effd640 1 --2- 172.21.15.5:0/4121801674 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faa6806b840 0x7faa6806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.791+0000 7faa8659c640 1 --2- 172.21.15.5:0/4121801674 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faa6806b840 0x7faa6806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.791+0000 7faa8659c640 1 --2- 172.21.15.5:0/4121801674 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faa6806b840 0x7faa6806dd00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faa78036000 tx=0x7faa78030000 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.793+0000 7faa7effd640 1 -- 172.21.15.5:0/4121801674 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7faa800590a0 con 0x7faa880cafd0 2024-03-24T10:45:14.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.794+0000 7f56f17fa640 1 -- 172.21.15.5:0/1763748615 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f56e407be20 con 0x7f56f406fc70 2024-03-24T10:45:14.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.794+0000 7f56f17fa640 1 -- 172.21.15.5:0/1763748615 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f56e40779c0 con 0x7f56f406fc70 2024-03-24T10:45:14.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.801+0000 7f2b62d9d640 1 --2- 172.21.15.5:0/3031462177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2b640cf340 0x7f2b641e6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.803+0000 7faa7effd640 1 -- 172.21.15.5:0/4121801674 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7faa80055340 con 0x7faa880cafd0 2024-03-24T10:45:14.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.805+0000 7f2ae0f8f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2adc14a300 0x7f2adc1f13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.805+0000 7fa4fb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4fc14d950 0x7fa4fc1edb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.806+0000 7f2ae0f8f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2adc14a300 0x7f2adc1f13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36632/0 (socket says 172.21.15.5:36632) 2024-03-24T10:45:14.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.806+0000 7f2ae0f8f640 1 -- 172.21.15.5:0/3396716020 learned_addr learned my addr 172.21.15.5:0/3396716020 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:14.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.806+0000 7fa4fb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4fc14d950 0x7fa4fc1edb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36640/0 (socket says 172.21.15.5:36640) 2024-03-24T10:45:14.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.806+0000 7fa4fb7fe640 1 -- 172.21.15.5:0/3969894825 learned_addr learned my addr 172.21.15.5:0/3969894825 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:14.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.808+0000 7f2b62d9d640 1 -- 172.21.15.5:0/3031462177 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2b6406bf50 msgr2=0x7f2b6415c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.808+0000 7f2b62d9d640 1 --2- 172.21.15.5:0/3031462177 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2b6406bf50 0x7f2b6415c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.808+0000 7f2b62d9d640 1 -- 172.21.15.5:0/3031462177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b640cc000 msgr2=0x7f2b640b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.808+0000 7f2b62d9d640 1 --2- 172.21.15.5:0/3031462177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b640cc000 0x7f2b640b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.808+0000 7f2b62d9d640 1 -- 172.21.15.5:0/3031462177 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b540094d0 con 0x7f2b640cf340 2024-03-24T10:45:14.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.810+0000 7f2ae0f8f640 1 -- 172.21.15.5:0/3396716020 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2ac80047c0 msgr2=0x7f2adc150510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.810+0000 7f2ae0f8f640 1 --2- 172.21.15.5:0/3396716020 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2ac80047c0 0x7f2adc150510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.810+0000 7f2ae0f8f640 1 -- 172.21.15.5:0/3396716020 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2adc1f1900 msgr2=0x7f2adc1496b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.810+0000 7f2ae0f8f640 1 --2- 172.21.15.5:0/3396716020 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2adc1f1900 0x7f2adc1496b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.810+0000 7f2ae0f8f640 1 -- 172.21.15.5:0/3396716020 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2adc14ef90 con 0x7f2adc14a300 2024-03-24T10:45:14.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.813+0000 7fa4fb7fe640 1 -- 172.21.15.5:0/3969894825 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa4fc0c1b60 msgr2=0x7fa4fc0c5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.813+0000 7fa4fb7fe640 1 --2- 172.21.15.5:0/3969894825 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa4fc0c1b60 0x7fa4fc0c5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.813+0000 7fa4fb7fe640 1 -- 172.21.15.5:0/3969894825 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4fc0c7600 msgr2=0x7fa4fc145940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:14.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.813+0000 7fa4fb7fe640 1 --2- 172.21.15.5:0/3969894825 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4fc0c7600 0x7fa4fc145940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:14.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.813+0000 7fa4fb7fe640 1 -- 172.21.15.5:0/3969894825 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f0009400 con 0x7fa4fc14d950 2024-03-24T10:45:14.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.813+0000 7fabe9ffb640 1 -- 172.21.15.5:0/2507962584 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf000c8e0 con 0x7fabec0c2450 2024-03-24T10:45:14.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.813+0000 7fabe9ffb640 1 -- 172.21.15.5:0/2507962584 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabf000ca40 con 0x7fabec0c2450 2024-03-24T10:45:14.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.813+0000 7fabfa6b1640 1 -- 172.21.15.5:0/2507962584 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabec1771f0 con 0x7fabec0c2450 2024-03-24T10:45:14.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.814+0000 7fabe9ffb640 1 -- 172.21.15.5:0/2507962584 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf000cdb0 con 0x7fabec0c2450 2024-03-24T10:45:14.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.814+0000 7fabfa6b1640 1 -- 172.21.15.5:0/2507962584 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabec172f40 con 0x7fabec0c2450 2024-03-24T10:45:14.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.815+0000 7f2b62d9d640 1 --2- 172.21.15.5:0/3031462177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2b640cf340 0x7f2b641e6f40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c00c780 tx=0x7f2b5c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.815+0000 7fabfa6b1640 1 -- 172.21.15.5:0/2507962584 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabec0052b0 con 0x7fabec0c2450 2024-03-24T10:45:14.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.818+0000 7f2ae0f8f640 1 --2- 172.21.15.5:0/3396716020 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2adc14a300 0x7f2adc1f13c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2acc02c570 tx=0x7f2acc001930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.818+0000 7fa4fb7fe640 1 --2- 172.21.15.5:0/3969894825 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4fc14d950 0x7fa4fc1edb90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400c780 tx=0x7fa4f400cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.823+0000 7f2b52ffd640 1 -- 172.21.15.5:0/3031462177 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c003b70 con 0x7f2b640cf340 2024-03-24T10:45:14.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.823+0000 7f2b52ffd640 1 -- 172.21.15.5:0/3031462177 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b5c003cf0 con 0x7f2b640cf340 2024-03-24T10:45:14.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.823+0000 7f2b52ffd640 1 -- 172.21.15.5:0/3031462177 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c003ea0 con 0x7f2b640cf340 2024-03-24T10:45:14.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.824+0000 7f2b68be2640 1 -- 172.21.15.5:0/3031462177 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b641f0a20 con 0x7f2b640cf340 2024-03-24T10:45:14.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.824+0000 7f2b68be2640 1 -- 172.21.15.5:0/3031462177 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b641f0fe0 con 0x7f2b640cf340 2024-03-24T10:45:14.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.825+0000 7f2b68be2640 1 -- 172.21.15.5:0/3031462177 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b641f1200 con 0x7f2b640cf340 2024-03-24T10:45:14.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.827+0000 7fa4e3fff640 1 -- 172.21.15.5:0/3969894825 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f4003d80 con 0x7fa4fc14d950 2024-03-24T10:45:14.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.827+0000 7fa4e3fff640 1 -- 172.21.15.5:0/3969894825 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4f400ce90 con 0x7fa4fc14d950 2024-03-24T10:45:14.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.827+0000 7fa4e3fff640 1 -- 172.21.15.5:0/3969894825 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f40052b0 con 0x7fa4fc14d950 2024-03-24T10:45:14.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.827+0000 7fa50195e640 1 -- 172.21.15.5:0/3969894825 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4fc159f70 con 0x7fa4fc14d950 2024-03-24T10:45:14.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.827+0000 7fa50195e640 1 -- 172.21.15.5:0/3969894825 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4fc1f1990 con 0x7fa4fc14d950 2024-03-24T10:45:14.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.828+0000 7f2ad1ffb640 1 -- 172.21.15.5:0/3396716020 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc03c070 con 0x7f2adc14a300 2024-03-24T10:45:14.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.828+0000 7f2ae39f4640 1 -- 172.21.15.5:0/3396716020 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2adc14f270 con 0x7f2adc14a300 2024-03-24T10:45:14.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.828+0000 7f2ae39f4640 1 -- 172.21.15.5:0/3396716020 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2adc14f830 con 0x7f2adc14a300 2024-03-24T10:45:14.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.829+0000 7fa50195e640 1 -- 172.21.15.5:0/3969894825 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4fc1f1b20 con 0x7fa4fc14d950 2024-03-24T10:45:14.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.829+0000 7f2ad1ffb640 1 -- 172.21.15.5:0/3396716020 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2acc004ab0 con 0x7f2adc14a300 2024-03-24T10:45:14.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.830+0000 7f2ad1ffb640 1 -- 172.21.15.5:0/3396716020 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc004e60 con 0x7f2adc14a300 2024-03-24T10:45:14.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.830+0000 7f2ac77fe640 1 -- 172.21.15.5:0/3396716020 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2adc0c8760 con 0x7f2adc14a300 2024-03-24T10:45:14.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.853+0000 7f2ad1ffb640 1 -- 172.21.15.5:0/3396716020 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2acc032070 con 0x7f2adc14a300 2024-03-24T10:45:14.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.854+0000 7f2ad1ffb640 1 --2- 172.21.15.5:0/3396716020 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2ac006b860 0x7f2ac006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.854+0000 7f2ae1790640 1 --2- 172.21.15.5:0/3396716020 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2ac006b860 0x7f2ac006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.854+0000 7f2ae1790640 1 --2- 172.21.15.5:0/3396716020 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2ac006b860 0x7f2ac006dd20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8007fd0 tx=0x7f2ad8009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.856+0000 7f2ad1ffb640 1 -- 172.21.15.5:0/3396716020 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2acc07a580 con 0x7f2adc14a300 2024-03-24T10:45:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.870+0000 7fa4e3fff640 1 -- 172.21.15.5:0/3969894825 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa4f4006b10 con 0x7fa4fc14d950 2024-03-24T10:45:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.870+0000 7fa4e3fff640 1 --2- 172.21.15.5:0/3969894825 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa4e806b810 0x7fa4e806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.871+0000 7fa4faffd640 1 --2- 172.21.15.5:0/3969894825 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa4e806b810 0x7fa4e806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.872+0000 7fa4faffd640 1 --2- 172.21.15.5:0/3969894825 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa4e806b810 0x7fa4e806dcd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec001550 tx=0x7fa4ec0078d0 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.871+0000 7fabe9ffb640 1 -- 172.21.15.5:0/2507962584 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fabf0013470 con 0x7fabec0c2450 2024-03-24T10:45:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.872+0000 7fabe9ffb640 1 --2- 172.21.15.5:0/2507962584 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fabd806b810 0x7fabd806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.872+0000 7fabf96af640 1 --2- 172.21.15.5:0/2507962584 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fabd806b810 0x7fabd806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.872+0000 7fabf96af640 1 --2- 172.21.15.5:0/2507962584 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fabd806b810 0x7fabd806dcd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fabf4064c00 tx=0x7fabf4069da0 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.884+0000 7f2b52ffd640 1 -- 172.21.15.5:0/3031462177 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2b5c0052b0 con 0x7f2b640cf340 2024-03-24T10:45:14.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.885+0000 7f2b52ffd640 1 --2- 172.21.15.5:0/3031462177 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2b4406b860 0x7f2b4406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:14.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.885+0000 7f2b6259c640 1 --2- 172.21.15.5:0/3031462177 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2b4406b860 0x7f2b4406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:14.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.886+0000 7f2b6259c640 1 --2- 172.21.15.5:0/3031462177 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2b4406b860 0x7f2b4406dd20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2b5402b9c0 tx=0x7f2b54008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:14.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:14.908+0000 7f2b52ffd640 1 -- 172.21.15.5:0/3031462177 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2b5c058d20 con 0x7f2b640cf340 2024-03-24T10:45:15.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.016+0000 7faa8cc6f640 1 -- 172.21.15.5:0/4121801674 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7faa880c9340 con 0x7faa880cafd0 2024-03-24T10:45:15.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.070+0000 7f6395e3b640 1 -- 172.21.15.5:0/3128326878 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f6370005810 con 0x7f63880f6d40 2024-03-24T10:45:15.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.164+0000 7f56fb042640 1 -- 172.21.15.5:0/1763748615 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f56f41535c0 con 0x7f56f406fc70 2024-03-24T10:45:15.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.185+0000 7f5bcffff640 1 -- 172.21.15.5:0/2088755695 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5bd004d3b0 con 0x7f5bdc0d9be0 2024-03-24T10:45:15.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.185+0000 7f5bcffff640 1 --2- 172.21.15.5:0/2088755695 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5bc806b810 0x7f5bc806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:15.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.185+0000 7f5be37fe640 1 --2- 172.21.15.5:0/2088755695 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5bc806b810 0x7f5bc806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:15.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.186+0000 7f5be37fe640 1 --2- 172.21.15.5:0/2088755695 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5bc806b810 0x7f5bc806dcd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8009710 tx=0x7f5bd8006c90 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:15.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.190+0000 7f5bcffff640 1 -- 172.21.15.5:0/2088755695 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5bd007bdc0 con 0x7f5bdc0d9be0 2024-03-24T10:45:15.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.209+0000 7f5bcffff640 1 -- 172.21.15.5:0/2088755695 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5bd0077960 con 0x7f5bdc0d9be0 2024-03-24T10:45:15.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.210+0000 7fa4e3fff640 1 -- 172.21.15.5:0/3969894825 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa4f4058800 con 0x7fa4fc14d950 2024-03-24T10:45:15.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.222+0000 7f638d7fa640 1 -- 172.21.15.5:0/3128326878 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6384052020 con 0x7f63880f6d40 2024-03-24T10:45:15.222 INFO:teuthology.orchestra.run.smithi005.stdout:60129542153 2024-03-24T10:45:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.224+0000 7f6395e3b640 1 -- 172.21.15.5:0/3128326878 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f637406b860 msgr2=0x7f637406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.224+0000 7f6395e3b640 1 --2- 172.21.15.5:0/3128326878 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f637406b860 0x7f637406dd20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f637c001560 tx=0x7f637c007990 comp rx=0 tx=0).stop 2024-03-24T10:45:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.224+0000 7f6395e3b640 1 -- 172.21.15.5:0/3128326878 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f63880f6d40 msgr2=0x7f63880ec2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.224+0000 7f6395e3b640 1 --2- 172.21.15.5:0/3128326878 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f63880f6d40 0x7f63880ec2c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f63840098c0 tx=0x7f63840046c0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.224+0000 7f6395e3b640 1 -- 172.21.15.5:0/3128326878 shutdown_connections 2024-03-24T10:45:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.224+0000 7f6395e3b640 1 --2- 172.21.15.5:0/3128326878 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f637406b860 0x7f637406dd20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.224+0000 7f6395e3b640 1 --2- 172.21.15.5:0/3128326878 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f63880fc970 0x7f63880f5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.224+0000 7f6395e3b640 1 --2- 172.21.15.5:0/3128326878 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f63880f6d40 0x7f63880ec2c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.224+0000 7f6395e3b640 1 --2- 172.21.15.5:0/3128326878 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f638804b5c0 0x7f63880f26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.224+0000 7f6395e3b640 1 -- 172.21.15.5:0/3128326878 >> 172.21.15.5:0/3128326878 conn(0x7f6388050170 msgr2=0x7f63880ecf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.224+0000 7f6395e3b640 1 -- 172.21.15.5:0/3128326878 shutdown_connections 2024-03-24T10:45:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.224+0000 7f6395e3b640 1 -- 172.21.15.5:0/3128326878 wait complete. 2024-03-24T10:45:15.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.230+0000 7fa4e3fff640 1 -- 172.21.15.5:0/3969894825 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa4f40543a0 con 0x7fa4fc14d950 2024-03-24T10:45:15.236 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542153 for osd.1 2024-03-24T10:45:15.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.425+0000 7f5be8c7d640 1 -- 172.21.15.5:0/2088755695 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f5bdc048e30 con 0x7f5bdc0d9be0 2024-03-24T10:45:15.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.431+0000 7f5bcffff640 1 -- 172.21.15.5:0/2088755695 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5bd0034020 con 0x7f5bdc0d9be0 2024-03-24T10:45:15.432 INFO:teuthology.orchestra.run.smithi005.stdout:60129542153 2024-03-24T10:45:15.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.434+0000 7f5be8c7d640 1 -- 172.21.15.5:0/2088755695 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5bc806b810 msgr2=0x7f5bc806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:15.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.434+0000 7f5be8c7d640 1 --2- 172.21.15.5:0/2088755695 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5bc806b810 0x7f5bc806dcd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8009710 tx=0x7f5bd8006c90 comp rx=0 tx=0).stop 2024-03-24T10:45:15.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.434+0000 7f5be8c7d640 1 -- 172.21.15.5:0/2088755695 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5bdc0d9be0 msgr2=0x7f5bdc0da840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:15.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.434+0000 7f5be8c7d640 1 --2- 172.21.15.5:0/2088755695 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5bdc0d9be0 0x7f5bdc0da840 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0009860 tx=0x7f5bd0004730 comp rx=0 tx=0).stop 2024-03-24T10:45:15.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.434+0000 7f5be8c7d640 1 -- 172.21.15.5:0/2088755695 shutdown_connections 2024-03-24T10:45:15.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.434+0000 7f5be8c7d640 1 --2- 172.21.15.5:0/2088755695 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5bc806b810 0x7f5bc806dcd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.434+0000 7f5be8c7d640 1 --2- 172.21.15.5:0/2088755695 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5bdc04bc00 0x7f5bdc0dbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.434+0000 7f5be8c7d640 1 --2- 172.21.15.5:0/2088755695 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5bdc0d9be0 0x7f5bdc0da840 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.434+0000 7f5be8c7d640 1 --2- 172.21.15.5:0/2088755695 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5bdc046160 0x7f5bdc0494a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.434+0000 7f5be8c7d640 1 -- 172.21.15.5:0/2088755695 >> 172.21.15.5:0/2088755695 conn(0x7f5bdc051280 msgr2=0x7f5bdc046e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:15.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.435+0000 7f5be8c7d640 1 -- 172.21.15.5:0/2088755695 shutdown_connections 2024-03-24T10:45:15.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.435+0000 7f5be8c7d640 1 -- 172.21.15.5:0/2088755695 wait complete. 2024-03-24T10:45:15.450 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542153 for osd.2 2024-03-24T10:45:15.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.520+0000 7fabe9ffb640 1 -- 172.21.15.5:0/2507962584 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fabf0059860 con 0x7fabec0c2450 2024-03-24T10:45:15.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.520+0000 7fabe9ffb640 1 -- 172.21.15.5:0/2507962584 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fabf0055400 con 0x7fabec0c2450 2024-03-24T10:45:15.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.542+0000 7f2b52ffd640 1 -- 172.21.15.5:0/3031462177 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2b5c0548c0 con 0x7f2b640cf340 2024-03-24T10:45:15.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.545+0000 7f2ad1ffb640 1 -- 172.21.15.5:0/3396716020 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2acc076120 con 0x7f2adc14a300 2024-03-24T10:45:15.564 INFO:teuthology.orchestra.run.smithi005.stdout:55834574855 2024-03-24T10:45:15.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.561+0000 7faa7effd640 1 -- 172.21.15.5:0/4121801674 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7faa8001d030 con 0x7faa880cafd0 2024-03-24T10:45:15.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.564+0000 7faa8cc6f640 1 -- 172.21.15.5:0/4121801674 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faa6806b840 msgr2=0x7faa6806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:15.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.564+0000 7faa8cc6f640 1 --2- 172.21.15.5:0/4121801674 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faa6806b840 0x7faa6806dd00 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faa78036000 tx=0x7faa78030000 comp rx=0 tx=0).stop 2024-03-24T10:45:15.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.564+0000 7faa8cc6f640 1 -- 172.21.15.5:0/4121801674 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa880cafd0 msgr2=0x7faa881e1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:15.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.564+0000 7faa8cc6f640 1 --2- 172.21.15.5:0/4121801674 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa880cafd0 0x7faa881e1530 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faa8000ccb0 tx=0x7faa8000a5c0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.564+0000 7faa8cc6f640 1 -- 172.21.15.5:0/4121801674 shutdown_connections 2024-03-24T10:45:15.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.564+0000 7faa8cc6f640 1 --2- 172.21.15.5:0/4121801674 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faa6806b840 0x7faa6806dd00 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.564+0000 7faa8cc6f640 1 --2- 172.21.15.5:0/4121801674 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa880c8e50 0x7faa8814e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.564+0000 7faa8cc6f640 1 --2- 172.21.15.5:0/4121801674 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa880cafd0 0x7faa881e1530 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.564+0000 7faa8cc6f640 1 --2- 172.21.15.5:0/4121801674 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faa880c8a70 0x7faa881534f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.564+0000 7faa8cc6f640 1 -- 172.21.15.5:0/4121801674 >> 172.21.15.5:0/4121801674 conn(0x7faa8806fa80 msgr2=0x7faa88152040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:15.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.564+0000 7faa8cc6f640 1 -- 172.21.15.5:0/4121801674 shutdown_connections 2024-03-24T10:45:15.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.564+0000 7faa8cc6f640 1 -- 172.21.15.5:0/4121801674 wait complete. 2024-03-24T10:45:15.583 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.6 2024-03-24T10:45:15.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.583+0000 7fa50195e640 1 -- 172.21.15.5:0/3969894825 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fa4fc1ed8b0 con 0x7fa4fc14d950 2024-03-24T10:45:15.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.591+0000 7fa4e3fff640 1 -- 172.21.15.5:0/3969894825 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa4fc1ed8b0 con 0x7fa4fc14d950 2024-03-24T10:45:15.594 INFO:teuthology.orchestra.run.smithi005.stdout:60129542152 2024-03-24T10:45:15.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.594+0000 7fa50195e640 1 -- 172.21.15.5:0/3969894825 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa4e806b810 msgr2=0x7fa4e806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:15.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.594+0000 7fa50195e640 1 --2- 172.21.15.5:0/3969894825 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa4e806b810 0x7fa4e806dcd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec001550 tx=0x7fa4ec0078d0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.594+0000 7fa50195e640 1 -- 172.21.15.5:0/3969894825 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4fc14d950 msgr2=0x7fa4fc1edb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:15.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.594+0000 7fa50195e640 1 --2- 172.21.15.5:0/3969894825 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4fc14d950 0x7fa4fc1edb90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa4f400c780 tx=0x7fa4f400cc40 comp rx=0 tx=0).stop 2024-03-24T10:45:15.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.594+0000 7fa50195e640 1 -- 172.21.15.5:0/3969894825 shutdown_connections 2024-03-24T10:45:15.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.594+0000 7fa50195e640 1 --2- 172.21.15.5:0/3969894825 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa4e806b810 0x7fa4e806dcd0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.594+0000 7fa50195e640 1 --2- 172.21.15.5:0/3969894825 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa4fc0c7600 0x7fa4fc145940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.594+0000 7fa50195e640 1 --2- 172.21.15.5:0/3969894825 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4fc14d950 0x7fa4fc1edb90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.594+0000 7fa50195e640 1 --2- 172.21.15.5:0/3969894825 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa4fc0c1b60 0x7fa4fc0c5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.594+0000 7fa50195e640 1 -- 172.21.15.5:0/3969894825 >> 172.21.15.5:0/3969894825 conn(0x7fa4fc06e480 msgr2=0x7fa4fc14c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:15.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.595+0000 7fa50195e640 1 -- 172.21.15.5:0/3969894825 shutdown_connections 2024-03-24T10:45:15.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.603+0000 7fa50195e640 1 -- 172.21.15.5:0/3969894825 wait complete. 2024-03-24T10:45:15.612 INFO:teuthology.orchestra.run.smithi005.stdout:60129542153 2024-03-24T10:45:15.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.609+0000 7f56f17fa640 1 -- 172.21.15.5:0/1763748615 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f56f41535c0 con 0x7f56f406fc70 2024-03-24T10:45:15.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.612+0000 7f56fb042640 1 -- 172.21.15.5:0/1763748615 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f56e006b890 msgr2=0x7f56e006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:15.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.612+0000 7f56fb042640 1 --2- 172.21.15.5:0/1763748615 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f56e006b890 0x7f56e006dd50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f56f412edc0 tx=0x7f56ec009250 comp rx=0 tx=0).stop 2024-03-24T10:45:15.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.612+0000 7f56fb042640 1 -- 172.21.15.5:0/1763748615 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56f406fc70 msgr2=0x7f56f414f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:15.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.612+0000 7f56fb042640 1 --2- 172.21.15.5:0/1763748615 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56f406fc70 0x7f56f414f920 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f56e4009960 tx=0x7f56e4004730 comp rx=0 tx=0).stop 2024-03-24T10:45:15.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.613+0000 7f56fb042640 1 -- 172.21.15.5:0/1763748615 shutdown_connections 2024-03-24T10:45:15.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.613+0000 7f56fb042640 1 --2- 172.21.15.5:0/1763748615 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f56e006b890 0x7f56e006dd50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.613+0000 7f56fb042640 1 --2- 172.21.15.5:0/1763748615 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56f41509a0 0x7f56f414ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.613+0000 7f56fb042640 1 --2- 172.21.15.5:0/1763748615 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56f40c7d60 0x7f56f412bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.613+0000 7f56fb042640 1 --2- 172.21.15.5:0/1763748615 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56f406fc70 0x7f56f414f920 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.613+0000 7f56fb042640 1 -- 172.21.15.5:0/1763748615 >> 172.21.15.5:0/1763748615 conn(0x7f56f406f180 msgr2=0x7f56f40ca9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:15.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.613+0000 7f56fb042640 1 -- 172.21.15.5:0/1763748615 shutdown_connections 2024-03-24T10:45:15.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.613+0000 7f56fb042640 1 -- 172.21.15.5:0/1763748615 wait complete. 2024-03-24T10:45:15.625 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.3 2024-03-24T10:45:15.636 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542153 for osd.5 2024-03-24T10:45:15.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.751+0000 7fabfa6b1640 1 -- 172.21.15.5:0/2507962584 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fabec1771f0 con 0x7fabec0c2450 2024-03-24T10:45:15.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.779+0000 7fabe9ffb640 1 -- 172.21.15.5:0/2507962584 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fabec1771f0 con 0x7fabec0c2450 2024-03-24T10:45:15.780 INFO:teuthology.orchestra.run.smithi005.stdout:55834574855 2024-03-24T10:45:15.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.782+0000 7fabfa6b1640 1 -- 172.21.15.5:0/2507962584 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fabd806b810 msgr2=0x7fabd806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:15.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.782+0000 7fabfa6b1640 1 --2- 172.21.15.5:0/2507962584 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fabd806b810 0x7fabd806dcd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fabf4064c00 tx=0x7fabf4069da0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.782+0000 7fabfa6b1640 1 -- 172.21.15.5:0/2507962584 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabec0c2450 msgr2=0x7fabec0cb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:15.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.782+0000 7fabfa6b1640 1 --2- 172.21.15.5:0/2507962584 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabec0c2450 0x7fabec0cb8a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fabf000e840 tx=0x7fabf000ed00 comp rx=0 tx=0).stop 2024-03-24T10:45:15.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.782+0000 7fabfa6b1640 1 -- 172.21.15.5:0/2507962584 shutdown_connections 2024-03-24T10:45:15.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.782+0000 7fabfa6b1640 1 --2- 172.21.15.5:0/2507962584 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fabd806b810 0x7fabd806dcd0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.782+0000 7fabfa6b1640 1 --2- 172.21.15.5:0/2507962584 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fabec0c2450 0x7fabec0cb8a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.782+0000 7fabfa6b1640 1 --2- 172.21.15.5:0/2507962584 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fabec04b4f0 0x7fabec176cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.782+0000 7fabfa6b1640 1 --2- 172.21.15.5:0/2507962584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fabec006410 0x7fabec0cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.782+0000 7fabfa6b1640 1 -- 172.21.15.5:0/2507962584 >> 172.21.15.5:0/2507962584 conn(0x7fabec050170 msgr2=0x7fabec050550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:15.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.782+0000 7fabfa6b1640 1 -- 172.21.15.5:0/2507962584 shutdown_connections 2024-03-24T10:45:15.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.782+0000 7fabfa6b1640 1 -- 172.21.15.5:0/2507962584 wait complete. 2024-03-24T10:45:15.795 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.0 2024-03-24T10:45:15.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.819+0000 7f2b68be2640 1 -- 172.21.15.5:0/3031462177 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f2b640681d0 con 0x7f2b640cf340 2024-03-24T10:45:15.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.868+0000 7f2b52ffd640 1 -- 172.21.15.5:0/3031462177 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2b5c017070 con 0x7f2b640cf340 2024-03-24T10:45:15.868 INFO:teuthology.orchestra.run.smithi005.stdout:60129542153 2024-03-24T10:45:15.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.870+0000 7f2b68be2640 1 -- 172.21.15.5:0/3031462177 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2b4406b860 msgr2=0x7f2b4406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:15.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.870+0000 7f2b68be2640 1 --2- 172.21.15.5:0/3031462177 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2b4406b860 0x7f2b4406dd20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2b5402b9c0 tx=0x7f2b54008030 comp rx=0 tx=0).stop 2024-03-24T10:45:15.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.870+0000 7f2b68be2640 1 -- 172.21.15.5:0/3031462177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2b640cf340 msgr2=0x7f2b641e6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:15.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.870+0000 7f2b68be2640 1 --2- 172.21.15.5:0/3031462177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2b640cf340 0x7f2b641e6f40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c00c780 tx=0x7f2b5c00cc40 comp rx=0 tx=0).stop 2024-03-24T10:45:15.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.871+0000 7f2b68be2640 1 -- 172.21.15.5:0/3031462177 shutdown_connections 2024-03-24T10:45:15.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.871+0000 7f2b68be2640 1 --2- 172.21.15.5:0/3031462177 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2b4406b860 0x7f2b4406dd20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.871+0000 7f2b68be2640 1 --2- 172.21.15.5:0/3031462177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2b640cf340 0x7f2b641e6f40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.871+0000 7f2b68be2640 1 --2- 172.21.15.5:0/3031462177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2b640cc000 0x7f2b640b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.871+0000 7f2b68be2640 1 --2- 172.21.15.5:0/3031462177 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2b6406bf50 0x7f2b6415c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.871+0000 7f2b68be2640 1 -- 172.21.15.5:0/3031462177 >> 172.21.15.5:0/3031462177 conn(0x7f2b640cfc30 msgr2=0x7f2b640c8030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:15.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.871+0000 7f2b68be2640 1 -- 172.21.15.5:0/3031462177 shutdown_connections 2024-03-24T10:45:15.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.871+0000 7f2b68be2640 1 -- 172.21.15.5:0/3031462177 wait complete. 2024-03-24T10:45:15.882 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542153 for osd.7 2024-03-24T10:45:15.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.907+0000 7f2ac77fe640 1 -- 172.21.15.5:0/3396716020 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f2adc0688e0 con 0x7f2adc14a300 2024-03-24T10:45:15.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.910+0000 7f2ad1ffb640 1 -- 172.21.15.5:0/3396716020 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2acc045020 con 0x7f2adc14a300 2024-03-24T10:45:15.910 INFO:teuthology.orchestra.run.smithi005.stdout:55834574855 2024-03-24T10:45:15.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.913+0000 7f2ac77fe640 1 -- 172.21.15.5:0/3396716020 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2ac006b860 msgr2=0x7f2ac006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:15.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.913+0000 7f2ac77fe640 1 --2- 172.21.15.5:0/3396716020 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2ac006b860 0x7f2ac006dd20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2ad8007fd0 tx=0x7f2ad8009040 comp rx=0 tx=0).stop 2024-03-24T10:45:15.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.913+0000 7f2ac77fe640 1 -- 172.21.15.5:0/3396716020 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2adc14a300 msgr2=0x7f2adc1f13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:15.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.913+0000 7f2ac77fe640 1 --2- 172.21.15.5:0/3396716020 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2adc14a300 0x7f2adc1f13c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2acc02c570 tx=0x7f2acc001930 comp rx=0 tx=0).stop 2024-03-24T10:45:15.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.913+0000 7f2ac77fe640 1 -- 172.21.15.5:0/3396716020 shutdown_connections 2024-03-24T10:45:15.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.913+0000 7f2ac77fe640 1 --2- 172.21.15.5:0/3396716020 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2ac006b860 0x7f2ac006dd20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.913+0000 7f2ac77fe640 1 --2- 172.21.15.5:0/3396716020 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2adc1f1900 0x7f2adc1496b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.913+0000 7f2ac77fe640 1 --2- 172.21.15.5:0/3396716020 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2adc14a300 0x7f2adc1f13c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.913+0000 7f2ac77fe640 1 --2- 172.21.15.5:0/3396716020 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2ac80047c0 0x7f2adc150510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:15.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.913+0000 7f2ac77fe640 1 -- 172.21.15.5:0/3396716020 >> 172.21.15.5:0/3396716020 conn(0x7f2adc070130 msgr2=0x7f2adc15ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:15.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.913+0000 7f2ac77fe640 1 -- 172.21.15.5:0/3396716020 shutdown_connections 2024-03-24T10:45:15.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:15.913+0000 7f2ac77fe640 1 -- 172.21.15.5:0/3396716020 wait complete. 2024-03-24T10:45:15.928 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.4 2024-03-24T10:45:16.237 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-24T10:45:16.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.341+0000 7fac5dd6c640 1 Processor -- start 2024-03-24T10:45:16.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.341+0000 7fac5dd6c640 1 -- start start 2024-03-24T10:45:16.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.342+0000 7fac5dd6c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac580cf340 0x7fac580cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.342+0000 7fac5dd6c640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac580cc3e0 con 0x7fac580ccb80 2024-03-24T10:45:16.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.342+0000 7fac5dd6c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac5815a5f0 con 0x7fac580cc000 2024-03-24T10:45:16.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.342+0000 7fac5dd6c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fac5815a400 con 0x7fac580cf340 2024-03-24T10:45:16.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.342+0000 7fac577fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fac580cc000 0x7fac5815e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:47106/0 (socket says 172.21.15.5:47106) 2024-03-24T10:45:16.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.342+0000 7fac577fe640 1 -- 172.21.15.5:0/1322526927 learned_addr learned my addr 172.21.15.5:0/1322526927 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:16.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.342+0000 7fac56ffd640 1 --2- 172.21.15.5:0/1322526927 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac580cf340 0x7fac580cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:16.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.345+0000 7fac567fc640 1 -- 172.21.15.5:0/1322526927 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3556759576 0 0) 0x7fac580cc3e0 con 0x7fac580ccb80 2024-03-24T10:45:16.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.345+0000 7fac567fc640 1 -- 172.21.15.5:0/1322526927 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac40003520 con 0x7fac580ccb80 2024-03-24T10:45:16.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.345+0000 7fac567fc640 1 -- 172.21.15.5:0/1322526927 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 688066430 0 0) 0x7fac5815a5f0 con 0x7fac580cc000 2024-03-24T10:45:16.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.345+0000 7fac567fc640 1 -- 172.21.15.5:0/1322526927 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac580cc3e0 con 0x7fac580cc000 2024-03-24T10:45:16.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.346+0000 7fac56ffd640 1 -- 172.21.15.5:0/1322526927 >> v1:172.21.15.57:6789/0 conn(0x7fac580cc000 legacy=0x7fac5815e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:16.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.346+0000 7fac56ffd640 1 -- 172.21.15.5:0/1322526927 >> v1:172.21.15.5:6789/0 conn(0x7fac580ccb80 legacy=0x7fac5806bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:16.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.346+0000 7fac56ffd640 1 -- 172.21.15.5:0/1322526927 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5815a8b0 con 0x7fac580cf340 2024-03-24T10:45:16.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.348+0000 7fac56ffd640 1 --2- 172.21.15.5:0/1322526927 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac580cf340 0x7fac580cc640 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fac4c0097d0 tx=0x7fac4c02b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec8007fe611d4fcb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:16.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.352+0000 7fac567fc640 1 -- 172.21.15.5:0/1322526927 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c030880 con 0x7fac580cf340 2024-03-24T10:45:16.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.352+0000 7fac567fc640 1 -- 172.21.15.5:0/1322526927 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac4c0309e0 con 0x7fac580cf340 2024-03-24T10:45:16.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.352+0000 7fac567fc640 1 -- 172.21.15.5:0/1322526927 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c030d70 con 0x7fac580cf340 2024-03-24T10:45:16.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.353+0000 7fac5dd6c640 1 -- 172.21.15.5:0/1322526927 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac580cf340 msgr2=0x7fac580cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:16.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.353+0000 7fac5dd6c640 1 --2- 172.21.15.5:0/1322526927 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac580cf340 0x7fac580cc640 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fac4c0097d0 tx=0x7fac4c02b9c0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.353+0000 7fac5dd6c640 1 -- 172.21.15.5:0/1322526927 shutdown_connections 2024-03-24T10:45:16.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.353+0000 7fac5dd6c640 1 --2- 172.21.15.5:0/1322526927 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac580cf340 0x7fac580cc640 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.353+0000 7fac5dd6c640 1 -- 172.21.15.5:0/1322526927 >> 172.21.15.5:0/1322526927 conn(0x7fac580cfc30 msgr2=0x7fac580d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:16.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.353+0000 7fac5dd6c640 1 -- 172.21.15.5:0/1322526927 shutdown_connections 2024-03-24T10:45:16.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.353+0000 7fac5dd6c640 1 -- 172.21.15.5:0/1322526927 wait complete. 2024-03-24T10:45:16.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.354+0000 7fac5dd6c640 1 Processor -- start 2024-03-24T10:45:16.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.354+0000 7fac5dd6c640 1 -- start start 2024-03-24T10:45:16.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.355+0000 7fac5dd6c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac580cc000 0x7fac581efc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.355+0000 7fac5dd6c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac580ccb80 0x7fac5814de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.355+0000 7fac5dd6c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac580cf340 0x7fac5814ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.355+0000 7fac5dd6c640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fac581f0250 con 0x7fac580cf340 2024-03-24T10:45:16.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.355+0000 7fac5dd6c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fac581eddf0 con 0x7fac580cc000 2024-03-24T10:45:16.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.355+0000 7fac5dd6c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fac5814eeb0 con 0x7fac580ccb80 2024-03-24T10:45:16.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.356+0000 7fac57fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac580cf340 0x7fac5814ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:16.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.356+0000 7fac577fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac580cc000 0x7fac581efc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:16.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.356+0000 7fac56ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac580ccb80 0x7fac5814de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:16.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.356+0000 7fac57fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac580cf340 0x7fac5814ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60192/0 (socket says 172.21.15.5:60192) 2024-03-24T10:45:16.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.356+0000 7fac57fff640 1 -- 172.21.15.5:0/2740686377 learned_addr learned my addr 172.21.15.5:0/2740686377 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:16.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.360+0000 7fac57fff640 1 -- 172.21.15.5:0/2740686377 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac580ccb80 msgr2=0x7fac5814de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:16.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.360+0000 7fac57fff640 1 --2- 172.21.15.5:0/2740686377 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac580ccb80 0x7fac5814de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.360+0000 7fac57fff640 1 -- 172.21.15.5:0/2740686377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac580cc000 msgr2=0x7fac581efc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:16.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.360+0000 7fac57fff640 1 --2- 172.21.15.5:0/2740686377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac580cc000 0x7fac581efc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.360+0000 7fac57fff640 1 -- 172.21.15.5:0/2740686377 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4c009400 con 0x7fac580cf340 2024-03-24T10:45:16.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.363+0000 7fac57fff640 1 --2- 172.21.15.5:0/2740686377 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac580cf340 0x7fac5814ff50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fac5000c770 tx=0x7fac5000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:16.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.365+0000 7fac3ffff640 1 -- 172.21.15.5:0/2740686377 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac50003b70 con 0x7fac580cf340 2024-03-24T10:45:16.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.365+0000 7fac5dd6c640 1 -- 172.21.15.5:0/2740686377 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac5814f190 con 0x7fac580cf340 2024-03-24T10:45:16.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.365+0000 7fac5dd6c640 1 -- 172.21.15.5:0/2740686377 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac5814f750 con 0x7fac580cf340 2024-03-24T10:45:16.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.366+0000 7fac5dd6c640 1 -- 172.21.15.5:0/2740686377 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac580681d0 con 0x7fac580cf340 2024-03-24T10:45:16.450 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-24T10:45:16.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.553+0000 7fc183834640 1 Processor -- start 2024-03-24T10:45:16.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.553+0000 7fc183834640 1 -- start start 2024-03-24T10:45:16.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.554+0000 7fc183834640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc17c0717d0 0x7fc17c158be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.554+0000 7fc183834640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17c0c9210 con 0x7fc17c0ce300 2024-03-24T10:45:16.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.554+0000 7fc183834640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17c0c2c80 con 0x7fc17c0713f0 2024-03-24T10:45:16.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.554+0000 7fc183834640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc17c0c2f20 con 0x7fc17c0717d0 2024-03-24T10:45:16.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.554+0000 7fc180dcf640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc17c0717d0 0x7fc17c158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:16.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.555+0000 7fc1815d0640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fc17c0713f0 0x7fc17c151f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:47114/0 (socket says 172.21.15.5:47114) 2024-03-24T10:45:16.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.555+0000 7fc1815d0640 1 -- 172.21.15.5:0/1032088586 learned_addr learned my addr 172.21.15.5:0/1032088586 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:16.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.557+0000 7fc180dcf640 1 -- 172.21.15.5:0/1032088586 >> v1:172.21.15.57:6789/0 conn(0x7fc17c0713f0 legacy=0x7fc17c151f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:16.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.557+0000 7fc180dcf640 1 -- 172.21.15.5:0/1032088586 >> v1:172.21.15.5:6789/0 conn(0x7fc17c0ce300 legacy=0x7fc17c1596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:16.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.557+0000 7fc180dcf640 1 -- 172.21.15.5:0/1032088586 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc17c0c31e0 con 0x7fc17c0717d0 2024-03-24T10:45:16.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.559+0000 7fc180dcf640 1 --2- 172.21.15.5:0/1032088586 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc17c0717d0 0x7fc17c158be0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc16c0097d0 tx=0x7fc16c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=119b2e757fa1e22d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:16.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.563+0000 7fc173fff640 1 -- 172.21.15.5:0/1032088586 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16c031890 con 0x7fc17c0717d0 2024-03-24T10:45:16.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.563+0000 7fc173fff640 1 -- 172.21.15.5:0/1032088586 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc16c0319f0 con 0x7fc17c0717d0 2024-03-24T10:45:16.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.563+0000 7fc173fff640 1 -- 172.21.15.5:0/1032088586 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16c031d80 con 0x7fc17c0717d0 2024-03-24T10:45:16.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.564+0000 7fc183834640 1 -- 172.21.15.5:0/1032088586 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc17c0717d0 msgr2=0x7fc17c158be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:16.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.564+0000 7fc183834640 1 --2- 172.21.15.5:0/1032088586 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc17c0717d0 0x7fc17c158be0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc16c0097d0 tx=0x7fc16c02b930 comp rx=0 tx=0).stop 2024-03-24T10:45:16.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.564+0000 7fc183834640 1 -- 172.21.15.5:0/1032088586 shutdown_connections 2024-03-24T10:45:16.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.564+0000 7fc183834640 1 --2- 172.21.15.5:0/1032088586 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc17c0717d0 0x7fc17c158be0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.564+0000 7fc183834640 1 -- 172.21.15.5:0/1032088586 >> 172.21.15.5:0/1032088586 conn(0x7fc17c06b760 msgr2=0x7fc17c06f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:16.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.564+0000 7fc183834640 1 -- 172.21.15.5:0/1032088586 shutdown_connections 2024-03-24T10:45:16.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.564+0000 7fc183834640 1 -- 172.21.15.5:0/1032088586 wait complete. 2024-03-24T10:45:16.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.566+0000 7fc183834640 1 Processor -- start 2024-03-24T10:45:16.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.566+0000 7fc183834640 1 -- start start 2024-03-24T10:45:16.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.567+0000 7fc183834640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc17c0713f0 0x7fc17c06cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.567+0000 7fc183834640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc17c0717d0 0x7fc17c1f3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.567+0000 7fc183834640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc17c0ce300 0x7fc17c0ca660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.567+0000 7fc183834640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c0adc70 con 0x7fc17c0713f0 2024-03-24T10:45:16.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.567+0000 7fc183834640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c1f35d0 con 0x7fc17c0ce300 2024-03-24T10:45:16.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.567+0000 7fc183834640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc17c0cbb10 con 0x7fc17c0717d0 2024-03-24T10:45:16.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.567+0000 7fc180dcf640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc17c0717d0 0x7fc17c1f3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:16.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.567+0000 7fc181dd1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc17c0ce300 0x7fc17c0ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:16.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.568+0000 7fc180dcf640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc17c0717d0 0x7fc17c1f3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:51572/0 (socket says 172.21.15.5:51572) 2024-03-24T10:45:16.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.568+0000 7fc180dcf640 1 -- 172.21.15.5:0/4251742212 learned_addr learned my addr 172.21.15.5:0/4251742212 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:16.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.571+0000 7fc180dcf640 1 -- 172.21.15.5:0/4251742212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc17c0ce300 msgr2=0x7fc17c0ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:16.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.571+0000 7fc180dcf640 1 --2- 172.21.15.5:0/4251742212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc17c0ce300 0x7fc17c0ca660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.571+0000 7fc180dcf640 1 -- 172.21.15.5:0/4251742212 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc17c0713f0 msgr2=0x7fc17c06cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:16.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.571+0000 7fc180dcf640 1 --2- 172.21.15.5:0/4251742212 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc17c0713f0 0x7fc17c06cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.571+0000 7fc180dcf640 1 -- 172.21.15.5:0/4251742212 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc16c009400 con 0x7fc17c0717d0 2024-03-24T10:45:16.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.573+0000 7fc180dcf640 1 --2- 172.21.15.5:0/4251742212 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc17c0717d0 0x7fc17c1f3a60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc16c02bf40 tx=0x7fc16c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:16.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.577+0000 7fc171ffb640 1 -- 172.21.15.5:0/4251742212 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16c031830 con 0x7fc17c0717d0 2024-03-24T10:45:16.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.577+0000 7fc171ffb640 1 -- 172.21.15.5:0/4251742212 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc16c046450 con 0x7fc17c0717d0 2024-03-24T10:45:16.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.577+0000 7fc183834640 1 -- 172.21.15.5:0/4251742212 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc17c0cbdb0 con 0x7fc17c0717d0 2024-03-24T10:45:16.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.577+0000 7fc171ffb640 1 -- 172.21.15.5:0/4251742212 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc16c046800 con 0x7fc17c0717d0 2024-03-24T10:45:16.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.578+0000 7fc183834640 1 -- 172.21.15.5:0/4251742212 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc17c0cc350 con 0x7fc17c0717d0 2024-03-24T10:45:16.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.579+0000 7fc183834640 1 -- 172.21.15.5:0/4251742212 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc17c0c3d10 con 0x7fc17c0717d0 2024-03-24T10:45:16.583 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-24T10:45:16.626 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-24T10:45:16.637 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-24T10:45:16.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.653+0000 7fac3ffff640 1 -- 172.21.15.5:0/2740686377 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac5000ce80 con 0x7fac580cf340 2024-03-24T10:45:16.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.653+0000 7fac3ffff640 1 -- 172.21.15.5:0/2740686377 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac5000a850 con 0x7fac580cf340 2024-03-24T10:45:16.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.654+0000 7fac3ffff640 1 -- 172.21.15.5:0/2740686377 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fac5000aa70 con 0x7fac580cf340 2024-03-24T10:45:16.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.654+0000 7fac3ffff640 1 --2- 172.21.15.5:0/2740686377 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac4406b810 0x7fac4406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.655+0000 7fac577fe640 1 --2- 172.21.15.5:0/2740686377 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac4406b810 0x7fac4406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:16.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.655+0000 7fac577fe640 1 --2- 172.21.15.5:0/2740686377 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac4406b810 0x7fac4406dcd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fac480011e0 tx=0x7fac48007d30 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:16.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.680+0000 7fac3ffff640 1 -- 172.21.15.5:0/2740686377 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fac50059230 con 0x7fac580cf340 2024-03-24T10:45:16.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.680+0000 7fac3ffff640 1 -- 172.21.15.5:0/2740686377 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fac500554d0 con 0x7fac580cf340 2024-03-24T10:45:16.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.701+0000 7f84accaf640 1 Processor -- start 2024-03-24T10:45:16.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.701+0000 7f84accaf640 1 -- start start 2024-03-24T10:45:16.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.702+0000 7f84accaf640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f84a80c91b0 0x7f84a80c7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.702+0000 7f84accaf640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f84a80c7aa0 0x7f84a80c7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.703+0000 7f84accaf640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84a80c45b0 con 0x7f84a8151370 2024-03-24T10:45:16.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.703+0000 7f84accaf640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f84a80c95d0 con 0x7f84a80c7aa0 2024-03-24T10:45:16.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.703+0000 7f84accaf640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f84a80c97b0 con 0x7f84a80c91b0 2024-03-24T10:45:16.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.703+0000 7f84a659c640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f84a8151370 0x7f84a8070680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:35438/0 (socket says 172.21.15.5:35438) 2024-03-24T10:45:16.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.703+0000 7f84a6d9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f84a80c7aa0 0x7f84a80c7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:16.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.703+0000 7f84a659c640 1 -- 172.21.15.5:0/1994305417 learned_addr learned my addr 172.21.15.5:0/1994305417 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:16.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.706+0000 7f84a559a640 1 -- 172.21.15.5:0/1994305417 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1439909845 0 0) 0x7f84a80c45b0 con 0x7f84a8151370 2024-03-24T10:45:16.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.706+0000 7f84a559a640 1 -- 172.21.15.5:0/1994305417 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8490003520 con 0x7f84a8151370 2024-03-24T10:45:16.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.707+0000 7f84a6d9d640 1 -- 172.21.15.5:0/1994305417 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f84a80c91b0 msgr2=0x7f84a80c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:16.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.707+0000 7f84a6d9d640 1 --2- 172.21.15.5:0/1994305417 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f84a80c91b0 0x7f84a80c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.707+0000 7f84a6d9d640 1 -- 172.21.15.5:0/1994305417 >> v1:172.21.15.5:6789/0 conn(0x7f84a8151370 legacy=0x7f84a8070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:16.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.707+0000 7f84a6d9d640 1 -- 172.21.15.5:0/1994305417 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84a80c9a90 con 0x7f84a80c7aa0 2024-03-24T10:45:16.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.710+0000 7f84a6d9d640 1 --2- 172.21.15.5:0/1994305417 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a80c7aa0 0x7f84a80c7f30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f84a0009d90 tx=0x7f84a002e060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f019a65e8798700 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:16.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.714+0000 7f84a559a640 1 -- 172.21.15.5:0/1994305417 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a002edf0 con 0x7f84a80c7aa0 2024-03-24T10:45:16.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.714+0000 7f84a559a640 1 -- 172.21.15.5:0/1994305417 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84a0036440 con 0x7f84a80c7aa0 2024-03-24T10:45:16.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.714+0000 7f84a559a640 1 -- 172.21.15.5:0/1994305417 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a0036810 con 0x7f84a80c7aa0 2024-03-24T10:45:16.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.715+0000 7f84accaf640 1 -- 172.21.15.5:0/1994305417 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a80c7aa0 msgr2=0x7f84a80c7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:16.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.715+0000 7f84accaf640 1 --2- 172.21.15.5:0/1994305417 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a80c7aa0 0x7f84a80c7f30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f84a0009d90 tx=0x7f84a002e060 comp rx=0 tx=0).stop 2024-03-24T10:45:16.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.715+0000 7f84accaf640 1 -- 172.21.15.5:0/1994305417 shutdown_connections 2024-03-24T10:45:16.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.715+0000 7f84accaf640 1 --2- 172.21.15.5:0/1994305417 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a80c7aa0 0x7f84a80c7f30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.715+0000 7f84accaf640 1 --2- 172.21.15.5:0/1994305417 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f84a80c91b0 0x7f84a80c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.715+0000 7f84accaf640 1 -- 172.21.15.5:0/1994305417 >> 172.21.15.5:0/1994305417 conn(0x7f84a806c240 msgr2=0x7f84a806c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:16.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.716+0000 7f84accaf640 1 -- 172.21.15.5:0/1994305417 shutdown_connections 2024-03-24T10:45:16.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.716+0000 7f84accaf640 1 -- 172.21.15.5:0/1994305417 wait complete. 2024-03-24T10:45:16.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.718+0000 7f84accaf640 1 Processor -- start 2024-03-24T10:45:16.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.718+0000 7f84accaf640 1 -- start start 2024-03-24T10:45:16.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.719+0000 7f84accaf640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f84a80c7aa0 0x7f84a8152270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.719+0000 7f84accaf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a80c91b0 0x7f84a8151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.719+0000 7f84accaf640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f84a8151370 0x7f84a80cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.719+0000 7f84accaf640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f84a80ccd40 con 0x7f84a8151370 2024-03-24T10:45:16.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.719+0000 7f84accaf640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f84a8154200 con 0x7f84a80c91b0 2024-03-24T10:45:16.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.719+0000 7f84accaf640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f84a80cd710 con 0x7f84a80c7aa0 2024-03-24T10:45:16.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.719+0000 7f84a5d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a80c91b0 0x7f84a8151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:16.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.720+0000 7f84a6d9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f84a8151370 0x7f84a80cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:16.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.720+0000 7f84a5d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a80c91b0 0x7f84a8151bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36676/0 (socket says 172.21.15.5:36676) 2024-03-24T10:45:16.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.720+0000 7f84a5d9b640 1 -- 172.21.15.5:0/3032203476 learned_addr learned my addr 172.21.15.5:0/3032203476 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:16.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.725+0000 7f84a5d9b640 1 -- 172.21.15.5:0/3032203476 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f84a80c7aa0 msgr2=0x7f84a8152270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:16.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.725+0000 7f84a5d9b640 1 --2- 172.21.15.5:0/3032203476 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f84a80c7aa0 0x7f84a8152270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.725+0000 7f84a5d9b640 1 -- 172.21.15.5:0/3032203476 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f84a8151370 msgr2=0x7f84a80cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:16.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.725+0000 7f84a5d9b640 1 --2- 172.21.15.5:0/3032203476 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f84a8151370 0x7f84a80cd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.725+0000 7f84a5d9b640 1 -- 172.21.15.5:0/3032203476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84a0009a70 con 0x7f84a80c91b0 2024-03-24T10:45:16.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.728+0000 7f84a5d9b640 1 --2- 172.21.15.5:0/3032203476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a80c91b0 0x7f84a8151bc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f849c00b4a0 tx=0x7f849c00b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:16.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.731+0000 7fc171ffb640 1 -- 172.21.15.5:0/4251742212 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc16c046960 con 0x7fc17c0717d0 2024-03-24T10:45:16.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.732+0000 7fc171ffb640 1 --2- 172.21.15.5:0/4251742212 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc16006b890 0x7fc16006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.732+0000 7fc1815d0640 1 --2- 172.21.15.5:0/4251742212 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc16006b890 0x7fc16006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:16.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.732+0000 7f8496ffd640 1 -- 172.21.15.5:0/3032203476 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f849c010610 con 0x7f84a80c91b0 2024-03-24T10:45:16.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.732+0000 7f8496ffd640 1 -- 172.21.15.5:0/3032203476 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f849c010770 con 0x7f84a80c91b0 2024-03-24T10:45:16.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.732+0000 7f8496ffd640 1 -- 172.21.15.5:0/3032203476 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f849c010b20 con 0x7f84a80c91b0 2024-03-24T10:45:16.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.733+0000 7f84accaf640 1 -- 172.21.15.5:0/3032203476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84a81544a0 con 0x7f84a80c91b0 2024-03-24T10:45:16.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.733+0000 7fc1815d0640 1 --2- 172.21.15.5:0/4251742212 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc16006b890 0x7fc16006dd50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc178000f60 tx=0x7fc178007a80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:16.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.733+0000 7f84accaf640 1 -- 172.21.15.5:0/3032203476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84a80cc9d0 con 0x7f84a80c91b0 2024-03-24T10:45:16.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.734+0000 7f84accaf640 1 -- 172.21.15.5:0/3032203476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84a80ccb60 con 0x7f84a80c91b0 2024-03-24T10:45:16.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.746+0000 7f565d4bc640 1 Processor -- start 2024-03-24T10:45:16.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.746+0000 7f565d4bc640 1 -- start start 2024-03-24T10:45:16.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.747+0000 7f565d4bc640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f56500d5dc0 0x7f565004d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.747+0000 7f565d4bc640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5650047e60 0x7f56500dc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.747+0000 7f565d4bc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5650007c30 0x7f5650046230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.747+0000 7f565d4bc640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f565004cc20 con 0x7f56500d5dc0 2024-03-24T10:45:16.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.747+0000 7f565d4bc640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f5650047110 con 0x7f5650007c30 2024-03-24T10:45:16.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.747+0000 7f565d4bc640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f565004c280 con 0x7f5650047e60 2024-03-24T10:45:16.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.748+0000 7f5657fff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f56500d5dc0 0x7f565004d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:16.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.748+0000 7f5657fff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f56500d5dc0 0x7f565004d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60206/0 (socket says 172.21.15.5:60206) 2024-03-24T10:45:16.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.748+0000 7f5657fff640 1 -- 172.21.15.5:0/2150036096 learned_addr learned my addr 172.21.15.5:0/2150036096 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:16.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.752+0000 7f5657fff640 1 -- 172.21.15.5:0/2150036096 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5650047e60 msgr2=0x7f56500dc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:16.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.752+0000 7f5657fff640 1 --2- 172.21.15.5:0/2150036096 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5650047e60 0x7f56500dc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.752+0000 7f5657fff640 1 -- 172.21.15.5:0/2150036096 >> v2:172.21.15.57:3300/0 conn(0x7f5650007c30 msgr2=0x7f5650046230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:16.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.752+0000 7f5657fff640 1 --2- 172.21.15.5:0/2150036096 >> v2:172.21.15.57:3300/0 conn(0x7f5650007c30 0x7f5650046230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.752+0000 7f5657fff640 1 -- 172.21.15.5:0/2150036096 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f565004c520 con 0x7f56500d5dc0 2024-03-24T10:45:16.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.755+0000 7f5657fff640 1 --2- 172.21.15.5:0/2150036096 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56500d5dc0 0x7f565004d7b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f56500484e0 tx=0x7f564c02b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28db12aae9e193a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:16.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.759+0000 7f5656ffd640 1 -- 172.21.15.5:0/2150036096 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564c030880 con 0x7f56500d5dc0 2024-03-24T10:45:16.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.759+0000 7f5656ffd640 1 -- 172.21.15.5:0/2150036096 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f564c0309e0 con 0x7f56500d5dc0 2024-03-24T10:45:16.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.759+0000 7f5656ffd640 1 -- 172.21.15.5:0/2150036096 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564c030d70 con 0x7f56500d5dc0 2024-03-24T10:45:16.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.760+0000 7f565d4bc640 1 -- 172.21.15.5:0/2150036096 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56500d5dc0 msgr2=0x7f565004d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:16.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.760+0000 7f565d4bc640 1 --2- 172.21.15.5:0/2150036096 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56500d5dc0 0x7f565004d7b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f56500484e0 tx=0x7f564c02b9c0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.760+0000 7f565d4bc640 1 -- 172.21.15.5:0/2150036096 shutdown_connections 2024-03-24T10:45:16.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.760+0000 7f565d4bc640 1 --2- 172.21.15.5:0/2150036096 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5650047e60 0x7f56500dc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.760+0000 7f565d4bc640 1 --2- 172.21.15.5:0/2150036096 >> v2:172.21.15.57:3300/0 conn(0x7f5650007c30 0x7f5650046230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.760+0000 7f565d4bc640 1 --2- 172.21.15.5:0/2150036096 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56500d5dc0 0x7f565004d7b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.760+0000 7f565d4bc640 1 -- 172.21.15.5:0/2150036096 >> 172.21.15.5:0/2150036096 conn(0x7f5650051290 msgr2=0x7f56500516a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:16.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.760+0000 7f565d4bc640 1 -- 172.21.15.5:0/2150036096 shutdown_connections 2024-03-24T10:45:16.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.760+0000 7f565d4bc640 1 -- 172.21.15.5:0/2150036096 wait complete. 2024-03-24T10:45:16.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.762+0000 7f565d4bc640 1 Processor -- start 2024-03-24T10:45:16.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.762+0000 7f565d4bc640 1 -- start start 2024-03-24T10:45:16.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.763+0000 7f565d4bc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5650007c30 0x7f5650048780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.763+0000 7f565d4bc640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5650047e60 0x7f56500df6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.763+0000 7f565d4bc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56500d5dc0 0x7f56500368f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.763+0000 7f565d4bc640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f56500cbe90 con 0x7f5650007c30 2024-03-24T10:45:16.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.763+0000 7f565d4bc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5650172df0 con 0x7f56500d5dc0 2024-03-24T10:45:16.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.763+0000 7f565d4bc640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f5650173320 con 0x7f5650047e60 2024-03-24T10:45:16.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.764+0000 7f5657fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5650007c30 0x7f5650048780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:16.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.764+0000 7f5657fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5650007c30 0x7f5650048780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60220/0 (socket says 172.21.15.5:60220) 2024-03-24T10:45:16.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.764+0000 7f5657fff640 1 -- 172.21.15.5:0/1583496135 learned_addr learned my addr 172.21.15.5:0/1583496135 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:16.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.770+0000 7f5657fff640 1 -- 172.21.15.5:0/1583496135 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5650047e60 msgr2=0x7f56500df6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:16.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.770+0000 7f5657fff640 1 --2- 172.21.15.5:0/1583496135 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5650047e60 0x7f56500df6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.770+0000 7f5657fff640 1 -- 172.21.15.5:0/1583496135 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56500d5dc0 msgr2=0x7f56500368f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:16.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.770+0000 7f5657fff640 1 --2- 172.21.15.5:0/1583496135 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56500d5dc0 0x7f56500368f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.770+0000 7f5657fff640 1 -- 172.21.15.5:0/1583496135 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f564c0094d0 con 0x7f5650007c30 2024-03-24T10:45:16.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.773+0000 7f5657fff640 1 --2- 172.21.15.5:0/1583496135 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5650007c30 0x7f5650048780 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f564c02ba00 tx=0x7f564c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:16.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.778+0000 7f5654ff9640 1 -- 172.21.15.5:0/1583496135 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564c030830 con 0x7f5650007c30 2024-03-24T10:45:16.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.778+0000 7f5654ff9640 1 -- 172.21.15.5:0/1583496135 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f564c043400 con 0x7f5650007c30 2024-03-24T10:45:16.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.778+0000 7f565d4bc640 1 -- 172.21.15.5:0/1583496135 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56500dfbe0 con 0x7f5650007c30 2024-03-24T10:45:16.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.778+0000 7f5654ff9640 1 -- 172.21.15.5:0/1583496135 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f564c0437b0 con 0x7f5650007c30 2024-03-24T10:45:16.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.778+0000 7f565d4bc640 1 -- 172.21.15.5:0/1583496135 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56500dd180 con 0x7f5650007c30 2024-03-24T10:45:16.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.779+0000 7f565d4bc640 1 -- 172.21.15.5:0/1583496135 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56500c9a90 con 0x7f5650007c30 2024-03-24T10:45:16.796 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-24T10:45:16.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.824+0000 7fbb5c1d6640 1 Processor -- start 2024-03-24T10:45:16.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.824+0000 7fbb5c1d6640 1 -- start start 2024-03-24T10:45:16.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.825+0000 7fbb5c1d6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbb54139ac0 0x7fbb540701b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.825+0000 7fbb5c1d6640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbb540706f0 0x7fbb5413c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.825+0000 7fbb5c1d6640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb5413c4e0 con 0x7fbb5413ceb0 2024-03-24T10:45:16.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.825+0000 7fbb5c1d6640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fbb5413d340 con 0x7fbb540706f0 2024-03-24T10:45:16.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.825+0000 7fbb5c1d6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fbb540ca110 con 0x7fbb54139ac0 2024-03-24T10:45:16.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.849+0000 7fc171ffb640 1 -- 172.21.15.5:0/4251742212 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc16c07bfb0 con 0x7fc17c0717d0 2024-03-24T10:45:16.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.869+0000 7fc171ffb640 1 -- 172.21.15.5:0/4251742212 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc16c077b50 con 0x7fc17c0717d0 2024-03-24T10:45:16.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.879+0000 7f5654ff9640 1 -- 172.21.15.5:0/1583496135 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f564c043910 con 0x7f5650007c30 2024-03-24T10:45:16.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.880+0000 7fbb59f72640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbb54139ac0 0x7fbb540701b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:16.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.880+0000 7f5654ff9640 1 --2- 172.21.15.5:0/1583496135 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f563c06b860 0x7f563c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.880+0000 7f56577fe640 1 --2- 172.21.15.5:0/1583496135 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f563c06b860 0x7f563c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:16.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.881+0000 7f56577fe640 1 --2- 172.21.15.5:0/1583496135 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f563c06b860 0x7f563c06dd20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f56501769a0 tx=0x7f5644006c90 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:16.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.881+0000 7fbb59f72640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbb54139ac0 0x7fbb540701b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:51622/0 (socket says 172.21.15.5:51622) 2024-03-24T10:45:16.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.881+0000 7fbb59f72640 1 -- 172.21.15.5:0/246545687 learned_addr learned my addr 172.21.15.5:0/246545687 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:16.883 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-24T10:45:16.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.885+0000 7fbb59f72640 1 -- 172.21.15.5:0/246545687 >> v2:172.21.15.5:3300/0 conn(0x7fbb540706f0 msgr2=0x7fbb5413c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:16.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.885+0000 7fbb59f72640 1 --2- 172.21.15.5:0/246545687 >> v2:172.21.15.5:3300/0 conn(0x7fbb540706f0 0x7fbb5413c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.885+0000 7fbb59f72640 1 -- 172.21.15.5:0/246545687 >> v1:172.21.15.5:6789/0 conn(0x7fbb5413ceb0 legacy=0x7fbb540c6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:16.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.885+0000 7fbb59f72640 1 -- 172.21.15.5:0/246545687 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb540c4460 con 0x7fbb54139ac0 2024-03-24T10:45:16.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.888+0000 7fbb59f72640 1 --2- 172.21.15.5:0/246545687 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbb54139ac0 0x7fbb540701b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbb50009870 tx=0x7fbb5002b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98fef8a6b2f3bdc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:16.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.894+0000 7fbb58f70640 1 -- 172.21.15.5:0/246545687 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb50031870 con 0x7fbb54139ac0 2024-03-24T10:45:16.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.894+0000 7fbb58f70640 1 -- 172.21.15.5:0/246545687 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb500319d0 con 0x7fbb54139ac0 2024-03-24T10:45:16.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.894+0000 7fbb58f70640 1 -- 172.21.15.5:0/246545687 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb50031d60 con 0x7fbb54139ac0 2024-03-24T10:45:16.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.895+0000 7fbb5c1d6640 1 -- 172.21.15.5:0/246545687 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbb54139ac0 msgr2=0x7fbb540701b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:16.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.895+0000 7fbb5c1d6640 1 --2- 172.21.15.5:0/246545687 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbb54139ac0 0x7fbb540701b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbb50009870 tx=0x7fbb5002b910 comp rx=0 tx=0).stop 2024-03-24T10:45:16.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.896+0000 7fbb5c1d6640 1 -- 172.21.15.5:0/246545687 shutdown_connections 2024-03-24T10:45:16.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.896+0000 7fbb5c1d6640 1 --2- 172.21.15.5:0/246545687 >> v2:172.21.15.5:3300/0 conn(0x7fbb540706f0 0x7fbb5413c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.896+0000 7fbb5c1d6640 1 --2- 172.21.15.5:0/246545687 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbb54139ac0 0x7fbb540701b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.896+0000 7fbb5c1d6640 1 -- 172.21.15.5:0/246545687 >> 172.21.15.5:0/246545687 conn(0x7fbb54061d30 msgr2=0x7fbb54072680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:16.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.896+0000 7fbb5c1d6640 1 -- 172.21.15.5:0/246545687 shutdown_connections 2024-03-24T10:45:16.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.896+0000 7fbb5c1d6640 1 -- 172.21.15.5:0/246545687 wait complete. 2024-03-24T10:45:16.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.898+0000 7fbb5c1d6640 1 Processor -- start 2024-03-24T10:45:16.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.899+0000 7fbb5c1d6640 1 -- start start 2024-03-24T10:45:16.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.900+0000 7fbb5c1d6640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb540706f0 0x7fbb541ee7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.900+0000 7fbb5c1d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb54139ac0 0x7fbb5412c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.900+0000 7fbb5c1d6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbb5413ceb0 0x7fbb540cd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.901+0000 7fbb5c1d6640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fbb541eecf0 con 0x7fbb540706f0 2024-03-24T10:45:16.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.901+0000 7fbb5c1d6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbb541392d0 con 0x7fbb54139ac0 2024-03-24T10:45:16.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.901+0000 7fbb5c1d6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fbb54151160 con 0x7fbb5413ceb0 2024-03-24T10:45:16.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.901+0000 7fbb5a773640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbb5413ceb0 0x7fbb540cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:16.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.903+0000 7fbb5a773640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbb5413ceb0 0x7fbb540cd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:51624/0 (socket says 172.21.15.5:51624) 2024-03-24T10:45:16.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.903+0000 7fbb5a773640 1 -- 172.21.15.5:0/1514786852 learned_addr learned my addr 172.21.15.5:0/1514786852 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:16.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.908+0000 7fbb5a773640 1 -- 172.21.15.5:0/1514786852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb54139ac0 msgr2=0x7fbb5412c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:16.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.908+0000 7fbb5a773640 1 --2- 172.21.15.5:0/1514786852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb54139ac0 0x7fbb5412c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.908+0000 7fbb5a773640 1 -- 172.21.15.5:0/1514786852 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb540706f0 msgr2=0x7fbb541ee7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:16.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.908+0000 7fbb5a773640 1 --2- 172.21.15.5:0/1514786852 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb540706f0 0x7fbb541ee7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:16.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.908+0000 7fbb5a773640 1 -- 172.21.15.5:0/1514786852 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb500094d0 con 0x7fbb5413ceb0 2024-03-24T10:45:16.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.911+0000 7fbb5a773640 1 --2- 172.21.15.5:0/1514786852 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbb5413ceb0 0x7fbb540cd050 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c00b990 tx=0x7fbb4c00be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:16.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.916+0000 7fbb4a7fc640 1 -- 172.21.15.5:0/1514786852 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c00cba0 con 0x7fbb5413ceb0 2024-03-24T10:45:16.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.916+0000 7fbb4a7fc640 1 -- 172.21.15.5:0/1514786852 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb4c00cd00 con 0x7fbb5413ceb0 2024-03-24T10:45:16.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.916+0000 7fbb4a7fc640 1 -- 172.21.15.5:0/1514786852 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c014680 con 0x7fbb5413ceb0 2024-03-24T10:45:16.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.916+0000 7fbb5c1d6640 1 -- 172.21.15.5:0/1514786852 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb541315e0 con 0x7fbb5413ceb0 2024-03-24T10:45:16.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.917+0000 7fbb5c1d6640 1 -- 172.21.15.5:0/1514786852 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb54138d10 con 0x7fbb5413ceb0 2024-03-24T10:45:16.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.918+0000 7fbb5c1d6640 1 -- 172.21.15.5:0/1514786852 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb54138b70 con 0x7fbb5413ceb0 2024-03-24T10:45:16.929 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-24T10:45:16.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.941+0000 7f5654ff9640 1 -- 172.21.15.5:0/1583496135 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f564c03e070 con 0x7f5650007c30 2024-03-24T10:45:16.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.958+0000 7fbb4a7fc640 1 -- 172.21.15.5:0/1514786852 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbb4c01d3b0 con 0x7fbb5413ceb0 2024-03-24T10:45:16.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.958+0000 7fbb4a7fc640 1 --2- 172.21.15.5:0/1514786852 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbb3806b860 0x7fbb3806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.959+0000 7fbb59f72640 1 --2- 172.21.15.5:0/1514786852 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbb3806b860 0x7fbb3806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:16.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.959+0000 7fbb59f72640 1 --2- 172.21.15.5:0/1514786852 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbb3806b860 0x7fbb3806dd20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbb50009870 tx=0x7fbb50030000 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:16.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.960+0000 7fbb4a7fc640 1 -- 172.21.15.5:0/1514786852 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbb4c059380 con 0x7fbb5413ceb0 2024-03-24T10:45:16.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.970+0000 7f6929c53640 1 Processor -- start 2024-03-24T10:45:16.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.970+0000 7f6929c53640 1 -- start start 2024-03-24T10:45:16.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.971+0000 7f6929c53640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f691c04b500 0x7f691c0dac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.971+0000 7f6929c53640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f691c04b8e0 0x7f691c005ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:16.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.971+0000 7f6929c53640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f691c007760 con 0x7f691c006420 2024-03-24T10:45:16.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.971+0000 7f6929c53640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f691c006900 con 0x7f691c04b8e0 2024-03-24T10:45:16.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.971+0000 7f6929c53640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f691c006be0 con 0x7f691c04b500 2024-03-24T10:45:16.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:16.976+0000 7fac5dd6c640 1 -- 172.21.15.5:0/2740686377 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fac58154ee0 con 0x7fac580cf340 2024-03-24T10:45:17.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.015+0000 7f6923fff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f691c04b8e0 0x7f691c005ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:17.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.015+0000 7f6929452640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f691c006420 0x7f691c034a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:35444/0 (socket says 172.21.15.5:35444) 2024-03-24T10:45:17.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.015+0000 7f6929452640 1 -- 172.21.15.5:0/4041535392 learned_addr learned my addr 172.21.15.5:0/4041535392 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:17.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.055+0000 7f8496ffd640 1 -- 172.21.15.5:0/3032203476 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f849c010c80 con 0x7f84a80c91b0 2024-03-24T10:45:17.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.055+0000 7f8496ffd640 1 --2- 172.21.15.5:0/3032203476 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f848806b810 0x7f848806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:17.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.056+0000 7f84a659c640 1 --2- 172.21.15.5:0/3032203476 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f848806b810 0x7f848806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:17.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.057+0000 7f84a659c640 1 --2- 172.21.15.5:0/3032203476 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f848806b810 0x7f848806dcd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8498001550 tx=0x7f84980078d0 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:17.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.057+0000 7f10c0f3e640 1 Processor -- start 2024-03-24T10:45:17.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.057+0000 7f10c0f3e640 1 -- start start 2024-03-24T10:45:17.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.058+0000 7f10c0f3e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f10bc0c82c0 0x7f10bc0c8730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:17.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.058+0000 7f10c0f3e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10bc158f40 con 0x7f10bc0cbf20 2024-03-24T10:45:17.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.058+0000 7f10c0f3e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10bc0c8c70 con 0x7f10bc0cbb40 2024-03-24T10:45:17.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.058+0000 7f10c0f3e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f10bc0c36d0 con 0x7f10bc0c82c0 2024-03-24T10:45:17.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.060+0000 7f8496ffd640 1 -- 172.21.15.5:0/3032203476 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f849c058a40 con 0x7f84a80c91b0 2024-03-24T10:45:17.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.080+0000 7f8496ffd640 1 -- 172.21.15.5:0/3032203476 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f849c054ce0 con 0x7f84a80c91b0 2024-03-24T10:45:17.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.093+0000 7f6928c51640 1 --2- 172.21.15.5:0/4041535392 >> v2:172.21.15.57:3300/0 conn(0x7f691c04b500 0x7f691c0dac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:17.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.095+0000 7f10ba59c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f10bc0cbb40 0x7f10bc0c2e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:47126/0 (socket says 172.21.15.5:47126) 2024-03-24T10:45:17.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.095+0000 7f10ba59c640 1 -- 172.21.15.5:0/3731124214 learned_addr learned my addr 172.21.15.5:0/3731124214 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:17.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.099+0000 7f6928c51640 1 -- 172.21.15.5:0/4041535392 >> v2:172.21.15.5:3300/0 conn(0x7f691c04b8e0 msgr2=0x7f691c005ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:17.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.099+0000 7f6928c51640 1 --2- 172.21.15.5:0/4041535392 >> v2:172.21.15.5:3300/0 conn(0x7f691c04b8e0 0x7f691c005ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.099+0000 7f6928c51640 1 -- 172.21.15.5:0/4041535392 >> v1:172.21.15.5:6789/0 conn(0x7f691c006420 legacy=0x7f691c034a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.099+0000 7f6928c51640 1 -- 172.21.15.5:0/4041535392 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f691c006ec0 con 0x7f691c04b500 2024-03-24T10:45:17.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.101+0000 7f4926b8a640 1 Processor -- start 2024-03-24T10:45:17.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.101+0000 7f4926b8a640 1 -- start start 2024-03-24T10:45:17.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.103+0000 7f4926b8a640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f49200cb5a0 0x7f49200cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:17.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.103+0000 7f4926b8a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4920167120 0x7f4920169510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:17.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.103+0000 7f4926b8a640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49200ca850 con 0x7f49200cc640 2024-03-24T10:45:17.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.103+0000 7f4926b8a640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f49200ccf60 con 0x7f4920167120 2024-03-24T10:45:17.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.103+0000 7f4926b8a640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f49200ce9c0 con 0x7f49200cb5a0 2024-03-24T10:45:17.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.104+0000 7f6928c51640 1 --2- 172.21.15.5:0/4041535392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f691c04b500 0x7f691c0dac60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f691c04a7c0 tx=0x7f691802cca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59ec84560617a5c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:17.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.104+0000 7f491ffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f49200cb5a0 0x7f49200cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:17.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.105+0000 7f491ffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f49200cb5a0 0x7f49200cca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36712/0 (socket says 172.21.15.5:36712) 2024-03-24T10:45:17.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.105+0000 7f491ffff640 1 -- 172.21.15.5:0/2716378251 learned_addr learned my addr 172.21.15.5:0/2716378251 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:17.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.110+0000 7f69237fe640 1 -- 172.21.15.5:0/4041535392 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f691802da50 con 0x7f691c04b500 2024-03-24T10:45:17.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.110+0000 7f69237fe640 1 -- 172.21.15.5:0/4041535392 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f691802dbb0 con 0x7f691c04b500 2024-03-24T10:45:17.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.110+0000 7f69237fe640 1 -- 172.21.15.5:0/4041535392 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6918035440 con 0x7f691c04b500 2024-03-24T10:45:17.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.111+0000 7f6929c53640 1 -- 172.21.15.5:0/4041535392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f691c04b500 msgr2=0x7f691c0dac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.111+0000 7f6929c53640 1 --2- 172.21.15.5:0/4041535392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f691c04b500 0x7f691c0dac60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f691c04a7c0 tx=0x7f691802cca0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.111+0000 7f6929c53640 1 -- 172.21.15.5:0/4041535392 shutdown_connections 2024-03-24T10:45:17.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.111+0000 7f6929c53640 1 --2- 172.21.15.5:0/4041535392 >> v2:172.21.15.5:3300/0 conn(0x7f691c04b8e0 0x7f691c005ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.111+0000 7f6929c53640 1 --2- 172.21.15.5:0/4041535392 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f691c04b500 0x7f691c0dac60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.111+0000 7f6929c53640 1 -- 172.21.15.5:0/4041535392 >> 172.21.15.5:0/4041535392 conn(0x7f691c050180 msgr2=0x7f691c050590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:17.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.111+0000 7f6929c53640 1 -- 172.21.15.5:0/4041535392 shutdown_connections 2024-03-24T10:45:17.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.112+0000 7f6929c53640 1 -- 172.21.15.5:0/4041535392 wait complete. 2024-03-24T10:45:17.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.112+0000 7f491ffff640 1 -- 172.21.15.5:0/2716378251 >> v2:172.21.15.5:3300/0 conn(0x7f4920167120 msgr2=0x7f4920169510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:17.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.112+0000 7f491ffff640 1 --2- 172.21.15.5:0/2716378251 >> v2:172.21.15.5:3300/0 conn(0x7f4920167120 0x7f4920169510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.112+0000 7f491ffff640 1 -- 172.21.15.5:0/2716378251 >> v1:172.21.15.5:6789/0 conn(0x7f49200cc640 legacy=0x7f49200ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:17.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.112+0000 7f491ffff640 1 -- 172.21.15.5:0/2716378251 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49200ca5b0 con 0x7f49200cb5a0 2024-03-24T10:45:17.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.113+0000 7f6929c53640 1 Processor -- start 2024-03-24T10:45:17.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.113+0000 7f6929c53640 1 -- start start 2024-03-24T10:45:17.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.115+0000 7f6929c53640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f691c006420 0x7f691c175410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:17.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.115+0000 7f6929c53640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f691c04b500 0x7f691c049d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:17.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.115+0000 7f6929c53640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f691c04b8e0 0x7f691c0d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:17.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.115+0000 7f6929c53640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f691c175950 con 0x7f691c006420 2024-03-24T10:45:17.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.115+0000 7f6929c53640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f691c0cf720 con 0x7f691c04b500 2024-03-24T10:45:17.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.115+0000 7f6929c53640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f691c04a250 con 0x7f691c04b8e0 2024-03-24T10:45:17.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.117+0000 7f491ffff640 1 --2- 172.21.15.5:0/2716378251 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49200cb5a0 0x7f49200cca20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f49200cd640 tx=0x7f490c02b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8654107bd8be4b87 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:17.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.122+0000 7f6923fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f691c04b500 0x7f691c049d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.123+0000 7f6923fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f691c04b500 0x7f691c049d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36714/0 (socket says 172.21.15.5:36714) 2024-03-24T10:45:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.123+0000 7f6923fff640 1 -- 172.21.15.5:0/2776200064 learned_addr learned my addr 172.21.15.5:0/2776200064 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.124+0000 7f491f7fe640 1 -- 172.21.15.5:0/2716378251 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f490c0317e0 con 0x7f49200cb5a0 2024-03-24T10:45:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.124+0000 7f491f7fe640 1 -- 172.21.15.5:0/2716378251 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f490c031940 con 0x7f49200cb5a0 2024-03-24T10:45:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.124+0000 7f491f7fe640 1 -- 172.21.15.5:0/2716378251 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f490c031cd0 con 0x7f49200cb5a0 2024-03-24T10:45:17.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.125+0000 7f4926b8a640 1 -- 172.21.15.5:0/2716378251 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49200cb5a0 msgr2=0x7f49200cca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.125+0000 7f4926b8a640 1 --2- 172.21.15.5:0/2716378251 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49200cb5a0 0x7f49200cca20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f49200cd640 tx=0x7f490c02b880 comp rx=0 tx=0).stop 2024-03-24T10:45:17.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.125+0000 7f4926b8a640 1 -- 172.21.15.5:0/2716378251 shutdown_connections 2024-03-24T10:45:17.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.125+0000 7f4926b8a640 1 --2- 172.21.15.5:0/2716378251 >> v2:172.21.15.5:3300/0 conn(0x7f4920167120 0x7f4920169510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.125+0000 7f4926b8a640 1 --2- 172.21.15.5:0/2716378251 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49200cb5a0 0x7f49200cca20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.125+0000 7f4926b8a640 1 -- 172.21.15.5:0/2716378251 >> 172.21.15.5:0/2716378251 conn(0x7f49200cfc30 msgr2=0x7f49200d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:17.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.125+0000 7f4926b8a640 1 -- 172.21.15.5:0/2716378251 shutdown_connections 2024-03-24T10:45:17.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.125+0000 7f4926b8a640 1 -- 172.21.15.5:0/2716378251 wait complete. 2024-03-24T10:45:17.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.127+0000 7f4926b8a640 1 Processor -- start 2024-03-24T10:45:17.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.127+0000 7f4926b8a640 1 -- start start 2024-03-24T10:45:17.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.128+0000 7f4926b8a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f49200cb5a0 0x7f49200c3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:17.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.128+0000 7f4926b8a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49200cc640 0x7f49200b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:17.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.128+0000 7f4926b8a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4920167120 0x7f49200c4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:17.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.128+0000 7f4926b8a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f49201f85c0 con 0x7f4920167120 2024-03-24T10:45:17.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.128+0000 7f4926b8a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4920163300 con 0x7f49200cc640 2024-03-24T10:45:17.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.128+0000 7f4926b8a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f49200c4390 con 0x7f49200cb5a0 2024-03-24T10:45:17.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.130+0000 7f6923fff640 1 -- 172.21.15.5:0/2776200064 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f691c04b8e0 msgr2=0x7f691c0d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.130+0000 7f6923fff640 1 --2- 172.21.15.5:0/2776200064 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f691c04b8e0 0x7f691c0d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.130+0000 7f6923fff640 1 -- 172.21.15.5:0/2776200064 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f691c006420 msgr2=0x7f691c175410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.130+0000 7f6923fff640 1 --2- 172.21.15.5:0/2776200064 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f691c006420 0x7f691c175410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.130+0000 7f6923fff640 1 -- 172.21.15.5:0/2776200064 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6918009400 con 0x7f691c04b500 2024-03-24T10:45:17.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.132+0000 7f491ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49200cc640 0x7f49200b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:17.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.133+0000 7f491ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49200cc640 0x7f49200b4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36728/0 (socket says 172.21.15.5:36728) 2024-03-24T10:45:17.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.133+0000 7f491ffff640 1 -- 172.21.15.5:0/2660253985 learned_addr learned my addr 172.21.15.5:0/2660253985 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:17.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.136+0000 7f6923fff640 1 --2- 172.21.15.5:0/2776200064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f691c04b500 0x7f691c049d10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6924068880 tx=0x7f6924068cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:17.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.140+0000 7f491ffff640 1 -- 172.21.15.5:0/2660253985 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f49200cb5a0 msgr2=0x7f49200c3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.140+0000 7f491ffff640 1 --2- 172.21.15.5:0/2660253985 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f49200cb5a0 0x7f49200c3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.140+0000 7f491ffff640 1 -- 172.21.15.5:0/2660253985 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4920167120 msgr2=0x7f49200c4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.140+0000 7f491ffff640 1 --2- 172.21.15.5:0/2660253985 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4920167120 0x7f49200c4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.140+0000 7f491ffff640 1 -- 172.21.15.5:0/2660253985 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f490c0094d0 con 0x7f49200cc640 2024-03-24T10:45:17.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.146+0000 7fac3ffff640 1 -- 172.21.15.5:0/2740686377 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fac50058b90 con 0x7fac580cf340 2024-03-24T10:45:17.146 INFO:teuthology.orchestra.run.smithi005.stdout:60129542154 2024-03-24T10:45:17.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.147+0000 7f491ffff640 1 --2- 172.21.15.5:0/2660253985 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49200cc640 0x7f49200b4ec0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f490c000c00 tx=0x7f490c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:17.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.148+0000 7f69217fa640 1 -- 172.21.15.5:0/2776200064 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692406c070 con 0x7f691c04b500 2024-03-24T10:45:17.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.148+0000 7f69217fa640 1 -- 172.21.15.5:0/2776200064 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f692406d040 con 0x7f691c04b500 2024-03-24T10:45:17.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.148+0000 7f69217fa640 1 -- 172.21.15.5:0/2776200064 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6924071a20 con 0x7f691c04b500 2024-03-24T10:45:17.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.148+0000 7f6929c53640 1 -- 172.21.15.5:0/2776200064 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f691c04a4a0 con 0x7f691c04b500 2024-03-24T10:45:17.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.148+0000 7f6929c53640 1 -- 172.21.15.5:0/2776200064 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f691c0d09e0 con 0x7f691c04b500 2024-03-24T10:45:17.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.149+0000 7fac5dd6c640 1 -- 172.21.15.5:0/2740686377 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac4406b810 msgr2=0x7fac4406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.149+0000 7fac5dd6c640 1 --2- 172.21.15.5:0/2740686377 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac4406b810 0x7fac4406dcd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fac480011e0 tx=0x7fac48007d30 comp rx=0 tx=0).stop 2024-03-24T10:45:17.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.149+0000 7fac5dd6c640 1 -- 172.21.15.5:0/2740686377 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac580cf340 msgr2=0x7fac5814ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.149+0000 7fac5dd6c640 1 --2- 172.21.15.5:0/2740686377 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac580cf340 0x7fac5814ff50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fac5000c770 tx=0x7fac5000cc30 comp rx=0 tx=0).stop 2024-03-24T10:45:17.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.150+0000 7f6929c53640 1 -- 172.21.15.5:0/2776200064 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6904005290 con 0x7f691c04b500 2024-03-24T10:45:17.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.151+0000 7fac5dd6c640 1 -- 172.21.15.5:0/2740686377 shutdown_connections 2024-03-24T10:45:17.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.151+0000 7fac5dd6c640 1 --2- 172.21.15.5:0/2740686377 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac4406b810 0x7fac4406dcd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.151+0000 7fac5dd6c640 1 --2- 172.21.15.5:0/2740686377 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac580cf340 0x7fac5814ff50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.151+0000 7fac5dd6c640 1 --2- 172.21.15.5:0/2740686377 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac580ccb80 0x7fac5814de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.151+0000 7fac5dd6c640 1 --2- 172.21.15.5:0/2740686377 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac580cc000 0x7fac581efc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.151+0000 7fac5dd6c640 1 -- 172.21.15.5:0/2740686377 >> 172.21.15.5:0/2740686377 conn(0x7fac580cfc30 msgr2=0x7fac580cda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:17.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.151+0000 7fac5dd6c640 1 -- 172.21.15.5:0/2740686377 shutdown_connections 2024-03-24T10:45:17.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.151+0000 7fac5dd6c640 1 -- 172.21.15.5:0/2740686377 wait complete. 2024-03-24T10:45:17.162 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542154 for osd.1 2024-03-24T10:45:17.162 DEBUG:teuthology.parallel:result is None 2024-03-24T10:45:17.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.167+0000 7f5654ff9640 1 -- 172.21.15.5:0/1583496135 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f564c078950 con 0x7f5650007c30 2024-03-24T10:45:17.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.169+0000 7f10bad9d640 1 --2- 172.21.15.5:0/3731124214 >> v2:172.21.15.5:3300/0 conn(0x7f10bc0c82c0 0x7f10bc0c8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:17.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.178+0000 7f69217fa640 1 -- 172.21.15.5:0/2776200064 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6924071b80 con 0x7f691c04b500 2024-03-24T10:45:17.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.179+0000 7f69217fa640 1 --2- 172.21.15.5:0/2776200064 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f690806b860 0x7f690806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:17.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.179+0000 7f6928c51640 1 --2- 172.21.15.5:0/2776200064 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f690806b860 0x7f690806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:17.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.180+0000 7f6928c51640 1 --2- 172.21.15.5:0/2776200064 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f690806b860 0x7f690806dd20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6918037000 tx=0x7f691802f000 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:17.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.185+0000 7f10b9d9b640 1 -- 172.21.15.5:0/3731124214 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2284542486 0 0) 0x7f10bc158f40 con 0x7f10bc0cbf20 2024-03-24T10:45:17.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.185+0000 7f10b9d9b640 1 -- 172.21.15.5:0/3731124214 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10a8003520 con 0x7f10bc0cbf20 2024-03-24T10:45:17.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.187+0000 7f10bad9d640 1 -- 172.21.15.5:0/3731124214 >> v1:172.21.15.57:6789/0 conn(0x7f10bc0cbb40 legacy=0x7f10bc0c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.187+0000 7f10bad9d640 1 -- 172.21.15.5:0/3731124214 >> v1:172.21.15.5:6789/0 conn(0x7f10bc0cbf20 legacy=0x7f10bc0c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.187+0000 7f10bad9d640 1 -- 172.21.15.5:0/3731124214 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10bc158330 con 0x7f10bc0c82c0 2024-03-24T10:45:17.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.191+0000 7f10bad9d640 1 --2- 172.21.15.5:0/3731124214 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10bc0c82c0 0x7f10bc0c8730 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f10b4009d90 tx=0x7f10b402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9808345c258932 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:17.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.199+0000 7f10b9d9b640 1 -- 172.21.15.5:0/3731124214 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b402cdc0 con 0x7f10bc0c82c0 2024-03-24T10:45:17.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.199+0000 7f10b9d9b640 1 -- 172.21.15.5:0/3731124214 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10b4034440 con 0x7f10bc0c82c0 2024-03-24T10:45:17.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.199+0000 7f10b9d9b640 1 -- 172.21.15.5:0/3731124214 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b40347d0 con 0x7f10bc0c82c0 2024-03-24T10:45:17.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.200+0000 7f10c0f3e640 1 -- 172.21.15.5:0/3731124214 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10bc0c82c0 msgr2=0x7f10bc0c8730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.200+0000 7f10c0f3e640 1 --2- 172.21.15.5:0/3731124214 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10bc0c82c0 0x7f10bc0c8730 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f10b4009d90 tx=0x7f10b402c010 comp rx=0 tx=0).stop 2024-03-24T10:45:17.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.201+0000 7f10c0f3e640 1 -- 172.21.15.5:0/3731124214 shutdown_connections 2024-03-24T10:45:17.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.201+0000 7f10c0f3e640 1 --2- 172.21.15.5:0/3731124214 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10bc0c82c0 0x7f10bc0c8730 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.201+0000 7f10c0f3e640 1 -- 172.21.15.5:0/3731124214 >> 172.21.15.5:0/3731124214 conn(0x7f10bc06f180 msgr2=0x7f10bc06f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:17.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.201+0000 7f10c0f3e640 1 -- 172.21.15.5:0/3731124214 shutdown_connections 2024-03-24T10:45:17.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.201+0000 7f10c0f3e640 1 -- 172.21.15.5:0/3731124214 wait complete. 2024-03-24T10:45:17.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.203+0000 7f10c0f3e640 1 Processor -- start 2024-03-24T10:45:17.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.203+0000 7f10c0f3e640 1 -- start start 2024-03-24T10:45:17.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.204+0000 7f10c0f3e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10bc0c82c0 0x7f10bc14fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:17.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.204+0000 7f10c0f3e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f10bc0cbb40 0x7f10bc14c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:17.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.204+0000 7f10c0f3e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10bc0cbf20 0x7f10bc1f32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:17.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.204+0000 7f10c0f3e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f10bc15c420 con 0x7f10bc0c82c0 2024-03-24T10:45:17.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.205+0000 7f10c0f3e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f10bc15b880 con 0x7f10bc0cbf20 2024-03-24T10:45:17.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.205+0000 7f10c0f3e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f10bc15b460 con 0x7f10bc0cbb40 2024-03-24T10:45:17.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.205+0000 7f10ba59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10bc0c82c0 0x7f10bc14fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:17.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.205+0000 7f10ba59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10bc0c82c0 0x7f10bc14fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60304/0 (socket says 172.21.15.5:60304) 2024-03-24T10:45:17.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.205+0000 7f10ba59c640 1 -- 172.21.15.5:0/1293524197 learned_addr learned my addr 172.21.15.5:0/1293524197 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:17.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.206+0000 7fc183834640 1 -- 172.21.15.5:0/4251742212 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fc17c0c3d10 con 0x7fc17c0717d0 2024-03-24T10:45:17.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.211+0000 7f10ba59c640 1 -- 172.21.15.5:0/1293524197 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f10bc0cbb40 msgr2=0x7f10bc14c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.211+0000 7f10ba59c640 1 --2- 172.21.15.5:0/1293524197 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f10bc0cbb40 0x7f10bc14c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.211+0000 7f10ba59c640 1 -- 172.21.15.5:0/1293524197 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10bc0cbf20 msgr2=0x7f10bc1f32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.211+0000 7f10ba59c640 1 --2- 172.21.15.5:0/1293524197 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10bc0cbf20 0x7f10bc1f32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.211+0000 7f10ba59c640 1 -- 172.21.15.5:0/1293524197 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b4009a70 con 0x7f10bc0c82c0 2024-03-24T10:45:17.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.214+0000 7f10ba59c640 1 --2- 172.21.15.5:0/1293524197 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10bc0c82c0 0x7f10bc14fc60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f10ac00c4d0 tx=0x7f10ac00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:17.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.218+0000 7f10b37fe640 1 -- 172.21.15.5:0/1293524197 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ac005ce0 con 0x7f10bc0c82c0 2024-03-24T10:45:17.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.218+0000 7f10c0f3e640 1 -- 172.21.15.5:0/1293524197 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10bc071990 con 0x7f10bc0c82c0 2024-03-24T10:45:17.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.218+0000 7f10c0f3e640 1 -- 172.21.15.5:0/1293524197 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10bc15bdd0 con 0x7f10bc0c82c0 2024-03-24T10:45:17.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.219+0000 7f10c0f3e640 1 -- 172.21.15.5:0/1293524197 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10bc0681d0 con 0x7f10bc0c82c0 2024-03-24T10:45:17.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.262+0000 7fbb4a7fc640 1 -- 172.21.15.5:0/1514786852 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbb4c054f20 con 0x7fbb5413ceb0 2024-03-24T10:45:17.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.305+0000 7fc171ffb640 1 -- 172.21.15.5:0/4251742212 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc16c07b210 con 0x7fc17c0717d0 2024-03-24T10:45:17.305 INFO:teuthology.orchestra.run.smithi005.stdout:60129542154 2024-03-24T10:45:17.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.308+0000 7fc183834640 1 -- 172.21.15.5:0/4251742212 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc16006b890 msgr2=0x7fc16006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.308+0000 7fc183834640 1 --2- 172.21.15.5:0/4251742212 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc16006b890 0x7fc16006dd50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc178000f60 tx=0x7fc178007a80 comp rx=0 tx=0).stop 2024-03-24T10:45:17.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.308+0000 7fc183834640 1 -- 172.21.15.5:0/4251742212 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc17c0717d0 msgr2=0x7fc17c1f3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.308+0000 7fc183834640 1 --2- 172.21.15.5:0/4251742212 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc17c0717d0 0x7fc17c1f3a60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc16c02bf40 tx=0x7fc16c0046c0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.308+0000 7fc183834640 1 -- 172.21.15.5:0/4251742212 shutdown_connections 2024-03-24T10:45:17.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.308+0000 7fc183834640 1 --2- 172.21.15.5:0/4251742212 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc16006b890 0x7fc16006dd50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.308+0000 7fc183834640 1 --2- 172.21.15.5:0/4251742212 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc17c0ce300 0x7fc17c0ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.308+0000 7fc183834640 1 --2- 172.21.15.5:0/4251742212 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc17c0717d0 0x7fc17c1f3a60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.308+0000 7fc183834640 1 --2- 172.21.15.5:0/4251742212 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc17c0713f0 0x7fc17c06cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.308+0000 7fc183834640 1 -- 172.21.15.5:0/4251742212 >> 172.21.15.5:0/4251742212 conn(0x7fc17c06b760 msgr2=0x7fc17c150370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:17.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.308+0000 7fc183834640 1 -- 172.21.15.5:0/4251742212 shutdown_connections 2024-03-24T10:45:17.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.308+0000 7fc183834640 1 -- 172.21.15.5:0/4251742212 wait complete. 2024-03-24T10:45:17.325 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542154 for osd.2 2024-03-24T10:45:17.325 DEBUG:teuthology.parallel:result is None 2024-03-24T10:45:17.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.334+0000 7f10b37fe640 1 -- 172.21.15.5:0/1293524197 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10ac005e40 con 0x7f10bc0c82c0 2024-03-24T10:45:17.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.334+0000 7f10b37fe640 1 -- 172.21.15.5:0/1293524197 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ac007270 con 0x7f10bc0c82c0 2024-03-24T10:45:17.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.335+0000 7f10b37fe640 1 -- 172.21.15.5:0/1293524197 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f10ac007490 con 0x7f10bc0c82c0 2024-03-24T10:45:17.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.336+0000 7f10b37fe640 1 --2- 172.21.15.5:0/1293524197 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f109c06b810 0x7f109c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:17.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.336+0000 7f10b1d9b640 1 --2- 172.21.15.5:0/1293524197 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f109c06b810 0x7f109c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:17.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.337+0000 7f10b1d9b640 1 --2- 172.21.15.5:0/1293524197 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f109c06b810 0x7f109c06dcd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f10bc14b120 tx=0x7f10a40078a0 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:17.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.371+0000 7f491d7fa640 1 -- 172.21.15.5:0/2660253985 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f490c032880 con 0x7f49200cc640 2024-03-24T10:45:17.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.371+0000 7f4926b8a640 1 -- 172.21.15.5:0/2660253985 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49201f7c90 con 0x7f49200cc640 2024-03-24T10:45:17.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.371+0000 7f4926b8a640 1 -- 172.21.15.5:0/2660253985 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49201f8250 con 0x7f49200cc640 2024-03-24T10:45:17.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.372+0000 7f4926b8a640 1 -- 172.21.15.5:0/2660253985 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49200cc4b0 con 0x7f49200cc640 2024-03-24T10:45:17.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.435+0000 7f84accaf640 1 -- 172.21.15.5:0/3032203476 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f84a80cceb0 con 0x7f84a80c91b0 2024-03-24T10:45:17.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.436+0000 7f69217fa640 1 -- 172.21.15.5:0/2776200064 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6924071e30 con 0x7f691c04b500 2024-03-24T10:45:17.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.437+0000 7f491d7fa640 1 -- 172.21.15.5:0/2660253985 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f490c0329e0 con 0x7f49200cc640 2024-03-24T10:45:17.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.437+0000 7f491d7fa640 1 -- 172.21.15.5:0/2660253985 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f490c032d50 con 0x7f49200cc640 2024-03-24T10:45:17.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.447+0000 7f69217fa640 1 -- 172.21.15.5:0/2776200064 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f69240d08b0 con 0x7f691c04b500 2024-03-24T10:45:17.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.487+0000 7f565d4bc640 1 -- 172.21.15.5:0/1583496135 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f56500c9a90 con 0x7f5650007c30 2024-03-24T10:45:17.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.543+0000 7f10b37fe640 1 -- 172.21.15.5:0/1293524197 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f10ac058720 con 0x7f10bc0c82c0 2024-03-24T10:45:17.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.565+0000 7f10b37fe640 1 -- 172.21.15.5:0/1293524197 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f10ac0549c0 con 0x7f10bc0c82c0 2024-03-24T10:45:17.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.568+0000 7fbb5c1d6640 1 -- 172.21.15.5:0/1514786852 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fbb54159350 con 0x7fbb5413ceb0 2024-03-24T10:45:17.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.573+0000 7fbb4a7fc640 1 -- 172.21.15.5:0/1514786852 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbb54159350 con 0x7fbb5413ceb0 2024-03-24T10:45:17.573 INFO:teuthology.orchestra.run.smithi005.stdout:60129542154 2024-03-24T10:45:17.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.575+0000 7fbb5c1d6640 1 -- 172.21.15.5:0/1514786852 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbb3806b860 msgr2=0x7fbb3806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.575+0000 7fbb5c1d6640 1 --2- 172.21.15.5:0/1514786852 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbb3806b860 0x7fbb3806dd20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbb50009870 tx=0x7fbb50030000 comp rx=0 tx=0).stop 2024-03-24T10:45:17.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.575+0000 7fbb5c1d6640 1 -- 172.21.15.5:0/1514786852 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbb5413ceb0 msgr2=0x7fbb540cd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.575+0000 7fbb5c1d6640 1 --2- 172.21.15.5:0/1514786852 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbb5413ceb0 0x7fbb540cd050 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c00b990 tx=0x7fbb4c00be50 comp rx=0 tx=0).stop 2024-03-24T10:45:17.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.575+0000 7fbb5c1d6640 1 -- 172.21.15.5:0/1514786852 shutdown_connections 2024-03-24T10:45:17.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.575+0000 7fbb5c1d6640 1 --2- 172.21.15.5:0/1514786852 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbb3806b860 0x7fbb3806dd20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.575+0000 7fbb5c1d6640 1 --2- 172.21.15.5:0/1514786852 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbb5413ceb0 0x7fbb540cd050 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.575+0000 7fbb5c1d6640 1 --2- 172.21.15.5:0/1514786852 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb54139ac0 0x7fbb5412c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.575+0000 7fbb5c1d6640 1 --2- 172.21.15.5:0/1514786852 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb540706f0 0x7fbb541ee7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.575+0000 7fbb5c1d6640 1 -- 172.21.15.5:0/1514786852 >> 172.21.15.5:0/1514786852 conn(0x7fbb54061d30 msgr2=0x7fbb5406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:17.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.575+0000 7fbb5c1d6640 1 -- 172.21.15.5:0/1514786852 shutdown_connections 2024-03-24T10:45:17.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.575+0000 7fbb5c1d6640 1 -- 172.21.15.5:0/1514786852 wait complete. 2024-03-24T10:45:17.586 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542154 for osd.5 2024-03-24T10:45:17.586 DEBUG:teuthology.parallel:result is None 2024-03-24T10:45:17.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.601+0000 7f5654ff9640 1 -- 172.21.15.5:0/1583496135 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f56500c9a90 con 0x7f5650007c30 2024-03-24T10:45:17.601 INFO:teuthology.orchestra.run.smithi005.stdout:60129542153 2024-03-24T10:45:17.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.603+0000 7f565d4bc640 1 -- 172.21.15.5:0/1583496135 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f563c06b860 msgr2=0x7f563c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.603+0000 7f565d4bc640 1 --2- 172.21.15.5:0/1583496135 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f563c06b860 0x7f563c06dd20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f56501769a0 tx=0x7f5644006c90 comp rx=0 tx=0).stop 2024-03-24T10:45:17.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.603+0000 7f565d4bc640 1 -- 172.21.15.5:0/1583496135 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5650007c30 msgr2=0x7f5650048780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.603+0000 7f565d4bc640 1 --2- 172.21.15.5:0/1583496135 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5650007c30 0x7f5650048780 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f564c02ba00 tx=0x7f564c004730 comp rx=0 tx=0).stop 2024-03-24T10:45:17.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.603+0000 7f565d4bc640 1 -- 172.21.15.5:0/1583496135 shutdown_connections 2024-03-24T10:45:17.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.603+0000 7f565d4bc640 1 --2- 172.21.15.5:0/1583496135 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f563c06b860 0x7f563c06dd20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.603+0000 7f565d4bc640 1 --2- 172.21.15.5:0/1583496135 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56500d5dc0 0x7f56500368f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.603+0000 7f565d4bc640 1 --2- 172.21.15.5:0/1583496135 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5650047e60 0x7f56500df6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.603+0000 7f565d4bc640 1 --2- 172.21.15.5:0/1583496135 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5650007c30 0x7f5650048780 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.603+0000 7f565d4bc640 1 -- 172.21.15.5:0/1583496135 >> 172.21.15.5:0/1583496135 conn(0x7f5650051290 msgr2=0x7f56500d4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:17.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.603+0000 7f565d4bc640 1 -- 172.21.15.5:0/1583496135 shutdown_connections 2024-03-24T10:45:17.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.603+0000 7f565d4bc640 1 -- 172.21.15.5:0/1583496135 wait complete. 2024-03-24T10:45:17.616 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.3 2024-03-24T10:45:17.616 DEBUG:teuthology.parallel:result is None 2024-03-24T10:45:17.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.772+0000 7f491d7fa640 1 -- 172.21.15.5:0/2660253985 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f490c04d3f0 con 0x7f49200cc640 2024-03-24T10:45:17.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.773+0000 7f491d7fa640 1 --2- 172.21.15.5:0/2660253985 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f490406b810 0x7f490406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:17.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.773+0000 7f4924926640 1 --2- 172.21.15.5:0/2660253985 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f490406b810 0x7f490406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:17.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.774+0000 7f4924926640 1 --2- 172.21.15.5:0/2660253985 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f490406b810 0x7f490406dcd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4914001560 tx=0x7f4914007990 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:17.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.776+0000 7f491d7fa640 1 -- 172.21.15.5:0/2660253985 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f490c07b2f0 con 0x7f49200cc640 2024-03-24T10:45:17.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.779+0000 7f6929c53640 1 -- 172.21.15.5:0/2776200064 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f6904005810 con 0x7f691c04b500 2024-03-24T10:45:17.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.782+0000 7f10c0f3e640 1 -- 172.21.15.5:0/1293524197 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f10bc1575b0 con 0x7f10bc0c82c0 2024-03-24T10:45:17.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.790+0000 7f491d7fa640 1 -- 172.21.15.5:0/2660253985 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f490c076e90 con 0x7f49200cc640 2024-03-24T10:45:17.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.792+0000 7f8496ffd640 1 -- 172.21.15.5:0/3032203476 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f849c01e070 con 0x7f84a80c91b0 2024-03-24T10:45:17.792 INFO:teuthology.orchestra.run.smithi005.stdout:55834574857 2024-03-24T10:45:17.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.795+0000 7f84accaf640 1 -- 172.21.15.5:0/3032203476 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f848806b810 msgr2=0x7f848806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.795+0000 7f84accaf640 1 --2- 172.21.15.5:0/3032203476 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f848806b810 0x7f848806dcd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8498001550 tx=0x7f84980078d0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.795+0000 7f84accaf640 1 -- 172.21.15.5:0/3032203476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a80c91b0 msgr2=0x7f84a8151bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.795+0000 7f84accaf640 1 --2- 172.21.15.5:0/3032203476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a80c91b0 0x7f84a8151bc0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f849c00b4a0 tx=0x7f849c00b960 comp rx=0 tx=0).stop 2024-03-24T10:45:17.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.795+0000 7f84accaf640 1 -- 172.21.15.5:0/3032203476 shutdown_connections 2024-03-24T10:45:17.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.795+0000 7f84accaf640 1 --2- 172.21.15.5:0/3032203476 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f848806b810 0x7f848806dcd0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.795+0000 7f84accaf640 1 --2- 172.21.15.5:0/3032203476 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f84a8151370 0x7f84a80cd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.795+0000 7f84accaf640 1 --2- 172.21.15.5:0/3032203476 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f84a80c91b0 0x7f84a8151bc0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.795+0000 7f84accaf640 1 --2- 172.21.15.5:0/3032203476 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f84a80c7aa0 0x7f84a8152270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.795+0000 7f84accaf640 1 -- 172.21.15.5:0/3032203476 >> 172.21.15.5:0/3032203476 conn(0x7f84a806c240 msgr2=0x7f84a80c7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:17.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.795+0000 7f84accaf640 1 -- 172.21.15.5:0/3032203476 shutdown_connections 2024-03-24T10:45:17.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.795+0000 7f84accaf640 1 -- 172.21.15.5:0/3032203476 wait complete. 2024-03-24T10:45:17.810 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.6 2024-03-24T10:45:17.810 DEBUG:teuthology.parallel:result is None 2024-03-24T10:45:17.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.812+0000 7f10b37fe640 1 -- 172.21.15.5:0/1293524197 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f10ac017090 con 0x7f10bc0c82c0 2024-03-24T10:45:17.812 INFO:teuthology.orchestra.run.smithi005.stdout:60129542154 2024-03-24T10:45:17.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.814+0000 7f10c0f3e640 1 -- 172.21.15.5:0/1293524197 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f109c06b810 msgr2=0x7f109c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.814+0000 7f10c0f3e640 1 --2- 172.21.15.5:0/1293524197 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f109c06b810 0x7f109c06dcd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f10bc14b120 tx=0x7f10a40078a0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.814+0000 7f10c0f3e640 1 -- 172.21.15.5:0/1293524197 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10bc0c82c0 msgr2=0x7f10bc14fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.814+0000 7f10c0f3e640 1 --2- 172.21.15.5:0/1293524197 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10bc0c82c0 0x7f10bc14fc60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f10ac00c4d0 tx=0x7f10ac00c990 comp rx=0 tx=0).stop 2024-03-24T10:45:17.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.814+0000 7f10c0f3e640 1 -- 172.21.15.5:0/1293524197 shutdown_connections 2024-03-24T10:45:17.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.814+0000 7f10c0f3e640 1 --2- 172.21.15.5:0/1293524197 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f109c06b810 0x7f109c06dcd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.814+0000 7f10c0f3e640 1 --2- 172.21.15.5:0/1293524197 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10bc0cbf20 0x7f10bc1f32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.814+0000 7f10c0f3e640 1 --2- 172.21.15.5:0/1293524197 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f10bc0cbb40 0x7f10bc14c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.814+0000 7f10c0f3e640 1 --2- 172.21.15.5:0/1293524197 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f10bc0c82c0 0x7f10bc14fc60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.814+0000 7f10c0f3e640 1 -- 172.21.15.5:0/1293524197 >> 172.21.15.5:0/1293524197 conn(0x7f10bc06f180 msgr2=0x7f10bc06f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:17.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.814+0000 7f10c0f3e640 1 -- 172.21.15.5:0/1293524197 shutdown_connections 2024-03-24T10:45:17.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.814+0000 7f10c0f3e640 1 -- 172.21.15.5:0/1293524197 wait complete. 2024-03-24T10:45:17.826 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542154 for osd.7 2024-03-24T10:45:17.826 DEBUG:teuthology.parallel:result is None 2024-03-24T10:45:17.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.857+0000 7f69217fa640 1 -- 172.21.15.5:0/2776200064 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6924075030 con 0x7f691c04b500 2024-03-24T10:45:17.857 INFO:teuthology.orchestra.run.smithi005.stdout:55834574857 2024-03-24T10:45:17.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.860+0000 7f6929c53640 1 -- 172.21.15.5:0/2776200064 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f690806b860 msgr2=0x7f690806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.860+0000 7f6929c53640 1 --2- 172.21.15.5:0/2776200064 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f690806b860 0x7f690806dd20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6918037000 tx=0x7f691802f000 comp rx=0 tx=0).stop 2024-03-24T10:45:17.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.860+0000 7f6929c53640 1 -- 172.21.15.5:0/2776200064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f691c04b500 msgr2=0x7f691c049d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:17.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.860+0000 7f6929c53640 1 --2- 172.21.15.5:0/2776200064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f691c04b500 0x7f691c049d10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6924068880 tx=0x7f6924068cb0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.860+0000 7f6929c53640 1 -- 172.21.15.5:0/2776200064 shutdown_connections 2024-03-24T10:45:17.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.860+0000 7f6929c53640 1 --2- 172.21.15.5:0/2776200064 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f690806b860 0x7f690806dd20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.860+0000 7f6929c53640 1 --2- 172.21.15.5:0/2776200064 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f691c04b8e0 0x7f691c0d0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.860+0000 7f6929c53640 1 --2- 172.21.15.5:0/2776200064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f691c04b500 0x7f691c049d10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.860+0000 7f6929c53640 1 --2- 172.21.15.5:0/2776200064 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f691c006420 0x7f691c175410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:17.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.860+0000 7f6929c53640 1 -- 172.21.15.5:0/2776200064 >> 172.21.15.5:0/2776200064 conn(0x7f691c050180 msgr2=0x7f691c0d3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:17.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.860+0000 7f6929c53640 1 -- 172.21.15.5:0/2776200064 shutdown_connections 2024-03-24T10:45:17.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:17.860+0000 7f6929c53640 1 -- 172.21.15.5:0/2776200064 wait complete. 2024-03-24T10:45:17.872 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574857 for osd.0 2024-03-24T10:45:17.872 DEBUG:teuthology.parallel:result is None 2024-03-24T10:45:18.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.004+0000 7f4926b8a640 1 -- 172.21.15.5:0/2660253985 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f49200c8dc0 con 0x7f49200cc640 2024-03-24T10:45:18.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.009+0000 7f491d7fa640 1 -- 172.21.15.5:0/2660253985 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f490c07a550 con 0x7f49200cc640 2024-03-24T10:45:18.010 INFO:teuthology.orchestra.run.smithi005.stdout:55834574857 2024-03-24T10:45:18.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.012+0000 7f4926b8a640 1 -- 172.21.15.5:0/2660253985 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f490406b810 msgr2=0x7f490406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:18.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.012+0000 7f4926b8a640 1 --2- 172.21.15.5:0/2660253985 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f490406b810 0x7f490406dcd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4914001560 tx=0x7f4914007990 comp rx=0 tx=0).stop 2024-03-24T10:45:18.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.012+0000 7f4926b8a640 1 -- 172.21.15.5:0/2660253985 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49200cc640 msgr2=0x7f49200b4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:18.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.012+0000 7f4926b8a640 1 --2- 172.21.15.5:0/2660253985 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49200cc640 0x7f49200b4ec0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f490c000c00 tx=0x7f490c0046c0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.013+0000 7f4926b8a640 1 -- 172.21.15.5:0/2660253985 shutdown_connections 2024-03-24T10:45:18.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.013+0000 7f4926b8a640 1 --2- 172.21.15.5:0/2660253985 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f490406b810 0x7f490406dcd0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.013+0000 7f4926b8a640 1 --2- 172.21.15.5:0/2660253985 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4920167120 0x7f49200c4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.013+0000 7f4926b8a640 1 --2- 172.21.15.5:0/2660253985 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49200cc640 0x7f49200b4ec0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.013+0000 7f4926b8a640 1 --2- 172.21.15.5:0/2660253985 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f49200cb5a0 0x7f49200c3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.013+0000 7f4926b8a640 1 -- 172.21.15.5:0/2660253985 >> 172.21.15.5:0/2660253985 conn(0x7f49200cfc30 msgr2=0x7f4920162060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:18.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.013+0000 7f4926b8a640 1 -- 172.21.15.5:0/2660253985 shutdown_connections 2024-03-24T10:45:18.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.013+0000 7f4926b8a640 1 -- 172.21.15.5:0/2660253985 wait complete. 2024-03-24T10:45:18.027 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.4 2024-03-24T10:45:18.027 DEBUG:teuthology.parallel:result is None 2024-03-24T10:45:18.027 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-24T10:45:18.028 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:45:18.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.128+0000 7f32787c7640 1 Processor -- start 2024-03-24T10:45:18.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.128+0000 7f32787c7640 1 -- start start 2024-03-24T10:45:18.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.129+0000 7f32787c7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f327006bf50 0x7f327006c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.129+0000 7f32787c7640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32700cb2f0 con 0x7f32701543a0 2024-03-24T10:45:18.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.129+0000 7f32787c7640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f327015b070 con 0x7f327015ac00 2024-03-24T10:45:18.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.129+0000 7f32787c7640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f32700c6750 con 0x7f327006bf50 2024-03-24T10:45:18.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.130+0000 7f3276d64640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f327006bf50 0x7f327006c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.130+0000 7f3275d62640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f327015ac00 0x7f32700cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:47138/0 (socket says 172.21.15.5:47138) 2024-03-24T10:45:18.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.130+0000 7f3275d62640 1 -- 172.21.15.5:0/3255786413 learned_addr learned my addr 172.21.15.5:0/3255786413 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:18.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.133+0000 7f3276d64640 1 -- 172.21.15.5:0/3255786413 >> v1:172.21.15.57:6789/0 conn(0x7f327015ac00 legacy=0x7f32700cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:18.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.133+0000 7f3276d64640 1 -- 172.21.15.5:0/3255786413 >> v1:172.21.15.5:6789/0 conn(0x7f32701543a0 legacy=0x7f327006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:18.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.133+0000 7f3276d64640 1 -- 172.21.15.5:0/3255786413 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32700ca7e0 con 0x7f327006bf50 2024-03-24T10:45:18.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.135+0000 7f3276d64640 1 --2- 172.21.15.5:0/3255786413 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f327006bf50 0x7f327006c330 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3268009d90 tx=0x7f326802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cd76dc7937479a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:18.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.140+0000 7f3275561640 1 -- 172.21.15.5:0/3255786413 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326802cc60 con 0x7f327006bf50 2024-03-24T10:45:18.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.140+0000 7f3275561640 1 -- 172.21.15.5:0/3255786413 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f326802cdc0 con 0x7f327006bf50 2024-03-24T10:45:18.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.140+0000 7f3275561640 1 -- 172.21.15.5:0/3255786413 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3268034660 con 0x7f327006bf50 2024-03-24T10:45:18.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.141+0000 7f32787c7640 1 -- 172.21.15.5:0/3255786413 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f327006bf50 msgr2=0x7f327006c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:18.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.141+0000 7f32787c7640 1 --2- 172.21.15.5:0/3255786413 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f327006bf50 0x7f327006c330 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3268009d90 tx=0x7f326802beb0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.141+0000 7f32787c7640 1 -- 172.21.15.5:0/3255786413 shutdown_connections 2024-03-24T10:45:18.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.141+0000 7f32787c7640 1 --2- 172.21.15.5:0/3255786413 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f327006bf50 0x7f327006c330 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.141+0000 7f32787c7640 1 -- 172.21.15.5:0/3255786413 >> 172.21.15.5:0/3255786413 conn(0x7f3270072ef0 msgr2=0x7f3270061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:18.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.141+0000 7f32787c7640 1 -- 172.21.15.5:0/3255786413 shutdown_connections 2024-03-24T10:45:18.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.141+0000 7f32787c7640 1 -- 172.21.15.5:0/3255786413 wait complete. 2024-03-24T10:45:18.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.143+0000 7f32787c7640 1 Processor -- start 2024-03-24T10:45:18.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.143+0000 7f32787c7640 1 -- start start 2024-03-24T10:45:18.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.143+0000 7f32787c7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f327006bf50 0x7f3270155f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.143+0000 7f32787c7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32701543a0 0x7f3270149a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.144+0000 7f32787c7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f327015ac00 0x7f3270149f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.144+0000 7f32787c7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f32700b4a10 con 0x7f327006bf50 2024-03-24T10:45:18.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.144+0000 7f32787c7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f327014a4c0 con 0x7f32701543a0 2024-03-24T10:45:18.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.144+0000 7f32787c7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f327014b200 con 0x7f327015ac00 2024-03-24T10:45:18.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.144+0000 7f3276563640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f327006bf50 0x7f3270155f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.144+0000 7f3275d62640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32701543a0 0x7f3270149a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.144+0000 7f3276d64640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f327015ac00 0x7f3270149f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.145+0000 7f3275d62640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32701543a0 0x7f3270149a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36754/0 (socket says 172.21.15.5:36754) 2024-03-24T10:45:18.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.145+0000 7f3275d62640 1 -- 172.21.15.5:0/127656282 learned_addr learned my addr 172.21.15.5:0/127656282 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:18.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.148+0000 7f3276563640 1 -- 172.21.15.5:0/127656282 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f327015ac00 msgr2=0x7f3270149f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:18.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.148+0000 7f3276563640 1 --2- 172.21.15.5:0/127656282 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f327015ac00 0x7f3270149f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.148+0000 7f3276563640 1 -- 172.21.15.5:0/127656282 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32701543a0 msgr2=0x7f3270149a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:18.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.148+0000 7f3276563640 1 --2- 172.21.15.5:0/127656282 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32701543a0 0x7f3270149a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.148+0000 7f3276563640 1 -- 172.21.15.5:0/127656282 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3268009a70 con 0x7f327006bf50 2024-03-24T10:45:18.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.150+0000 7f3276563640 1 --2- 172.21.15.5:0/127656282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f327006bf50 0x7f3270155f10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f326c00c410 tx=0x7f326c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:18.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.154+0000 7f3266ffd640 1 -- 172.21.15.5:0/127656282 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c005230 con 0x7f327006bf50 2024-03-24T10:45:18.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.154+0000 7f3266ffd640 1 -- 172.21.15.5:0/127656282 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f326c005390 con 0x7f327006bf50 2024-03-24T10:45:18.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.154+0000 7f3266ffd640 1 -- 172.21.15.5:0/127656282 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326c004820 con 0x7f327006bf50 2024-03-24T10:45:18.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.154+0000 7f32787c7640 1 -- 172.21.15.5:0/127656282 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f327014b4c0 con 0x7f327006bf50 2024-03-24T10:45:18.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.154+0000 7f32787c7640 1 -- 172.21.15.5:0/127656282 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32700b3c50 con 0x7f327006bf50 2024-03-24T10:45:18.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.155+0000 7f32787c7640 1 -- 172.21.15.5:0/127656282 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32700681d0 con 0x7f327006bf50 2024-03-24T10:45:18.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.422+0000 7f3266ffd640 1 -- 172.21.15.5:0/127656282 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f326c004980 con 0x7f327006bf50 2024-03-24T10:45:18.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.422+0000 7f3266ffd640 1 --2- 172.21.15.5:0/127656282 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f325406b810 0x7f325406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.423+0000 7f3275d62640 1 --2- 172.21.15.5:0/127656282 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f325406b810 0x7f325406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.423+0000 7f3275d62640 1 --2- 172.21.15.5:0/127656282 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f325406b810 0x7f325406dcd0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f32700b2c20 tx=0x7f3260007f10 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:18.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.437+0000 7f3266ffd640 1 -- 172.21.15.5:0/127656282 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f326c058190 con 0x7f327006bf50 2024-03-24T10:45:18.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.437+0000 7f3266ffd640 1 -- 172.21.15.5:0/127656282 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f326c054430 con 0x7f327006bf50 2024-03-24T10:45:18.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.616+0000 7f32787c7640 1 -- 172.21.15.5:0/127656282 --> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f327015a7b0 con 0x7f325406b810 2024-03-24T10:45:18.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.629+0000 7f3266ffd640 1 -- 172.21.15.5:0/127656282 <== mgr.4120 v2:172.21.15.5:6832/1102439181 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252060 (secure 0 0 0) 0x7f327015a7b0 con 0x7f325406b810 2024-03-24T10:45:18.632 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:45:18.633 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-03-24T10:45:18.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.632+0000 7f32787c7640 1 -- 172.21.15.5:0/127656282 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f325406b810 msgr2=0x7f325406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:18.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.632+0000 7f32787c7640 1 --2- 172.21.15.5:0/127656282 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f325406b810 0x7f325406dcd0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f32700b2c20 tx=0x7f3260007f10 comp rx=0 tx=0).stop 2024-03-24T10:45:18.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.632+0000 7f32787c7640 1 -- 172.21.15.5:0/127656282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f327006bf50 msgr2=0x7f3270155f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:18.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.632+0000 7f32787c7640 1 --2- 172.21.15.5:0/127656282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f327006bf50 0x7f3270155f10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f326c00c410 tx=0x7f326c00c8d0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.633+0000 7f32787c7640 1 -- 172.21.15.5:0/127656282 shutdown_connections 2024-03-24T10:45:18.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.633+0000 7f32787c7640 1 --2- 172.21.15.5:0/127656282 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f325406b810 0x7f325406dcd0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.633+0000 7f32787c7640 1 --2- 172.21.15.5:0/127656282 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f327015ac00 0x7f3270149f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.633+0000 7f32787c7640 1 --2- 172.21.15.5:0/127656282 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f32701543a0 0x7f3270149a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.633+0000 7f32787c7640 1 --2- 172.21.15.5:0/127656282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f327006bf50 0x7f3270155f10 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.633+0000 7f32787c7640 1 -- 172.21.15.5:0/127656282 >> 172.21.15.5:0/127656282 conn(0x7f3270072ef0 msgr2=0x7f32700c5340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:18.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.633+0000 7f32787c7640 1 -- 172.21.15.5:0/127656282 shutdown_connections 2024-03-24T10:45:18.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.633+0000 7f32787c7640 1 -- 172.21.15.5:0/127656282 wait complete. 2024-03-24T10:45:18.644 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-03-24T10:45:17.373412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598312,"num_objects":93,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":351,"num_write_kb":2477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":285,"ondisk_log_size":285,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238952,"kb_used_data":2024,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754735768,"statfs":{"total":773094113280,"available":772849426432,"internally_reserved":0,"allocated":2072576,"data_stored":1789598,"data_compressed":65430,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":242535986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5694,"num_objects":81,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":175,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":56,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002522"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956091+0000","last_change":"2024-03-24T10:45:01.894006+0000","last_active":"2024-03-24T10:45:02.956091+0000","last_peered":"2024-03-24T10:45:02.956091+0000","last_clean":"2024-03-24T10:45:02.956091+0000","last_became_active":"2024-03-24T10:45:01.893734+0000","last_became_peered":"2024-03-24T10:45:01.893734+0000","last_unstale":"2024-03-24T10:45:02.956091+0000","last_undegraded":"2024-03-24T10:45:02.956091+0000","last_fullsized":"2024-03-24T10:45:02.956091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:05:51.579388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.932998+0000","last_change":"2024-03-24T10:45:01.931707+0000","last_active":"2024-03-24T10:45:02.932998+0000","last_peered":"2024-03-24T10:45:02.932998+0000","last_clean":"2024-03-24T10:45:02.932998+0000","last_became_active":"2024-03-24T10:45:01.931449+0000","last_became_peered":"2024-03-24T10:45:01.931449+0000","last_unstale":"2024-03-24T10:45:02.932998+0000","last_undegraded":"2024-03-24T10:45:02.932998+0000","last_fullsized":"2024-03-24T10:45:02.932998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953238+0000","last_change":"2024-03-24T10:45:01.895929+0000","last_active":"2024-03-24T10:45:02.953238+0000","last_peered":"2024-03-24T10:45:02.953238+0000","last_clean":"2024-03-24T10:45:02.953238+0000","last_became_active":"2024-03-24T10:45:01.895654+0000","last_became_peered":"2024-03-24T10:45:01.895654+0000","last_unstale":"2024-03-24T10:45:02.953238+0000","last_undegraded":"2024-03-24T10:45:02.953238+0000","last_fullsized":"2024-03-24T10:45:02.953238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954093+0000","last_change":"2024-03-24T10:45:01.952468+0000","last_active":"2024-03-24T10:45:02.954093+0000","last_peered":"2024-03-24T10:45:02.954093+0000","last_clean":"2024-03-24T10:45:02.954093+0000","last_became_active":"2024-03-24T10:45:01.952219+0000","last_became_peered":"2024-03-24T10:45:01.952219+0000","last_unstale":"2024-03-24T10:45:02.954093+0000","last_undegraded":"2024-03-24T10:45:02.954093+0000","last_fullsized":"2024-03-24T10:45:02.954093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:27:12.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955739+0000","last_change":"2024-03-24T10:45:01.895885+0000","last_active":"2024-03-24T10:45:02.955739+0000","last_peered":"2024-03-24T10:45:02.955739+0000","last_clean":"2024-03-24T10:45:02.955739+0000","last_became_active":"2024-03-24T10:45:01.895622+0000","last_became_peered":"2024-03-24T10:45:01.895622+0000","last_unstale":"2024-03-24T10:45:02.955739+0000","last_undegraded":"2024-03-24T10:45:02.955739+0000","last_fullsized":"2024-03-24T10:45:02.955739+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:16.523177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954340+0000","last_change":"2024-03-24T10:45:01.958421+0000","last_active":"2024-03-24T10:45:02.954340+0000","last_peered":"2024-03-24T10:45:02.954340+0000","last_clean":"2024-03-24T10:45:02.954340+0000","last_became_active":"2024-03-24T10:45:01.957972+0000","last_became_peered":"2024-03-24T10:45:01.957972+0000","last_unstale":"2024-03-24T10:45:02.954340+0000","last_undegraded":"2024-03-24T10:45:02.954340+0000","last_fullsized":"2024-03-24T10:45:02.954340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:16.523177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955433+0000","last_change":"2024-03-24T10:45:01.892745+0000","last_active":"2024-03-24T10:45:02.955433+0000","last_peered":"2024-03-24T10:45:02.955433+0000","last_clean":"2024-03-24T10:45:02.955433+0000","last_became_active":"2024-03-24T10:45:01.892452+0000","last_became_peered":"2024-03-24T10:45:01.892452+0000","last_unstale":"2024-03-24T10:45:02.955433+0000","last_undegraded":"2024-03-24T10:45:02.955433+0000","last_fullsized":"2024-03-24T10:45:02.955433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954102+0000","last_change":"2024-03-24T10:45:01.953432+0000","last_active":"2024-03-24T10:45:02.954102+0000","last_peered":"2024-03-24T10:45:02.954102+0000","last_clean":"2024-03-24T10:45:02.954102+0000","last_became_active":"2024-03-24T10:45:01.953149+0000","last_became_peered":"2024-03-24T10:45:01.953149+0000","last_unstale":"2024-03-24T10:45:02.954102+0000","last_undegraded":"2024-03-24T10:45:02.954102+0000","last_fullsized":"2024-03-24T10:45:02.954102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956172+0000","last_change":"2024-03-24T10:45:01.956488+0000","last_active":"2024-03-24T10:45:02.956172+0000","last_peered":"2024-03-24T10:45:02.956172+0000","last_clean":"2024-03-24T10:45:02.956172+0000","last_became_active":"2024-03-24T10:45:01.956207+0000","last_became_peered":"2024-03-24T10:45:01.956207+0000","last_unstale":"2024-03-24T10:45:02.956172+0000","last_undegraded":"2024-03-24T10:45:02.956172+0000","last_fullsized":"2024-03-24T10:45:02.956172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:23:26.523809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953720+0000","last_change":"2024-03-24T10:45:01.956484+0000","last_active":"2024-03-24T10:45:02.953720+0000","last_peered":"2024-03-24T10:45:02.953720+0000","last_clean":"2024-03-24T10:45:02.953720+0000","last_became_active":"2024-03-24T10:45:01.956209+0000","last_became_peered":"2024-03-24T10:45:01.956209+0000","last_unstale":"2024-03-24T10:45:02.953720+0000","last_undegraded":"2024-03-24T10:45:02.953720+0000","last_fullsized":"2024-03-24T10:45:02.953720+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953838+0000","last_change":"2024-03-24T10:45:01.957103+0000","last_active":"2024-03-24T10:45:02.953838+0000","last_peered":"2024-03-24T10:45:02.953838+0000","last_clean":"2024-03-24T10:45:02.953838+0000","last_became_active":"2024-03-24T10:45:01.956849+0000","last_became_peered":"2024-03-24T10:45:01.956849+0000","last_unstale":"2024-03-24T10:45:02.953838+0000","last_undegraded":"2024-03-24T10:45:02.953838+0000","last_fullsized":"2024-03-24T10:45:02.953838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954691+0000","last_change":"2024-03-24T10:45:01.896701+0000","last_active":"2024-03-24T10:45:02.954691+0000","last_peered":"2024-03-24T10:45:02.954691+0000","last_clean":"2024-03-24T10:45:02.954691+0000","last_became_active":"2024-03-24T10:45:01.896400+0000","last_became_peered":"2024-03-24T10:45:01.896400+0000","last_unstale":"2024-03-24T10:45:02.954691+0000","last_undegraded":"2024-03-24T10:45:02.954691+0000","last_fullsized":"2024-03-24T10:45:02.954691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:44.380352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953999+0000","last_change":"2024-03-24T10:45:01.894954+0000","last_active":"2024-03-24T10:45:02.953999+0000","last_peered":"2024-03-24T10:45:02.953999+0000","last_clean":"2024-03-24T10:45:02.953999+0000","last_became_active":"2024-03-24T10:45:01.894632+0000","last_became_peered":"2024-03-24T10:45:01.894632+0000","last_unstale":"2024-03-24T10:45:02.953999+0000","last_undegraded":"2024-03-24T10:45:02.953999+0000","last_fullsized":"2024-03-24T10:45:02.953999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.952924+0000","last_change":"2024-03-24T10:45:01.929588+0000","last_active":"2024-03-24T10:45:02.952924+0000","last_peered":"2024-03-24T10:45:02.952924+0000","last_clean":"2024-03-24T10:45:02.952924+0000","last_became_active":"2024-03-24T10:45:01.928460+0000","last_became_peered":"2024-03-24T10:45:01.928460+0000","last_unstale":"2024-03-24T10:45:02.952924+0000","last_undegraded":"2024-03-24T10:45:02.952924+0000","last_fullsized":"2024-03-24T10:45:02.952924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:47:48.896914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933000+0000","last_change":"2024-03-24T10:45:01.928721+0000","last_active":"2024-03-24T10:45:02.933000+0000","last_peered":"2024-03-24T10:45:02.933000+0000","last_clean":"2024-03-24T10:45:02.933000+0000","last_became_active":"2024-03-24T10:45:01.928467+0000","last_became_peered":"2024-03-24T10:45:01.928467+0000","last_unstale":"2024-03-24T10:45:02.933000+0000","last_undegraded":"2024-03-24T10:45:02.933000+0000","last_fullsized":"2024-03-24T10:45:02.933000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953333+0000","last_change":"2024-03-24T10:45:01.956030+0000","last_active":"2024-03-24T10:45:02.953333+0000","last_peered":"2024-03-24T10:45:02.953333+0000","last_clean":"2024-03-24T10:45:02.953333+0000","last_became_active":"2024-03-24T10:45:01.955768+0000","last_became_peered":"2024-03-24T10:45:01.955768+0000","last_unstale":"2024-03-24T10:45:02.953333+0000","last_undegraded":"2024-03-24T10:45:02.953333+0000","last_fullsized":"2024-03-24T10:45:02.953333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954695+0000","last_change":"2024-03-24T10:45:01.956181+0000","last_active":"2024-03-24T10:45:02.954695+0000","last_peered":"2024-03-24T10:45:02.954695+0000","last_clean":"2024-03-24T10:45:02.954695+0000","last_became_active":"2024-03-24T10:45:01.955890+0000","last_became_peered":"2024-03-24T10:45:01.955890+0000","last_unstale":"2024-03-24T10:45:02.954695+0000","last_undegraded":"2024-03-24T10:45:02.954695+0000","last_fullsized":"2024-03-24T10:45:02.954695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:15:11.107915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.936626+0000","last_change":"2024-03-24T10:45:01.896663+0000","last_active":"2024-03-24T10:45:02.936626+0000","last_peered":"2024-03-24T10:45:02.936626+0000","last_clean":"2024-03-24T10:45:02.936626+0000","last_became_active":"2024-03-24T10:45:01.896391+0000","last_became_peered":"2024-03-24T10:45:01.896391+0000","last_unstale":"2024-03-24T10:45:02.936626+0000","last_undegraded":"2024-03-24T10:45:02.936626+0000","last_fullsized":"2024-03-24T10:45:02.936626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:30.942758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956053+0000","last_change":"2024-03-24T10:45:01.895951+0000","last_active":"2024-03-24T10:45:02.956053+0000","last_peered":"2024-03-24T10:45:02.956053+0000","last_clean":"2024-03-24T10:45:02.956053+0000","last_became_active":"2024-03-24T10:45:01.895668+0000","last_became_peered":"2024-03-24T10:45:01.895668+0000","last_unstale":"2024-03-24T10:45:02.956053+0000","last_undegraded":"2024-03-24T10:45:02.956053+0000","last_fullsized":"2024-03-24T10:45:02.956053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:56:43.462458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953964+0000","last_change":"2024-03-24T10:45:01.898126+0000","last_active":"2024-03-24T10:45:02.953964+0000","last_peered":"2024-03-24T10:45:02.953964+0000","last_clean":"2024-03-24T10:45:02.953964+0000","last_became_active":"2024-03-24T10:45:01.897878+0000","last_became_peered":"2024-03-24T10:45:01.897878+0000","last_unstale":"2024-03-24T10:45:02.953964+0000","last_undegraded":"2024-03-24T10:45:02.953964+0000","last_fullsized":"2024-03-24T10:45:02.953964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953247+0000","last_change":"2024-03-24T10:45:01.896109+0000","last_active":"2024-03-24T10:45:02.953247+0000","last_peered":"2024-03-24T10:45:02.953247+0000","last_clean":"2024-03-24T10:45:02.953247+0000","last_became_active":"2024-03-24T10:45:01.895696+0000","last_became_peered":"2024-03-24T10:45:01.895696+0000","last_unstale":"2024-03-24T10:45:02.953247+0000","last_undegraded":"2024-03-24T10:45:02.953247+0000","last_fullsized":"2024-03-24T10:45:02.953247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954483+0000","last_change":"2024-03-24T10:45:01.958119+0000","last_active":"2024-03-24T10:45:02.954483+0000","last_peered":"2024-03-24T10:45:02.954483+0000","last_clean":"2024-03-24T10:45:02.954483+0000","last_became_active":"2024-03-24T10:45:01.957832+0000","last_became_peered":"2024-03-24T10:45:01.957832+0000","last_unstale":"2024-03-24T10:45:02.954483+0000","last_undegraded":"2024-03-24T10:45:02.954483+0000","last_fullsized":"2024-03-24T10:45:02.954483+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:12.476490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953638+0000","last_change":"2024-03-24T10:45:01.842442+0000","last_active":"2024-03-24T10:45:02.953638+0000","last_peered":"2024-03-24T10:45:02.953638+0000","last_clean":"2024-03-24T10:45:02.953638+0000","last_became_active":"2024-03-24T10:45:01.842199+0000","last_became_peered":"2024-03-24T10:45:01.842199+0000","last_unstale":"2024-03-24T10:45:02.953638+0000","last_undegraded":"2024-03-24T10:45:02.953638+0000","last_fullsized":"2024-03-24T10:45:02.953638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953764+0000","last_change":"2024-03-24T10:45:01.952378+0000","last_active":"2024-03-24T10:45:02.953764+0000","last_peered":"2024-03-24T10:45:02.953764+0000","last_clean":"2024-03-24T10:45:02.953764+0000","last_became_active":"2024-03-24T10:45:01.952123+0000","last_became_peered":"2024-03-24T10:45:01.952123+0000","last_unstale":"2024-03-24T10:45:02.953764+0000","last_undegraded":"2024-03-24T10:45:02.953764+0000","last_fullsized":"2024-03-24T10:45:02.953764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954628+0000","last_change":"2024-03-24T10:45:01.842299+0000","last_active":"2024-03-24T10:45:02.954628+0000","last_peered":"2024-03-24T10:45:02.954628+0000","last_clean":"2024-03-24T10:45:02.954628+0000","last_became_active":"2024-03-24T10:45:01.842097+0000","last_became_peered":"2024-03-24T10:45:01.842097+0000","last_unstale":"2024-03-24T10:45:02.954628+0000","last_undegraded":"2024-03-24T10:45:02.954628+0000","last_fullsized":"2024-03-24T10:45:02.954628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:16.341360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953886+0000","last_change":"2024-03-24T10:45:01.893579+0000","last_active":"2024-03-24T10:45:02.953886+0000","last_peered":"2024-03-24T10:45:02.953886+0000","last_clean":"2024-03-24T10:45:02.953886+0000","last_became_active":"2024-03-24T10:45:01.893273+0000","last_became_peered":"2024-03-24T10:45:01.893273+0000","last_unstale":"2024-03-24T10:45:02.953886+0000","last_undegraded":"2024-03-24T10:45:02.953886+0000","last_fullsized":"2024-03-24T10:45:02.953886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933308+0000","last_change":"2024-03-24T10:45:01.930032+0000","last_active":"2024-03-24T10:45:02.933308+0000","last_peered":"2024-03-24T10:45:02.933308+0000","last_clean":"2024-03-24T10:45:02.933308+0000","last_became_active":"2024-03-24T10:45:01.929770+0000","last_became_peered":"2024-03-24T10:45:01.929770+0000","last_unstale":"2024-03-24T10:45:02.933308+0000","last_undegraded":"2024-03-24T10:45:02.933308+0000","last_fullsized":"2024-03-24T10:45:02.933308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954962+0000","last_change":"2024-03-24T10:45:01.842460+0000","last_active":"2024-03-24T10:45:02.954962+0000","last_peered":"2024-03-24T10:45:02.954962+0000","last_clean":"2024-03-24T10:45:02.954962+0000","last_became_active":"2024-03-24T10:45:01.842165+0000","last_became_peered":"2024-03-24T10:45:01.842165+0000","last_unstale":"2024-03-24T10:45:02.954962+0000","last_undegraded":"2024-03-24T10:45:02.954962+0000","last_fullsized":"2024-03-24T10:45:02.954962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953401+0000","last_change":"2024-03-24T10:45:01.895868+0000","last_active":"2024-03-24T10:45:02.953401+0000","last_peered":"2024-03-24T10:45:02.953401+0000","last_clean":"2024-03-24T10:45:02.953401+0000","last_became_active":"2024-03-24T10:45:01.895290+0000","last_became_peered":"2024-03-24T10:45:01.895290+0000","last_unstale":"2024-03-24T10:45:02.953401+0000","last_undegraded":"2024-03-24T10:45:02.953401+0000","last_fullsized":"2024-03-24T10:45:02.953401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954615+0000","last_change":"2024-03-24T10:45:01.957724+0000","last_active":"2024-03-24T10:45:02.954615+0000","last_peered":"2024-03-24T10:45:02.954615+0000","last_clean":"2024-03-24T10:45:02.954615+0000","last_became_active":"2024-03-24T10:45:01.957429+0000","last_became_peered":"2024-03-24T10:45:01.957429+0000","last_unstale":"2024-03-24T10:45:02.954615+0000","last_undegraded":"2024-03-24T10:45:02.954615+0000","last_fullsized":"2024-03-24T10:45:02.954615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:22.089070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.936465+0000","last_change":"2024-03-24T10:45:01.952159+0000","last_active":"2024-03-24T10:45:02.936465+0000","last_peered":"2024-03-24T10:45:02.936465+0000","last_clean":"2024-03-24T10:45:02.936465+0000","last_became_active":"2024-03-24T10:45:01.951961+0000","last_became_peered":"2024-03-24T10:45:01.951961+0000","last_unstale":"2024-03-24T10:45:02.936465+0000","last_undegraded":"2024-03-24T10:45:02.936465+0000","last_fullsized":"2024-03-24T10:45:02.936465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:44.380352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953610+0000","last_change":"2024-03-24T10:45:01.893452+0000","last_active":"2024-03-24T10:45:02.953610+0000","last_peered":"2024-03-24T10:45:02.953610+0000","last_clean":"2024-03-24T10:45:02.953610+0000","last_became_active":"2024-03-24T10:45:01.893139+0000","last_became_peered":"2024-03-24T10:45:01.893139+0000","last_unstale":"2024-03-24T10:45:02.953610+0000","last_undegraded":"2024-03-24T10:45:02.953610+0000","last_fullsized":"2024-03-24T10:45:02.953610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953542+0000","last_change":"2024-03-24T10:45:01.928827+0000","last_active":"2024-03-24T10:45:02.953542+0000","last_peered":"2024-03-24T10:45:02.953542+0000","last_clean":"2024-03-24T10:45:02.953542+0000","last_became_active":"2024-03-24T10:45:01.928521+0000","last_became_peered":"2024-03-24T10:45:01.928521+0000","last_unstale":"2024-03-24T10:45:02.953542+0000","last_undegraded":"2024-03-24T10:45:02.953542+0000","last_fullsized":"2024-03-24T10:45:02.953542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:59.643246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.699015+0000","last_change":"2024-03-24T10:45:00.637993+0000","last_active":"2024-03-24T10:45:07.699015+0000","last_peered":"2024-03-24T10:45:07.699015+0000","last_clean":"2024-03-24T10:45:07.699015+0000","last_became_active":"2024-03-24T10:45:00.637803+0000","last_became_peered":"2024-03-24T10:45:00.637803+0000","last_unstale":"2024-03-24T10:45:07.699015+0000","last_undegraded":"2024-03-24T10:45:07.699015+0000","last_fullsized":"2024-03-24T10:45:07.699015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:28:28.854541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953550+0000","last_change":"2024-03-24T10:45:01.897686+0000","last_active":"2024-03-24T10:45:02.953550+0000","last_peered":"2024-03-24T10:45:02.953550+0000","last_clean":"2024-03-24T10:45:02.953550+0000","last_became_active":"2024-03-24T10:45:01.897320+0000","last_became_peered":"2024-03-24T10:45:01.897320+0000","last_unstale":"2024-03-24T10:45:02.953550+0000","last_undegraded":"2024-03-24T10:45:02.953550+0000","last_fullsized":"2024-03-24T10:45:02.953550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.776260+0000","last_change":"2024-03-24T10:45:00.638329+0000","last_active":"2024-03-24T10:45:12.776260+0000","last_peered":"2024-03-24T10:45:12.776260+0000","last_clean":"2024-03-24T10:45:12.776260+0000","last_became_active":"2024-03-24T10:45:00.638006+0000","last_became_peered":"2024-03-24T10:45:00.638006+0000","last_unstale":"2024-03-24T10:45:12.776260+0000","last_undegraded":"2024-03-24T10:45:12.776260+0000","last_fullsized":"2024-03-24T10:45:12.776260+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:10.355399+0000","last_change":"2024-03-24T10:45:00.633703+0000","last_active":"2024-03-24T10:45:10.355399+0000","last_peered":"2024-03-24T10:45:10.355399+0000","last_clean":"2024-03-24T10:45:10.355399+0000","last_became_active":"2024-03-24T10:45:00.633427+0000","last_became_peered":"2024-03-24T10:45:00.633427+0000","last_unstale":"2024-03-24T10:45:10.355399+0000","last_undegraded":"2024-03-24T10:45:10.355399+0000","last_fullsized":"2024-03-24T10:45:10.355399+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:30.350191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954240+0000","last_change":"2024-03-24T10:45:01.893902+0000","last_active":"2024-03-24T10:45:02.954240+0000","last_peered":"2024-03-24T10:45:02.954240+0000","last_clean":"2024-03-24T10:45:02.954240+0000","last_became_active":"2024-03-24T10:45:01.893617+0000","last_became_peered":"2024-03-24T10:45:01.893617+0000","last_unstale":"2024-03-24T10:45:02.954240+0000","last_undegraded":"2024-03-24T10:45:02.954240+0000","last_fullsized":"2024-03-24T10:45:02.954240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.780177+0000","last_change":"2024-03-24T10:45:00.638778+0000","last_active":"2024-03-24T10:45:12.780177+0000","last_peered":"2024-03-24T10:45:12.780177+0000","last_clean":"2024-03-24T10:45:12.780177+0000","last_became_active":"2024-03-24T10:45:00.638089+0000","last_became_peered":"2024-03-24T10:45:00.638089+0000","last_unstale":"2024-03-24T10:45:12.780177+0000","last_undegraded":"2024-03-24T10:45:12.780177+0000","last_fullsized":"2024-03-24T10:45:12.780177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.932839+0000","last_change":"2024-03-24T10:45:01.930063+0000","last_active":"2024-03-24T10:45:02.932839+0000","last_peered":"2024-03-24T10:45:02.932839+0000","last_clean":"2024-03-24T10:45:02.932839+0000","last_became_active":"2024-03-24T10:45:01.929786+0000","last_became_peered":"2024-03-24T10:45:01.929786+0000","last_unstale":"2024-03-24T10:45:02.932839+0000","last_undegraded":"2024-03-24T10:45:02.932839+0000","last_fullsized":"2024-03-24T10:45:02.932839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.773173+0000","last_change":"2024-03-24T10:45:00.639822+0000","last_active":"2024-03-24T10:45:07.773173+0000","last_peered":"2024-03-24T10:45:07.773173+0000","last_clean":"2024-03-24T10:45:07.773173+0000","last_became_active":"2024-03-24T10:45:00.639600+0000","last_became_peered":"2024-03-24T10:45:00.639600+0000","last_unstale":"2024-03-24T10:45:07.773173+0000","last_undegraded":"2024-03-24T10:45:07.773173+0000","last_fullsized":"2024-03-24T10:45:07.773173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953735+0000","last_change":"2024-03-24T10:45:01.954015+0000","last_active":"2024-03-24T10:45:02.953735+0000","last_peered":"2024-03-24T10:45:02.953735+0000","last_clean":"2024-03-24T10:45:02.953735+0000","last_became_active":"2024-03-24T10:45:01.953767+0000","last_became_peered":"2024-03-24T10:45:01.953767+0000","last_unstale":"2024-03-24T10:45:02.953735+0000","last_undegraded":"2024-03-24T10:45:02.953735+0000","last_fullsized":"2024-03-24T10:45:02.953735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:15.553426+0000","last_change":"2024-03-24T10:45:00.636553+0000","last_active":"2024-03-24T10:45:15.553426+0000","last_peered":"2024-03-24T10:45:15.553426+0000","last_clean":"2024-03-24T10:45:15.553426+0000","last_became_active":"2024-03-24T10:45:00.636282+0000","last_became_peered":"2024-03-24T10:45:00.636282+0000","last_unstale":"2024-03-24T10:45:15.553426+0000","last_undegraded":"2024-03-24T10:45:15.553426+0000","last_fullsized":"2024-03-24T10:45:15.553426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953458+0000","last_change":"2024-03-24T10:45:01.957905+0000","last_active":"2024-03-24T10:45:02.953458+0000","last_peered":"2024-03-24T10:45:02.953458+0000","last_clean":"2024-03-24T10:45:02.953458+0000","last_became_active":"2024-03-24T10:45:01.957676+0000","last_became_peered":"2024-03-24T10:45:01.957676+0000","last_unstale":"2024-03-24T10:45:02.953458+0000","last_undegraded":"2024-03-24T10:45:02.953458+0000","last_fullsized":"2024-03-24T10:45:02.953458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.798829+0000","last_change":"2024-03-24T10:45:00.635197+0000","last_active":"2024-03-24T10:45:07.798829+0000","last_peered":"2024-03-24T10:45:07.798829+0000","last_clean":"2024-03-24T10:45:07.798829+0000","last_became_active":"2024-03-24T10:45:00.634989+0000","last_became_peered":"2024-03-24T10:45:00.634989+0000","last_unstale":"2024-03-24T10:45:07.798829+0000","last_undegraded":"2024-03-24T10:45:07.798829+0000","last_fullsized":"2024-03-24T10:45:07.798829+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954821+0000","last_change":"2024-03-24T10:45:01.853785+0000","last_active":"2024-03-24T10:45:02.954821+0000","last_peered":"2024-03-24T10:45:02.954821+0000","last_clean":"2024-03-24T10:45:02.954821+0000","last_became_active":"2024-03-24T10:45:01.853504+0000","last_became_peered":"2024-03-24T10:45:01.853504+0000","last_unstale":"2024-03-24T10:45:02.954821+0000","last_undegraded":"2024-03-24T10:45:02.954821+0000","last_fullsized":"2024-03-24T10:45:02.954821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:30.942758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.773017+0000","last_change":"2024-03-24T10:45:00.632381+0000","last_active":"2024-03-24T10:45:07.773017+0000","last_peered":"2024-03-24T10:45:07.773017+0000","last_clean":"2024-03-24T10:45:07.773017+0000","last_became_active":"2024-03-24T10:45:00.632108+0000","last_became_peered":"2024-03-24T10:45:00.632108+0000","last_unstale":"2024-03-24T10:45:07.773017+0000","last_undegraded":"2024-03-24T10:45:07.773017+0000","last_fullsized":"2024-03-24T10:45:07.773017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:06.142451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955746+0000","last_change":"2024-03-24T10:45:01.853591+0000","last_active":"2024-03-24T10:45:02.955746+0000","last_peered":"2024-03-24T10:45:02.955746+0000","last_clean":"2024-03-24T10:45:02.955746+0000","last_became_active":"2024-03-24T10:45:01.853279+0000","last_became_peered":"2024-03-24T10:45:01.853279+0000","last_unstale":"2024-03-24T10:45:02.955746+0000","last_undegraded":"2024-03-24T10:45:02.955746+0000","last_fullsized":"2024-03-24T10:45:02.955746+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:07:24.149961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:10.356293+0000","last_change":"2024-03-24T10:45:00.634192+0000","last_active":"2024-03-24T10:45:10.356293+0000","last_peered":"2024-03-24T10:45:10.356293+0000","last_clean":"2024-03-24T10:45:10.356293+0000","last_became_active":"2024-03-24T10:45:00.633893+0000","last_became_peered":"2024-03-24T10:45:00.633893+0000","last_unstale":"2024-03-24T10:45:10.356293+0000","last_undegraded":"2024-03-24T10:45:10.356293+0000","last_fullsized":"2024-03-24T10:45:10.356293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:30.350191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953075+0000","last_change":"2024-03-24T10:45:01.893426+0000","last_active":"2024-03-24T10:45:02.953075+0000","last_peered":"2024-03-24T10:45:02.953075+0000","last_clean":"2024-03-24T10:45:02.953075+0000","last_became_active":"2024-03-24T10:45:01.893178+0000","last_became_peered":"2024-03-24T10:45:01.893178+0000","last_unstale":"2024-03-24T10:45:02.953075+0000","last_undegraded":"2024-03-24T10:45:02.953075+0000","last_fullsized":"2024-03-24T10:45:02.953075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:13.887947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.780472+0000","last_change":"2024-03-24T10:45:00.638214+0000","last_active":"2024-03-24T10:45:12.780472+0000","last_peered":"2024-03-24T10:45:12.780472+0000","last_clean":"2024-03-24T10:45:12.780472+0000","last_became_active":"2024-03-24T10:45:00.637973+0000","last_became_peered":"2024-03-24T10:45:00.637973+0000","last_unstale":"2024-03-24T10:45:12.780472+0000","last_undegraded":"2024-03-24T10:45:12.780472+0000","last_fullsized":"2024-03-24T10:45:12.780472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:15.749451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955767+0000","last_change":"2024-03-24T10:45:01.856756+0000","last_active":"2024-03-24T10:45:02.955767+0000","last_peered":"2024-03-24T10:45:02.955767+0000","last_clean":"2024-03-24T10:45:02.955767+0000","last_became_active":"2024-03-24T10:45:01.856482+0000","last_became_peered":"2024-03-24T10:45:01.856482+0000","last_unstale":"2024-03-24T10:45:02.955767+0000","last_undegraded":"2024-03-24T10:45:02.955767+0000","last_fullsized":"2024-03-24T10:45:02.955767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:12.476490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.773462+0000","last_change":"2024-03-24T10:45:00.636481+0000","last_active":"2024-03-24T10:45:12.773462+0000","last_peered":"2024-03-24T10:45:12.773462+0000","last_clean":"2024-03-24T10:45:12.773462+0000","last_became_active":"2024-03-24T10:45:00.636206+0000","last_became_peered":"2024-03-24T10:45:00.636206+0000","last_unstale":"2024-03-24T10:45:12.773462+0000","last_undegraded":"2024-03-24T10:45:12.773462+0000","last_fullsized":"2024-03-24T10:45:12.773462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954153+0000","last_change":"2024-03-24T10:45:01.897279+0000","last_active":"2024-03-24T10:45:02.954153+0000","last_peered":"2024-03-24T10:45:02.954153+0000","last_clean":"2024-03-24T10:45:02.954153+0000","last_became_active":"2024-03-24T10:45:01.896736+0000","last_became_peered":"2024-03-24T10:45:01.896736+0000","last_unstale":"2024-03-24T10:45:02.954153+0000","last_undegraded":"2024-03-24T10:45:02.954153+0000","last_fullsized":"2024-03-24T10:45:02.954153+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":31,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.794043+0000","last_change":"2024-03-24T10:45:00.564066+0000","last_active":"2024-03-24T10:45:12.794043+0000","last_peered":"2024-03-24T10:45:12.794043+0000","last_clean":"2024-03-24T10:45:12.794043+0000","last_became_active":"2024-03-24T10:45:00.563812+0000","last_became_peered":"2024-03-24T10:45:00.563812+0000","last_unstale":"2024-03-24T10:45:12.794043+0000","last_undegraded":"2024-03-24T10:45:12.794043+0000","last_fullsized":"2024-03-24T10:45:12.794043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:28:28.854541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955401+0000","last_change":"2024-03-24T10:45:01.854429+0000","last_active":"2024-03-24T10:45:02.955401+0000","last_peered":"2024-03-24T10:45:02.955401+0000","last_clean":"2024-03-24T10:45:02.955401+0000","last_became_active":"2024-03-24T10:45:01.854179+0000","last_became_peered":"2024-03-24T10:45:01.854179+0000","last_unstale":"2024-03-24T10:45:02.955401+0000","last_undegraded":"2024-03-24T10:45:02.955401+0000","last_fullsized":"2024-03-24T10:45:02.955401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:03.132419+0000","last_change":"2024-03-24T10:45:00.634223+0000","last_active":"2024-03-24T10:45:03.132419+0000","last_peered":"2024-03-24T10:45:03.132419+0000","last_clean":"2024-03-24T10:45:03.132419+0000","last_became_active":"2024-03-24T10:45:00.633355+0000","last_became_peered":"2024-03-24T10:45:00.633355+0000","last_unstale":"2024-03-24T10:45:03.132419+0000","last_undegraded":"2024-03-24T10:45:03.132419+0000","last_fullsized":"2024-03-24T10:45:03.132419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:04.930115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956031+0000","last_change":"2024-03-24T10:45:01.856935+0000","last_active":"2024-03-24T10:45:02.956031+0000","last_peered":"2024-03-24T10:45:02.956031+0000","last_clean":"2024-03-24T10:45:02.956031+0000","last_became_active":"2024-03-24T10:45:01.856653+0000","last_became_peered":"2024-03-24T10:45:01.856653+0000","last_unstale":"2024-03-24T10:45:02.956031+0000","last_undegraded":"2024-03-24T10:45:02.956031+0000","last_fullsized":"2024-03-24T10:45:02.956031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:15:11.107915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:15.622360+0000","last_change":"2024-03-24T10:45:00.626484+0000","last_active":"2024-03-24T10:45:15.622360+0000","last_peered":"2024-03-24T10:45:15.622360+0000","last_clean":"2024-03-24T10:45:15.622360+0000","last_became_active":"2024-03-24T10:45:00.626195+0000","last_became_peered":"2024-03-24T10:45:00.626195+0000","last_unstale":"2024-03-24T10:45:15.622360+0000","last_undegraded":"2024-03-24T10:45:15.622360+0000","last_fullsized":"2024-03-24T10:45:15.622360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956227+0000","last_change":"2024-03-24T10:45:01.856995+0000","last_active":"2024-03-24T10:45:02.956227+0000","last_peered":"2024-03-24T10:45:02.956227+0000","last_clean":"2024-03-24T10:45:02.956227+0000","last_became_active":"2024-03-24T10:45:01.856531+0000","last_became_peered":"2024-03-24T10:45:01.856531+0000","last_unstale":"2024-03-24T10:45:02.956227+0000","last_undegraded":"2024-03-24T10:45:02.956227+0000","last_fullsized":"2024-03-24T10:45:02.956227+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:29:31.508911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.746930+0000","last_change":"2024-03-24T10:45:00.628438+0000","last_active":"2024-03-24T10:45:12.746930+0000","last_peered":"2024-03-24T10:45:12.746930+0000","last_clean":"2024-03-24T10:45:12.746930+0000","last_became_active":"2024-03-24T10:45:00.628158+0000","last_became_peered":"2024-03-24T10:45:00.628158+0000","last_unstale":"2024-03-24T10:45:12.746930+0000","last_undegraded":"2024-03-24T10:45:12.746930+0000","last_fullsized":"2024-03-24T10:45:12.746930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:29.826202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954370+0000","last_change":"2024-03-24T10:45:01.951927+0000","last_active":"2024-03-24T10:45:02.954370+0000","last_peered":"2024-03-24T10:45:02.954370+0000","last_clean":"2024-03-24T10:45:02.954370+0000","last_became_active":"2024-03-24T10:45:01.951165+0000","last_became_peered":"2024-03-24T10:45:01.951165+0000","last_unstale":"2024-03-24T10:45:02.954370+0000","last_undegraded":"2024-03-24T10:45:02.954370+0000","last_fullsized":"2024-03-24T10:45:02.954370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:07:24.149961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.793997+0000","last_change":"2024-03-24T10:45:00.639093+0000","last_active":"2024-03-24T10:45:12.793997+0000","last_peered":"2024-03-24T10:45:12.793997+0000","last_clean":"2024-03-24T10:45:12.793997+0000","last_became_active":"2024-03-24T10:45:00.638785+0000","last_became_peered":"2024-03-24T10:45:00.638785+0000","last_unstale":"2024-03-24T10:45:12.793997+0000","last_undegraded":"2024-03-24T10:45:12.793997+0000","last_fullsized":"2024-03-24T10:45:12.793997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955920+0000","last_change":"2024-03-24T10:45:01.953772+0000","last_active":"2024-03-24T10:45:02.955920+0000","last_peered":"2024-03-24T10:45:02.955920+0000","last_clean":"2024-03-24T10:45:02.955920+0000","last_became_active":"2024-03-24T10:45:01.953402+0000","last_became_peered":"2024-03-24T10:45:01.953402+0000","last_unstale":"2024-03-24T10:45:02.955920+0000","last_undegraded":"2024-03-24T10:45:02.955920+0000","last_fullsized":"2024-03-24T10:45:02.955920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:22.089070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955402+0000","last_change":"2024-03-24T10:44:48.241074+0000","last_active":"2024-03-24T10:45:02.955402+0000","last_peered":"2024-03-24T10:45:02.955402+0000","last_clean":"2024-03-24T10:45:02.955402+0000","last_became_active":"2024-03-24T10:44:48.239986+0000","last_became_peered":"2024-03-24T10:44:48.239986+0000","last_unstale":"2024-03-24T10:45:02.955402+0000","last_undegraded":"2024-03-24T10:45:02.955402+0000","last_fullsized":"2024-03-24T10:45:02.955402+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:46.142247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:46.142247+0000","last_clean_scrub_stamp":"2024-03-24T10:44:46.142247+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:05:50.110100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.780173+0000","last_change":"2024-03-24T10:45:00.637276+0000","last_active":"2024-03-24T10:45:12.780173+0000","last_peered":"2024-03-24T10:45:12.780173+0000","last_clean":"2024-03-24T10:45:12.780173+0000","last_became_active":"2024-03-24T10:45:00.636734+0000","last_became_peered":"2024-03-24T10:45:00.636734+0000","last_unstale":"2024-03-24T10:45:12.780173+0000","last_undegraded":"2024-03-24T10:45:12.780173+0000","last_fullsized":"2024-03-24T10:45:12.780173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954655+0000","last_change":"2024-03-24T10:45:01.958505+0000","last_active":"2024-03-24T10:45:02.954655+0000","last_peered":"2024-03-24T10:45:02.954655+0000","last_clean":"2024-03-24T10:45:02.954655+0000","last_became_active":"2024-03-24T10:45:01.958268+0000","last_became_peered":"2024-03-24T10:45:01.958268+0000","last_unstale":"2024-03-24T10:45:02.954655+0000","last_undegraded":"2024-03-24T10:45:02.954655+0000","last_fullsized":"2024-03-24T10:45:02.954655+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:27:12.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:10.261218+0000","last_change":"2024-03-24T10:45:00.597856+0000","last_active":"2024-03-24T10:45:10.261218+0000","last_peered":"2024-03-24T10:45:10.261218+0000","last_clean":"2024-03-24T10:45:10.261218+0000","last_became_active":"2024-03-24T10:45:00.597640+0000","last_became_peered":"2024-03-24T10:45:00.597640+0000","last_unstale":"2024-03-24T10:45:10.261218+0000","last_undegraded":"2024-03-24T10:45:10.261218+0000","last_fullsized":"2024-03-24T10:45:10.261218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:29.826202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.952842+0000","last_change":"2024-03-24T10:45:01.897502+0000","last_active":"2024-03-24T10:45:02.952842+0000","last_peered":"2024-03-24T10:45:02.952842+0000","last_clean":"2024-03-24T10:45:02.952842+0000","last_became_active":"2024-03-24T10:45:01.897250+0000","last_became_peered":"2024-03-24T10:45:01.897250+0000","last_unstale":"2024-03-24T10:45:02.952842+0000","last_undegraded":"2024-03-24T10:45:02.952842+0000","last_fullsized":"2024-03-24T10:45:02.952842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:09:46.341672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:10.390365+0000","last_change":"2024-03-24T10:45:00.627166+0000","last_active":"2024-03-24T10:45:10.390365+0000","last_peered":"2024-03-24T10:45:10.390365+0000","last_clean":"2024-03-24T10:45:10.390365+0000","last_became_active":"2024-03-24T10:45:00.626881+0000","last_became_peered":"2024-03-24T10:45:00.626881+0000","last_unstale":"2024-03-24T10:45:10.390365+0000","last_undegraded":"2024-03-24T10:45:10.390365+0000","last_fullsized":"2024-03-24T10:45:10.390365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.932815+0000","last_change":"2024-03-24T10:45:01.930271+0000","last_active":"2024-03-24T10:45:02.932815+0000","last_peered":"2024-03-24T10:45:02.932815+0000","last_clean":"2024-03-24T10:45:02.932815+0000","last_became_active":"2024-03-24T10:45:01.929998+0000","last_became_peered":"2024-03-24T10:45:01.929998+0000","last_unstale":"2024-03-24T10:45:02.932815+0000","last_undegraded":"2024-03-24T10:45:02.932815+0000","last_fullsized":"2024-03-24T10:45:02.932815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:15.554272+0000","last_change":"2024-03-24T10:45:00.638309+0000","last_active":"2024-03-24T10:45:15.554272+0000","last_peered":"2024-03-24T10:45:15.554272+0000","last_clean":"2024-03-24T10:45:15.554272+0000","last_became_active":"2024-03-24T10:45:00.638041+0000","last_became_peered":"2024-03-24T10:45:00.638041+0000","last_unstale":"2024-03-24T10:45:15.554272+0000","last_undegraded":"2024-03-24T10:45:15.554272+0000","last_fullsized":"2024-03-24T10:45:15.554272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:14.049099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953823+0000","last_change":"2024-03-24T10:45:01.897722+0000","last_active":"2024-03-24T10:45:02.953823+0000","last_peered":"2024-03-24T10:45:02.953823+0000","last_clean":"2024-03-24T10:45:02.953823+0000","last_became_active":"2024-03-24T10:45:01.897159+0000","last_became_peered":"2024-03-24T10:45:01.897159+0000","last_unstale":"2024-03-24T10:45:02.953823+0000","last_undegraded":"2024-03-24T10:45:02.953823+0000","last_fullsized":"2024-03-24T10:45:02.953823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:42.293342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933241+0000","last_change":"2024-03-24T10:45:00.629267+0000","last_active":"2024-03-24T10:45:02.933241+0000","last_peered":"2024-03-24T10:45:02.933241+0000","last_clean":"2024-03-24T10:45:02.933241+0000","last_became_active":"2024-03-24T10:45:00.628971+0000","last_became_peered":"2024-03-24T10:45:00.628971+0000","last_unstale":"2024-03-24T10:45:02.933241+0000","last_undegraded":"2024-03-24T10:45:02.933241+0000","last_fullsized":"2024-03-24T10:45:02.933241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:41.176786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954115+0000","last_change":"2024-03-24T10:45:01.952527+0000","last_active":"2024-03-24T10:45:02.954115+0000","last_peered":"2024-03-24T10:45:02.954115+0000","last_clean":"2024-03-24T10:45:02.954115+0000","last_became_active":"2024-03-24T10:45:01.952192+0000","last_became_peered":"2024-03-24T10:45:01.952192+0000","last_unstale":"2024-03-24T10:45:02.954115+0000","last_undegraded":"2024-03-24T10:45:02.954115+0000","last_fullsized":"2024-03-24T10:45:02.954115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.749212+0000","last_change":"2024-03-24T10:45:00.631941+0000","last_active":"2024-03-24T10:45:12.749212+0000","last_peered":"2024-03-24T10:45:12.749212+0000","last_clean":"2024-03-24T10:45:12.749212+0000","last_became_active":"2024-03-24T10:45:00.631670+0000","last_became_peered":"2024-03-24T10:45:00.631670+0000","last_unstale":"2024-03-24T10:45:12.749212+0000","last_undegraded":"2024-03-24T10:45:12.749212+0000","last_fullsized":"2024-03-24T10:45:12.749212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:09:45.225115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933116+0000","last_change":"2024-03-24T10:45:01.930483+0000","last_active":"2024-03-24T10:45:02.933116+0000","last_peered":"2024-03-24T10:45:02.933116+0000","last_clean":"2024-03-24T10:45:02.933116+0000","last_became_active":"2024-03-24T10:45:01.930220+0000","last_became_peered":"2024-03-24T10:45:01.930220+0000","last_unstale":"2024-03-24T10:45:02.933116+0000","last_undegraded":"2024-03-24T10:45:02.933116+0000","last_fullsized":"2024-03-24T10:45:02.933116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.748135+0000","last_change":"2024-03-24T10:45:00.633650+0000","last_active":"2024-03-24T10:45:12.748135+0000","last_peered":"2024-03-24T10:45:12.748135+0000","last_clean":"2024-03-24T10:45:12.748135+0000","last_became_active":"2024-03-24T10:45:00.633329+0000","last_became_peered":"2024-03-24T10:45:00.633329+0000","last_unstale":"2024-03-24T10:45:12.748135+0000","last_undegraded":"2024-03-24T10:45:12.748135+0000","last_fullsized":"2024-03-24T10:45:12.748135+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954759+0000","last_change":"2024-03-24T10:45:01.853487+0000","last_active":"2024-03-24T10:45:02.954759+0000","last_peered":"2024-03-24T10:45:02.954759+0000","last_clean":"2024-03-24T10:45:02.954759+0000","last_became_active":"2024-03-24T10:45:01.853257+0000","last_became_peered":"2024-03-24T10:45:01.853257+0000","last_unstale":"2024-03-24T10:45:02.954759+0000","last_undegraded":"2024-03-24T10:45:02.954759+0000","last_fullsized":"2024-03-24T10:45:02.954759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.773005+0000","last_change":"2024-03-24T10:45:00.639282+0000","last_active":"2024-03-24T10:45:07.773005+0000","last_peered":"2024-03-24T10:45:07.773005+0000","last_clean":"2024-03-24T10:45:07.773005+0000","last_became_active":"2024-03-24T10:45:00.638999+0000","last_became_peered":"2024-03-24T10:45:00.638999+0000","last_unstale":"2024-03-24T10:45:07.773005+0000","last_undegraded":"2024-03-24T10:45:07.773005+0000","last_fullsized":"2024-03-24T10:45:07.773005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:41.176786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953535+0000","last_change":"2024-03-24T10:45:01.928900+0000","last_active":"2024-03-24T10:45:02.953535+0000","last_peered":"2024-03-24T10:45:02.953535+0000","last_clean":"2024-03-24T10:45:02.953535+0000","last_became_active":"2024-03-24T10:45:01.928610+0000","last_became_peered":"2024-03-24T10:45:01.928610+0000","last_unstale":"2024-03-24T10:45:02.953535+0000","last_undegraded":"2024-03-24T10:45:02.953535+0000","last_fullsized":"2024-03-24T10:45:02.953535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.772043+0000","last_change":"2024-03-24T10:45:00.631327+0000","last_active":"2024-03-24T10:45:07.772043+0000","last_peered":"2024-03-24T10:45:07.772043+0000","last_clean":"2024-03-24T10:45:07.772043+0000","last_became_active":"2024-03-24T10:45:00.631078+0000","last_became_peered":"2024-03-24T10:45:00.631078+0000","last_unstale":"2024-03-24T10:45:07.772043+0000","last_undegraded":"2024-03-24T10:45:07.772043+0000","last_fullsized":"2024-03-24T10:45:07.772043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:43.263796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953696+0000","last_change":"2024-03-24T10:45:01.897329+0000","last_active":"2024-03-24T10:45:02.953696+0000","last_peered":"2024-03-24T10:45:02.953696+0000","last_clean":"2024-03-24T10:45:02.953696+0000","last_became_active":"2024-03-24T10:45:01.897063+0000","last_became_peered":"2024-03-24T10:45:01.897063+0000","last_unstale":"2024-03-24T10:45:02.953696+0000","last_undegraded":"2024-03-24T10:45:02.953696+0000","last_fullsized":"2024-03-24T10:45:02.953696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:07.259007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:15.511661+0000","last_change":"2024-03-24T10:45:00.597948+0000","last_active":"2024-03-24T10:45:15.511661+0000","last_peered":"2024-03-24T10:45:15.511661+0000","last_clean":"2024-03-24T10:45:15.511661+0000","last_became_active":"2024-03-24T10:45:00.597671+0000","last_became_peered":"2024-03-24T10:45:00.597671+0000","last_unstale":"2024-03-24T10:45:15.511661+0000","last_undegraded":"2024-03-24T10:45:15.511661+0000","last_fullsized":"2024-03-24T10:45:15.511661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:06.142451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954368+0000","last_change":"2024-03-24T10:45:01.927822+0000","last_active":"2024-03-24T10:45:02.954368+0000","last_peered":"2024-03-24T10:45:02.954368+0000","last_clean":"2024-03-24T10:45:02.954368+0000","last_became_active":"2024-03-24T10:45:01.927538+0000","last_became_peered":"2024-03-24T10:45:01.927538+0000","last_unstale":"2024-03-24T10:45:02.954368+0000","last_undegraded":"2024-03-24T10:45:02.954368+0000","last_fullsized":"2024-03-24T10:45:02.954368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.746638+0000","last_change":"2024-03-24T10:45:00.626818+0000","last_active":"2024-03-24T10:45:12.746638+0000","last_peered":"2024-03-24T10:45:12.746638+0000","last_clean":"2024-03-24T10:45:12.746638+0000","last_became_active":"2024-03-24T10:45:00.626557+0000","last_became_peered":"2024-03-24T10:45:00.626557+0000","last_unstale":"2024-03-24T10:45:12.746638+0000","last_undegraded":"2024-03-24T10:45:12.746638+0000","last_fullsized":"2024-03-24T10:45:12.746638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:43.263796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954131+0000","last_change":"2024-03-24T10:45:01.893192+0000","last_active":"2024-03-24T10:45:02.954131+0000","last_peered":"2024-03-24T10:45:02.954131+0000","last_clean":"2024-03-24T10:45:02.954131+0000","last_became_active":"2024-03-24T10:45:01.892934+0000","last_became_peered":"2024-03-24T10:45:01.892934+0000","last_unstale":"2024-03-24T10:45:02.954131+0000","last_undegraded":"2024-03-24T10:45:02.954131+0000","last_fullsized":"2024-03-24T10:45:02.954131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.772069+0000","last_change":"2024-03-24T10:45:00.630147+0000","last_active":"2024-03-24T10:45:07.772069+0000","last_peered":"2024-03-24T10:45:07.772069+0000","last_clean":"2024-03-24T10:45:07.772069+0000","last_became_active":"2024-03-24T10:45:00.629850+0000","last_became_peered":"2024-03-24T10:45:00.629850+0000","last_unstale":"2024-03-24T10:45:07.772069+0000","last_undegraded":"2024-03-24T10:45:07.772069+0000","last_fullsized":"2024-03-24T10:45:07.772069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:01:12.843222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953404+0000","last_change":"2024-03-24T10:45:01.897333+0000","last_active":"2024-03-24T10:45:02.953404+0000","last_peered":"2024-03-24T10:45:02.953404+0000","last_clean":"2024-03-24T10:45:02.953404+0000","last_became_active":"2024-03-24T10:45:01.897080+0000","last_became_peered":"2024-03-24T10:45:01.897080+0000","last_unstale":"2024-03-24T10:45:02.953404+0000","last_undegraded":"2024-03-24T10:45:02.953404+0000","last_fullsized":"2024-03-24T10:45:02.953404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:13.887947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953968+0000","last_change":"2024-03-24T10:45:01.956041+0000","last_active":"2024-03-24T10:45:02.953968+0000","last_peered":"2024-03-24T10:45:02.953968+0000","last_clean":"2024-03-24T10:45:02.953968+0000","last_became_active":"2024-03-24T10:45:01.955766+0000","last_became_peered":"2024-03-24T10:45:01.955766+0000","last_unstale":"2024-03-24T10:45:02.953968+0000","last_undegraded":"2024-03-24T10:45:02.953968+0000","last_fullsized":"2024-03-24T10:45:02.953968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:16.341360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:15.640266+0000","last_change":"2024-03-24T10:45:00.639913+0000","last_active":"2024-03-24T10:45:15.640266+0000","last_peered":"2024-03-24T10:45:15.640266+0000","last_clean":"2024-03-24T10:45:15.640266+0000","last_became_active":"2024-03-24T10:45:00.639653+0000","last_became_peered":"2024-03-24T10:45:00.639653+0000","last_unstale":"2024-03-24T10:45:15.640266+0000","last_undegraded":"2024-03-24T10:45:15.640266+0000","last_fullsized":"2024-03-24T10:45:15.640266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:15.749451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954233+0000","last_change":"2024-03-24T10:45:01.952340+0000","last_active":"2024-03-24T10:45:02.954233+0000","last_peered":"2024-03-24T10:45:02.954233+0000","last_clean":"2024-03-24T10:45:02.954233+0000","last_became_active":"2024-03-24T10:45:01.952074+0000","last_became_peered":"2024-03-24T10:45:01.952074+0000","last_unstale":"2024-03-24T10:45:02.954233+0000","last_undegraded":"2024-03-24T10:45:02.954233+0000","last_fullsized":"2024-03-24T10:45:02.954233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'11","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:15.640576+0000","last_change":"2024-03-24T10:45:00.633790+0000","last_active":"2024-03-24T10:45:15.640576+0000","last_peered":"2024-03-24T10:45:15.640576+0000","last_clean":"2024-03-24T10:45:15.640576+0000","last_became_active":"2024-03-24T10:45:00.633494+0000","last_became_peered":"2024-03-24T10:45:00.633494+0000","last_unstale":"2024-03-24T10:45:15.640576+0000","last_undegraded":"2024-03-24T10:45:15.640576+0000","last_fullsized":"2024-03-24T10:45:15.640576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:14.049099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953320+0000","last_change":"2024-03-24T10:45:01.957958+0000","last_active":"2024-03-24T10:45:02.953320+0000","last_peered":"2024-03-24T10:45:02.953320+0000","last_clean":"2024-03-24T10:45:02.953320+0000","last_became_active":"2024-03-24T10:45:01.957676+0000","last_became_peered":"2024-03-24T10:45:01.957676+0000","last_unstale":"2024-03-24T10:45:02.953320+0000","last_undegraded":"2024-03-24T10:45:02.953320+0000","last_fullsized":"2024-03-24T10:45:02.953320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:03.159944+0000","last_change":"2024-03-24T10:45:00.629120+0000","last_active":"2024-03-24T10:45:03.159944+0000","last_peered":"2024-03-24T10:45:03.159944+0000","last_clean":"2024-03-24T10:45:03.159944+0000","last_became_active":"2024-03-24T10:45:00.628712+0000","last_became_peered":"2024-03-24T10:45:00.628712+0000","last_unstale":"2024-03-24T10:45:03.159944+0000","last_undegraded":"2024-03-24T10:45:03.159944+0000","last_fullsized":"2024-03-24T10:45:03.159944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954346+0000","last_change":"2024-03-24T10:45:01.894602+0000","last_active":"2024-03-24T10:45:02.954346+0000","last_peered":"2024-03-24T10:45:02.954346+0000","last_clean":"2024-03-24T10:45:02.954346+0000","last_became_active":"2024-03-24T10:45:01.894362+0000","last_became_peered":"2024-03-24T10:45:01.894362+0000","last_unstale":"2024-03-24T10:45:02.954346+0000","last_undegraded":"2024-03-24T10:45:02.954346+0000","last_fullsized":"2024-03-24T10:45:02.954346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'11","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.684496+0000","last_change":"2024-03-24T10:45:00.639171+0000","last_active":"2024-03-24T10:45:12.684496+0000","last_peered":"2024-03-24T10:45:12.684496+0000","last_clean":"2024-03-24T10:45:12.684496+0000","last_became_active":"2024-03-24T10:45:00.638871+0000","last_became_peered":"2024-03-24T10:45:00.638871+0000","last_unstale":"2024-03-24T10:45:12.684496+0000","last_undegraded":"2024-03-24T10:45:12.684496+0000","last_fullsized":"2024-03-24T10:45:12.684496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:52:46.322196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7944,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":15888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":168,"ondisk_log_size":168,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65430,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542155,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31484,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340356,"statfs":{"total":96636764160,"available":96604524544,"internally_reserved":0,"allocated":315392,"data_stored":666673,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542155,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":262144,"data_stored":78137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542154,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":76619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542155,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":75611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542155,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":237568,"data_stored":75149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":241664,"data_stored":76607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":237568,"data_stored":75535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31468,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340372,"statfs":{"total":96636764160,"available":96604540928,"internally_reserved":0,"allocated":299008,"data_stored":665267,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:45:18.648 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:45:18.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.789+0000 7fd3d8d1e640 1 Processor -- start 2024-03-24T10:45:18.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.789+0000 7fd3d8d1e640 1 -- start start 2024-03-24T10:45:18.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.790+0000 7fd3d8d1e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd3d41543a0 0x7fd3d415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.790+0000 7fd3d8d1e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd3d406d4f0 0x7fd3d40cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.790+0000 7fd3d8d1e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3d40cb2f0 con 0x7fd3d40ccaa0 2024-03-24T10:45:18.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.790+0000 7fd3d8d1e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fd3d40c6750 con 0x7fd3d41543a0 2024-03-24T10:45:18.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.790+0000 7fd3d8d1e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fd3d40cc380 con 0x7fd3d406d4f0 2024-03-24T10:45:18.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.791+0000 7fd3d1d9b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd3d406d4f0 0x7fd3d40cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.792+0000 7fd3d1d9b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd3d406d4f0 0x7fd3d40cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36768/0 (socket says 172.21.15.5:36768) 2024-03-24T10:45:18.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.792+0000 7fd3d1d9b640 1 -- 172.21.15.5:0/2642547185 learned_addr learned my addr 172.21.15.5:0/2642547185 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:18.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.795+0000 7fd3d1d9b640 1 -- 172.21.15.5:0/2642547185 >> v2:172.21.15.5:3300/0 conn(0x7fd3d41543a0 msgr2=0x7fd3d415ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:18.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.795+0000 7fd3d1d9b640 1 --2- 172.21.15.5:0/2642547185 >> v2:172.21.15.5:3300/0 conn(0x7fd3d41543a0 0x7fd3d415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.795+0000 7fd3d1d9b640 1 -- 172.21.15.5:0/2642547185 >> v1:172.21.15.5:6789/0 conn(0x7fd3d40ccaa0 legacy=0x7fd3d406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:18.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.795+0000 7fd3d1d9b640 1 -- 172.21.15.5:0/2642547185 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d40ca7e0 con 0x7fd3d406d4f0 2024-03-24T10:45:18.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.799+0000 7fd3d1d9b640 1 --2- 172.21.15.5:0/2642547185 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3d406d4f0 0x7fd3d40cc560 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd3d40c6990 tx=0x7fd3c802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7d6ec927f321333 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:18.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.882+0000 7fd3d159a640 1 -- 172.21.15.5:0/2642547185 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8030860 con 0x7fd3d406d4f0 2024-03-24T10:45:18.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.882+0000 7fd3d159a640 1 -- 172.21.15.5:0/2642547185 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3c80309c0 con 0x7fd3d406d4f0 2024-03-24T10:45:18.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.882+0000 7fd3d159a640 1 -- 172.21.15.5:0/2642547185 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8030d50 con 0x7fd3d406d4f0 2024-03-24T10:45:18.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.883+0000 7fd3d8d1e640 1 -- 172.21.15.5:0/2642547185 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3d406d4f0 msgr2=0x7fd3d40cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:18.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.883+0000 7fd3d8d1e640 1 --2- 172.21.15.5:0/2642547185 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3d406d4f0 0x7fd3d40cc560 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd3d40c6990 tx=0x7fd3c802b9a0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.883+0000 7fd3d8d1e640 1 -- 172.21.15.5:0/2642547185 shutdown_connections 2024-03-24T10:45:18.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.883+0000 7fd3d8d1e640 1 --2- 172.21.15.5:0/2642547185 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3d406d4f0 0x7fd3d40cc560 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.883+0000 7fd3d8d1e640 1 --2- 172.21.15.5:0/2642547185 >> v2:172.21.15.5:3300/0 conn(0x7fd3d41543a0 0x7fd3d415ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.883+0000 7fd3d8d1e640 1 -- 172.21.15.5:0/2642547185 >> 172.21.15.5:0/2642547185 conn(0x7fd3d4072ef0 msgr2=0x7fd3d4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:18.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.883+0000 7fd3d8d1e640 1 -- 172.21.15.5:0/2642547185 shutdown_connections 2024-03-24T10:45:18.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.883+0000 7fd3d8d1e640 1 -- 172.21.15.5:0/2642547185 wait complete. 2024-03-24T10:45:18.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.884+0000 7fd3d8d1e640 1 Processor -- start 2024-03-24T10:45:18.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.884+0000 7fd3d8d1e640 1 -- start start 2024-03-24T10:45:18.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.885+0000 7fd3d8d1e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd3d406d4f0 0x7fd3d41f0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.885+0000 7fd3d8d1e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3d40ccaa0 0x7fd3d40cef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.885+0000 7fd3d8d1e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3d41543a0 0x7fd3d40cf440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.885+0000 7fd3d8d1e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d41f11d0 con 0x7fd3d41543a0 2024-03-24T10:45:18.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.885+0000 7fd3d8d1e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d40cea70 con 0x7fd3d40ccaa0 2024-03-24T10:45:18.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.885+0000 7fd3d8d1e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd3d40cda50 con 0x7fd3d406d4f0 2024-03-24T10:45:18.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.886+0000 7fd3d1d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3d40ccaa0 0x7fd3d40cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.886+0000 7fd3d2d9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3d41543a0 0x7fd3d40cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.886+0000 7fd3d1d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3d40ccaa0 0x7fd3d40cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36774/0 (socket says 172.21.15.5:36774) 2024-03-24T10:45:18.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.886+0000 7fd3d1d9b640 1 -- 172.21.15.5:0/3347988355 learned_addr learned my addr 172.21.15.5:0/3347988355 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:18.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.886+0000 7fd3d259c640 1 --2- 172.21.15.5:0/3347988355 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd3d406d4f0 0x7fd3d41f0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.890+0000 7fd3d2d9d640 1 -- 172.21.15.5:0/3347988355 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd3d406d4f0 msgr2=0x7fd3d41f0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:18.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.890+0000 7fd3d2d9d640 1 --2- 172.21.15.5:0/3347988355 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd3d406d4f0 0x7fd3d41f0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.890+0000 7fd3d2d9d640 1 -- 172.21.15.5:0/3347988355 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3d40ccaa0 msgr2=0x7fd3d40cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:18.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.890+0000 7fd3d2d9d640 1 --2- 172.21.15.5:0/3347988355 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3d40ccaa0 0x7fd3d40cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:18.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.890+0000 7fd3d2d9d640 1 -- 172.21.15.5:0/3347988355 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3c80094d0 con 0x7fd3d41543a0 2024-03-24T10:45:18.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.892+0000 7fd3d2d9d640 1 --2- 172.21.15.5:0/3347988355 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3d41543a0 0x7fd3d40cf440 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc00c780 tx=0x7fd3cc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:18.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.895+0000 7fd3c2ffd640 1 -- 172.21.15.5:0/3347988355 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc003b70 con 0x7fd3d41543a0 2024-03-24T10:45:18.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.895+0000 7fd3c2ffd640 1 -- 172.21.15.5:0/3347988355 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3cc003cf0 con 0x7fd3d41543a0 2024-03-24T10:45:18.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.895+0000 7fd3d8d1e640 1 -- 172.21.15.5:0/3347988355 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3d40cdcf0 con 0x7fd3d41543a0 2024-03-24T10:45:18.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.896+0000 7fd3c2ffd640 1 -- 172.21.15.5:0/3347988355 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc003ea0 con 0x7fd3d41543a0 2024-03-24T10:45:18.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.896+0000 7fd3d8d1e640 1 -- 172.21.15.5:0/3347988355 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3d40ce2b0 con 0x7fd3d41543a0 2024-03-24T10:45:18.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.897+0000 7fd3d8d1e640 1 -- 172.21.15.5:0/3347988355 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3d40681d0 con 0x7fd3d41543a0 2024-03-24T10:45:18.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.915+0000 7fd3c2ffd640 1 -- 172.21.15.5:0/3347988355 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd3cc0052b0 con 0x7fd3d41543a0 2024-03-24T10:45:18.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.915+0000 7fd3c2ffd640 1 --2- 172.21.15.5:0/3347988355 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd3b406b860 0x7fd3b406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:18.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.916+0000 7fd3d259c640 1 --2- 172.21.15.5:0/3347988355 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd3b406b860 0x7fd3b406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:18.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.916+0000 7fd3d259c640 1 --2- 172.21.15.5:0/3347988355 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd3b406b860 0x7fd3b406dd20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd3c40098e0 tx=0x7fd3c4009060 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:18.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:18.919+0000 7fd3c2ffd640 1 -- 172.21.15.5:0/3347988355 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd3cc058680 con 0x7fd3d41543a0 2024-03-24T10:45:19.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.177+0000 7fd3c2ffd640 1 -- 172.21.15.5:0/3347988355 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd3cc054920 con 0x7fd3d41543a0 2024-03-24T10:45:19.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.355+0000 7fd3d8d1e640 1 -- 172.21.15.5:0/3347988355 --> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd3d4159080 con 0x7fd3b406b860 2024-03-24T10:45:19.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.361+0000 7fd3c2ffd640 1 -- 172.21.15.5:0/3347988355 <== mgr.4120 v2:172.21.15.5:6832/1102439181 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252060 (secure 0 0 0) 0x7fd3d4159080 con 0x7fd3b406b860 2024-03-24T10:45:19.364 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:45:19.365 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-03-24T10:45:19.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.364+0000 7fd3d8d1e640 1 -- 172.21.15.5:0/3347988355 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd3b406b860 msgr2=0x7fd3b406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:19.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.364+0000 7fd3d8d1e640 1 --2- 172.21.15.5:0/3347988355 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd3b406b860 0x7fd3b406dd20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd3c40098e0 tx=0x7fd3c4009060 comp rx=0 tx=0).stop 2024-03-24T10:45:19.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.364+0000 7fd3d8d1e640 1 -- 172.21.15.5:0/3347988355 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3d41543a0 msgr2=0x7fd3d40cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:19.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.364+0000 7fd3d8d1e640 1 --2- 172.21.15.5:0/3347988355 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3d41543a0 0x7fd3d40cf440 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc00c780 tx=0x7fd3cc00cc40 comp rx=0 tx=0).stop 2024-03-24T10:45:19.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.364+0000 7fd3d8d1e640 1 -- 172.21.15.5:0/3347988355 shutdown_connections 2024-03-24T10:45:19.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.364+0000 7fd3d8d1e640 1 --2- 172.21.15.5:0/3347988355 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd3b406b860 0x7fd3b406dd20 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:19.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.364+0000 7fd3d8d1e640 1 --2- 172.21.15.5:0/3347988355 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd3d41543a0 0x7fd3d40cf440 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:19.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.364+0000 7fd3d8d1e640 1 --2- 172.21.15.5:0/3347988355 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd3d40ccaa0 0x7fd3d40cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:19.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.364+0000 7fd3d8d1e640 1 --2- 172.21.15.5:0/3347988355 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd3d406d4f0 0x7fd3d41f0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:19.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.364+0000 7fd3d8d1e640 1 -- 172.21.15.5:0/3347988355 >> 172.21.15.5:0/3347988355 conn(0x7fd3d4072ef0 msgr2=0x7fd3d4149430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:19.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.365+0000 7fd3d8d1e640 1 -- 172.21.15.5:0/3347988355 shutdown_connections 2024-03-24T10:45:19.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.365+0000 7fd3d8d1e640 1 -- 172.21.15.5:0/3347988355 wait complete. 2024-03-24T10:45:19.376 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-03-24T10:45:17.373412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598312,"num_objects":93,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":351,"num_write_kb":2477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":285,"ondisk_log_size":285,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238952,"kb_used_data":2024,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754735768,"statfs":{"total":773094113280,"available":772849426432,"internally_reserved":0,"allocated":2072576,"data_stored":1789598,"data_compressed":65430,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":242535986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5694,"num_objects":81,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":175,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":56,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002522"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956091+0000","last_change":"2024-03-24T10:45:01.894006+0000","last_active":"2024-03-24T10:45:02.956091+0000","last_peered":"2024-03-24T10:45:02.956091+0000","last_clean":"2024-03-24T10:45:02.956091+0000","last_became_active":"2024-03-24T10:45:01.893734+0000","last_became_peered":"2024-03-24T10:45:01.893734+0000","last_unstale":"2024-03-24T10:45:02.956091+0000","last_undegraded":"2024-03-24T10:45:02.956091+0000","last_fullsized":"2024-03-24T10:45:02.956091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:05:51.579388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.932998+0000","last_change":"2024-03-24T10:45:01.931707+0000","last_active":"2024-03-24T10:45:02.932998+0000","last_peered":"2024-03-24T10:45:02.932998+0000","last_clean":"2024-03-24T10:45:02.932998+0000","last_became_active":"2024-03-24T10:45:01.931449+0000","last_became_peered":"2024-03-24T10:45:01.931449+0000","last_unstale":"2024-03-24T10:45:02.932998+0000","last_undegraded":"2024-03-24T10:45:02.932998+0000","last_fullsized":"2024-03-24T10:45:02.932998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953238+0000","last_change":"2024-03-24T10:45:01.895929+0000","last_active":"2024-03-24T10:45:02.953238+0000","last_peered":"2024-03-24T10:45:02.953238+0000","last_clean":"2024-03-24T10:45:02.953238+0000","last_became_active":"2024-03-24T10:45:01.895654+0000","last_became_peered":"2024-03-24T10:45:01.895654+0000","last_unstale":"2024-03-24T10:45:02.953238+0000","last_undegraded":"2024-03-24T10:45:02.953238+0000","last_fullsized":"2024-03-24T10:45:02.953238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954093+0000","last_change":"2024-03-24T10:45:01.952468+0000","last_active":"2024-03-24T10:45:02.954093+0000","last_peered":"2024-03-24T10:45:02.954093+0000","last_clean":"2024-03-24T10:45:02.954093+0000","last_became_active":"2024-03-24T10:45:01.952219+0000","last_became_peered":"2024-03-24T10:45:01.952219+0000","last_unstale":"2024-03-24T10:45:02.954093+0000","last_undegraded":"2024-03-24T10:45:02.954093+0000","last_fullsized":"2024-03-24T10:45:02.954093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:27:12.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955739+0000","last_change":"2024-03-24T10:45:01.895885+0000","last_active":"2024-03-24T10:45:02.955739+0000","last_peered":"2024-03-24T10:45:02.955739+0000","last_clean":"2024-03-24T10:45:02.955739+0000","last_became_active":"2024-03-24T10:45:01.895622+0000","last_became_peered":"2024-03-24T10:45:01.895622+0000","last_unstale":"2024-03-24T10:45:02.955739+0000","last_undegraded":"2024-03-24T10:45:02.955739+0000","last_fullsized":"2024-03-24T10:45:02.955739+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:16.523177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954340+0000","last_change":"2024-03-24T10:45:01.958421+0000","last_active":"2024-03-24T10:45:02.954340+0000","last_peered":"2024-03-24T10:45:02.954340+0000","last_clean":"2024-03-24T10:45:02.954340+0000","last_became_active":"2024-03-24T10:45:01.957972+0000","last_became_peered":"2024-03-24T10:45:01.957972+0000","last_unstale":"2024-03-24T10:45:02.954340+0000","last_undegraded":"2024-03-24T10:45:02.954340+0000","last_fullsized":"2024-03-24T10:45:02.954340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:16.523177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955433+0000","last_change":"2024-03-24T10:45:01.892745+0000","last_active":"2024-03-24T10:45:02.955433+0000","last_peered":"2024-03-24T10:45:02.955433+0000","last_clean":"2024-03-24T10:45:02.955433+0000","last_became_active":"2024-03-24T10:45:01.892452+0000","last_became_peered":"2024-03-24T10:45:01.892452+0000","last_unstale":"2024-03-24T10:45:02.955433+0000","last_undegraded":"2024-03-24T10:45:02.955433+0000","last_fullsized":"2024-03-24T10:45:02.955433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954102+0000","last_change":"2024-03-24T10:45:01.953432+0000","last_active":"2024-03-24T10:45:02.954102+0000","last_peered":"2024-03-24T10:45:02.954102+0000","last_clean":"2024-03-24T10:45:02.954102+0000","last_became_active":"2024-03-24T10:45:01.953149+0000","last_became_peered":"2024-03-24T10:45:01.953149+0000","last_unstale":"2024-03-24T10:45:02.954102+0000","last_undegraded":"2024-03-24T10:45:02.954102+0000","last_fullsized":"2024-03-24T10:45:02.954102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956172+0000","last_change":"2024-03-24T10:45:01.956488+0000","last_active":"2024-03-24T10:45:02.956172+0000","last_peered":"2024-03-24T10:45:02.956172+0000","last_clean":"2024-03-24T10:45:02.956172+0000","last_became_active":"2024-03-24T10:45:01.956207+0000","last_became_peered":"2024-03-24T10:45:01.956207+0000","last_unstale":"2024-03-24T10:45:02.956172+0000","last_undegraded":"2024-03-24T10:45:02.956172+0000","last_fullsized":"2024-03-24T10:45:02.956172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:23:26.523809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953720+0000","last_change":"2024-03-24T10:45:01.956484+0000","last_active":"2024-03-24T10:45:02.953720+0000","last_peered":"2024-03-24T10:45:02.953720+0000","last_clean":"2024-03-24T10:45:02.953720+0000","last_became_active":"2024-03-24T10:45:01.956209+0000","last_became_peered":"2024-03-24T10:45:01.956209+0000","last_unstale":"2024-03-24T10:45:02.953720+0000","last_undegraded":"2024-03-24T10:45:02.953720+0000","last_fullsized":"2024-03-24T10:45:02.953720+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953838+0000","last_change":"2024-03-24T10:45:01.957103+0000","last_active":"2024-03-24T10:45:02.953838+0000","last_peered":"2024-03-24T10:45:02.953838+0000","last_clean":"2024-03-24T10:45:02.953838+0000","last_became_active":"2024-03-24T10:45:01.956849+0000","last_became_peered":"2024-03-24T10:45:01.956849+0000","last_unstale":"2024-03-24T10:45:02.953838+0000","last_undegraded":"2024-03-24T10:45:02.953838+0000","last_fullsized":"2024-03-24T10:45:02.953838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954691+0000","last_change":"2024-03-24T10:45:01.896701+0000","last_active":"2024-03-24T10:45:02.954691+0000","last_peered":"2024-03-24T10:45:02.954691+0000","last_clean":"2024-03-24T10:45:02.954691+0000","last_became_active":"2024-03-24T10:45:01.896400+0000","last_became_peered":"2024-03-24T10:45:01.896400+0000","last_unstale":"2024-03-24T10:45:02.954691+0000","last_undegraded":"2024-03-24T10:45:02.954691+0000","last_fullsized":"2024-03-24T10:45:02.954691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:44.380352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953999+0000","last_change":"2024-03-24T10:45:01.894954+0000","last_active":"2024-03-24T10:45:02.953999+0000","last_peered":"2024-03-24T10:45:02.953999+0000","last_clean":"2024-03-24T10:45:02.953999+0000","last_became_active":"2024-03-24T10:45:01.894632+0000","last_became_peered":"2024-03-24T10:45:01.894632+0000","last_unstale":"2024-03-24T10:45:02.953999+0000","last_undegraded":"2024-03-24T10:45:02.953999+0000","last_fullsized":"2024-03-24T10:45:02.953999+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.952924+0000","last_change":"2024-03-24T10:45:01.929588+0000","last_active":"2024-03-24T10:45:02.952924+0000","last_peered":"2024-03-24T10:45:02.952924+0000","last_clean":"2024-03-24T10:45:02.952924+0000","last_became_active":"2024-03-24T10:45:01.928460+0000","last_became_peered":"2024-03-24T10:45:01.928460+0000","last_unstale":"2024-03-24T10:45:02.952924+0000","last_undegraded":"2024-03-24T10:45:02.952924+0000","last_fullsized":"2024-03-24T10:45:02.952924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:47:48.896914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933000+0000","last_change":"2024-03-24T10:45:01.928721+0000","last_active":"2024-03-24T10:45:02.933000+0000","last_peered":"2024-03-24T10:45:02.933000+0000","last_clean":"2024-03-24T10:45:02.933000+0000","last_became_active":"2024-03-24T10:45:01.928467+0000","last_became_peered":"2024-03-24T10:45:01.928467+0000","last_unstale":"2024-03-24T10:45:02.933000+0000","last_undegraded":"2024-03-24T10:45:02.933000+0000","last_fullsized":"2024-03-24T10:45:02.933000+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953333+0000","last_change":"2024-03-24T10:45:01.956030+0000","last_active":"2024-03-24T10:45:02.953333+0000","last_peered":"2024-03-24T10:45:02.953333+0000","last_clean":"2024-03-24T10:45:02.953333+0000","last_became_active":"2024-03-24T10:45:01.955768+0000","last_became_peered":"2024-03-24T10:45:01.955768+0000","last_unstale":"2024-03-24T10:45:02.953333+0000","last_undegraded":"2024-03-24T10:45:02.953333+0000","last_fullsized":"2024-03-24T10:45:02.953333+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954695+0000","last_change":"2024-03-24T10:45:01.956181+0000","last_active":"2024-03-24T10:45:02.954695+0000","last_peered":"2024-03-24T10:45:02.954695+0000","last_clean":"2024-03-24T10:45:02.954695+0000","last_became_active":"2024-03-24T10:45:01.955890+0000","last_became_peered":"2024-03-24T10:45:01.955890+0000","last_unstale":"2024-03-24T10:45:02.954695+0000","last_undegraded":"2024-03-24T10:45:02.954695+0000","last_fullsized":"2024-03-24T10:45:02.954695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:15:11.107915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.936626+0000","last_change":"2024-03-24T10:45:01.896663+0000","last_active":"2024-03-24T10:45:02.936626+0000","last_peered":"2024-03-24T10:45:02.936626+0000","last_clean":"2024-03-24T10:45:02.936626+0000","last_became_active":"2024-03-24T10:45:01.896391+0000","last_became_peered":"2024-03-24T10:45:01.896391+0000","last_unstale":"2024-03-24T10:45:02.936626+0000","last_undegraded":"2024-03-24T10:45:02.936626+0000","last_fullsized":"2024-03-24T10:45:02.936626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:30.942758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956053+0000","last_change":"2024-03-24T10:45:01.895951+0000","last_active":"2024-03-24T10:45:02.956053+0000","last_peered":"2024-03-24T10:45:02.956053+0000","last_clean":"2024-03-24T10:45:02.956053+0000","last_became_active":"2024-03-24T10:45:01.895668+0000","last_became_peered":"2024-03-24T10:45:01.895668+0000","last_unstale":"2024-03-24T10:45:02.956053+0000","last_undegraded":"2024-03-24T10:45:02.956053+0000","last_fullsized":"2024-03-24T10:45:02.956053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:56:43.462458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953964+0000","last_change":"2024-03-24T10:45:01.898126+0000","last_active":"2024-03-24T10:45:02.953964+0000","last_peered":"2024-03-24T10:45:02.953964+0000","last_clean":"2024-03-24T10:45:02.953964+0000","last_became_active":"2024-03-24T10:45:01.897878+0000","last_became_peered":"2024-03-24T10:45:01.897878+0000","last_unstale":"2024-03-24T10:45:02.953964+0000","last_undegraded":"2024-03-24T10:45:02.953964+0000","last_fullsized":"2024-03-24T10:45:02.953964+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953247+0000","last_change":"2024-03-24T10:45:01.896109+0000","last_active":"2024-03-24T10:45:02.953247+0000","last_peered":"2024-03-24T10:45:02.953247+0000","last_clean":"2024-03-24T10:45:02.953247+0000","last_became_active":"2024-03-24T10:45:01.895696+0000","last_became_peered":"2024-03-24T10:45:01.895696+0000","last_unstale":"2024-03-24T10:45:02.953247+0000","last_undegraded":"2024-03-24T10:45:02.953247+0000","last_fullsized":"2024-03-24T10:45:02.953247+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954483+0000","last_change":"2024-03-24T10:45:01.958119+0000","last_active":"2024-03-24T10:45:02.954483+0000","last_peered":"2024-03-24T10:45:02.954483+0000","last_clean":"2024-03-24T10:45:02.954483+0000","last_became_active":"2024-03-24T10:45:01.957832+0000","last_became_peered":"2024-03-24T10:45:01.957832+0000","last_unstale":"2024-03-24T10:45:02.954483+0000","last_undegraded":"2024-03-24T10:45:02.954483+0000","last_fullsized":"2024-03-24T10:45:02.954483+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:12.476490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953638+0000","last_change":"2024-03-24T10:45:01.842442+0000","last_active":"2024-03-24T10:45:02.953638+0000","last_peered":"2024-03-24T10:45:02.953638+0000","last_clean":"2024-03-24T10:45:02.953638+0000","last_became_active":"2024-03-24T10:45:01.842199+0000","last_became_peered":"2024-03-24T10:45:01.842199+0000","last_unstale":"2024-03-24T10:45:02.953638+0000","last_undegraded":"2024-03-24T10:45:02.953638+0000","last_fullsized":"2024-03-24T10:45:02.953638+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953764+0000","last_change":"2024-03-24T10:45:01.952378+0000","last_active":"2024-03-24T10:45:02.953764+0000","last_peered":"2024-03-24T10:45:02.953764+0000","last_clean":"2024-03-24T10:45:02.953764+0000","last_became_active":"2024-03-24T10:45:01.952123+0000","last_became_peered":"2024-03-24T10:45:01.952123+0000","last_unstale":"2024-03-24T10:45:02.953764+0000","last_undegraded":"2024-03-24T10:45:02.953764+0000","last_fullsized":"2024-03-24T10:45:02.953764+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954628+0000","last_change":"2024-03-24T10:45:01.842299+0000","last_active":"2024-03-24T10:45:02.954628+0000","last_peered":"2024-03-24T10:45:02.954628+0000","last_clean":"2024-03-24T10:45:02.954628+0000","last_became_active":"2024-03-24T10:45:01.842097+0000","last_became_peered":"2024-03-24T10:45:01.842097+0000","last_unstale":"2024-03-24T10:45:02.954628+0000","last_undegraded":"2024-03-24T10:45:02.954628+0000","last_fullsized":"2024-03-24T10:45:02.954628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:16.341360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953886+0000","last_change":"2024-03-24T10:45:01.893579+0000","last_active":"2024-03-24T10:45:02.953886+0000","last_peered":"2024-03-24T10:45:02.953886+0000","last_clean":"2024-03-24T10:45:02.953886+0000","last_became_active":"2024-03-24T10:45:01.893273+0000","last_became_peered":"2024-03-24T10:45:01.893273+0000","last_unstale":"2024-03-24T10:45:02.953886+0000","last_undegraded":"2024-03-24T10:45:02.953886+0000","last_fullsized":"2024-03-24T10:45:02.953886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933308+0000","last_change":"2024-03-24T10:45:01.930032+0000","last_active":"2024-03-24T10:45:02.933308+0000","last_peered":"2024-03-24T10:45:02.933308+0000","last_clean":"2024-03-24T10:45:02.933308+0000","last_became_active":"2024-03-24T10:45:01.929770+0000","last_became_peered":"2024-03-24T10:45:01.929770+0000","last_unstale":"2024-03-24T10:45:02.933308+0000","last_undegraded":"2024-03-24T10:45:02.933308+0000","last_fullsized":"2024-03-24T10:45:02.933308+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954962+0000","last_change":"2024-03-24T10:45:01.842460+0000","last_active":"2024-03-24T10:45:02.954962+0000","last_peered":"2024-03-24T10:45:02.954962+0000","last_clean":"2024-03-24T10:45:02.954962+0000","last_became_active":"2024-03-24T10:45:01.842165+0000","last_became_peered":"2024-03-24T10:45:01.842165+0000","last_unstale":"2024-03-24T10:45:02.954962+0000","last_undegraded":"2024-03-24T10:45:02.954962+0000","last_fullsized":"2024-03-24T10:45:02.954962+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953401+0000","last_change":"2024-03-24T10:45:01.895868+0000","last_active":"2024-03-24T10:45:02.953401+0000","last_peered":"2024-03-24T10:45:02.953401+0000","last_clean":"2024-03-24T10:45:02.953401+0000","last_became_active":"2024-03-24T10:45:01.895290+0000","last_became_peered":"2024-03-24T10:45:01.895290+0000","last_unstale":"2024-03-24T10:45:02.953401+0000","last_undegraded":"2024-03-24T10:45:02.953401+0000","last_fullsized":"2024-03-24T10:45:02.953401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954615+0000","last_change":"2024-03-24T10:45:01.957724+0000","last_active":"2024-03-24T10:45:02.954615+0000","last_peered":"2024-03-24T10:45:02.954615+0000","last_clean":"2024-03-24T10:45:02.954615+0000","last_became_active":"2024-03-24T10:45:01.957429+0000","last_became_peered":"2024-03-24T10:45:01.957429+0000","last_unstale":"2024-03-24T10:45:02.954615+0000","last_undegraded":"2024-03-24T10:45:02.954615+0000","last_fullsized":"2024-03-24T10:45:02.954615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:22.089070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.936465+0000","last_change":"2024-03-24T10:45:01.952159+0000","last_active":"2024-03-24T10:45:02.936465+0000","last_peered":"2024-03-24T10:45:02.936465+0000","last_clean":"2024-03-24T10:45:02.936465+0000","last_became_active":"2024-03-24T10:45:01.951961+0000","last_became_peered":"2024-03-24T10:45:01.951961+0000","last_unstale":"2024-03-24T10:45:02.936465+0000","last_undegraded":"2024-03-24T10:45:02.936465+0000","last_fullsized":"2024-03-24T10:45:02.936465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:44.380352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953610+0000","last_change":"2024-03-24T10:45:01.893452+0000","last_active":"2024-03-24T10:45:02.953610+0000","last_peered":"2024-03-24T10:45:02.953610+0000","last_clean":"2024-03-24T10:45:02.953610+0000","last_became_active":"2024-03-24T10:45:01.893139+0000","last_became_peered":"2024-03-24T10:45:01.893139+0000","last_unstale":"2024-03-24T10:45:02.953610+0000","last_undegraded":"2024-03-24T10:45:02.953610+0000","last_fullsized":"2024-03-24T10:45:02.953610+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953542+0000","last_change":"2024-03-24T10:45:01.928827+0000","last_active":"2024-03-24T10:45:02.953542+0000","last_peered":"2024-03-24T10:45:02.953542+0000","last_clean":"2024-03-24T10:45:02.953542+0000","last_became_active":"2024-03-24T10:45:01.928521+0000","last_became_peered":"2024-03-24T10:45:01.928521+0000","last_unstale":"2024-03-24T10:45:02.953542+0000","last_undegraded":"2024-03-24T10:45:02.953542+0000","last_fullsized":"2024-03-24T10:45:02.953542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:59.643246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.699015+0000","last_change":"2024-03-24T10:45:00.637993+0000","last_active":"2024-03-24T10:45:07.699015+0000","last_peered":"2024-03-24T10:45:07.699015+0000","last_clean":"2024-03-24T10:45:07.699015+0000","last_became_active":"2024-03-24T10:45:00.637803+0000","last_became_peered":"2024-03-24T10:45:00.637803+0000","last_unstale":"2024-03-24T10:45:07.699015+0000","last_undegraded":"2024-03-24T10:45:07.699015+0000","last_fullsized":"2024-03-24T10:45:07.699015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:28:28.854541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953550+0000","last_change":"2024-03-24T10:45:01.897686+0000","last_active":"2024-03-24T10:45:02.953550+0000","last_peered":"2024-03-24T10:45:02.953550+0000","last_clean":"2024-03-24T10:45:02.953550+0000","last_became_active":"2024-03-24T10:45:01.897320+0000","last_became_peered":"2024-03-24T10:45:01.897320+0000","last_unstale":"2024-03-24T10:45:02.953550+0000","last_undegraded":"2024-03-24T10:45:02.953550+0000","last_fullsized":"2024-03-24T10:45:02.953550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.776260+0000","last_change":"2024-03-24T10:45:00.638329+0000","last_active":"2024-03-24T10:45:12.776260+0000","last_peered":"2024-03-24T10:45:12.776260+0000","last_clean":"2024-03-24T10:45:12.776260+0000","last_became_active":"2024-03-24T10:45:00.638006+0000","last_became_peered":"2024-03-24T10:45:00.638006+0000","last_unstale":"2024-03-24T10:45:12.776260+0000","last_undegraded":"2024-03-24T10:45:12.776260+0000","last_fullsized":"2024-03-24T10:45:12.776260+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:10.355399+0000","last_change":"2024-03-24T10:45:00.633703+0000","last_active":"2024-03-24T10:45:10.355399+0000","last_peered":"2024-03-24T10:45:10.355399+0000","last_clean":"2024-03-24T10:45:10.355399+0000","last_became_active":"2024-03-24T10:45:00.633427+0000","last_became_peered":"2024-03-24T10:45:00.633427+0000","last_unstale":"2024-03-24T10:45:10.355399+0000","last_undegraded":"2024-03-24T10:45:10.355399+0000","last_fullsized":"2024-03-24T10:45:10.355399+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:30.350191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954240+0000","last_change":"2024-03-24T10:45:01.893902+0000","last_active":"2024-03-24T10:45:02.954240+0000","last_peered":"2024-03-24T10:45:02.954240+0000","last_clean":"2024-03-24T10:45:02.954240+0000","last_became_active":"2024-03-24T10:45:01.893617+0000","last_became_peered":"2024-03-24T10:45:01.893617+0000","last_unstale":"2024-03-24T10:45:02.954240+0000","last_undegraded":"2024-03-24T10:45:02.954240+0000","last_fullsized":"2024-03-24T10:45:02.954240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.780177+0000","last_change":"2024-03-24T10:45:00.638778+0000","last_active":"2024-03-24T10:45:12.780177+0000","last_peered":"2024-03-24T10:45:12.780177+0000","last_clean":"2024-03-24T10:45:12.780177+0000","last_became_active":"2024-03-24T10:45:00.638089+0000","last_became_peered":"2024-03-24T10:45:00.638089+0000","last_unstale":"2024-03-24T10:45:12.780177+0000","last_undegraded":"2024-03-24T10:45:12.780177+0000","last_fullsized":"2024-03-24T10:45:12.780177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.932839+0000","last_change":"2024-03-24T10:45:01.930063+0000","last_active":"2024-03-24T10:45:02.932839+0000","last_peered":"2024-03-24T10:45:02.932839+0000","last_clean":"2024-03-24T10:45:02.932839+0000","last_became_active":"2024-03-24T10:45:01.929786+0000","last_became_peered":"2024-03-24T10:45:01.929786+0000","last_unstale":"2024-03-24T10:45:02.932839+0000","last_undegraded":"2024-03-24T10:45:02.932839+0000","last_fullsized":"2024-03-24T10:45:02.932839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.773173+0000","last_change":"2024-03-24T10:45:00.639822+0000","last_active":"2024-03-24T10:45:07.773173+0000","last_peered":"2024-03-24T10:45:07.773173+0000","last_clean":"2024-03-24T10:45:07.773173+0000","last_became_active":"2024-03-24T10:45:00.639600+0000","last_became_peered":"2024-03-24T10:45:00.639600+0000","last_unstale":"2024-03-24T10:45:07.773173+0000","last_undegraded":"2024-03-24T10:45:07.773173+0000","last_fullsized":"2024-03-24T10:45:07.773173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953735+0000","last_change":"2024-03-24T10:45:01.954015+0000","last_active":"2024-03-24T10:45:02.953735+0000","last_peered":"2024-03-24T10:45:02.953735+0000","last_clean":"2024-03-24T10:45:02.953735+0000","last_became_active":"2024-03-24T10:45:01.953767+0000","last_became_peered":"2024-03-24T10:45:01.953767+0000","last_unstale":"2024-03-24T10:45:02.953735+0000","last_undegraded":"2024-03-24T10:45:02.953735+0000","last_fullsized":"2024-03-24T10:45:02.953735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:15.553426+0000","last_change":"2024-03-24T10:45:00.636553+0000","last_active":"2024-03-24T10:45:15.553426+0000","last_peered":"2024-03-24T10:45:15.553426+0000","last_clean":"2024-03-24T10:45:15.553426+0000","last_became_active":"2024-03-24T10:45:00.636282+0000","last_became_peered":"2024-03-24T10:45:00.636282+0000","last_unstale":"2024-03-24T10:45:15.553426+0000","last_undegraded":"2024-03-24T10:45:15.553426+0000","last_fullsized":"2024-03-24T10:45:15.553426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953458+0000","last_change":"2024-03-24T10:45:01.957905+0000","last_active":"2024-03-24T10:45:02.953458+0000","last_peered":"2024-03-24T10:45:02.953458+0000","last_clean":"2024-03-24T10:45:02.953458+0000","last_became_active":"2024-03-24T10:45:01.957676+0000","last_became_peered":"2024-03-24T10:45:01.957676+0000","last_unstale":"2024-03-24T10:45:02.953458+0000","last_undegraded":"2024-03-24T10:45:02.953458+0000","last_fullsized":"2024-03-24T10:45:02.953458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.798829+0000","last_change":"2024-03-24T10:45:00.635197+0000","last_active":"2024-03-24T10:45:07.798829+0000","last_peered":"2024-03-24T10:45:07.798829+0000","last_clean":"2024-03-24T10:45:07.798829+0000","last_became_active":"2024-03-24T10:45:00.634989+0000","last_became_peered":"2024-03-24T10:45:00.634989+0000","last_unstale":"2024-03-24T10:45:07.798829+0000","last_undegraded":"2024-03-24T10:45:07.798829+0000","last_fullsized":"2024-03-24T10:45:07.798829+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954821+0000","last_change":"2024-03-24T10:45:01.853785+0000","last_active":"2024-03-24T10:45:02.954821+0000","last_peered":"2024-03-24T10:45:02.954821+0000","last_clean":"2024-03-24T10:45:02.954821+0000","last_became_active":"2024-03-24T10:45:01.853504+0000","last_became_peered":"2024-03-24T10:45:01.853504+0000","last_unstale":"2024-03-24T10:45:02.954821+0000","last_undegraded":"2024-03-24T10:45:02.954821+0000","last_fullsized":"2024-03-24T10:45:02.954821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:30.942758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.773017+0000","last_change":"2024-03-24T10:45:00.632381+0000","last_active":"2024-03-24T10:45:07.773017+0000","last_peered":"2024-03-24T10:45:07.773017+0000","last_clean":"2024-03-24T10:45:07.773017+0000","last_became_active":"2024-03-24T10:45:00.632108+0000","last_became_peered":"2024-03-24T10:45:00.632108+0000","last_unstale":"2024-03-24T10:45:07.773017+0000","last_undegraded":"2024-03-24T10:45:07.773017+0000","last_fullsized":"2024-03-24T10:45:07.773017+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:06.142451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955746+0000","last_change":"2024-03-24T10:45:01.853591+0000","last_active":"2024-03-24T10:45:02.955746+0000","last_peered":"2024-03-24T10:45:02.955746+0000","last_clean":"2024-03-24T10:45:02.955746+0000","last_became_active":"2024-03-24T10:45:01.853279+0000","last_became_peered":"2024-03-24T10:45:01.853279+0000","last_unstale":"2024-03-24T10:45:02.955746+0000","last_undegraded":"2024-03-24T10:45:02.955746+0000","last_fullsized":"2024-03-24T10:45:02.955746+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:07:24.149961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:10.356293+0000","last_change":"2024-03-24T10:45:00.634192+0000","last_active":"2024-03-24T10:45:10.356293+0000","last_peered":"2024-03-24T10:45:10.356293+0000","last_clean":"2024-03-24T10:45:10.356293+0000","last_became_active":"2024-03-24T10:45:00.633893+0000","last_became_peered":"2024-03-24T10:45:00.633893+0000","last_unstale":"2024-03-24T10:45:10.356293+0000","last_undegraded":"2024-03-24T10:45:10.356293+0000","last_fullsized":"2024-03-24T10:45:10.356293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:30.350191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953075+0000","last_change":"2024-03-24T10:45:01.893426+0000","last_active":"2024-03-24T10:45:02.953075+0000","last_peered":"2024-03-24T10:45:02.953075+0000","last_clean":"2024-03-24T10:45:02.953075+0000","last_became_active":"2024-03-24T10:45:01.893178+0000","last_became_peered":"2024-03-24T10:45:01.893178+0000","last_unstale":"2024-03-24T10:45:02.953075+0000","last_undegraded":"2024-03-24T10:45:02.953075+0000","last_fullsized":"2024-03-24T10:45:02.953075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:13.887947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.780472+0000","last_change":"2024-03-24T10:45:00.638214+0000","last_active":"2024-03-24T10:45:12.780472+0000","last_peered":"2024-03-24T10:45:12.780472+0000","last_clean":"2024-03-24T10:45:12.780472+0000","last_became_active":"2024-03-24T10:45:00.637973+0000","last_became_peered":"2024-03-24T10:45:00.637973+0000","last_unstale":"2024-03-24T10:45:12.780472+0000","last_undegraded":"2024-03-24T10:45:12.780472+0000","last_fullsized":"2024-03-24T10:45:12.780472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:15.749451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955767+0000","last_change":"2024-03-24T10:45:01.856756+0000","last_active":"2024-03-24T10:45:02.955767+0000","last_peered":"2024-03-24T10:45:02.955767+0000","last_clean":"2024-03-24T10:45:02.955767+0000","last_became_active":"2024-03-24T10:45:01.856482+0000","last_became_peered":"2024-03-24T10:45:01.856482+0000","last_unstale":"2024-03-24T10:45:02.955767+0000","last_undegraded":"2024-03-24T10:45:02.955767+0000","last_fullsized":"2024-03-24T10:45:02.955767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:12.476490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.773462+0000","last_change":"2024-03-24T10:45:00.636481+0000","last_active":"2024-03-24T10:45:12.773462+0000","last_peered":"2024-03-24T10:45:12.773462+0000","last_clean":"2024-03-24T10:45:12.773462+0000","last_became_active":"2024-03-24T10:45:00.636206+0000","last_became_peered":"2024-03-24T10:45:00.636206+0000","last_unstale":"2024-03-24T10:45:12.773462+0000","last_undegraded":"2024-03-24T10:45:12.773462+0000","last_fullsized":"2024-03-24T10:45:12.773462+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954153+0000","last_change":"2024-03-24T10:45:01.897279+0000","last_active":"2024-03-24T10:45:02.954153+0000","last_peered":"2024-03-24T10:45:02.954153+0000","last_clean":"2024-03-24T10:45:02.954153+0000","last_became_active":"2024-03-24T10:45:01.896736+0000","last_became_peered":"2024-03-24T10:45:01.896736+0000","last_unstale":"2024-03-24T10:45:02.954153+0000","last_undegraded":"2024-03-24T10:45:02.954153+0000","last_fullsized":"2024-03-24T10:45:02.954153+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":31,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.794043+0000","last_change":"2024-03-24T10:45:00.564066+0000","last_active":"2024-03-24T10:45:12.794043+0000","last_peered":"2024-03-24T10:45:12.794043+0000","last_clean":"2024-03-24T10:45:12.794043+0000","last_became_active":"2024-03-24T10:45:00.563812+0000","last_became_peered":"2024-03-24T10:45:00.563812+0000","last_unstale":"2024-03-24T10:45:12.794043+0000","last_undegraded":"2024-03-24T10:45:12.794043+0000","last_fullsized":"2024-03-24T10:45:12.794043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:28:28.854541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955401+0000","last_change":"2024-03-24T10:45:01.854429+0000","last_active":"2024-03-24T10:45:02.955401+0000","last_peered":"2024-03-24T10:45:02.955401+0000","last_clean":"2024-03-24T10:45:02.955401+0000","last_became_active":"2024-03-24T10:45:01.854179+0000","last_became_peered":"2024-03-24T10:45:01.854179+0000","last_unstale":"2024-03-24T10:45:02.955401+0000","last_undegraded":"2024-03-24T10:45:02.955401+0000","last_fullsized":"2024-03-24T10:45:02.955401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:03.132419+0000","last_change":"2024-03-24T10:45:00.634223+0000","last_active":"2024-03-24T10:45:03.132419+0000","last_peered":"2024-03-24T10:45:03.132419+0000","last_clean":"2024-03-24T10:45:03.132419+0000","last_became_active":"2024-03-24T10:45:00.633355+0000","last_became_peered":"2024-03-24T10:45:00.633355+0000","last_unstale":"2024-03-24T10:45:03.132419+0000","last_undegraded":"2024-03-24T10:45:03.132419+0000","last_fullsized":"2024-03-24T10:45:03.132419+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:04.930115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956031+0000","last_change":"2024-03-24T10:45:01.856935+0000","last_active":"2024-03-24T10:45:02.956031+0000","last_peered":"2024-03-24T10:45:02.956031+0000","last_clean":"2024-03-24T10:45:02.956031+0000","last_became_active":"2024-03-24T10:45:01.856653+0000","last_became_peered":"2024-03-24T10:45:01.856653+0000","last_unstale":"2024-03-24T10:45:02.956031+0000","last_undegraded":"2024-03-24T10:45:02.956031+0000","last_fullsized":"2024-03-24T10:45:02.956031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:15:11.107915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:15.622360+0000","last_change":"2024-03-24T10:45:00.626484+0000","last_active":"2024-03-24T10:45:15.622360+0000","last_peered":"2024-03-24T10:45:15.622360+0000","last_clean":"2024-03-24T10:45:15.622360+0000","last_became_active":"2024-03-24T10:45:00.626195+0000","last_became_peered":"2024-03-24T10:45:00.626195+0000","last_unstale":"2024-03-24T10:45:15.622360+0000","last_undegraded":"2024-03-24T10:45:15.622360+0000","last_fullsized":"2024-03-24T10:45:15.622360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.956227+0000","last_change":"2024-03-24T10:45:01.856995+0000","last_active":"2024-03-24T10:45:02.956227+0000","last_peered":"2024-03-24T10:45:02.956227+0000","last_clean":"2024-03-24T10:45:02.956227+0000","last_became_active":"2024-03-24T10:45:01.856531+0000","last_became_peered":"2024-03-24T10:45:01.856531+0000","last_unstale":"2024-03-24T10:45:02.956227+0000","last_undegraded":"2024-03-24T10:45:02.956227+0000","last_fullsized":"2024-03-24T10:45:02.956227+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:29:31.508911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.746930+0000","last_change":"2024-03-24T10:45:00.628438+0000","last_active":"2024-03-24T10:45:12.746930+0000","last_peered":"2024-03-24T10:45:12.746930+0000","last_clean":"2024-03-24T10:45:12.746930+0000","last_became_active":"2024-03-24T10:45:00.628158+0000","last_became_peered":"2024-03-24T10:45:00.628158+0000","last_unstale":"2024-03-24T10:45:12.746930+0000","last_undegraded":"2024-03-24T10:45:12.746930+0000","last_fullsized":"2024-03-24T10:45:12.746930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:29.826202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954370+0000","last_change":"2024-03-24T10:45:01.951927+0000","last_active":"2024-03-24T10:45:02.954370+0000","last_peered":"2024-03-24T10:45:02.954370+0000","last_clean":"2024-03-24T10:45:02.954370+0000","last_became_active":"2024-03-24T10:45:01.951165+0000","last_became_peered":"2024-03-24T10:45:01.951165+0000","last_unstale":"2024-03-24T10:45:02.954370+0000","last_undegraded":"2024-03-24T10:45:02.954370+0000","last_fullsized":"2024-03-24T10:45:02.954370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:07:24.149961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.793997+0000","last_change":"2024-03-24T10:45:00.639093+0000","last_active":"2024-03-24T10:45:12.793997+0000","last_peered":"2024-03-24T10:45:12.793997+0000","last_clean":"2024-03-24T10:45:12.793997+0000","last_became_active":"2024-03-24T10:45:00.638785+0000","last_became_peered":"2024-03-24T10:45:00.638785+0000","last_unstale":"2024-03-24T10:45:12.793997+0000","last_undegraded":"2024-03-24T10:45:12.793997+0000","last_fullsized":"2024-03-24T10:45:12.793997+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955920+0000","last_change":"2024-03-24T10:45:01.953772+0000","last_active":"2024-03-24T10:45:02.955920+0000","last_peered":"2024-03-24T10:45:02.955920+0000","last_clean":"2024-03-24T10:45:02.955920+0000","last_became_active":"2024-03-24T10:45:01.953402+0000","last_became_peered":"2024-03-24T10:45:01.953402+0000","last_unstale":"2024-03-24T10:45:02.955920+0000","last_undegraded":"2024-03-24T10:45:02.955920+0000","last_fullsized":"2024-03-24T10:45:02.955920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:22.089070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.955402+0000","last_change":"2024-03-24T10:44:48.241074+0000","last_active":"2024-03-24T10:45:02.955402+0000","last_peered":"2024-03-24T10:45:02.955402+0000","last_clean":"2024-03-24T10:45:02.955402+0000","last_became_active":"2024-03-24T10:44:48.239986+0000","last_became_peered":"2024-03-24T10:44:48.239986+0000","last_unstale":"2024-03-24T10:45:02.955402+0000","last_undegraded":"2024-03-24T10:45:02.955402+0000","last_fullsized":"2024-03-24T10:45:02.955402+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:46.142247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:46.142247+0000","last_clean_scrub_stamp":"2024-03-24T10:44:46.142247+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:05:50.110100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.780173+0000","last_change":"2024-03-24T10:45:00.637276+0000","last_active":"2024-03-24T10:45:12.780173+0000","last_peered":"2024-03-24T10:45:12.780173+0000","last_clean":"2024-03-24T10:45:12.780173+0000","last_became_active":"2024-03-24T10:45:00.636734+0000","last_became_peered":"2024-03-24T10:45:00.636734+0000","last_unstale":"2024-03-24T10:45:12.780173+0000","last_undegraded":"2024-03-24T10:45:12.780173+0000","last_fullsized":"2024-03-24T10:45:12.780173+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954655+0000","last_change":"2024-03-24T10:45:01.958505+0000","last_active":"2024-03-24T10:45:02.954655+0000","last_peered":"2024-03-24T10:45:02.954655+0000","last_clean":"2024-03-24T10:45:02.954655+0000","last_became_active":"2024-03-24T10:45:01.958268+0000","last_became_peered":"2024-03-24T10:45:01.958268+0000","last_unstale":"2024-03-24T10:45:02.954655+0000","last_undegraded":"2024-03-24T10:45:02.954655+0000","last_fullsized":"2024-03-24T10:45:02.954655+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:27:12.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:10.261218+0000","last_change":"2024-03-24T10:45:00.597856+0000","last_active":"2024-03-24T10:45:10.261218+0000","last_peered":"2024-03-24T10:45:10.261218+0000","last_clean":"2024-03-24T10:45:10.261218+0000","last_became_active":"2024-03-24T10:45:00.597640+0000","last_became_peered":"2024-03-24T10:45:00.597640+0000","last_unstale":"2024-03-24T10:45:10.261218+0000","last_undegraded":"2024-03-24T10:45:10.261218+0000","last_fullsized":"2024-03-24T10:45:10.261218+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:29.826202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.952842+0000","last_change":"2024-03-24T10:45:01.897502+0000","last_active":"2024-03-24T10:45:02.952842+0000","last_peered":"2024-03-24T10:45:02.952842+0000","last_clean":"2024-03-24T10:45:02.952842+0000","last_became_active":"2024-03-24T10:45:01.897250+0000","last_became_peered":"2024-03-24T10:45:01.897250+0000","last_unstale":"2024-03-24T10:45:02.952842+0000","last_undegraded":"2024-03-24T10:45:02.952842+0000","last_fullsized":"2024-03-24T10:45:02.952842+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:09:46.341672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:10.390365+0000","last_change":"2024-03-24T10:45:00.627166+0000","last_active":"2024-03-24T10:45:10.390365+0000","last_peered":"2024-03-24T10:45:10.390365+0000","last_clean":"2024-03-24T10:45:10.390365+0000","last_became_active":"2024-03-24T10:45:00.626881+0000","last_became_peered":"2024-03-24T10:45:00.626881+0000","last_unstale":"2024-03-24T10:45:10.390365+0000","last_undegraded":"2024-03-24T10:45:10.390365+0000","last_fullsized":"2024-03-24T10:45:10.390365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.932815+0000","last_change":"2024-03-24T10:45:01.930271+0000","last_active":"2024-03-24T10:45:02.932815+0000","last_peered":"2024-03-24T10:45:02.932815+0000","last_clean":"2024-03-24T10:45:02.932815+0000","last_became_active":"2024-03-24T10:45:01.929998+0000","last_became_peered":"2024-03-24T10:45:01.929998+0000","last_unstale":"2024-03-24T10:45:02.932815+0000","last_undegraded":"2024-03-24T10:45:02.932815+0000","last_fullsized":"2024-03-24T10:45:02.932815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:15.554272+0000","last_change":"2024-03-24T10:45:00.638309+0000","last_active":"2024-03-24T10:45:15.554272+0000","last_peered":"2024-03-24T10:45:15.554272+0000","last_clean":"2024-03-24T10:45:15.554272+0000","last_became_active":"2024-03-24T10:45:00.638041+0000","last_became_peered":"2024-03-24T10:45:00.638041+0000","last_unstale":"2024-03-24T10:45:15.554272+0000","last_undegraded":"2024-03-24T10:45:15.554272+0000","last_fullsized":"2024-03-24T10:45:15.554272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:14.049099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953823+0000","last_change":"2024-03-24T10:45:01.897722+0000","last_active":"2024-03-24T10:45:02.953823+0000","last_peered":"2024-03-24T10:45:02.953823+0000","last_clean":"2024-03-24T10:45:02.953823+0000","last_became_active":"2024-03-24T10:45:01.897159+0000","last_became_peered":"2024-03-24T10:45:01.897159+0000","last_unstale":"2024-03-24T10:45:02.953823+0000","last_undegraded":"2024-03-24T10:45:02.953823+0000","last_fullsized":"2024-03-24T10:45:02.953823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:42.293342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933241+0000","last_change":"2024-03-24T10:45:00.629267+0000","last_active":"2024-03-24T10:45:02.933241+0000","last_peered":"2024-03-24T10:45:02.933241+0000","last_clean":"2024-03-24T10:45:02.933241+0000","last_became_active":"2024-03-24T10:45:00.628971+0000","last_became_peered":"2024-03-24T10:45:00.628971+0000","last_unstale":"2024-03-24T10:45:02.933241+0000","last_undegraded":"2024-03-24T10:45:02.933241+0000","last_fullsized":"2024-03-24T10:45:02.933241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:41.176786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954115+0000","last_change":"2024-03-24T10:45:01.952527+0000","last_active":"2024-03-24T10:45:02.954115+0000","last_peered":"2024-03-24T10:45:02.954115+0000","last_clean":"2024-03-24T10:45:02.954115+0000","last_became_active":"2024-03-24T10:45:01.952192+0000","last_became_peered":"2024-03-24T10:45:01.952192+0000","last_unstale":"2024-03-24T10:45:02.954115+0000","last_undegraded":"2024-03-24T10:45:02.954115+0000","last_fullsized":"2024-03-24T10:45:02.954115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.749212+0000","last_change":"2024-03-24T10:45:00.631941+0000","last_active":"2024-03-24T10:45:12.749212+0000","last_peered":"2024-03-24T10:45:12.749212+0000","last_clean":"2024-03-24T10:45:12.749212+0000","last_became_active":"2024-03-24T10:45:00.631670+0000","last_became_peered":"2024-03-24T10:45:00.631670+0000","last_unstale":"2024-03-24T10:45:12.749212+0000","last_undegraded":"2024-03-24T10:45:12.749212+0000","last_fullsized":"2024-03-24T10:45:12.749212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:09:45.225115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.933116+0000","last_change":"2024-03-24T10:45:01.930483+0000","last_active":"2024-03-24T10:45:02.933116+0000","last_peered":"2024-03-24T10:45:02.933116+0000","last_clean":"2024-03-24T10:45:02.933116+0000","last_became_active":"2024-03-24T10:45:01.930220+0000","last_became_peered":"2024-03-24T10:45:01.930220+0000","last_unstale":"2024-03-24T10:45:02.933116+0000","last_undegraded":"2024-03-24T10:45:02.933116+0000","last_fullsized":"2024-03-24T10:45:02.933116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.748135+0000","last_change":"2024-03-24T10:45:00.633650+0000","last_active":"2024-03-24T10:45:12.748135+0000","last_peered":"2024-03-24T10:45:12.748135+0000","last_clean":"2024-03-24T10:45:12.748135+0000","last_became_active":"2024-03-24T10:45:00.633329+0000","last_became_peered":"2024-03-24T10:45:00.633329+0000","last_unstale":"2024-03-24T10:45:12.748135+0000","last_undegraded":"2024-03-24T10:45:12.748135+0000","last_fullsized":"2024-03-24T10:45:12.748135+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954759+0000","last_change":"2024-03-24T10:45:01.853487+0000","last_active":"2024-03-24T10:45:02.954759+0000","last_peered":"2024-03-24T10:45:02.954759+0000","last_clean":"2024-03-24T10:45:02.954759+0000","last_became_active":"2024-03-24T10:45:01.853257+0000","last_became_peered":"2024-03-24T10:45:01.853257+0000","last_unstale":"2024-03-24T10:45:02.954759+0000","last_undegraded":"2024-03-24T10:45:02.954759+0000","last_fullsized":"2024-03-24T10:45:02.954759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.773005+0000","last_change":"2024-03-24T10:45:00.639282+0000","last_active":"2024-03-24T10:45:07.773005+0000","last_peered":"2024-03-24T10:45:07.773005+0000","last_clean":"2024-03-24T10:45:07.773005+0000","last_became_active":"2024-03-24T10:45:00.638999+0000","last_became_peered":"2024-03-24T10:45:00.638999+0000","last_unstale":"2024-03-24T10:45:07.773005+0000","last_undegraded":"2024-03-24T10:45:07.773005+0000","last_fullsized":"2024-03-24T10:45:07.773005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:41.176786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953535+0000","last_change":"2024-03-24T10:45:01.928900+0000","last_active":"2024-03-24T10:45:02.953535+0000","last_peered":"2024-03-24T10:45:02.953535+0000","last_clean":"2024-03-24T10:45:02.953535+0000","last_became_active":"2024-03-24T10:45:01.928610+0000","last_became_peered":"2024-03-24T10:45:01.928610+0000","last_unstale":"2024-03-24T10:45:02.953535+0000","last_undegraded":"2024-03-24T10:45:02.953535+0000","last_fullsized":"2024-03-24T10:45:02.953535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.772043+0000","last_change":"2024-03-24T10:45:00.631327+0000","last_active":"2024-03-24T10:45:07.772043+0000","last_peered":"2024-03-24T10:45:07.772043+0000","last_clean":"2024-03-24T10:45:07.772043+0000","last_became_active":"2024-03-24T10:45:00.631078+0000","last_became_peered":"2024-03-24T10:45:00.631078+0000","last_unstale":"2024-03-24T10:45:07.772043+0000","last_undegraded":"2024-03-24T10:45:07.772043+0000","last_fullsized":"2024-03-24T10:45:07.772043+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:43.263796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953696+0000","last_change":"2024-03-24T10:45:01.897329+0000","last_active":"2024-03-24T10:45:02.953696+0000","last_peered":"2024-03-24T10:45:02.953696+0000","last_clean":"2024-03-24T10:45:02.953696+0000","last_became_active":"2024-03-24T10:45:01.897063+0000","last_became_peered":"2024-03-24T10:45:01.897063+0000","last_unstale":"2024-03-24T10:45:02.953696+0000","last_undegraded":"2024-03-24T10:45:02.953696+0000","last_fullsized":"2024-03-24T10:45:02.953696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:07.259007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:15.511661+0000","last_change":"2024-03-24T10:45:00.597948+0000","last_active":"2024-03-24T10:45:15.511661+0000","last_peered":"2024-03-24T10:45:15.511661+0000","last_clean":"2024-03-24T10:45:15.511661+0000","last_became_active":"2024-03-24T10:45:00.597671+0000","last_became_peered":"2024-03-24T10:45:00.597671+0000","last_unstale":"2024-03-24T10:45:15.511661+0000","last_undegraded":"2024-03-24T10:45:15.511661+0000","last_fullsized":"2024-03-24T10:45:15.511661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:06.142451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954368+0000","last_change":"2024-03-24T10:45:01.927822+0000","last_active":"2024-03-24T10:45:02.954368+0000","last_peered":"2024-03-24T10:45:02.954368+0000","last_clean":"2024-03-24T10:45:02.954368+0000","last_became_active":"2024-03-24T10:45:01.927538+0000","last_became_peered":"2024-03-24T10:45:01.927538+0000","last_unstale":"2024-03-24T10:45:02.954368+0000","last_undegraded":"2024-03-24T10:45:02.954368+0000","last_fullsized":"2024-03-24T10:45:02.954368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.746638+0000","last_change":"2024-03-24T10:45:00.626818+0000","last_active":"2024-03-24T10:45:12.746638+0000","last_peered":"2024-03-24T10:45:12.746638+0000","last_clean":"2024-03-24T10:45:12.746638+0000","last_became_active":"2024-03-24T10:45:00.626557+0000","last_became_peered":"2024-03-24T10:45:00.626557+0000","last_unstale":"2024-03-24T10:45:12.746638+0000","last_undegraded":"2024-03-24T10:45:12.746638+0000","last_fullsized":"2024-03-24T10:45:12.746638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:43.263796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954131+0000","last_change":"2024-03-24T10:45:01.893192+0000","last_active":"2024-03-24T10:45:02.954131+0000","last_peered":"2024-03-24T10:45:02.954131+0000","last_clean":"2024-03-24T10:45:02.954131+0000","last_became_active":"2024-03-24T10:45:01.892934+0000","last_became_peered":"2024-03-24T10:45:01.892934+0000","last_unstale":"2024-03-24T10:45:02.954131+0000","last_undegraded":"2024-03-24T10:45:02.954131+0000","last_fullsized":"2024-03-24T10:45:02.954131+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:07.772069+0000","last_change":"2024-03-24T10:45:00.630147+0000","last_active":"2024-03-24T10:45:07.772069+0000","last_peered":"2024-03-24T10:45:07.772069+0000","last_clean":"2024-03-24T10:45:07.772069+0000","last_became_active":"2024-03-24T10:45:00.629850+0000","last_became_peered":"2024-03-24T10:45:00.629850+0000","last_unstale":"2024-03-24T10:45:07.772069+0000","last_undegraded":"2024-03-24T10:45:07.772069+0000","last_fullsized":"2024-03-24T10:45:07.772069+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:01:12.843222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953404+0000","last_change":"2024-03-24T10:45:01.897333+0000","last_active":"2024-03-24T10:45:02.953404+0000","last_peered":"2024-03-24T10:45:02.953404+0000","last_clean":"2024-03-24T10:45:02.953404+0000","last_became_active":"2024-03-24T10:45:01.897080+0000","last_became_peered":"2024-03-24T10:45:01.897080+0000","last_unstale":"2024-03-24T10:45:02.953404+0000","last_undegraded":"2024-03-24T10:45:02.953404+0000","last_fullsized":"2024-03-24T10:45:02.953404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:13.887947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953968+0000","last_change":"2024-03-24T10:45:01.956041+0000","last_active":"2024-03-24T10:45:02.953968+0000","last_peered":"2024-03-24T10:45:02.953968+0000","last_clean":"2024-03-24T10:45:02.953968+0000","last_became_active":"2024-03-24T10:45:01.955766+0000","last_became_peered":"2024-03-24T10:45:01.955766+0000","last_unstale":"2024-03-24T10:45:02.953968+0000","last_undegraded":"2024-03-24T10:45:02.953968+0000","last_fullsized":"2024-03-24T10:45:02.953968+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:16.341360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:15.640266+0000","last_change":"2024-03-24T10:45:00.639913+0000","last_active":"2024-03-24T10:45:15.640266+0000","last_peered":"2024-03-24T10:45:15.640266+0000","last_clean":"2024-03-24T10:45:15.640266+0000","last_became_active":"2024-03-24T10:45:00.639653+0000","last_became_peered":"2024-03-24T10:45:00.639653+0000","last_unstale":"2024-03-24T10:45:15.640266+0000","last_undegraded":"2024-03-24T10:45:15.640266+0000","last_fullsized":"2024-03-24T10:45:15.640266+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:15.749451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954233+0000","last_change":"2024-03-24T10:45:01.952340+0000","last_active":"2024-03-24T10:45:02.954233+0000","last_peered":"2024-03-24T10:45:02.954233+0000","last_clean":"2024-03-24T10:45:02.954233+0000","last_became_active":"2024-03-24T10:45:01.952074+0000","last_became_peered":"2024-03-24T10:45:01.952074+0000","last_unstale":"2024-03-24T10:45:02.954233+0000","last_undegraded":"2024-03-24T10:45:02.954233+0000","last_fullsized":"2024-03-24T10:45:02.954233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'11","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:15.640576+0000","last_change":"2024-03-24T10:45:00.633790+0000","last_active":"2024-03-24T10:45:15.640576+0000","last_peered":"2024-03-24T10:45:15.640576+0000","last_clean":"2024-03-24T10:45:15.640576+0000","last_became_active":"2024-03-24T10:45:00.633494+0000","last_became_peered":"2024-03-24T10:45:00.633494+0000","last_unstale":"2024-03-24T10:45:15.640576+0000","last_undegraded":"2024-03-24T10:45:15.640576+0000","last_fullsized":"2024-03-24T10:45:15.640576+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:14.049099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.953320+0000","last_change":"2024-03-24T10:45:01.957958+0000","last_active":"2024-03-24T10:45:02.953320+0000","last_peered":"2024-03-24T10:45:02.953320+0000","last_clean":"2024-03-24T10:45:02.953320+0000","last_became_active":"2024-03-24T10:45:01.957676+0000","last_became_peered":"2024-03-24T10:45:01.957676+0000","last_unstale":"2024-03-24T10:45:02.953320+0000","last_undegraded":"2024-03-24T10:45:02.953320+0000","last_fullsized":"2024-03-24T10:45:02.953320+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:03.159944+0000","last_change":"2024-03-24T10:45:00.629120+0000","last_active":"2024-03-24T10:45:03.159944+0000","last_peered":"2024-03-24T10:45:03.159944+0000","last_clean":"2024-03-24T10:45:03.159944+0000","last_became_active":"2024-03-24T10:45:00.628712+0000","last_became_peered":"2024-03-24T10:45:00.628712+0000","last_unstale":"2024-03-24T10:45:03.159944+0000","last_undegraded":"2024-03-24T10:45:03.159944+0000","last_fullsized":"2024-03-24T10:45:03.159944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:02.954346+0000","last_change":"2024-03-24T10:45:01.894602+0000","last_active":"2024-03-24T10:45:02.954346+0000","last_peered":"2024-03-24T10:45:02.954346+0000","last_clean":"2024-03-24T10:45:02.954346+0000","last_became_active":"2024-03-24T10:45:01.894362+0000","last_became_peered":"2024-03-24T10:45:01.894362+0000","last_unstale":"2024-03-24T10:45:02.954346+0000","last_undegraded":"2024-03-24T10:45:02.954346+0000","last_fullsized":"2024-03-24T10:45:02.954346+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'11","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-03-24T10:45:12.684496+0000","last_change":"2024-03-24T10:45:00.639171+0000","last_active":"2024-03-24T10:45:12.684496+0000","last_peered":"2024-03-24T10:45:12.684496+0000","last_clean":"2024-03-24T10:45:12.684496+0000","last_became_active":"2024-03-24T10:45:00.638871+0000","last_became_peered":"2024-03-24T10:45:00.638871+0000","last_unstale":"2024-03-24T10:45:12.684496+0000","last_undegraded":"2024-03-24T10:45:12.684496+0000","last_fullsized":"2024-03-24T10:45:12.684496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:52:46.322196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7944,"num_objects":91,"num_object_clones":0,"num_object_copies":182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":61,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":60,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":15888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":168,"ondisk_log_size":168,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65430,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542155,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31484,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340356,"statfs":{"total":96636764160,"available":96604524544,"internally_reserved":0,"allocated":315392,"data_stored":666673,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542155,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":262144,"data_stored":78137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542154,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":76619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542155,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":75611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542155,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":237568,"data_stored":75149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":241664,"data_stored":76607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":237568,"data_stored":75535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31468,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340372,"statfs":{"total":96636764160,"available":96604540928,"internally_reserved":0,"allocated":299008,"data_stored":665267,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:45:19.380 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-24T10:45:19.380 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T10:45:19.380 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-24T10:45:19.380 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-24T10:45:19.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.517+0000 7f104d823640 1 Processor -- start 2024-03-24T10:45:19.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.517+0000 7f104d823640 1 -- start start 2024-03-24T10:45:19.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.518+0000 7f104d823640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f104815ac00 0x7f10480cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:19.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.518+0000 7f104d823640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f10480ccaa0 0x7f104806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:19.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.518+0000 7f104d823640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10480cb2f0 con 0x7f10481543a0 2024-03-24T10:45:19.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.518+0000 7f104d823640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f10480c6750 con 0x7f10480ccaa0 2024-03-24T10:45:19.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.518+0000 7f104d823640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f10480cc380 con 0x7f104815ac00 2024-03-24T10:45:19.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.518+0000 7f103ffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f104815ac00 0x7f10480cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:19.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.519+0000 7f103ffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f104815ac00 0x7f10480cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:51688/0 (socket says 172.21.15.5:51688) 2024-03-24T10:45:19.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.519+0000 7f103ffff640 1 -- 172.21.15.5:0/3381610542 learned_addr learned my addr 172.21.15.5:0/3381610542 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:19.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.521+0000 7f103ffff640 1 -- 172.21.15.5:0/3381610542 >> v2:172.21.15.5:3300/0 conn(0x7f10480ccaa0 msgr2=0x7f104806bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:19.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.521+0000 7f103ffff640 1 --2- 172.21.15.5:0/3381610542 >> v2:172.21.15.5:3300/0 conn(0x7f10480ccaa0 0x7f104806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:19.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.521+0000 7f103ffff640 1 -- 172.21.15.5:0/3381610542 >> v1:172.21.15.5:6789/0 conn(0x7f10481543a0 legacy=0x7f104806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:19.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.521+0000 7f103ffff640 1 -- 172.21.15.5:0/3381610542 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104815b070 con 0x7f104815ac00 2024-03-24T10:45:19.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.523+0000 7f103ffff640 1 --2- 172.21.15.5:0/3381610542 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f104815ac00 0x7f10480cc560 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1030009870 tx=0x7f103002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f35a416f87a9be8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:19.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.528+0000 7f10467fc640 1 -- 172.21.15.5:0/3381610542 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1030031640 con 0x7f104815ac00 2024-03-24T10:45:19.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.528+0000 7f10467fc640 1 -- 172.21.15.5:0/3381610542 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10300317a0 con 0x7f104815ac00 2024-03-24T10:45:19.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.528+0000 7f10467fc640 1 -- 172.21.15.5:0/3381610542 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1030031b30 con 0x7f104815ac00 2024-03-24T10:45:19.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.529+0000 7f104d823640 1 -- 172.21.15.5:0/3381610542 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f104815ac00 msgr2=0x7f10480cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:19.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.529+0000 7f104d823640 1 --2- 172.21.15.5:0/3381610542 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f104815ac00 0x7f10480cc560 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1030009870 tx=0x7f103002b8f0 comp rx=0 tx=0).stop 2024-03-24T10:45:19.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.529+0000 7f104d823640 1 -- 172.21.15.5:0/3381610542 shutdown_connections 2024-03-24T10:45:19.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.529+0000 7f104d823640 1 --2- 172.21.15.5:0/3381610542 >> v2:172.21.15.5:3300/0 conn(0x7f10480ccaa0 0x7f104806bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:19.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.529+0000 7f104d823640 1 --2- 172.21.15.5:0/3381610542 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f104815ac00 0x7f10480cc560 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:19.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.529+0000 7f104d823640 1 -- 172.21.15.5:0/3381610542 >> 172.21.15.5:0/3381610542 conn(0x7f1048072ef0 msgr2=0x7f1048061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:19.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.529+0000 7f104d823640 1 -- 172.21.15.5:0/3381610542 shutdown_connections 2024-03-24T10:45:19.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.529+0000 7f104d823640 1 -- 172.21.15.5:0/3381610542 wait complete. 2024-03-24T10:45:19.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.530+0000 7f104d823640 1 Processor -- start 2024-03-24T10:45:19.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.530+0000 7f104d823640 1 -- start start 2024-03-24T10:45:19.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.531+0000 7f104d823640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f10480ccaa0 0x7f10481e5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:19.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.531+0000 7f104d823640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10481543a0 0x7f1048152560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:19.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.531+0000 7f104d823640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f104815ac00 0x7f104814e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:19.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.531+0000 7f104d823640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f104814ff20 con 0x7f104815ac00 2024-03-24T10:45:19.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.531+0000 7f104d823640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1048155300 con 0x7f10481543a0 2024-03-24T10:45:19.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.531+0000 7f104d823640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f10480ce710 con 0x7f10480ccaa0 2024-03-24T10:45:19.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.531+0000 7f10477fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f104815ac00 0x7f104814e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:19.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.531+0000 7f1046ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f10480ccaa0 0x7f10481e5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:19.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.532+0000 7f1046ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f10480ccaa0 0x7f10481e5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:51704/0 (socket says 172.21.15.5:51704) 2024-03-24T10:45:19.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.532+0000 7f1046ffd640 1 -- 172.21.15.5:0/3848185713 learned_addr learned my addr 172.21.15.5:0/3848185713 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:19.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.532+0000 7f103ffff640 1 --2- 172.21.15.5:0/3848185713 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10481543a0 0x7f1048152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:19.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.535+0000 7f10477fe640 1 -- 172.21.15.5:0/3848185713 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f10480ccaa0 msgr2=0x7f10481e5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:19.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.535+0000 7f10477fe640 1 --2- 172.21.15.5:0/3848185713 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f10480ccaa0 0x7f10481e5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:19.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.535+0000 7f10477fe640 1 -- 172.21.15.5:0/3848185713 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10481543a0 msgr2=0x7f1048152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:19.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.535+0000 7f10477fe640 1 --2- 172.21.15.5:0/3848185713 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10481543a0 0x7f1048152560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:19.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.535+0000 7f10477fe640 1 -- 172.21.15.5:0/3848185713 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10300094d0 con 0x7f104815ac00 2024-03-24T10:45:19.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.537+0000 7f10477fe640 1 --2- 172.21.15.5:0/3848185713 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f104815ac00 0x7f104814e8e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f104000be20 tx=0x7f104000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:19.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.541+0000 7f1044ff9640 1 -- 172.21.15.5:0/3848185713 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104000cdb0 con 0x7f104815ac00 2024-03-24T10:45:19.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.541+0000 7f1044ff9640 1 -- 172.21.15.5:0/3848185713 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1040014440 con 0x7f104815ac00 2024-03-24T10:45:19.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.541+0000 7f104d823640 1 -- 172.21.15.5:0/3848185713 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10481e5670 con 0x7f104815ac00 2024-03-24T10:45:19.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.541+0000 7f1044ff9640 1 -- 172.21.15.5:0/3848185713 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10400147f0 con 0x7f104815ac00 2024-03-24T10:45:19.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.541+0000 7f104d823640 1 -- 172.21.15.5:0/3848185713 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10480cdb00 con 0x7f104815ac00 2024-03-24T10:45:19.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.542+0000 7f104d823640 1 -- 172.21.15.5:0/3848185713 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10480cbac0 con 0x7f104815ac00 2024-03-24T10:45:19.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.811+0000 7f1044ff9640 1 -- 172.21.15.5:0/3848185713 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f104001d3b0 con 0x7f104815ac00 2024-03-24T10:45:19.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.811+0000 7f1044ff9640 1 --2- 172.21.15.5:0/3848185713 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f102806b890 0x7f102806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:19.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.811+0000 7f1046ffd640 1 --2- 172.21.15.5:0/3848185713 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f102806b890 0x7f102806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:19.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.812+0000 7f1046ffd640 1 --2- 172.21.15.5:0/3848185713 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f102806b890 0x7f102806dd50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f10380018b0 tx=0x7f1038009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:19.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.830+0000 7f1044ff9640 1 -- 172.21.15.5:0/3848185713 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1040058db0 con 0x7f104815ac00 2024-03-24T10:45:19.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:19.830+0000 7f1044ff9640 1 -- 172.21.15.5:0/3848185713 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1040055050 con 0x7f104815ac00 2024-03-24T10:45:20.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.077+0000 7f104d823640 1 -- 172.21.15.5:0/3848185713 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f10480cbac0 con 0x7f104815ac00 2024-03-24T10:45:20.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.084+0000 7f1044ff9640 1 -- 172.21.15.5:0/3848185713 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f10480cbac0 con 0x7f104815ac00 2024-03-24T10:45:20.084 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:45:20.084 INFO:teuthology.orchestra.run.smithi005.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T10:45:20.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.086+0000 7f104d823640 1 -- 172.21.15.5:0/3848185713 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f102806b890 msgr2=0x7f102806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:20.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.086+0000 7f104d823640 1 --2- 172.21.15.5:0/3848185713 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f102806b890 0x7f102806dd50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f10380018b0 tx=0x7f1038009040 comp rx=0 tx=0).stop 2024-03-24T10:45:20.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.086+0000 7f104d823640 1 -- 172.21.15.5:0/3848185713 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f104815ac00 msgr2=0x7f104814e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:20.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.086+0000 7f104d823640 1 --2- 172.21.15.5:0/3848185713 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f104815ac00 0x7f104814e8e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f104000be20 tx=0x7f104000bf10 comp rx=0 tx=0).stop 2024-03-24T10:45:20.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.086+0000 7f104d823640 1 -- 172.21.15.5:0/3848185713 shutdown_connections 2024-03-24T10:45:20.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.086+0000 7f104d823640 1 --2- 172.21.15.5:0/3848185713 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f102806b890 0x7f102806dd50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:20.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.086+0000 7f104d823640 1 --2- 172.21.15.5:0/3848185713 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f104815ac00 0x7f104814e8e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:20.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.086+0000 7f104d823640 1 --2- 172.21.15.5:0/3848185713 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f10481543a0 0x7f1048152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:20.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.086+0000 7f104d823640 1 --2- 172.21.15.5:0/3848185713 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f10480ccaa0 0x7f10481e5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:20.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.086+0000 7f104d823640 1 -- 172.21.15.5:0/3848185713 >> 172.21.15.5:0/3848185713 conn(0x7f1048072ef0 msgr2=0x7f1048158880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:20.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.086+0000 7f104d823640 1 -- 172.21.15.5:0/3848185713 shutdown_connections 2024-03-24T10:45:20.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.086+0000 7f104d823640 1 -- 172.21.15.5:0/3848185713 wait complete. 2024-03-24T10:45:20.097 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-24T10:45:20.098 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:45:20.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.199+0000 7f0906855640 1 Processor -- start 2024-03-24T10:45:20.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.199+0000 7f0906855640 1 -- start start 2024-03-24T10:45:20.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.199+0000 7f0906855640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f09001543a0 0x7f090015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:20.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.199+0000 7f0906855640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f090006d4f0 0x7f09000cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:20.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.200+0000 7f0906855640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09000cb2f0 con 0x7f09000ccaa0 2024-03-24T10:45:20.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.200+0000 7f0906855640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f09000c6750 con 0x7f09001543a0 2024-03-24T10:45:20.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.200+0000 7f0906855640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f09000cc380 con 0x7f090006d4f0 2024-03-24T10:45:20.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.200+0000 7f08ff7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f090006d4f0 0x7f09000cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:20.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.200+0000 7f08fffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f09001543a0 0x7f090015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:20.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.201+0000 7f08ff7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f090006d4f0 0x7f09000cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:51706/0 (socket says 172.21.15.5:51706) 2024-03-24T10:45:20.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.201+0000 7f0904df2640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f09000ccaa0 0x7f090006bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:47146/0 (socket says 172.21.15.5:47146) 2024-03-24T10:45:20.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.201+0000 7f08ff7fe640 1 -- 172.21.15.5:0/2478177236 learned_addr learned my addr 172.21.15.5:0/2478177236 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:20.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.204+0000 7f08ff7fe640 1 -- 172.21.15.5:0/2478177236 >> v2:172.21.15.57:3300/0 conn(0x7f09001543a0 msgr2=0x7f090015ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:20.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.204+0000 7f08ff7fe640 1 --2- 172.21.15.5:0/2478177236 >> v2:172.21.15.57:3300/0 conn(0x7f09001543a0 0x7f090015ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:20.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.204+0000 7f08ff7fe640 1 -- 172.21.15.5:0/2478177236 >> v1:172.21.15.57:6789/0 conn(0x7f09000ccaa0 legacy=0x7f090006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:20.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.204+0000 7f08ff7fe640 1 -- 172.21.15.5:0/2478177236 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09000ca7e0 con 0x7f090006d4f0 2024-03-24T10:45:20.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.207+0000 7f08ff7fe640 1 --2- 172.21.15.5:0/2478177236 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f090006d4f0 0x7f09000cc560 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f08f4009870 tx=0x7f08f402ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7368d2c2005bff6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:20.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.211+0000 7f08feffd640 1 -- 172.21.15.5:0/2478177236 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f402f840 con 0x7f090006d4f0 2024-03-24T10:45:20.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.211+0000 7f08feffd640 1 -- 172.21.15.5:0/2478177236 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08f402f9a0 con 0x7f090006d4f0 2024-03-24T10:45:20.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.211+0000 7f08feffd640 1 -- 172.21.15.5:0/2478177236 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f402fd30 con 0x7f090006d4f0 2024-03-24T10:45:20.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.212+0000 7f0906855640 1 -- 172.21.15.5:0/2478177236 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f090006d4f0 msgr2=0x7f09000cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:20.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.212+0000 7f0906855640 1 --2- 172.21.15.5:0/2478177236 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f090006d4f0 0x7f09000cc560 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f08f4009870 tx=0x7f08f402ba20 comp rx=0 tx=0).stop 2024-03-24T10:45:20.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.212+0000 7f0906855640 1 -- 172.21.15.5:0/2478177236 shutdown_connections 2024-03-24T10:45:20.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.212+0000 7f0906855640 1 --2- 172.21.15.5:0/2478177236 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f090006d4f0 0x7f09000cc560 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:20.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.212+0000 7f0906855640 1 --2- 172.21.15.5:0/2478177236 >> v2:172.21.15.57:3300/0 conn(0x7f09001543a0 0x7f090015ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:20.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.212+0000 7f0906855640 1 -- 172.21.15.5:0/2478177236 >> 172.21.15.5:0/2478177236 conn(0x7f0900072ef0 msgr2=0x7f0900061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:20.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.212+0000 7f0906855640 1 -- 172.21.15.5:0/2478177236 shutdown_connections 2024-03-24T10:45:20.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.212+0000 7f0906855640 1 -- 172.21.15.5:0/2478177236 wait complete. 2024-03-24T10:45:20.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.214+0000 7f0906855640 1 Processor -- start 2024-03-24T10:45:20.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.214+0000 7f0906855640 1 -- start start 2024-03-24T10:45:20.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.215+0000 7f0906855640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f090006d4f0 0x7f09000b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:20.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.215+0000 7f0906855640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09000ccaa0 0x7f090014da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:20.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.215+0000 7f0906855640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09001543a0 0x7f09000cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:20.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.215+0000 7f0906855640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f09001efc10 con 0x7f09001543a0 2024-03-24T10:45:20.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.215+0000 7f0906855640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0900156a00 con 0x7f09000ccaa0 2024-03-24T10:45:20.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.215+0000 7f0906855640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0900156ce0 con 0x7f090006d4f0 2024-03-24T10:45:20.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.216+0000 7f08fffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f090006d4f0 0x7f09000b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:20.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.216+0000 7f0904df2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09001543a0 0x7f09000cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:20.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.216+0000 7f08ff7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09000ccaa0 0x7f090014da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:20.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.216+0000 7f08fffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f090006d4f0 0x7f09000b2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:51708/0 (socket says 172.21.15.5:51708) 2024-03-24T10:45:20.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.216+0000 7f08fffff640 1 -- 172.21.15.5:0/2761024855 learned_addr learned my addr 172.21.15.5:0/2761024855 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:20.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.221+0000 7f08fffff640 1 -- 172.21.15.5:0/2761024855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09000ccaa0 msgr2=0x7f090014da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:20.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.221+0000 7f08fffff640 1 --2- 172.21.15.5:0/2761024855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09000ccaa0 0x7f090014da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:20.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.221+0000 7f08fffff640 1 -- 172.21.15.5:0/2761024855 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09001543a0 msgr2=0x7f09000cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:20.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.221+0000 7f08fffff640 1 --2- 172.21.15.5:0/2761024855 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09001543a0 0x7f09000cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:20.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.221+0000 7f08fffff640 1 -- 172.21.15.5:0/2761024855 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08f40094d0 con 0x7f090006d4f0 2024-03-24T10:45:20.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.224+0000 7f08fffff640 1 --2- 172.21.15.5:0/2761024855 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f090006d4f0 0x7f09000b2470 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f08f000e870 tx=0x7f08f000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:20.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.229+0000 7f08fcff9640 1 -- 172.21.15.5:0/2761024855 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f00094f0 con 0x7f090006d4f0 2024-03-24T10:45:20.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.229+0000 7f08fcff9640 1 -- 172.21.15.5:0/2761024855 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08f000c940 con 0x7f090006d4f0 2024-03-24T10:45:20.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.229+0000 7f0906855640 1 -- 172.21.15.5:0/2761024855 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0900156fc0 con 0x7f090006d4f0 2024-03-24T10:45:20.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.229+0000 7f08fcff9640 1 -- 172.21.15.5:0/2761024855 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f000ccf0 con 0x7f090006d4f0 2024-03-24T10:45:20.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.229+0000 7f0906855640 1 -- 172.21.15.5:0/2761024855 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f090014e0a0 con 0x7f090006d4f0 2024-03-24T10:45:20.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.230+0000 7f0906855640 1 -- 172.21.15.5:0/2761024855 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09001efd80 con 0x7f090006d4f0 2024-03-24T10:45:20.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.262+0000 7f08fcff9640 1 -- 172.21.15.5:0/2761024855 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f08f0013470 con 0x7f090006d4f0 2024-03-24T10:45:20.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.262+0000 7f08fcff9640 1 --2- 172.21.15.5:0/2761024855 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f08ec06b810 0x7f08ec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:20.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.262+0000 7f08ff7fe640 1 --2- 172.21.15.5:0/2761024855 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f08ec06b810 0x7f08ec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:20.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.263+0000 7f08ff7fe640 1 --2- 172.21.15.5:0/2761024855 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f08ec06b810 0x7f08ec06dcd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f08f4036000 tx=0x7f08f4008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:20.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.362+0000 7f08fcff9640 1 -- 172.21.15.5:0/2761024855 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f08f00590b0 con 0x7f090006d4f0 2024-03-24T10:45:20.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.535+0000 7f08fcff9640 1 -- 172.21.15.5:0/2761024855 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f08f0055350 con 0x7f090006d4f0 2024-03-24T10:45:20.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.718+0000 7f0906855640 1 -- 172.21.15.5:0/2761024855 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f09001efd80 con 0x7f090006d4f0 2024-03-24T10:45:20.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.761+0000 7f08fcff9640 1 -- 172.21.15.5:0/2761024855 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16485 (secure 0 0 0) 0x7f09001efd80 con 0x7f090006d4f0 2024-03-24T10:45:20.762 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:45:20.762 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":21,"fsid":"7d9848db-2fe7-4d95-a6e2-b04f4f639ebc","created":"2024-03-24T10:44:30.435176+0000","modified":"2024-03-24T10:45:02.781257+0000","last_up_change":"2024-03-24T10:44:44.634047+0000","last_in_change":"2024-03-24T10:44:35.986391+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:44:45.829320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:44:58.435543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:45:00.129313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50f590b3-f5ec-4e1b-ac46-25cab307a27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6825","nonce":1991336051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6827","nonce":1991336051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6831","nonce":1991336051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6829","nonce":1991336051}]},"public_addr":"172.21.15.5:6825/1991336051","cluster_addr":"172.21.15.5:6827/1991336051","heartbeat_back_addr":"172.21.15.5:6831/1991336051","heartbeat_front_addr":"172.21.15.5:6829/1991336051","state":["exists","up"]},{"osd":1,"uuid":"ccba9e93-c704-472b-a832-8d4eeb456c47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6801","nonce":2140667501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6803","nonce":2140667501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6807","nonce":2140667501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6805","nonce":2140667501}]},"public_addr":"172.21.15.5:6801/2140667501","cluster_addr":"172.21.15.5:6803/2140667501","heartbeat_back_addr":"172.21.15.5:6807/2140667501","heartbeat_front_addr":"172.21.15.5:6805/2140667501","state":["exists","up"]},{"osd":2,"uuid":"e614c5b1-6504-452e-82e0-95c7ca271737","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6817","nonce":90414679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6819","nonce":90414679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6823","nonce":90414679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6821","nonce":90414679}]},"public_addr":"172.21.15.5:6817/90414679","cluster_addr":"172.21.15.5:6819/90414679","heartbeat_back_addr":"172.21.15.5:6823/90414679","heartbeat_front_addr":"172.21.15.5:6821/90414679","state":["exists","up"]},{"osd":3,"uuid":"c60154d0-85e0-46a1-8273-4863bd8ad2dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6809","nonce":4212473119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6811","nonce":4212473119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6815","nonce":4212473119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6813","nonce":4212473119}]},"public_addr":"172.21.15.5:6809/4212473119","cluster_addr":"172.21.15.5:6811/4212473119","heartbeat_back_addr":"172.21.15.5:6815/4212473119","heartbeat_front_addr":"172.21.15.5:6813/4212473119","state":["exists","up"]},{"osd":4,"uuid":"21d74109-da3a-4fc0-b2d8-93e9ae3ed30c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6817","nonce":2193986505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6819","nonce":2193986505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6823","nonce":2193986505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6821","nonce":2193986505}]},"public_addr":"172.21.15.57:6817/2193986505","cluster_addr":"172.21.15.57:6819/2193986505","heartbeat_back_addr":"172.21.15.57:6823/2193986505","heartbeat_front_addr":"172.21.15.57:6821/2193986505","state":["exists","up"]},{"osd":5,"uuid":"5cf19cc7-215f-4d69-a30a-80543c0293e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6809","nonce":3044987466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6811","nonce":3044987466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6815","nonce":3044987466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6813","nonce":3044987466}]},"public_addr":"172.21.15.57:6809/3044987466","cluster_addr":"172.21.15.57:6811/3044987466","heartbeat_back_addr":"172.21.15.57:6815/3044987466","heartbeat_front_addr":"172.21.15.57:6813/3044987466","state":["exists","up"]},{"osd":6,"uuid":"a5e76c69-c3fc-47b2-8f92-e560cf503e45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6801","nonce":4290897445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6803","nonce":4290897445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6807","nonce":4290897445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6805","nonce":4290897445}]},"public_addr":"172.21.15.57:6801/4290897445","cluster_addr":"172.21.15.57:6803/4290897445","heartbeat_back_addr":"172.21.15.57:6807/4290897445","heartbeat_front_addr":"172.21.15.57:6805/4290897445","state":["exists","up"]},{"osd":7,"uuid":"47d0b873-e3ac-4f40-a372-681436e45d0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6825","nonce":212605698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6827","nonce":212605698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6831","nonce":212605698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6829","nonce":212605698}]},"public_addr":"172.21.15.57:6825/212605698","cluster_addr":"172.21.15.57:6827/212605698","heartbeat_back_addr":"172.21.15.57:6831/212605698","heartbeat_front_addr":"172.21.15.57:6829/212605698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.211237+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.192822+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.273429+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.199212+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.277082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.218768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.190915+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.225294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:45:20.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.764+0000 7f0906855640 1 -- 172.21.15.5:0/2761024855 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f08ec06b810 msgr2=0x7f08ec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:20.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.764+0000 7f0906855640 1 --2- 172.21.15.5:0/2761024855 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f08ec06b810 0x7f08ec06dcd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f08f4036000 tx=0x7f08f4008030 comp rx=0 tx=0).stop 2024-03-24T10:45:20.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.764+0000 7f0906855640 1 -- 172.21.15.5:0/2761024855 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f090006d4f0 msgr2=0x7f09000b2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:20.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.764+0000 7f0906855640 1 --2- 172.21.15.5:0/2761024855 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f090006d4f0 0x7f09000b2470 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f08f000e870 tx=0x7f08f000ed30 comp rx=0 tx=0).stop 2024-03-24T10:45:20.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.764+0000 7f0906855640 1 -- 172.21.15.5:0/2761024855 shutdown_connections 2024-03-24T10:45:20.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.764+0000 7f0906855640 1 --2- 172.21.15.5:0/2761024855 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f08ec06b810 0x7f08ec06dcd0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:20.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.764+0000 7f0906855640 1 --2- 172.21.15.5:0/2761024855 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09001543a0 0x7f09000cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:20.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.764+0000 7f0906855640 1 --2- 172.21.15.5:0/2761024855 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09000ccaa0 0x7f090014da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:20.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.764+0000 7f0906855640 1 --2- 172.21.15.5:0/2761024855 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f090006d4f0 0x7f09000b2470 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:20.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.764+0000 7f0906855640 1 -- 172.21.15.5:0/2761024855 >> 172.21.15.5:0/2761024855 conn(0x7f0900072ef0 msgr2=0x7f09001528f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:20.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.764+0000 7f0906855640 1 -- 172.21.15.5:0/2761024855 shutdown_connections 2024-03-24T10:45:20.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.764+0000 7f0906855640 1 -- 172.21.15.5:0/2761024855 wait complete. 2024-03-24T10:45:20.776 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:44:45.829320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:44:58.435543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:45:00.129313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:45:20.776 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:45:20.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.883+0000 7f4e43255640 1 Processor -- start 2024-03-24T10:45:20.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.883+0000 7f4e43255640 1 -- start start 2024-03-24T10:45:20.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.883+0000 7f4e43255640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4e3c0cf340 0x7f4e3c0cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:20.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.883+0000 7f4e43255640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e3c0ccb80 0x7f4e3c06bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:20.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.883+0000 7f4e43255640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e3c0cc3e0 con 0x7f4e3c0cc000 2024-03-24T10:45:20.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.883+0000 7f4e43255640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f4e3c15a400 con 0x7f4e3c0cf340 2024-03-24T10:45:20.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.883+0000 7f4e43255640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4e3c154bd0 con 0x7f4e3c0ccb80 2024-03-24T10:45:20.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.884+0000 7f4e417f2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e3c0ccb80 0x7f4e3c06bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:20.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.884+0000 7f4e40ff1640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f4e3c0cc000 0x7f4e3c15e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:47152/0 (socket says 172.21.15.5:47152) 2024-03-24T10:45:20.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.884+0000 7f4e40ff1640 1 -- 172.21.15.5:0/2855014441 learned_addr learned my addr 172.21.15.5:0/2855014441 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:20.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.884+0000 7f4e3bfff640 1 --2- 172.21.15.5:0/2855014441 >> v2:172.21.15.57:3300/0 conn(0x7f4e3c0cf340 0x7f4e3c0cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:20.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.888+0000 7f4e417f2640 1 -- 172.21.15.5:0/2855014441 >> v2:172.21.15.57:3300/0 conn(0x7f4e3c0cf340 msgr2=0x7f4e3c0cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:20.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.888+0000 7f4e417f2640 1 --2- 172.21.15.5:0/2855014441 >> v2:172.21.15.57:3300/0 conn(0x7f4e3c0cf340 0x7f4e3c0cc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:20.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.888+0000 7f4e417f2640 1 -- 172.21.15.5:0/2855014441 >> v1:172.21.15.57:6789/0 conn(0x7f4e3c0cc000 legacy=0x7f4e3c15e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:20.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.888+0000 7f4e417f2640 1 -- 172.21.15.5:0/2855014441 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e3c15a5f0 con 0x7f4e3c0ccb80 2024-03-24T10:45:20.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.891+0000 7f4e417f2640 1 --2- 172.21.15.5:0/2855014441 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e3c0ccb80 0x7f4e3c06bf50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009d90 tx=0x7f4e3402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b38bc831eddc39d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:20.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.896+0000 7f4e3b7fe640 1 -- 172.21.15.5:0/2855014441 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3402cc50 con 0x7f4e3c0ccb80 2024-03-24T10:45:20.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.896+0000 7f4e3b7fe640 1 -- 172.21.15.5:0/2855014441 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e3402cdb0 con 0x7f4e3c0ccb80 2024-03-24T10:45:20.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.896+0000 7f4e3b7fe640 1 -- 172.21.15.5:0/2855014441 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e34034660 con 0x7f4e3c0ccb80 2024-03-24T10:45:20.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.897+0000 7f4e43255640 1 -- 172.21.15.5:0/2855014441 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e3c0ccb80 msgr2=0x7f4e3c06bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:20.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.897+0000 7f4e43255640 1 --2- 172.21.15.5:0/2855014441 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e3c0ccb80 0x7f4e3c06bf50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009d90 tx=0x7f4e3402bea0 comp rx=0 tx=0).stop 2024-03-24T10:45:20.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.897+0000 7f4e43255640 1 -- 172.21.15.5:0/2855014441 shutdown_connections 2024-03-24T10:45:20.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.897+0000 7f4e43255640 1 --2- 172.21.15.5:0/2855014441 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e3c0ccb80 0x7f4e3c06bf50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:20.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.897+0000 7f4e43255640 1 --2- 172.21.15.5:0/2855014441 >> v2:172.21.15.57:3300/0 conn(0x7f4e3c0cf340 0x7f4e3c0cc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:20.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.897+0000 7f4e43255640 1 -- 172.21.15.5:0/2855014441 >> 172.21.15.5:0/2855014441 conn(0x7f4e3c0cfc30 msgr2=0x7f4e3c0d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:20.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.897+0000 7f4e43255640 1 -- 172.21.15.5:0/2855014441 shutdown_connections 2024-03-24T10:45:20.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.897+0000 7f4e43255640 1 -- 172.21.15.5:0/2855014441 wait complete. 2024-03-24T10:45:20.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.898+0000 7f4e43255640 1 Processor -- start 2024-03-24T10:45:20.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.898+0000 7f4e43255640 1 -- start start 2024-03-24T10:45:20.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.899+0000 7f4e43255640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e3c0cc000 0x7f4e3c1e6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:20.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.899+0000 7f4e43255640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e3c0ccb80 0x7f4e3c1f00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:20.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.899+0000 7f4e43255640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e3c0cf340 0x7f4e3c1f05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:20.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.899+0000 7f4e43255640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c0c4940 con 0x7f4e3c0cf340 2024-03-24T10:45:20.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.899+0000 7f4e43255640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c15cd40 con 0x7f4e3c0cc000 2024-03-24T10:45:20.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.899+0000 7f4e43255640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4e3c0c3900 con 0x7f4e3c0ccb80 2024-03-24T10:45:20.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.899+0000 7f4e40ff1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e3c0cc000 0x7f4e3c1e6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:20.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.900+0000 7f4e3bfff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e3c0ccb80 0x7f4e3c1f00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:20.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.900+0000 7f4e417f2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e3c0cf340 0x7f4e3c1f05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:20.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.900+0000 7f4e40ff1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e3c0cc000 0x7f4e3c1e6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36818/0 (socket says 172.21.15.5:36818) 2024-03-24T10:45:20.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.900+0000 7f4e40ff1640 1 -- 172.21.15.5:0/2211147419 learned_addr learned my addr 172.21.15.5:0/2211147419 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:20.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.903+0000 7f4e40ff1640 1 -- 172.21.15.5:0/2211147419 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e3c0ccb80 msgr2=0x7f4e3c1f00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:20.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.903+0000 7f4e40ff1640 1 --2- 172.21.15.5:0/2211147419 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e3c0ccb80 0x7f4e3c1f00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:20.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.903+0000 7f4e40ff1640 1 -- 172.21.15.5:0/2211147419 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e3c0cf340 msgr2=0x7f4e3c1f05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:20.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.903+0000 7f4e40ff1640 1 --2- 172.21.15.5:0/2211147419 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e3c0cf340 0x7f4e3c1f05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:20.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.903+0000 7f4e40ff1640 1 -- 172.21.15.5:0/2211147419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e34009a70 con 0x7f4e3c0cc000 2024-03-24T10:45:20.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.906+0000 7f4e40ff1640 1 --2- 172.21.15.5:0/2211147419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e3c0cc000 0x7f4e3c1e6bd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4e3000c3f0 tx=0x7f4e3000c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:20.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.911+0000 7f4e397fa640 1 -- 172.21.15.5:0/2211147419 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e30005ce0 con 0x7f4e3c0cc000 2024-03-24T10:45:20.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.911+0000 7f4e397fa640 1 -- 172.21.15.5:0/2211147419 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e30005e40 con 0x7f4e3c0cc000 2024-03-24T10:45:20.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.911+0000 7f4e397fa640 1 -- 172.21.15.5:0/2211147419 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e30007310 con 0x7f4e3c0cc000 2024-03-24T10:45:20.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.911+0000 7f4e43255640 1 -- 172.21.15.5:0/2211147419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e3c0c4440 con 0x7f4e3c0cc000 2024-03-24T10:45:20.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.911+0000 7f4e43255640 1 -- 172.21.15.5:0/2211147419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e3c0c3e80 con 0x7f4e3c0cc000 2024-03-24T10:45:20.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.912+0000 7f4e43255640 1 -- 172.21.15.5:0/2211147419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e3c0681d0 con 0x7f4e3c0cc000 2024-03-24T10:45:20.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.936+0000 7f4e397fa640 1 -- 172.21.15.5:0/2211147419 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4e30007470 con 0x7f4e3c0cc000 2024-03-24T10:45:20.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.937+0000 7f4e397fa640 1 --2- 172.21.15.5:0/2211147419 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e2006b860 0x7f4e2006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:20.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.937+0000 7f4e3bfff640 1 --2- 172.21.15.5:0/2211147419 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e2006b860 0x7f4e2006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:20.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:20.937+0000 7f4e3bfff640 1 --2- 172.21.15.5:0/2211147419 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e2006b860 0x7f4e2006dd20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4e28009b40 tx=0x7f4e28009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:21.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.245+0000 7f4e397fa640 1 -- 172.21.15.5:0/2211147419 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4e30007720 con 0x7f4e3c0cc000 2024-03-24T10:45:21.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.257+0000 7f4e397fa640 1 -- 172.21.15.5:0/2211147419 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4e300545c0 con 0x7f4e3c0cc000 2024-03-24T10:45:21.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.478+0000 7f4e43255640 1 -- 172.21.15.5:0/2211147419 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4e3c0c30b0 con 0x7f4e3c0cc000 2024-03-24T10:45:21.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.517+0000 7f4e397fa640 1 -- 172.21.15.5:0/2211147419 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f4e300170a0 con 0x7f4e3c0cc000 2024-03-24T10:45:21.517 INFO:teuthology.orchestra.run.smithi005.stdout:pg_num: 1 2024-03-24T10:45:21.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.519+0000 7f4e43255640 1 -- 172.21.15.5:0/2211147419 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e2006b860 msgr2=0x7f4e2006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:21.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.519+0000 7f4e43255640 1 --2- 172.21.15.5:0/2211147419 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e2006b860 0x7f4e2006dd20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4e28009b40 tx=0x7f4e28009190 comp rx=0 tx=0).stop 2024-03-24T10:45:21.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.519+0000 7f4e43255640 1 -- 172.21.15.5:0/2211147419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e3c0cc000 msgr2=0x7f4e3c1e6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:21.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.519+0000 7f4e43255640 1 --2- 172.21.15.5:0/2211147419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e3c0cc000 0x7f4e3c1e6bd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4e3000c3f0 tx=0x7f4e3000c8b0 comp rx=0 tx=0).stop 2024-03-24T10:45:21.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.519+0000 7f4e43255640 1 -- 172.21.15.5:0/2211147419 shutdown_connections 2024-03-24T10:45:21.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.519+0000 7f4e43255640 1 --2- 172.21.15.5:0/2211147419 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e2006b860 0x7f4e2006dd20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:21.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.519+0000 7f4e43255640 1 --2- 172.21.15.5:0/2211147419 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e3c0cf340 0x7f4e3c1f05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:21.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.519+0000 7f4e43255640 1 --2- 172.21.15.5:0/2211147419 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e3c0ccb80 0x7f4e3c1f00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:21.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.519+0000 7f4e43255640 1 --2- 172.21.15.5:0/2211147419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e3c0cc000 0x7f4e3c1e6bd0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:21.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.519+0000 7f4e43255640 1 -- 172.21.15.5:0/2211147419 >> 172.21.15.5:0/2211147419 conn(0x7f4e3c0cfc30 msgr2=0x7f4e3c158d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:21.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.519+0000 7f4e43255640 1 -- 172.21.15.5:0/2211147419 shutdown_connections 2024-03-24T10:45:21.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.520+0000 7f4e43255640 1 -- 172.21.15.5:0/2211147419 wait complete. 2024-03-24T10:45:21.531 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:45:21.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.636+0000 7f6a46be9640 1 Processor -- start 2024-03-24T10:45:21.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.636+0000 7f6a46be9640 1 -- start start 2024-03-24T10:45:21.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.637+0000 7f6a46be9640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6a400cc000 0x7f6a4015e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:21.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.637+0000 7f6a46be9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6a400cf340 0x7f6a400cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:21.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.637+0000 7f6a46be9640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a400cc3e0 con 0x7f6a400ccb80 2024-03-24T10:45:21.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.637+0000 7f6a46be9640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f6a4015a400 con 0x7f6a400cc000 2024-03-24T10:45:21.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.637+0000 7f6a46be9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f6a40154bd0 con 0x7f6a400cf340 2024-03-24T10:45:21.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.638+0000 7f6a44985640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6a400cc000 0x7f6a4015e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:21.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.638+0000 7f6a3ffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6a400cf340 0x7f6a400cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:21.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.638+0000 7f6a44985640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6a400cc000 0x7f6a4015e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:60374/0 (socket says 172.21.15.5:60374) 2024-03-24T10:45:21.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.638+0000 7f6a3ffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6a400cf340 0x7f6a400cc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:51732/0 (socket says 172.21.15.5:51732) 2024-03-24T10:45:21.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.638+0000 7f6a44985640 1 -- 172.21.15.5:0/3816639482 learned_addr learned my addr 172.21.15.5:0/3816639482 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.641+0000 7f6a3ffff640 1 -- 172.21.15.5:0/3816639482 >> v2:172.21.15.5:3300/0 conn(0x7f6a400cc000 msgr2=0x7f6a4015e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.641+0000 7f6a3ffff640 1 --2- 172.21.15.5:0/3816639482 >> v2:172.21.15.5:3300/0 conn(0x7f6a400cc000 0x7f6a4015e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.641+0000 7f6a3ffff640 1 -- 172.21.15.5:0/3816639482 >> v1:172.21.15.5:6789/0 conn(0x7f6a400ccb80 legacy=0x7f6a4006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:21.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.641+0000 7f6a3ffff640 1 -- 172.21.15.5:0/3816639482 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a4015a5f0 con 0x7f6a400cf340 2024-03-24T10:45:21.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.646+0000 7f6a3ffff640 1 --2- 172.21.15.5:0/3816639482 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6a400cf340 0x7f6a400cc640 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c0097a0 tx=0x7f6a2c02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd56acec158dd145 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:21.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.651+0000 7f6a3f7fe640 1 -- 172.21.15.5:0/3816639482 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c030610 con 0x7f6a400cf340 2024-03-24T10:45:21.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.651+0000 7f6a3f7fe640 1 -- 172.21.15.5:0/3816639482 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a2c030770 con 0x7f6a400cf340 2024-03-24T10:45:21.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.651+0000 7f6a3f7fe640 1 -- 172.21.15.5:0/3816639482 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c030b00 con 0x7f6a400cf340 2024-03-24T10:45:21.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.652+0000 7f6a46be9640 1 -- 172.21.15.5:0/3816639482 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6a400cf340 msgr2=0x7f6a400cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:21.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.652+0000 7f6a46be9640 1 --2- 172.21.15.5:0/3816639482 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6a400cf340 0x7f6a400cc640 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c0097a0 tx=0x7f6a2c02b960 comp rx=0 tx=0).stop 2024-03-24T10:45:21.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.652+0000 7f6a46be9640 1 -- 172.21.15.5:0/3816639482 shutdown_connections 2024-03-24T10:45:21.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.652+0000 7f6a46be9640 1 --2- 172.21.15.5:0/3816639482 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6a400cf340 0x7f6a400cc640 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:21.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.652+0000 7f6a46be9640 1 --2- 172.21.15.5:0/3816639482 >> v2:172.21.15.5:3300/0 conn(0x7f6a400cc000 0x7f6a4015e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:21.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.652+0000 7f6a46be9640 1 -- 172.21.15.5:0/3816639482 >> 172.21.15.5:0/3816639482 conn(0x7f6a400cfc30 msgr2=0x7f6a400d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:21.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.652+0000 7f6a46be9640 1 -- 172.21.15.5:0/3816639482 shutdown_connections 2024-03-24T10:45:21.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.652+0000 7f6a46be9640 1 -- 172.21.15.5:0/3816639482 wait complete. 2024-03-24T10:45:21.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.653+0000 7f6a46be9640 1 Processor -- start 2024-03-24T10:45:21.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.653+0000 7f6a46be9640 1 -- start start 2024-03-24T10:45:21.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.654+0000 7f6a46be9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6a400cc000 0x7f6a400c5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:21.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.654+0000 7f6a46be9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a400ccb80 0x7f6a40154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:21.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.654+0000 7f6a46be9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a400cf340 0x7f6a401efc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:21.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.654+0000 7f6a46be9640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f6a400c3d90 con 0x7f6a400ccb80 2024-03-24T10:45:21.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.654+0000 7f6a46be9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6a400c5bd0 con 0x7f6a400cf340 2024-03-24T10:45:21.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.654+0000 7f6a46be9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f6a400c5eb0 con 0x7f6a400cc000 2024-03-24T10:45:21.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.655+0000 7f6a3ffff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a400ccb80 0x7f6a40154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:21.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.655+0000 7f6a44985640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6a400cc000 0x7f6a400c5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:21.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.655+0000 7f6a44985640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6a400cc000 0x7f6a400c5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:51740/0 (socket says 172.21.15.5:51740) 2024-03-24T10:45:21.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.655+0000 7f6a44985640 1 -- 172.21.15.5:0/353937288 learned_addr learned my addr 172.21.15.5:0/353937288 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:21.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.655+0000 7f6a45186640 1 --2- 172.21.15.5:0/353937288 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a400cf340 0x7f6a401efc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:21.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.658+0000 7f6a44985640 1 -- 172.21.15.5:0/353937288 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a400cf340 msgr2=0x7f6a401efc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:21.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.658+0000 7f6a44985640 1 --2- 172.21.15.5:0/353937288 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a400cf340 0x7f6a401efc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:21.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.658+0000 7f6a44985640 1 -- 172.21.15.5:0/353937288 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a400ccb80 msgr2=0x7f6a40154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:21.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.658+0000 7f6a44985640 1 --2- 172.21.15.5:0/353937288 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a400ccb80 0x7f6a40154fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:21.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.658+0000 7f6a44985640 1 -- 172.21.15.5:0/353937288 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a2c009400 con 0x7f6a400cc000 2024-03-24T10:45:21.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.660+0000 7f6a44985640 1 --2- 172.21.15.5:0/353937288 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6a400cc000 0x7f6a400c5390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6a34009fd0 tx=0x7f6a3400ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:21.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.664+0000 7f6a3d7fa640 1 -- 172.21.15.5:0/353937288 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3400c940 con 0x7f6a400cc000 2024-03-24T10:45:21.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.665+0000 7f6a3d7fa640 1 -- 172.21.15.5:0/353937288 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a3400caa0 con 0x7f6a400cc000 2024-03-24T10:45:21.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.665+0000 7f6a46be9640 1 -- 172.21.15.5:0/353937288 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a400c4050 con 0x7f6a400cc000 2024-03-24T10:45:21.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.665+0000 7f6a3d7fa640 1 -- 172.21.15.5:0/353937288 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3400ce30 con 0x7f6a400cc000 2024-03-24T10:45:21.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.665+0000 7f6a46be9640 1 -- 172.21.15.5:0/353937288 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a400c4610 con 0x7f6a400cc000 2024-03-24T10:45:21.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.666+0000 7f6a46be9640 1 -- 172.21.15.5:0/353937288 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a400681d0 con 0x7f6a400cc000 2024-03-24T10:45:21.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.936+0000 7f6a3d7fa640 1 -- 172.21.15.5:0/353937288 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6a34013470 con 0x7f6a400cc000 2024-03-24T10:45:21.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.936+0000 7f6a3d7fa640 1 --2- 172.21.15.5:0/353937288 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6a2406b860 0x7f6a2406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:21.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.937+0000 7f6a3ffff640 1 --2- 172.21.15.5:0/353937288 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6a2406b860 0x7f6a2406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:21.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.937+0000 7f6a3ffff640 1 --2- 172.21.15.5:0/353937288 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6a2406b860 0x7f6a2406dd20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c036000 tx=0x7f6a2c03b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:21.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.941+0000 7f6a3d7fa640 1 -- 172.21.15.5:0/353937288 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6a34059330 con 0x7f6a400cc000 2024-03-24T10:45:21.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:21.952+0000 7f6a3d7fa640 1 -- 172.21.15.5:0/353937288 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6a340555d0 con 0x7f6a400cc000 2024-03-24T10:45:21.966 INFO:tasks.ceph.mds.a.smithi005.stderr:2024-03-24T10:45:21.522+0000 f5f1640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-24T10:45:21.966 INFO:tasks.ceph.mds.a.smithi005.stderr:2024-03-24T10:45:21.523+0000 f5f1640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-24T10:45:22.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.135+0000 7f6a46be9640 1 -- 172.21.15.5:0/353937288 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6a400c30b0 con 0x7f6a400cc000 2024-03-24T10:45:22.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.171+0000 7f6a3d7fa640 1 -- 172.21.15.5:0/353937288 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f6a340170a0 con 0x7f6a400cc000 2024-03-24T10:45:22.172 INFO:teuthology.orchestra.run.smithi005.stdout:pg_num: 32 2024-03-24T10:45:22.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.173+0000 7f6a46be9640 1 -- 172.21.15.5:0/353937288 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6a2406b860 msgr2=0x7f6a2406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:22.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.173+0000 7f6a46be9640 1 --2- 172.21.15.5:0/353937288 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6a2406b860 0x7f6a2406dd20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c036000 tx=0x7f6a2c03b040 comp rx=0 tx=0).stop 2024-03-24T10:45:22.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.173+0000 7f6a46be9640 1 -- 172.21.15.5:0/353937288 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6a400cc000 msgr2=0x7f6a400c5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:22.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.173+0000 7f6a46be9640 1 --2- 172.21.15.5:0/353937288 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6a400cc000 0x7f6a400c5390 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6a34009fd0 tx=0x7f6a3400ed40 comp rx=0 tx=0).stop 2024-03-24T10:45:22.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.174+0000 7f6a46be9640 1 -- 172.21.15.5:0/353937288 shutdown_connections 2024-03-24T10:45:22.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.174+0000 7f6a46be9640 1 --2- 172.21.15.5:0/353937288 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6a2406b860 0x7f6a2406dd20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:22.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.174+0000 7f6a46be9640 1 --2- 172.21.15.5:0/353937288 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6a400ccb80 0x7f6a40154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:22.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.174+0000 7f6a46be9640 1 --2- 172.21.15.5:0/353937288 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6a400cf340 0x7f6a401efc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:22.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.174+0000 7f6a46be9640 1 --2- 172.21.15.5:0/353937288 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6a400cc000 0x7f6a400c5390 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:22.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.174+0000 7f6a46be9640 1 -- 172.21.15.5:0/353937288 >> 172.21.15.5:0/353937288 conn(0x7f6a400cfc30 msgr2=0x7f6a400c6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:22.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.174+0000 7f6a46be9640 1 -- 172.21.15.5:0/353937288 shutdown_connections 2024-03-24T10:45:22.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.174+0000 7f6a46be9640 1 -- 172.21.15.5:0/353937288 wait complete. 2024-03-24T10:45:22.191 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:45:22.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.298+0000 7fe00fb4f640 1 Processor -- start 2024-03-24T10:45:22.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.298+0000 7fe00fb4f640 1 -- start start 2024-03-24T10:45:22.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.299+0000 7fe00fb4f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe0080cc000 0x7fe00815e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:22.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.299+0000 7fe00fb4f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe00806bf50 0x7fe00806c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:22.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.299+0000 7fe00fb4f640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0080cc3e0 con 0x7fe0080cf340 2024-03-24T10:45:22.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.299+0000 7fe00fb4f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fe00815a400 con 0x7fe0080cc000 2024-03-24T10:45:22.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.299+0000 7fe00fb4f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe008154bd0 con 0x7fe00806bf50 2024-03-24T10:45:22.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.337+0000 7fe00d8eb640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe0080cc000 0x7fe00815e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:22.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.338+0000 7fe00d8eb640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe0080cc000 0x7fe00815e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36830/0 (socket says 172.21.15.5:36830) 2024-03-24T10:45:22.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.338+0000 7fe00d8eb640 1 -- 172.21.15.5:0/77687934 learned_addr learned my addr 172.21.15.5:0/77687934 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:22.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.341+0000 7fe00d8eb640 1 -- 172.21.15.5:0/77687934 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe00806bf50 msgr2=0x7fe00806c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:22.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.341+0000 7fe00d8eb640 1 --2- 172.21.15.5:0/77687934 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe00806bf50 0x7fe00806c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:22.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.341+0000 7fe00d8eb640 1 -- 172.21.15.5:0/77687934 >> v1:172.21.15.5:6789/0 conn(0x7fe0080cf340 legacy=0x7fe0080cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:22.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.341+0000 7fe00d8eb640 1 -- 172.21.15.5:0/77687934 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe00806c8e0 con 0x7fe0080cc000 2024-03-24T10:45:22.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.344+0000 7fe00d8eb640 1 --2- 172.21.15.5:0/77687934 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0080cc000 0x7fe00815e3b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe0080b27b0 tx=0x7fe00402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fa5d3d30154961b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:22.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.348+0000 7fe00c8e9640 1 -- 172.21.15.5:0/77687934 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe004031640 con 0x7fe0080cc000 2024-03-24T10:45:22.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.348+0000 7fe00c8e9640 1 -- 172.21.15.5:0/77687934 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0040317a0 con 0x7fe0080cc000 2024-03-24T10:45:22.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.348+0000 7fe00c8e9640 1 -- 172.21.15.5:0/77687934 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe004031b30 con 0x7fe0080cc000 2024-03-24T10:45:22.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.349+0000 7fe00fb4f640 1 -- 172.21.15.5:0/77687934 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0080cc000 msgr2=0x7fe00815e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:22.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.349+0000 7fe00fb4f640 1 --2- 172.21.15.5:0/77687934 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0080cc000 0x7fe00815e3b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe0080b27b0 tx=0x7fe00402b8f0 comp rx=0 tx=0).stop 2024-03-24T10:45:22.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.349+0000 7fe00fb4f640 1 -- 172.21.15.5:0/77687934 shutdown_connections 2024-03-24T10:45:22.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.349+0000 7fe00fb4f640 1 --2- 172.21.15.5:0/77687934 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe00806bf50 0x7fe00806c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:22.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.349+0000 7fe00fb4f640 1 --2- 172.21.15.5:0/77687934 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe0080cc000 0x7fe00815e3b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:22.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.349+0000 7fe00fb4f640 1 -- 172.21.15.5:0/77687934 >> 172.21.15.5:0/77687934 conn(0x7fe0080cfc30 msgr2=0x7fe0080d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:22.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.349+0000 7fe00fb4f640 1 -- 172.21.15.5:0/77687934 shutdown_connections 2024-03-24T10:45:22.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.349+0000 7fe00fb4f640 1 -- 172.21.15.5:0/77687934 wait complete. 2024-03-24T10:45:22.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.350+0000 7fe00fb4f640 1 Processor -- start 2024-03-24T10:45:22.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.351+0000 7fe00fb4f640 1 -- start start 2024-03-24T10:45:22.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.351+0000 7fe00fb4f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe00806bf50 0x7fe0080ab210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:22.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.351+0000 7fe00fb4f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe0080cc000 0x7fe0080c4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:22.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.351+0000 7fe00fb4f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe0080cf340 0x7fe0080ae000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:22.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.351+0000 7fe00fb4f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fe0080ad720 con 0x7fe0080cc000 2024-03-24T10:45:22.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.351+0000 7fe00fb4f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe0080ae540 con 0x7fe00806bf50 2024-03-24T10:45:22.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.351+0000 7fe00fb4f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe0080aed00 con 0x7fe0080cf340 2024-03-24T10:45:22.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.352+0000 7fe00d8eb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe00806bf50 0x7fe0080ab210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:22.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.352+0000 7fe00d8eb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe00806bf50 0x7fe0080ab210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36844/0 (socket says 172.21.15.5:36844) 2024-03-24T10:45:22.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.352+0000 7fe00d8eb640 1 -- 172.21.15.5:0/3436593556 learned_addr learned my addr 172.21.15.5:0/3436593556 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:22.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.357+0000 7fe00d8eb640 1 -- 172.21.15.5:0/3436593556 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe0080cf340 msgr2=0x7fe0080ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:22.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.357+0000 7fe00d8eb640 1 --2- 172.21.15.5:0/3436593556 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe0080cf340 0x7fe0080ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:22.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.357+0000 7fe00d8eb640 1 -- 172.21.15.5:0/3436593556 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe0080cc000 msgr2=0x7fe0080c4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:22.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.357+0000 7fe00d8eb640 1 --2- 172.21.15.5:0/3436593556 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe0080cc000 0x7fe0080c4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:22.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.357+0000 7fe00d8eb640 1 -- 172.21.15.5:0/3436593556 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0040094d0 con 0x7fe00806bf50 2024-03-24T10:45:22.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.361+0000 7fe00d8eb640 1 --2- 172.21.15.5:0/3436593556 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe00806bf50 0x7fe0080ab210 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe004009960 tx=0x7fe0040046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:22.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.366+0000 7fdffe7fc640 1 -- 172.21.15.5:0/3436593556 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00403ac60 con 0x7fe00806bf50 2024-03-24T10:45:22.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.366+0000 7fdffe7fc640 1 -- 172.21.15.5:0/3436593556 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe00403ade0 con 0x7fe00806bf50 2024-03-24T10:45:22.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.366+0000 7fe00fb4f640 1 -- 172.21.15.5:0/3436593556 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0080ae760 con 0x7fe00806bf50 2024-03-24T10:45:22.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.366+0000 7fdffe7fc640 1 -- 172.21.15.5:0/3436593556 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe004044630 con 0x7fe00806bf50 2024-03-24T10:45:22.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.366+0000 7fe00fb4f640 1 -- 172.21.15.5:0/3436593556 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0080c3890 con 0x7fe00806bf50 2024-03-24T10:45:22.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.367+0000 7fe00fb4f640 1 -- 172.21.15.5:0/3436593556 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0080c7e10 con 0x7fe00806bf50 2024-03-24T10:45:22.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.391+0000 7fdffe7fc640 1 -- 172.21.15.5:0/3436593556 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe00402f050 con 0x7fe00806bf50 2024-03-24T10:45:22.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.392+0000 7fdffe7fc640 1 --2- 172.21.15.5:0/3436593556 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdfec06b810 0x7fdfec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:22.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.392+0000 7fe00d0ea640 1 --2- 172.21.15.5:0/3436593556 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdfec06b810 0x7fdfec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:22.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.392+0000 7fe00d0ea640 1 --2- 172.21.15.5:0/3436593556 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdfec06b810 0x7fdfec06dcd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe00815a0d0 tx=0x7fdff8007990 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:22.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.648+0000 7fdffe7fc640 1 -- 172.21.15.5:0/3436593556 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe00407b8c0 con 0x7fe00806bf50 2024-03-24T10:45:22.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.667+0000 7fdffe7fc640 1 -- 172.21.15.5:0/3436593556 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe004077460 con 0x7fe00806bf50 2024-03-24T10:45:22.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.853+0000 7fe00fb4f640 1 -- 172.21.15.5:0/3436593556 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe00815a1e0 con 0x7fe00806bf50 2024-03-24T10:45:22.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.861+0000 7fdffe7fc640 1 -- 172.21.15.5:0/3436593556 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fe0040520a0 con 0x7fe00806bf50 2024-03-24T10:45:22.861 INFO:teuthology.orchestra.run.smithi005.stdout:pg_num: 64 2024-03-24T10:45:22.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.863+0000 7fe00fb4f640 1 -- 172.21.15.5:0/3436593556 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdfec06b810 msgr2=0x7fdfec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:22.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.863+0000 7fe00fb4f640 1 --2- 172.21.15.5:0/3436593556 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdfec06b810 0x7fdfec06dcd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe00815a0d0 tx=0x7fdff8007990 comp rx=0 tx=0).stop 2024-03-24T10:45:22.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.863+0000 7fe00fb4f640 1 -- 172.21.15.5:0/3436593556 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe00806bf50 msgr2=0x7fe0080ab210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:22.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.863+0000 7fe00fb4f640 1 --2- 172.21.15.5:0/3436593556 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe00806bf50 0x7fe0080ab210 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe004009960 tx=0x7fe0040046c0 comp rx=0 tx=0).stop 2024-03-24T10:45:22.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.863+0000 7fe00fb4f640 1 -- 172.21.15.5:0/3436593556 shutdown_connections 2024-03-24T10:45:22.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.863+0000 7fe00fb4f640 1 --2- 172.21.15.5:0/3436593556 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdfec06b810 0x7fdfec06dcd0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:22.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.863+0000 7fe00fb4f640 1 --2- 172.21.15.5:0/3436593556 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe0080cc000 0x7fe0080c4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:22.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.863+0000 7fe00fb4f640 1 --2- 172.21.15.5:0/3436593556 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe0080cf340 0x7fe0080ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:22.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.863+0000 7fe00fb4f640 1 --2- 172.21.15.5:0/3436593556 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe00806bf50 0x7fe0080ab210 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:22.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.863+0000 7fe00fb4f640 1 -- 172.21.15.5:0/3436593556 >> 172.21.15.5:0/3436593556 conn(0x7fe0080cfc30 msgr2=0x7fe0081e5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:22.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.863+0000 7fe00fb4f640 1 -- 172.21.15.5:0/3436593556 shutdown_connections 2024-03-24T10:45:22.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.863+0000 7fe00fb4f640 1 -- 172.21.15.5:0/3436593556 wait complete. 2024-03-24T10:45:22.876 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:45:22.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.978+0000 7f79e306d640 1 Processor -- start 2024-03-24T10:45:22.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.978+0000 7f79e306d640 1 -- start start 2024-03-24T10:45:22.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.979+0000 7f79e306d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f79dc06bf50 0x7f79dc06c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:22.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.979+0000 7f79e306d640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79dc0cb2f0 con 0x7f79dc15ac00 2024-03-24T10:45:22.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.979+0000 7f79e306d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79dc15b070 con 0x7f79dc1543a0 2024-03-24T10:45:22.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.979+0000 7f79e306d640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f79dc0c6750 con 0x7f79dc06bf50 2024-03-24T10:45:22.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.979+0000 7f79dbfff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f79dc15ac00 0x7f79dc0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:41784/0 (socket says 172.21.15.5:41784) 2024-03-24T10:45:22.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.979+0000 7f79dbfff640 1 -- 172.21.15.5:0/4116799869 learned_addr learned my addr 172.21.15.5:0/4116799869 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:22.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.980+0000 7f79e160a640 1 --2- 172.21.15.5:0/4116799869 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f79dc06bf50 0x7f79dc06c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:22.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.982+0000 7f79db7fe640 1 -- 172.21.15.5:0/4116799869 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1059142078 0 0) 0x7f79dc0cb2f0 con 0x7f79dc15ac00 2024-03-24T10:45:22.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.982+0000 7f79db7fe640 1 -- 172.21.15.5:0/4116799869 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79cc003520 con 0x7f79dc15ac00 2024-03-24T10:45:22.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.983+0000 7f79e160a640 1 -- 172.21.15.5:0/4116799869 >> v1:172.21.15.57:6789/0 conn(0x7f79dc1543a0 legacy=0x7f79dc06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:22.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.983+0000 7f79e160a640 1 -- 172.21.15.5:0/4116799869 >> v1:172.21.15.5:6789/0 conn(0x7f79dc15ac00 legacy=0x7f79dc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:22.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.983+0000 7f79e160a640 1 -- 172.21.15.5:0/4116799869 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79dc0ca7e0 con 0x7f79dc06bf50 2024-03-24T10:45:22.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.985+0000 7f79e160a640 1 --2- 172.21.15.5:0/4116799869 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f79dc06bf50 0x7f79dc06c360 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009dc0 tx=0x7f79d402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f4889790953067e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:22.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.989+0000 7f79db7fe640 1 -- 172.21.15.5:0/4116799869 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d402ccd0 con 0x7f79dc06bf50 2024-03-24T10:45:22.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.989+0000 7f79db7fe640 1 -- 172.21.15.5:0/4116799869 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79d402ce30 con 0x7f79dc06bf50 2024-03-24T10:45:22.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.990+0000 7f79db7fe640 1 -- 172.21.15.5:0/4116799869 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d4034660 con 0x7f79dc06bf50 2024-03-24T10:45:22.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.990+0000 7f79e306d640 1 -- 172.21.15.5:0/4116799869 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f79dc06bf50 msgr2=0x7f79dc06c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:22.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.990+0000 7f79e306d640 1 --2- 172.21.15.5:0/4116799869 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f79dc06bf50 0x7f79dc06c360 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009dc0 tx=0x7f79d402bf20 comp rx=0 tx=0).stop 2024-03-24T10:45:22.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.990+0000 7f79e306d640 1 -- 172.21.15.5:0/4116799869 shutdown_connections 2024-03-24T10:45:22.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.990+0000 7f79e306d640 1 --2- 172.21.15.5:0/4116799869 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f79dc06bf50 0x7f79dc06c360 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:22.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.990+0000 7f79e306d640 1 -- 172.21.15.5:0/4116799869 >> 172.21.15.5:0/4116799869 conn(0x7f79dc072ef0 msgr2=0x7f79dc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:22.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.990+0000 7f79e306d640 1 -- 172.21.15.5:0/4116799869 shutdown_connections 2024-03-24T10:45:22.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.990+0000 7f79e306d640 1 -- 172.21.15.5:0/4116799869 wait complete. 2024-03-24T10:45:22.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.992+0000 7f79e306d640 1 Processor -- start 2024-03-24T10:45:22.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.992+0000 7f79e306d640 1 -- start start 2024-03-24T10:45:22.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.992+0000 7f79e306d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f79dc06bf50 0x7f79dc14e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:22.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.992+0000 7f79e306d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79dc1543a0 0x7f79dc14da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:22.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.992+0000 7f79e306d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79dc15ac00 0x7f79dc159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:22.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.993+0000 7f79e306d640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc14ffc0 con 0x7f79dc15ac00 2024-03-24T10:45:22.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.993+0000 7f79e306d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc158ca0 con 0x7f79dc1543a0 2024-03-24T10:45:22.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.993+0000 7f79e306d640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f79dc158090 con 0x7f79dc06bf50 2024-03-24T10:45:22.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.993+0000 7f79e160a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79dc15ac00 0x7f79dc159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:22.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.993+0000 7f79e0e09640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f79dc06bf50 0x7f79dc14e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:22.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.993+0000 7f79dbfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79dc1543a0 0x7f79dc14da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:22.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.993+0000 7f79e0e09640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f79dc06bf50 0x7f79dc14e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:44034/0 (socket says 172.21.15.5:44034) 2024-03-24T10:45:22.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.993+0000 7f79e0e09640 1 -- 172.21.15.5:0/3018920272 learned_addr learned my addr 172.21.15.5:0/3018920272 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:22.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.997+0000 7f79e0e09640 1 -- 172.21.15.5:0/3018920272 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79dc1543a0 msgr2=0x7f79dc14da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:22.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.997+0000 7f79e0e09640 1 --2- 172.21.15.5:0/3018920272 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79dc1543a0 0x7f79dc14da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:22.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.997+0000 7f79e0e09640 1 -- 172.21.15.5:0/3018920272 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79dc15ac00 msgr2=0x7f79dc159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:22.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.997+0000 7f79e0e09640 1 --2- 172.21.15.5:0/3018920272 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79dc15ac00 0x7f79dc159130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:22.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.997+0000 7f79e0e09640 1 -- 172.21.15.5:0/3018920272 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79d4009a70 con 0x7f79dc06bf50 2024-03-24T10:45:22.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:22.999+0000 7f79e0e09640 1 --2- 172.21.15.5:0/3018920272 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f79dc06bf50 0x7f79dc14e640 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f79d000c450 tx=0x7f79d000c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:23.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.004+0000 7f79d97fa640 1 -- 172.21.15.5:0/3018920272 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d00052d0 con 0x7f79dc06bf50 2024-03-24T10:45:23.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.004+0000 7f79d97fa640 1 -- 172.21.15.5:0/3018920272 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79d0005430 con 0x7f79dc06bf50 2024-03-24T10:45:23.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.004+0000 7f79d97fa640 1 -- 172.21.15.5:0/3018920272 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d0012870 con 0x7f79dc06bf50 2024-03-24T10:45:23.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.004+0000 7f79e306d640 1 -- 172.21.15.5:0/3018920272 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79dc158370 con 0x7f79dc06bf50 2024-03-24T10:45:23.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.004+0000 7f79e306d640 1 -- 172.21.15.5:0/3018920272 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f79dc158930 con 0x7f79dc06bf50 2024-03-24T10:45:23.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.005+0000 7f79e306d640 1 -- 172.21.15.5:0/3018920272 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79dc0681d0 con 0x7f79dc06bf50 2024-03-24T10:45:23.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.030+0000 7f79d97fa640 1 -- 172.21.15.5:0/3018920272 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f79d00129d0 con 0x7f79dc06bf50 2024-03-24T10:45:23.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.031+0000 7f79d97fa640 1 --2- 172.21.15.5:0/3018920272 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f79c006b810 0x7f79c006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:23.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.031+0000 7f79dbfff640 1 --2- 172.21.15.5:0/3018920272 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f79c006b810 0x7f79c006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:23.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.031+0000 7f79dbfff640 1 --2- 172.21.15.5:0/3018920272 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f79c006b810 0x7f79c006dcd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f79dc153370 tx=0x7f79c8009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:23.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.303+0000 7f79d97fa640 1 -- 172.21.15.5:0/3018920272 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f79d0058f20 con 0x7f79dc06bf50 2024-03-24T10:45:23.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.315+0000 7f79d97fa640 1 -- 172.21.15.5:0/3018920272 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f79d00551c0 con 0x7f79dc06bf50 2024-03-24T10:45:23.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.577+0000 7f79e306d640 1 -- 172.21.15.5:0/3018920272 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79dc15e580 con 0x7f79dc06bf50 2024-03-24T10:45:23.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.632+0000 7f79d97fa640 1 -- 172.21.15.5:0/3018920272 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6890 (secure 0 0 0) 0x7f79d0058880 con 0x7f79dc06bf50 2024-03-24T10:45:23.633 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:45:23.633 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:45:16.791534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4257,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4257":{"gid":4257,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.57:6837/1426147484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":1426147484},{"type":"v1","addr":"172.21.15.57:6837","nonce":1426147484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4257,4258]},"id":1}]} 2024-03-24T10:45:23.633 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 17 2024-03-24T10:45:23.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.635+0000 7f79e306d640 1 -- 172.21.15.5:0/3018920272 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f79c006b810 msgr2=0x7f79c006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:23.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.635+0000 7f79e306d640 1 --2- 172.21.15.5:0/3018920272 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f79c006b810 0x7f79c006dcd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f79dc153370 tx=0x7f79c8009040 comp rx=0 tx=0).stop 2024-03-24T10:45:23.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.635+0000 7f79e306d640 1 -- 172.21.15.5:0/3018920272 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f79dc06bf50 msgr2=0x7f79dc14e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:23.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.635+0000 7f79e306d640 1 --2- 172.21.15.5:0/3018920272 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f79dc06bf50 0x7f79dc14e640 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f79d000c450 tx=0x7f79d000c910 comp rx=0 tx=0).stop 2024-03-24T10:45:23.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.635+0000 7f79e306d640 1 -- 172.21.15.5:0/3018920272 shutdown_connections 2024-03-24T10:45:23.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.635+0000 7f79e306d640 1 --2- 172.21.15.5:0/3018920272 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f79c006b810 0x7f79c006dcd0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:23.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.635+0000 7f79e306d640 1 --2- 172.21.15.5:0/3018920272 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f79dc15ac00 0x7f79dc159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:23.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.635+0000 7f79e306d640 1 --2- 172.21.15.5:0/3018920272 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f79dc1543a0 0x7f79dc14da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:23.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.635+0000 7f79e306d640 1 --2- 172.21.15.5:0/3018920272 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f79dc06bf50 0x7f79dc14e640 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:23.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.635+0000 7f79e306d640 1 -- 172.21.15.5:0/3018920272 >> 172.21.15.5:0/3018920272 conn(0x7f79dc072ef0 msgr2=0x7f79dc06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:23.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.635+0000 7f79e306d640 1 -- 172.21.15.5:0/3018920272 shutdown_connections 2024-03-24T10:45:23.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.635+0000 7f79e306d640 1 -- 172.21.15.5:0/3018920272 wait complete. 2024-03-24T10:45:23.648 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:45:23.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.751+0000 7f475cfd7640 1 Processor -- start 2024-03-24T10:45:23.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.751+0000 7f475cfd7640 1 -- start start 2024-03-24T10:45:23.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.752+0000 7f475cfd7640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f475806bf50 0x7f475806c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:23.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.752+0000 7f475cfd7640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47580cb2f0 con 0x7f47581543a0 2024-03-24T10:45:23.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.752+0000 7f475cfd7640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f475815b070 con 0x7f475815ac00 2024-03-24T10:45:23.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.752+0000 7f475cfd7640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f47580c6750 con 0x7f475806bf50 2024-03-24T10:45:23.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.753+0000 7f4756d9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f475806bf50 0x7f475806c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:23.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.753+0000 7f474ffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f475815ac00 0x7f47580cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:58748/0 (socket says 172.21.15.5:58748) 2024-03-24T10:45:23.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.753+0000 7f474ffff640 1 -- 172.21.15.5:0/2820441089 learned_addr learned my addr 172.21.15.5:0/2820441089 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:23.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.760+0000 7f4756d9d640 1 -- 172.21.15.5:0/2820441089 >> v1:172.21.15.57:6789/0 conn(0x7f475815ac00 legacy=0x7f47580cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:23.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.760+0000 7f4756d9d640 1 -- 172.21.15.5:0/2820441089 >> v1:172.21.15.5:6789/0 conn(0x7f47581543a0 legacy=0x7f475806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:23.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.761+0000 7f4756d9d640 1 -- 172.21.15.5:0/2820441089 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47580ca7e0 con 0x7f475806bf50 2024-03-24T10:45:23.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.767+0000 7f4756d9d640 1 --2- 172.21.15.5:0/2820441089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f475806bf50 0x7f475806c330 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4750009d90 tx=0x7f475002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a2997f878cac5e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:23.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.775+0000 7f4755d9b640 1 -- 172.21.15.5:0/2820441089 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475002cd00 con 0x7f475806bf50 2024-03-24T10:45:23.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.775+0000 7f4755d9b640 1 -- 172.21.15.5:0/2820441089 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f475002ce60 con 0x7f475806bf50 2024-03-24T10:45:23.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.775+0000 7f4755d9b640 1 -- 172.21.15.5:0/2820441089 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4750034660 con 0x7f475806bf50 2024-03-24T10:45:23.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.776+0000 7f475cfd7640 1 -- 172.21.15.5:0/2820441089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f475806bf50 msgr2=0x7f475806c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:23.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.776+0000 7f475cfd7640 1 --2- 172.21.15.5:0/2820441089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f475806bf50 0x7f475806c330 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4750009d90 tx=0x7f475002bf50 comp rx=0 tx=0).stop 2024-03-24T10:45:23.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.776+0000 7f475cfd7640 1 -- 172.21.15.5:0/2820441089 shutdown_connections 2024-03-24T10:45:23.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.776+0000 7f475cfd7640 1 --2- 172.21.15.5:0/2820441089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f475806bf50 0x7f475806c330 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:23.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.776+0000 7f475cfd7640 1 -- 172.21.15.5:0/2820441089 >> 172.21.15.5:0/2820441089 conn(0x7f4758072ef0 msgr2=0x7f4758061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:23.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.776+0000 7f475cfd7640 1 -- 172.21.15.5:0/2820441089 shutdown_connections 2024-03-24T10:45:23.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.776+0000 7f475cfd7640 1 -- 172.21.15.5:0/2820441089 wait complete. 2024-03-24T10:45:23.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.778+0000 7f475cfd7640 1 Processor -- start 2024-03-24T10:45:23.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.778+0000 7f475cfd7640 1 -- start start 2024-03-24T10:45:23.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.779+0000 7f475cfd7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f475806bf50 0x7f47580cfd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:23.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.779+0000 7f475cfd7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f47581543a0 0x7f47581f3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:23.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.779+0000 7f475cfd7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f475815ac00 0x7f47580cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:23.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.779+0000 7f475cfd7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f47581e64d0 con 0x7f475806bf50 2024-03-24T10:45:23.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.779+0000 7f475cfd7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f47581f3500 con 0x7f475815ac00 2024-03-24T10:45:23.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.779+0000 7f475cfd7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f47580b2ed0 con 0x7f47581543a0 2024-03-24T10:45:23.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.780+0000 7f474ffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f47581543a0 0x7f47581f3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:23.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.780+0000 7f474ffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f47581543a0 0x7f47581f3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:44046/0 (socket says 172.21.15.5:44046) 2024-03-24T10:45:23.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.780+0000 7f474ffff640 1 -- 172.21.15.5:0/2689344149 learned_addr learned my addr 172.21.15.5:0/2689344149 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:23.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.782+0000 7f4756d9d640 1 --2- 172.21.15.5:0/2689344149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f475815ac00 0x7f47580cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:23.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.785+0000 7f474ffff640 1 -- 172.21.15.5:0/2689344149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f475815ac00 msgr2=0x7f47580cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:23.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.785+0000 7f474ffff640 1 --2- 172.21.15.5:0/2689344149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f475815ac00 0x7f47580cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:23.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.785+0000 7f474ffff640 1 -- 172.21.15.5:0/2689344149 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f475806bf50 msgr2=0x7f47580cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:23.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.785+0000 7f474ffff640 1 --2- 172.21.15.5:0/2689344149 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f475806bf50 0x7f47580cfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:23.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.785+0000 7f474ffff640 1 -- 172.21.15.5:0/2689344149 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4750009a70 con 0x7f47581543a0 2024-03-24T10:45:23.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.795+0000 7f474ffff640 1 --2- 172.21.15.5:0/2689344149 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f47581543a0 0x7f47581f3920 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f474000c480 tx=0x7f474000c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:23.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.889+0000 7f474effd640 1 -- 172.21.15.5:0/2689344149 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4740005ce0 con 0x7f47581543a0 2024-03-24T10:45:23.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.889+0000 7f474effd640 1 -- 172.21.15.5:0/2689344149 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4740005e40 con 0x7f47581543a0 2024-03-24T10:45:23.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.889+0000 7f474effd640 1 -- 172.21.15.5:0/2689344149 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47400072f0 con 0x7f47581543a0 2024-03-24T10:45:23.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.889+0000 7f475cfd7640 1 -- 172.21.15.5:0/2689344149 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4758159c00 con 0x7f47581543a0 2024-03-24T10:45:23.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.890+0000 7f475cfd7640 1 -- 172.21.15.5:0/2689344149 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f47580b4b00 con 0x7f47581543a0 2024-03-24T10:45:23.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.891+0000 7f475cfd7640 1 -- 172.21.15.5:0/2689344149 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47580681d0 con 0x7f47581543a0 2024-03-24T10:45:23.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.912+0000 7f474effd640 1 -- 172.21.15.5:0/2689344149 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4740007450 con 0x7f47581543a0 2024-03-24T10:45:23.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.912+0000 7f474effd640 1 --2- 172.21.15.5:0/2689344149 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f473806b890 0x7f473806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:23.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.912+0000 7f475659c640 1 --2- 172.21.15.5:0/2689344149 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f473806b890 0x7f473806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:23.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.913+0000 7f475659c640 1 --2- 172.21.15.5:0/2689344149 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f473806b890 0x7f473806dd50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4748000f60 tx=0x7f47480079d0 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:23.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:23.915+0000 7f474effd640 1 -- 172.21.15.5:0/2689344149 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f47400583e0 con 0x7f47581543a0 2024-03-24T10:45:24.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.175+0000 7f474effd640 1 -- 172.21.15.5:0/2689344149 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4740054680 con 0x7f47581543a0 2024-03-24T10:45:24.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.374+0000 7f475cfd7640 1 -- 172.21.15.5:0/2689344149 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f47580cc070 con 0x7f47581543a0 2024-03-24T10:45:24.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.412+0000 7f474effd640 1 -- 172.21.15.5:0/2689344149 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16485 (secure 0 0 0) 0x7f4740021090 con 0x7f47581543a0 2024-03-24T10:45:24.413 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:45:24.413 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":21,"fsid":"7d9848db-2fe7-4d95-a6e2-b04f4f639ebc","created":"2024-03-24T10:44:30.435176+0000","modified":"2024-03-24T10:45:02.781257+0000","last_up_change":"2024-03-24T10:44:44.634047+0000","last_in_change":"2024-03-24T10:44:35.986391+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:44:45.829320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:44:58.435543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:45:00.129313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50f590b3-f5ec-4e1b-ac46-25cab307a27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6825","nonce":1991336051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6827","nonce":1991336051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6831","nonce":1991336051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6829","nonce":1991336051}]},"public_addr":"172.21.15.5:6825/1991336051","cluster_addr":"172.21.15.5:6827/1991336051","heartbeat_back_addr":"172.21.15.5:6831/1991336051","heartbeat_front_addr":"172.21.15.5:6829/1991336051","state":["exists","up"]},{"osd":1,"uuid":"ccba9e93-c704-472b-a832-8d4eeb456c47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6801","nonce":2140667501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6803","nonce":2140667501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6807","nonce":2140667501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6805","nonce":2140667501}]},"public_addr":"172.21.15.5:6801/2140667501","cluster_addr":"172.21.15.5:6803/2140667501","heartbeat_back_addr":"172.21.15.5:6807/2140667501","heartbeat_front_addr":"172.21.15.5:6805/2140667501","state":["exists","up"]},{"osd":2,"uuid":"e614c5b1-6504-452e-82e0-95c7ca271737","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6817","nonce":90414679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6819","nonce":90414679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6823","nonce":90414679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6821","nonce":90414679}]},"public_addr":"172.21.15.5:6817/90414679","cluster_addr":"172.21.15.5:6819/90414679","heartbeat_back_addr":"172.21.15.5:6823/90414679","heartbeat_front_addr":"172.21.15.5:6821/90414679","state":["exists","up"]},{"osd":3,"uuid":"c60154d0-85e0-46a1-8273-4863bd8ad2dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6809","nonce":4212473119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6811","nonce":4212473119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6815","nonce":4212473119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6813","nonce":4212473119}]},"public_addr":"172.21.15.5:6809/4212473119","cluster_addr":"172.21.15.5:6811/4212473119","heartbeat_back_addr":"172.21.15.5:6815/4212473119","heartbeat_front_addr":"172.21.15.5:6813/4212473119","state":["exists","up"]},{"osd":4,"uuid":"21d74109-da3a-4fc0-b2d8-93e9ae3ed30c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6817","nonce":2193986505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6819","nonce":2193986505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6823","nonce":2193986505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6821","nonce":2193986505}]},"public_addr":"172.21.15.57:6817/2193986505","cluster_addr":"172.21.15.57:6819/2193986505","heartbeat_back_addr":"172.21.15.57:6823/2193986505","heartbeat_front_addr":"172.21.15.57:6821/2193986505","state":["exists","up"]},{"osd":5,"uuid":"5cf19cc7-215f-4d69-a30a-80543c0293e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6809","nonce":3044987466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6811","nonce":3044987466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6815","nonce":3044987466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6813","nonce":3044987466}]},"public_addr":"172.21.15.57:6809/3044987466","cluster_addr":"172.21.15.57:6811/3044987466","heartbeat_back_addr":"172.21.15.57:6815/3044987466","heartbeat_front_addr":"172.21.15.57:6813/3044987466","state":["exists","up"]},{"osd":6,"uuid":"a5e76c69-c3fc-47b2-8f92-e560cf503e45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6801","nonce":4290897445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6803","nonce":4290897445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6807","nonce":4290897445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6805","nonce":4290897445}]},"public_addr":"172.21.15.57:6801/4290897445","cluster_addr":"172.21.15.57:6803/4290897445","heartbeat_back_addr":"172.21.15.57:6807/4290897445","heartbeat_front_addr":"172.21.15.57:6805/4290897445","state":["exists","up"]},{"osd":7,"uuid":"47d0b873-e3ac-4f40-a372-681436e45d0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6825","nonce":212605698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6827","nonce":212605698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6831","nonce":212605698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6829","nonce":212605698}]},"public_addr":"172.21.15.57:6825/212605698","cluster_addr":"172.21.15.57:6827/212605698","heartbeat_back_addr":"172.21.15.57:6831/212605698","heartbeat_front_addr":"172.21.15.57:6829/212605698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.211237+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.192822+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.273429+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.199212+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.277082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.218768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.190915+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.225294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:45:24.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.414+0000 7f475cfd7640 1 -- 172.21.15.5:0/2689344149 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f473806b890 msgr2=0x7f473806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:24.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.414+0000 7f475cfd7640 1 --2- 172.21.15.5:0/2689344149 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f473806b890 0x7f473806dd50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4748000f60 tx=0x7f47480079d0 comp rx=0 tx=0).stop 2024-03-24T10:45:24.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.414+0000 7f475cfd7640 1 -- 172.21.15.5:0/2689344149 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f47581543a0 msgr2=0x7f47581f3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:24.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.414+0000 7f475cfd7640 1 --2- 172.21.15.5:0/2689344149 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f47581543a0 0x7f47581f3920 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f474000c480 tx=0x7f474000c940 comp rx=0 tx=0).stop 2024-03-24T10:45:24.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.414+0000 7f475cfd7640 1 -- 172.21.15.5:0/2689344149 shutdown_connections 2024-03-24T10:45:24.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.414+0000 7f475cfd7640 1 --2- 172.21.15.5:0/2689344149 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f473806b890 0x7f473806dd50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:24.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.414+0000 7f475cfd7640 1 --2- 172.21.15.5:0/2689344149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f475815ac00 0x7f47580cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:24.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.414+0000 7f475cfd7640 1 --2- 172.21.15.5:0/2689344149 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f47581543a0 0x7f47581f3920 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:24.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.414+0000 7f475cfd7640 1 --2- 172.21.15.5:0/2689344149 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f475806bf50 0x7f47580cfd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:24.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.414+0000 7f475cfd7640 1 -- 172.21.15.5:0/2689344149 >> 172.21.15.5:0/2689344149 conn(0x7f4758072ef0 msgr2=0x7f4758150e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:24.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.414+0000 7f475cfd7640 1 -- 172.21.15.5:0/2689344149 shutdown_connections 2024-03-24T10:45:24.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.414+0000 7f475cfd7640 1 -- 172.21.15.5:0/2689344149 wait complete. 2024-03-24T10:45:24.426 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T10:44:45.829320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T10:44:58.435543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T10:45:00.129313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T10:45:24.426 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T10:45:24.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.530+0000 7fba95ec2640 1 Processor -- start 2024-03-24T10:45:24.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.530+0000 7fba95ec2640 1 -- start start 2024-03-24T10:45:24.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.531+0000 7fba95ec2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fba900cf340 0x7fba900cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:24.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.531+0000 7fba95ec2640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fba900ccb80 0x7fba9006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:24.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.531+0000 7fba95ec2640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba900cc3e0 con 0x7fba900cc000 2024-03-24T10:45:24.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.531+0000 7fba95ec2640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fba9015a400 con 0x7fba900ccb80 2024-03-24T10:45:24.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.531+0000 7fba95ec2640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fba90154bd0 con 0x7fba900cf340 2024-03-24T10:45:24.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.532+0000 7fba8effd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fba900cf340 0x7fba900cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:24.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.532+0000 7fba8effd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fba900cf340 0x7fba900cc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:44062/0 (socket says 172.21.15.5:44062) 2024-03-24T10:45:24.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.532+0000 7fba8effd640 1 -- 172.21.15.5:0/33442000 learned_addr learned my addr 172.21.15.5:0/33442000 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:24.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.532+0000 7fba8ffff640 1 --2- 172.21.15.5:0/33442000 >> v2:172.21.15.57:3300/0 conn(0x7fba900ccb80 0x7fba9006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:24.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.535+0000 7fba8effd640 1 -- 172.21.15.5:0/33442000 >> v2:172.21.15.57:3300/0 conn(0x7fba900ccb80 msgr2=0x7fba9006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:24.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.535+0000 7fba8effd640 1 --2- 172.21.15.5:0/33442000 >> v2:172.21.15.57:3300/0 conn(0x7fba900ccb80 0x7fba9006bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:24.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.535+0000 7fba8effd640 1 -- 172.21.15.5:0/33442000 >> v1:172.21.15.57:6789/0 conn(0x7fba900cc000 legacy=0x7fba9015e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:24.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.535+0000 7fba8effd640 1 -- 172.21.15.5:0/33442000 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba9015a5f0 con 0x7fba900cf340 2024-03-24T10:45:24.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.538+0000 7fba8effd640 1 --2- 172.21.15.5:0/33442000 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fba900cf340 0x7fba900cc640 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fba84009870 tx=0x7fba8402ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=679bf1aa3fbf8ba9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:24.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.541+0000 7fba8e7fc640 1 -- 172.21.15.5:0/33442000 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8402f840 con 0x7fba900cf340 2024-03-24T10:45:24.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.543+0000 7fba8e7fc640 1 -- 172.21.15.5:0/33442000 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba8402f9a0 con 0x7fba900cf340 2024-03-24T10:45:24.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.543+0000 7fba8e7fc640 1 -- 172.21.15.5:0/33442000 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8402fd30 con 0x7fba900cf340 2024-03-24T10:45:24.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.544+0000 7fba95ec2640 1 -- 172.21.15.5:0/33442000 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fba900cf340 msgr2=0x7fba900cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:24.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.544+0000 7fba95ec2640 1 --2- 172.21.15.5:0/33442000 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fba900cf340 0x7fba900cc640 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fba84009870 tx=0x7fba8402ba20 comp rx=0 tx=0).stop 2024-03-24T10:45:24.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.544+0000 7fba95ec2640 1 -- 172.21.15.5:0/33442000 shutdown_connections 2024-03-24T10:45:24.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.544+0000 7fba95ec2640 1 --2- 172.21.15.5:0/33442000 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fba900cf340 0x7fba900cc640 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:24.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.544+0000 7fba95ec2640 1 --2- 172.21.15.5:0/33442000 >> v2:172.21.15.57:3300/0 conn(0x7fba900ccb80 0x7fba9006bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:24.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.544+0000 7fba95ec2640 1 -- 172.21.15.5:0/33442000 >> 172.21.15.5:0/33442000 conn(0x7fba900cfc30 msgr2=0x7fba900d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:24.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.544+0000 7fba95ec2640 1 -- 172.21.15.5:0/33442000 shutdown_connections 2024-03-24T10:45:24.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.544+0000 7fba95ec2640 1 -- 172.21.15.5:0/33442000 wait complete. 2024-03-24T10:45:24.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.546+0000 7fba95ec2640 1 Processor -- start 2024-03-24T10:45:24.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.546+0000 7fba95ec2640 1 -- start start 2024-03-24T10:45:24.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.547+0000 7fba95ec2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba900cc000 0x7fba90157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:24.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.547+0000 7fba95ec2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba900ccb80 0x7fba90155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:24.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.547+0000 7fba95ec2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fba900cf340 0x7fba901f36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:24.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.547+0000 7fba95ec2640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fba901f32a0 con 0x7fba900cc000 2024-03-24T10:45:24.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.547+0000 7fba95ec2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fba901f2d70 con 0x7fba900ccb80 2024-03-24T10:45:24.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.547+0000 7fba95ec2640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fba90153f60 con 0x7fba900cf340 2024-03-24T10:45:24.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.548+0000 7fba8effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba900ccb80 0x7fba90155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:24.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.548+0000 7fba8f7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba900cc000 0x7fba90157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:24.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.548+0000 7fba8ffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fba900cf340 0x7fba901f36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:24.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.548+0000 7fba8effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba900ccb80 0x7fba90155300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:48012/0 (socket says 172.21.15.5:48012) 2024-03-24T10:45:24.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.548+0000 7fba8effd640 1 -- 172.21.15.5:0/3594105215 learned_addr learned my addr 172.21.15.5:0/3594105215 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:24.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.552+0000 7fba8effd640 1 -- 172.21.15.5:0/3594105215 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fba900cf340 msgr2=0x7fba901f36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:24.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.552+0000 7fba8effd640 1 --2- 172.21.15.5:0/3594105215 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fba900cf340 0x7fba901f36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:24.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.552+0000 7fba8effd640 1 -- 172.21.15.5:0/3594105215 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba900cc000 msgr2=0x7fba90157b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:24.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.552+0000 7fba8effd640 1 --2- 172.21.15.5:0/3594105215 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba900cc000 0x7fba90157b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:24.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.552+0000 7fba8effd640 1 -- 172.21.15.5:0/3594105215 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba840094d0 con 0x7fba900ccb80 2024-03-24T10:45:24.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.554+0000 7fba8effd640 1 --2- 172.21.15.5:0/3594105215 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba900ccb80 0x7fba90155300 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fba8402ba60 tx=0x7fba84004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:24.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.559+0000 7fba77fff640 1 -- 172.21.15.5:0/3594105215 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8403c040 con 0x7fba900ccb80 2024-03-24T10:45:24.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.559+0000 7fba77fff640 1 -- 172.21.15.5:0/3594105215 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba84037c60 con 0x7fba900ccb80 2024-03-24T10:45:24.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.559+0000 7fba95ec2640 1 -- 172.21.15.5:0/3594105215 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba901e39e0 con 0x7fba900ccb80 2024-03-24T10:45:24.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.559+0000 7fba77fff640 1 -- 172.21.15.5:0/3594105215 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba840423f0 con 0x7fba900ccb80 2024-03-24T10:45:24.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.559+0000 7fba95ec2640 1 -- 172.21.15.5:0/3594105215 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba901f41c0 con 0x7fba900ccb80 2024-03-24T10:45:24.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.560+0000 7fba95ec2640 1 -- 172.21.15.5:0/3594105215 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba900681d0 con 0x7fba900ccb80 2024-03-24T10:45:24.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.890+0000 7fba77fff640 1 -- 172.21.15.5:0/3594105215 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fba8402d3d0 con 0x7fba900ccb80 2024-03-24T10:45:24.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.891+0000 7fba77fff640 1 --2- 172.21.15.5:0/3594105215 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fba7c06b860 0x7fba7c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:24.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.891+0000 7fba8f7fe640 1 --2- 172.21.15.5:0/3594105215 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fba7c06b860 0x7fba7c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:24.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.892+0000 7fba8f7fe640 1 --2- 172.21.15.5:0/3594105215 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fba7c06b860 0x7fba7c06dd20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fba800012e0 tx=0x7fba80009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:24.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.897+0000 7fba77fff640 1 -- 172.21.15.5:0/3594105215 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fba8407b0e0 con 0x7fba900ccb80 2024-03-24T10:45:24.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:24.916+0000 7fba77fff640 1 -- 172.21.15.5:0/3594105215 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fba84077380 con 0x7fba900ccb80 2024-03-24T10:45:25.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.139+0000 7fba95ec2640 1 -- 172.21.15.5:0/3594105215 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fba900cf900 con 0x7fba900ccb80 2024-03-24T10:45:25.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.145+0000 7fba77fff640 1 -- 172.21.15.5:0/3594105215 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fba84047070 con 0x7fba900ccb80 2024-03-24T10:45:25.146 INFO:teuthology.orchestra.run.smithi005.stdout:pg_num: 1 2024-03-24T10:45:25.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.147+0000 7fba95ec2640 1 -- 172.21.15.5:0/3594105215 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fba7c06b860 msgr2=0x7fba7c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:25.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.147+0000 7fba95ec2640 1 --2- 172.21.15.5:0/3594105215 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fba7c06b860 0x7fba7c06dd20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fba800012e0 tx=0x7fba80009040 comp rx=0 tx=0).stop 2024-03-24T10:45:25.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.147+0000 7fba95ec2640 1 -- 172.21.15.5:0/3594105215 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba900ccb80 msgr2=0x7fba90155300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:25.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.147+0000 7fba95ec2640 1 --2- 172.21.15.5:0/3594105215 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba900ccb80 0x7fba90155300 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fba8402ba60 tx=0x7fba84004730 comp rx=0 tx=0).stop 2024-03-24T10:45:25.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.148+0000 7fba95ec2640 1 -- 172.21.15.5:0/3594105215 shutdown_connections 2024-03-24T10:45:25.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.148+0000 7fba95ec2640 1 --2- 172.21.15.5:0/3594105215 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fba7c06b860 0x7fba7c06dd20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:25.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.148+0000 7fba95ec2640 1 --2- 172.21.15.5:0/3594105215 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fba900cf340 0x7fba901f36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:25.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.148+0000 7fba95ec2640 1 --2- 172.21.15.5:0/3594105215 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fba900ccb80 0x7fba90155300 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:25.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.148+0000 7fba95ec2640 1 --2- 172.21.15.5:0/3594105215 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fba900cc000 0x7fba90157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:25.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.148+0000 7fba95ec2640 1 -- 172.21.15.5:0/3594105215 >> 172.21.15.5:0/3594105215 conn(0x7fba900cfc30 msgr2=0x7fba9006bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:25.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.148+0000 7fba95ec2640 1 -- 172.21.15.5:0/3594105215 shutdown_connections 2024-03-24T10:45:25.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.148+0000 7fba95ec2640 1 -- 172.21.15.5:0/3594105215 wait complete. 2024-03-24T10:45:25.159 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T10:45:25.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.264+0000 7ff5832fb640 1 Processor -- start 2024-03-24T10:45:25.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.264+0000 7ff5832fb640 1 -- start start 2024-03-24T10:45:25.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.265+0000 7ff5832fb640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff57c06bf50 0x7ff57c06c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:25.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.265+0000 7ff5832fb640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff57c0cc3e0 con 0x7ff57c0cc000 2024-03-24T10:45:25.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.265+0000 7ff5832fb640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff57c06c8e0 con 0x7ff57c0cf340 2024-03-24T10:45:25.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.265+0000 7ff5832fb640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7ff57c15a400 con 0x7ff57c06bf50 2024-03-24T10:45:25.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.266+0000 7ff580896640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff57c0cf340 0x7ff57c0cc640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:58772/0 (socket says 172.21.15.5:58772) 2024-03-24T10:45:25.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.266+0000 7ff580896640 1 -- 172.21.15.5:0/3848350408 learned_addr learned my addr 172.21.15.5:0/3848350408 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:25.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.266+0000 7ff581898640 1 --2- 172.21.15.5:0/3848350408 >> v2:172.21.15.5:3300/0 conn(0x7ff57c06bf50 0x7ff57c06c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:25.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.268+0000 7ff573fff640 1 -- 172.21.15.5:0/3848350408 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3446941937 0 0) 0x7ff57c06c8e0 con 0x7ff57c0cf340 2024-03-24T10:45:25.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.268+0000 7ff573fff640 1 -- 172.21.15.5:0/3848350408 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff568003520 con 0x7ff57c0cf340 2024-03-24T10:45:25.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.269+0000 7ff581898640 1 -- 172.21.15.5:0/3848350408 >> v1:172.21.15.57:6789/0 conn(0x7ff57c0cf340 legacy=0x7ff57c0cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:25.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.270+0000 7ff581898640 1 -- 172.21.15.5:0/3848350408 >> v1:172.21.15.5:6789/0 conn(0x7ff57c0cc000 legacy=0x7ff57c15e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:25.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.270+0000 7ff581898640 1 -- 172.21.15.5:0/3848350408 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff57c15a5f0 con 0x7ff57c06bf50 2024-03-24T10:45:25.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.275+0000 7ff581898640 1 --2- 172.21.15.5:0/3848350408 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff57c06bf50 0x7ff57c06c3a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff574009d90 tx=0x7ff57402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=955c8c8108a9050a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:25.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.278+0000 7ff573fff640 1 -- 172.21.15.5:0/3848350408 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57402cd00 con 0x7ff57c06bf50 2024-03-24T10:45:25.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.280+0000 7ff573fff640 1 -- 172.21.15.5:0/3848350408 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff57402ce60 con 0x7ff57c06bf50 2024-03-24T10:45:25.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.281+0000 7ff573fff640 1 -- 172.21.15.5:0/3848350408 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff574034640 con 0x7ff57c06bf50 2024-03-24T10:45:25.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.281+0000 7ff5832fb640 1 -- 172.21.15.5:0/3848350408 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff57c06bf50 msgr2=0x7ff57c06c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:25.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.281+0000 7ff5832fb640 1 --2- 172.21.15.5:0/3848350408 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff57c06bf50 0x7ff57c06c3a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff574009d90 tx=0x7ff57402bf50 comp rx=0 tx=0).stop 2024-03-24T10:45:25.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.281+0000 7ff5832fb640 1 -- 172.21.15.5:0/3848350408 shutdown_connections 2024-03-24T10:45:25.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.282+0000 7ff5832fb640 1 --2- 172.21.15.5:0/3848350408 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff57c06bf50 0x7ff57c06c3a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:25.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.282+0000 7ff5832fb640 1 -- 172.21.15.5:0/3848350408 >> 172.21.15.5:0/3848350408 conn(0x7ff57c0cfc30 msgr2=0x7ff57c0d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:25.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.282+0000 7ff5832fb640 1 -- 172.21.15.5:0/3848350408 shutdown_connections 2024-03-24T10:45:25.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.282+0000 7ff5832fb640 1 -- 172.21.15.5:0/3848350408 wait complete. 2024-03-24T10:45:25.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.283+0000 7ff5832fb640 1 Processor -- start 2024-03-24T10:45:25.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.283+0000 7ff5832fb640 1 -- start start 2024-03-24T10:45:25.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.284+0000 7ff5832fb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff57c06bf50 0x7ff57c155060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:25.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.284+0000 7ff5832fb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff57c0cc000 0x7ff57c0b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:25.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.284+0000 7ff5832fb640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff57c0cf340 0x7ff57c0b2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:25.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.284+0000 7ff5832fb640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff57c1555a0 con 0x7ff57c06bf50 2024-03-24T10:45:25.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.284+0000 7ff5832fb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff57c14f300 con 0x7ff57c0cc000 2024-03-24T10:45:25.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.284+0000 7ff5832fb640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff57c14f5e0 con 0x7ff57c0cf340 2024-03-24T10:45:25.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.284+0000 7ff581097640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff57c06bf50 0x7ff57c155060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:25.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.284+0000 7ff580896640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff57c0cc000 0x7ff57c0b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:25.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.285+0000 7ff580896640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff57c0cc000 0x7ff57c0b1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:48026/0 (socket says 172.21.15.5:48026) 2024-03-24T10:45:25.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.285+0000 7ff580896640 1 -- 172.21.15.5:0/4056518678 learned_addr learned my addr 172.21.15.5:0/4056518678 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:25.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.288+0000 7ff581097640 1 -- 172.21.15.5:0/4056518678 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff57c0cf340 msgr2=0x7ff57c0b2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:25.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.288+0000 7ff581097640 1 --2- 172.21.15.5:0/4056518678 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff57c0cf340 0x7ff57c0b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:25.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.288+0000 7ff581097640 1 -- 172.21.15.5:0/4056518678 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff57c0cc000 msgr2=0x7ff57c0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:25.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.288+0000 7ff581097640 1 --2- 172.21.15.5:0/4056518678 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff57c0cc000 0x7ff57c0b1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:25.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.288+0000 7ff581097640 1 -- 172.21.15.5:0/4056518678 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff574009a70 con 0x7ff57c06bf50 2024-03-24T10:45:25.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.291+0000 7ff581097640 1 --2- 172.21.15.5:0/4056518678 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff57c06bf50 0x7ff57c155060 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff57800c340 tx=0x7ff57800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:25.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.295+0000 7ff571ffb640 1 -- 172.21.15.5:0/4056518678 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5780052d0 con 0x7ff57c06bf50 2024-03-24T10:45:25.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.295+0000 7ff571ffb640 1 -- 172.21.15.5:0/4056518678 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff578005450 con 0x7ff57c06bf50 2024-03-24T10:45:25.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.295+0000 7ff571ffb640 1 -- 172.21.15.5:0/4056518678 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff578007260 con 0x7ff57c06bf50 2024-03-24T10:45:25.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.295+0000 7ff5832fb640 1 -- 172.21.15.5:0/4056518678 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff57c14f8c0 con 0x7ff57c06bf50 2024-03-24T10:45:25.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.296+0000 7ff5832fb640 1 -- 172.21.15.5:0/4056518678 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff57c0c8e60 con 0x7ff57c06bf50 2024-03-24T10:45:25.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.297+0000 7ff5832fb640 1 -- 172.21.15.5:0/4056518678 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff57c15e2b0 con 0x7ff57c06bf50 2024-03-24T10:45:25.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.316+0000 7ff571ffb640 1 -- 172.21.15.5:0/4056518678 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff5780073c0 con 0x7ff57c06bf50 2024-03-24T10:45:25.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.317+0000 7ff571ffb640 1 --2- 172.21.15.5:0/4056518678 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff56006b890 0x7ff56006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:25.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.317+0000 7ff580896640 1 --2- 172.21.15.5:0/4056518678 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff56006b890 0x7ff56006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:25.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.317+0000 7ff580896640 1 --2- 172.21.15.5:0/4056518678 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff56006b890 0x7ff56006dd50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff57c150230 tx=0x7ff56c009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:25.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.319+0000 7ff571ffb640 1 -- 172.21.15.5:0/4056518678 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff5780581a0 con 0x7ff57c06bf50 2024-03-24T10:45:25.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.577+0000 7ff571ffb640 1 -- 172.21.15.5:0/4056518678 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff578054440 con 0x7ff57c06bf50 2024-03-24T10:45:25.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.777+0000 7ff5832fb640 1 -- 172.21.15.5:0/4056518678 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff57c0681d0 con 0x7ff57c06bf50 2024-03-24T10:45:25.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.782+0000 7ff571ffb640 1 -- 172.21.15.5:0/4056518678 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7ff5780200a0 con 0x7ff57c06bf50 2024-03-24T10:45:25.782 INFO:teuthology.orchestra.run.smithi005.stdout:pg_num: 32 2024-03-24T10:45:25.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.785+0000 7ff5832fb640 1 -- 172.21.15.5:0/4056518678 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff56006b890 msgr2=0x7ff56006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:25.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.785+0000 7ff5832fb640 1 --2- 172.21.15.5:0/4056518678 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff56006b890 0x7ff56006dd50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff57c150230 tx=0x7ff56c009040 comp rx=0 tx=0).stop 2024-03-24T10:45:25.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.785+0000 7ff5832fb640 1 -- 172.21.15.5:0/4056518678 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff57c06bf50 msgr2=0x7ff57c155060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:25.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.785+0000 7ff5832fb640 1 --2- 172.21.15.5:0/4056518678 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff57c06bf50 0x7ff57c155060 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff57800c340 tx=0x7ff57800c800 comp rx=0 tx=0).stop 2024-03-24T10:45:25.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.785+0000 7ff5832fb640 1 -- 172.21.15.5:0/4056518678 shutdown_connections 2024-03-24T10:45:25.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.785+0000 7ff5832fb640 1 --2- 172.21.15.5:0/4056518678 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff56006b890 0x7ff56006dd50 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:25.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.785+0000 7ff5832fb640 1 --2- 172.21.15.5:0/4056518678 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff57c0cf340 0x7ff57c0b2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:25.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.785+0000 7ff5832fb640 1 --2- 172.21.15.5:0/4056518678 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff57c0cc000 0x7ff57c0b1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:25.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.785+0000 7ff5832fb640 1 --2- 172.21.15.5:0/4056518678 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff57c06bf50 0x7ff57c155060 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:25.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.785+0000 7ff5832fb640 1 -- 172.21.15.5:0/4056518678 >> 172.21.15.5:0/4056518678 conn(0x7ff57c0cfc30 msgr2=0x7ff57c0c7240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:25.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.785+0000 7ff5832fb640 1 -- 172.21.15.5:0/4056518678 shutdown_connections 2024-03-24T10:45:25.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.785+0000 7ff5832fb640 1 -- 172.21.15.5:0/4056518678 wait complete. 2024-03-24T10:45:25.800 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T10:45:25.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.906+0000 7fd457a42640 1 Processor -- start 2024-03-24T10:45:25.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.906+0000 7fd457a42640 1 -- start start 2024-03-24T10:45:25.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.906+0000 7fd457a42640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd4500cc000 0x7fd45015e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:25.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.906+0000 7fd457a42640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4500cf340 0x7fd4500cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:25.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.907+0000 7fd457a42640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4500cc3e0 con 0x7fd4500ccb80 2024-03-24T10:45:25.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.907+0000 7fd457a42640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fd45015a400 con 0x7fd4500cc000 2024-03-24T10:45:25.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.907+0000 7fd457a42640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd450154bd0 con 0x7fd4500cf340 2024-03-24T10:45:25.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.907+0000 7fd455fdf640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fd4500ccb80 0x7fd45006bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:41818/0 (socket says 172.21.15.5:41818) 2024-03-24T10:45:25.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.907+0000 7fd455fdf640 1 -- 172.21.15.5:0/1644215717 learned_addr learned my addr 172.21.15.5:0/1644215717 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:25.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.907+0000 7fd4557de640 1 --2- 172.21.15.5:0/1644215717 >> v2:172.21.15.57:3300/0 conn(0x7fd4500cc000 0x7fd45015e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:25.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.907+0000 7fd454fdd640 1 --2- 172.21.15.5:0/1644215717 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4500cf340 0x7fd4500cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:25.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.910+0000 7fd4557de640 1 -- 172.21.15.5:0/1644215717 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4500cf340 msgr2=0x7fd4500cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:25.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.910+0000 7fd4557de640 1 --2- 172.21.15.5:0/1644215717 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4500cf340 0x7fd4500cc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:25.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.910+0000 7fd4557de640 1 -- 172.21.15.5:0/1644215717 >> v1:172.21.15.5:6789/0 conn(0x7fd4500ccb80 legacy=0x7fd45006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:25.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.910+0000 7fd4557de640 1 -- 172.21.15.5:0/1644215717 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd45015a5f0 con 0x7fd4500cc000 2024-03-24T10:45:25.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.912+0000 7fd4557de640 1 --2- 172.21.15.5:0/1644215717 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4500cc000 0x7fd45015e3b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd44c0097a0 tx=0x7fd44c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=106082be9ecd9904 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:25.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.917+0000 7fd447fff640 1 -- 172.21.15.5:0/1644215717 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd44c030610 con 0x7fd4500cc000 2024-03-24T10:45:25.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.917+0000 7fd447fff640 1 -- 172.21.15.5:0/1644215717 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd44c030770 con 0x7fd4500cc000 2024-03-24T10:45:25.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.917+0000 7fd447fff640 1 -- 172.21.15.5:0/1644215717 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd44c030b00 con 0x7fd4500cc000 2024-03-24T10:45:25.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.918+0000 7fd457a42640 1 -- 172.21.15.5:0/1644215717 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4500cc000 msgr2=0x7fd45015e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:25.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.918+0000 7fd457a42640 1 --2- 172.21.15.5:0/1644215717 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4500cc000 0x7fd45015e3b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd44c0097a0 tx=0x7fd44c02b960 comp rx=0 tx=0).stop 2024-03-24T10:45:25.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.918+0000 7fd457a42640 1 -- 172.21.15.5:0/1644215717 shutdown_connections 2024-03-24T10:45:25.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.918+0000 7fd457a42640 1 --2- 172.21.15.5:0/1644215717 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4500cf340 0x7fd4500cc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:25.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.918+0000 7fd457a42640 1 --2- 172.21.15.5:0/1644215717 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4500cc000 0x7fd45015e3b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:25.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.918+0000 7fd457a42640 1 -- 172.21.15.5:0/1644215717 >> 172.21.15.5:0/1644215717 conn(0x7fd4500cfc30 msgr2=0x7fd4500d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:25.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.918+0000 7fd457a42640 1 -- 172.21.15.5:0/1644215717 shutdown_connections 2024-03-24T10:45:25.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.918+0000 7fd457a42640 1 -- 172.21.15.5:0/1644215717 wait complete. 2024-03-24T10:45:25.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.919+0000 7fd457a42640 1 Processor -- start 2024-03-24T10:45:25.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.919+0000 7fd457a42640 1 -- start start 2024-03-24T10:45:25.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.920+0000 7fd457a42640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4500cc000 0x7fd4500c3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:25.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.920+0000 7fd457a42640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4500ccb80 0x7fd4501e6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:25.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.920+0000 7fd457a42640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4500cf340 0x7fd4501f3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:25.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.920+0000 7fd457a42640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd4501f0630 con 0x7fd4500cc000 2024-03-24T10:45:25.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.920+0000 7fd457a42640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd4501576d0 con 0x7fd4500ccb80 2024-03-24T10:45:25.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.920+0000 7fd457a42640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd4501efcf0 con 0x7fd4500cf340 2024-03-24T10:45:25.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.921+0000 7fd455fdf640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4500cf340 0x7fd4501f3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:25.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.921+0000 7fd454fdd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4500ccb80 0x7fd4501e6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:25.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.921+0000 7fd455fdf640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4500cf340 0x7fd4501f3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:44116/0 (socket says 172.21.15.5:44116) 2024-03-24T10:45:25.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.921+0000 7fd455fdf640 1 -- 172.21.15.5:0/2342286033 learned_addr learned my addr 172.21.15.5:0/2342286033 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:25.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.923+0000 7fd4557de640 1 --2- 172.21.15.5:0/2342286033 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4500cc000 0x7fd4500c3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:25.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.926+0000 7fd455fdf640 1 -- 172.21.15.5:0/2342286033 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4500ccb80 msgr2=0x7fd4501e6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:25.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.926+0000 7fd455fdf640 1 --2- 172.21.15.5:0/2342286033 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4500ccb80 0x7fd4501e6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:25.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.926+0000 7fd455fdf640 1 -- 172.21.15.5:0/2342286033 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4500cc000 msgr2=0x7fd4500c3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:25.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.926+0000 7fd455fdf640 1 --2- 172.21.15.5:0/2342286033 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4500cc000 0x7fd4500c3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:25.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.926+0000 7fd455fdf640 1 -- 172.21.15.5:0/2342286033 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd44c009400 con 0x7fd4500cf340 2024-03-24T10:45:25.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.929+0000 7fd455fdf640 1 --2- 172.21.15.5:0/2342286033 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4500cf340 0x7fd4501f3260 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd44800cb10 tx=0x7fd44800a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:25.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.939+0000 7fd445ffb640 1 -- 172.21.15.5:0/2342286033 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd44800ac70 con 0x7fd4500cf340 2024-03-24T10:45:25.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.939+0000 7fd445ffb640 1 -- 172.21.15.5:0/2342286033 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd44800add0 con 0x7fd4500cf340 2024-03-24T10:45:25.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.939+0000 7fd457a42640 1 -- 172.21.15.5:0/2342286033 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd450157900 con 0x7fd4500cf340 2024-03-24T10:45:25.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.939+0000 7fd445ffb640 1 -- 172.21.15.5:0/2342286033 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd448012620 con 0x7fd4500cf340 2024-03-24T10:45:25.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.939+0000 7fd457a42640 1 -- 172.21.15.5:0/2342286033 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4501eff70 con 0x7fd4500cf340 2024-03-24T10:45:25.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:25.940+0000 7fd457a42640 1 -- 172.21.15.5:0/2342286033 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4500681d0 con 0x7fd4500cf340 2024-03-24T10:45:26.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.261+0000 7fd445ffb640 1 -- 172.21.15.5:0/2342286033 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd448007070 con 0x7fd4500cf340 2024-03-24T10:45:26.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.261+0000 7fd445ffb640 1 --2- 172.21.15.5:0/2342286033 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd43406b840 0x7fd43406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:26.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.262+0000 7fd4557de640 1 --2- 172.21.15.5:0/2342286033 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd43406b840 0x7fd43406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:26.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.262+0000 7fd4557de640 1 --2- 172.21.15.5:0/2342286033 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd43406b840 0x7fd43406dd00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd44c036000 tx=0x7fd44c008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:26.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.266+0000 7fd445ffb640 1 -- 172.21.15.5:0/2342286033 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd448058d40 con 0x7fd4500cf340 2024-03-24T10:45:26.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.286+0000 7fd445ffb640 1 -- 172.21.15.5:0/2342286033 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd448054fe0 con 0x7fd4500cf340 2024-03-24T10:45:26.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.472+0000 7fd457a42640 1 -- 172.21.15.5:0/2342286033 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd4500c30b0 con 0x7fd4500cf340 2024-03-24T10:45:26.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.492+0000 7fd445ffb640 1 -- 172.21.15.5:0/2342286033 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fd448017070 con 0x7fd4500cf340 2024-03-24T10:45:26.492 INFO:teuthology.orchestra.run.smithi005.stdout:pg_num: 64 2024-03-24T10:45:26.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.494+0000 7fd457a42640 1 -- 172.21.15.5:0/2342286033 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd43406b840 msgr2=0x7fd43406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:26.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.494+0000 7fd457a42640 1 --2- 172.21.15.5:0/2342286033 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd43406b840 0x7fd43406dd00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd44c036000 tx=0x7fd44c008030 comp rx=0 tx=0).stop 2024-03-24T10:45:26.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.494+0000 7fd457a42640 1 -- 172.21.15.5:0/2342286033 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4500cf340 msgr2=0x7fd4501f3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:26.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.494+0000 7fd457a42640 1 --2- 172.21.15.5:0/2342286033 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4500cf340 0x7fd4501f3260 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd44800cb10 tx=0x7fd44800a060 comp rx=0 tx=0).stop 2024-03-24T10:45:26.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.494+0000 7fd457a42640 1 -- 172.21.15.5:0/2342286033 shutdown_connections 2024-03-24T10:45:26.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.494+0000 7fd457a42640 1 --2- 172.21.15.5:0/2342286033 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd43406b840 0x7fd43406dd00 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:26.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.494+0000 7fd457a42640 1 --2- 172.21.15.5:0/2342286033 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4500cf340 0x7fd4501f3260 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:26.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.494+0000 7fd457a42640 1 --2- 172.21.15.5:0/2342286033 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4500ccb80 0x7fd4501e6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:26.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.494+0000 7fd457a42640 1 --2- 172.21.15.5:0/2342286033 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4500cc000 0x7fd4500c3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:26.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.494+0000 7fd457a42640 1 -- 172.21.15.5:0/2342286033 >> 172.21.15.5:0/2342286033 conn(0x7fd4500cfc30 msgr2=0x7fd4500c8610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:26.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.494+0000 7fd457a42640 1 -- 172.21.15.5:0/2342286033 shutdown_connections 2024-03-24T10:45:26.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.494+0000 7fd457a42640 1 -- 172.21.15.5:0/2342286033 wait complete. 2024-03-24T10:45:26.506 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:45:26.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.607+0000 7f8a3336c640 1 Processor -- start 2024-03-24T10:45:26.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.607+0000 7f8a3336c640 1 -- start start 2024-03-24T10:45:26.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.609+0000 7f8a3336c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8a2c1543a0 0x7f8a2c15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:26.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.609+0000 7f8a3336c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8a2c06bf50 0x7f8a2c06c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:26.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.609+0000 7f8a3336c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a2c0cb2f0 con 0x7f8a2c06d4f0 2024-03-24T10:45:26.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.609+0000 7f8a3336c640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f8a2c0c6750 con 0x7f8a2c1543a0 2024-03-24T10:45:26.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.609+0000 7f8a3336c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f8a2c0cc380 con 0x7f8a2c06bf50 2024-03-24T10:45:26.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.609+0000 7f8a31108640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8a2c1543a0 0x7f8a2c15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:26.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.610+0000 7f8a31108640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8a2c1543a0 0x7f8a2c15ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44784/0 (socket says 172.21.15.5:44784) 2024-03-24T10:45:26.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.610+0000 7f8a31108640 1 -- 172.21.15.5:0/3380562591 learned_addr learned my addr 172.21.15.5:0/3380562591 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:26.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.610+0000 7f8a31909640 1 --2- 172.21.15.5:0/3380562591 >> v2:172.21.15.57:3300/0 conn(0x7f8a2c06bf50 0x7f8a2c06c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:26.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.613+0000 7f8a31108640 1 -- 172.21.15.5:0/3380562591 >> v2:172.21.15.57:3300/0 conn(0x7f8a2c06bf50 msgr2=0x7f8a2c06c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:26.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.613+0000 7f8a31108640 1 --2- 172.21.15.5:0/3380562591 >> v2:172.21.15.57:3300/0 conn(0x7f8a2c06bf50 0x7f8a2c06c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:26.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.613+0000 7f8a31108640 1 -- 172.21.15.5:0/3380562591 >> v1:172.21.15.57:6789/0 conn(0x7f8a2c06d4f0 legacy=0x7f8a2c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:26.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.613+0000 7f8a31108640 1 -- 172.21.15.5:0/3380562591 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a2c0ca7e0 con 0x7f8a2c1543a0 2024-03-24T10:45:26.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.616+0000 7f8a31108640 1 --2- 172.21.15.5:0/3380562591 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a2c1543a0 0x7f8a2c15ac00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c0b51c0 tx=0x7f8a2802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2574a37c8360a135 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:26.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.621+0000 7f8a23fff640 1 -- 172.21.15.5:0/3380562591 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2802f850 con 0x7f8a2c1543a0 2024-03-24T10:45:26.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.621+0000 7f8a23fff640 1 -- 172.21.15.5:0/3380562591 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a2802f9b0 con 0x7f8a2c1543a0 2024-03-24T10:45:26.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.621+0000 7f8a23fff640 1 -- 172.21.15.5:0/3380562591 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2802fd40 con 0x7f8a2c1543a0 2024-03-24T10:45:26.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.622+0000 7f8a3336c640 1 -- 172.21.15.5:0/3380562591 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a2c1543a0 msgr2=0x7f8a2c15ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:26.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.622+0000 7f8a3336c640 1 --2- 172.21.15.5:0/3380562591 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a2c1543a0 0x7f8a2c15ac00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c0b51c0 tx=0x7f8a2802ba30 comp rx=0 tx=0).stop 2024-03-24T10:45:26.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.622+0000 7f8a3336c640 1 -- 172.21.15.5:0/3380562591 shutdown_connections 2024-03-24T10:45:26.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.622+0000 7f8a3336c640 1 --2- 172.21.15.5:0/3380562591 >> v2:172.21.15.57:3300/0 conn(0x7f8a2c06bf50 0x7f8a2c06c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:26.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.622+0000 7f8a3336c640 1 --2- 172.21.15.5:0/3380562591 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a2c1543a0 0x7f8a2c15ac00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:26.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.622+0000 7f8a3336c640 1 -- 172.21.15.5:0/3380562591 >> 172.21.15.5:0/3380562591 conn(0x7f8a2c072ef0 msgr2=0x7f8a2c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:26.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.622+0000 7f8a3336c640 1 -- 172.21.15.5:0/3380562591 shutdown_connections 2024-03-24T10:45:26.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.622+0000 7f8a3336c640 1 -- 172.21.15.5:0/3380562591 wait complete. 2024-03-24T10:45:26.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.624+0000 7f8a3336c640 1 Processor -- start 2024-03-24T10:45:26.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.624+0000 7f8a3336c640 1 -- start start 2024-03-24T10:45:26.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.625+0000 7f8a3336c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8a2c06bf50 0x7f8a2c14a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:26.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.625+0000 7f8a3336c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a2c06d4f0 0x7f8a2c1e3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:26.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.625+0000 7f8a3336c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a2c1543a0 0x7f8a2c1e3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:26.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.625+0000 7f8a3336c640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2c14adb0 con 0x7f8a2c06d4f0 2024-03-24T10:45:26.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.625+0000 7f8a3336c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2c1e4bf0 con 0x7f8a2c1543a0 2024-03-24T10:45:26.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.625+0000 7f8a3336c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f8a2c1f0ad0 con 0x7f8a2c06bf50 2024-03-24T10:45:26.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.626+0000 7f8a30907640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a2c06d4f0 0x7f8a2c1e3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:26.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.626+0000 7f8a31909640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a2c1543a0 0x7f8a2c1e3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:26.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.626+0000 7f8a30907640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a2c06d4f0 0x7f8a2c1e3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44798/0 (socket says 172.21.15.5:44798) 2024-03-24T10:45:26.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.626+0000 7f8a30907640 1 -- 172.21.15.5:0/1038939873 learned_addr learned my addr 172.21.15.5:0/1038939873 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:26.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.626+0000 7f8a31108640 1 --2- 172.21.15.5:0/1038939873 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8a2c06bf50 0x7f8a2c14a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:26.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.631+0000 7f8a31909640 1 -- 172.21.15.5:0/1038939873 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8a2c06bf50 msgr2=0x7f8a2c14a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:26.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.631+0000 7f8a31909640 1 --2- 172.21.15.5:0/1038939873 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8a2c06bf50 0x7f8a2c14a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:26.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.631+0000 7f8a31909640 1 -- 172.21.15.5:0/1038939873 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a2c06d4f0 msgr2=0x7f8a2c1e3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:26.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.631+0000 7f8a31909640 1 --2- 172.21.15.5:0/1038939873 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a2c06d4f0 0x7f8a2c1e3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:26.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.631+0000 7f8a31909640 1 -- 172.21.15.5:0/1038939873 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a280094d0 con 0x7f8a2c1543a0 2024-03-24T10:45:26.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.634+0000 7f8a31909640 1 --2- 172.21.15.5:0/1038939873 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a2c1543a0 0x7f8a2c1e3a40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8a2400efc0 tx=0x7f8a2400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:26.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.639+0000 7f8a21ffb640 1 -- 172.21.15.5:0/1038939873 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24009c00 con 0x7f8a2c1543a0 2024-03-24T10:45:26.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.639+0000 7f8a21ffb640 1 -- 172.21.15.5:0/1038939873 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a2400f040 con 0x7f8a2c1543a0 2024-03-24T10:45:26.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.639+0000 7f8a3336c640 1 -- 172.21.15.5:0/1038939873 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a2c159720 con 0x7f8a2c1543a0 2024-03-24T10:45:26.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.639+0000 7f8a21ffb640 1 -- 172.21.15.5:0/1038939873 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a240146c0 con 0x7f8a2c1543a0 2024-03-24T10:45:26.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.640+0000 7f8a3336c640 1 -- 172.21.15.5:0/1038939873 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a2c159ce0 con 0x7f8a2c1543a0 2024-03-24T10:45:26.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.640+0000 7f8a3336c640 1 -- 172.21.15.5:0/1038939873 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a2c1e4d60 con 0x7f8a2c1543a0 2024-03-24T10:45:26.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.670+0000 7f8a21ffb640 1 -- 172.21.15.5:0/1038939873 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8a24014820 con 0x7f8a2c1543a0 2024-03-24T10:45:26.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.670+0000 7f8a21ffb640 1 --2- 172.21.15.5:0/1038939873 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8a1006b890 0x7f8a1006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:26.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.670+0000 7f8a31108640 1 --2- 172.21.15.5:0/1038939873 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8a1006b890 0x7f8a1006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:26.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.671+0000 7f8a31108640 1 --2- 172.21.15.5:0/1038939873 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8a1006b890 0x7f8a1006dd50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8a28036000 tx=0x7f8a28008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:26.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.672+0000 7f8a21ffb640 1 -- 172.21.15.5:0/1038939873 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8a24059ce0 con 0x7f8a2c1543a0 2024-03-24T10:45:26.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:26.948+0000 7f8a21ffb640 1 -- 172.21.15.5:0/1038939873 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8a24055f80 con 0x7f8a2c1543a0 2024-03-24T10:45:27.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.175+0000 7f8a3336c640 1 -- 172.21.15.5:0/1038939873 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a2c0681d0 con 0x7f8a2c1543a0 2024-03-24T10:45:27.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.229+0000 7f8a21ffb640 1 -- 172.21.15.5:0/1038939873 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6890 (secure 0 0 0) 0x7f8a24018070 con 0x7f8a2c1543a0 2024-03-24T10:45:27.229 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:45:27.229 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:45:16.791534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4257,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4257":{"gid":4257,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.57:6837/1426147484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":1426147484},{"type":"v1","addr":"172.21.15.57:6837","nonce":1426147484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4257,4258]},"id":1}]} 2024-03-24T10:45:27.230 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 17 2024-03-24T10:45:27.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.231+0000 7f8a3336c640 1 -- 172.21.15.5:0/1038939873 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8a1006b890 msgr2=0x7f8a1006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:27.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.231+0000 7f8a3336c640 1 --2- 172.21.15.5:0/1038939873 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8a1006b890 0x7f8a1006dd50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8a28036000 tx=0x7f8a28008030 comp rx=0 tx=0).stop 2024-03-24T10:45:27.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.231+0000 7f8a3336c640 1 -- 172.21.15.5:0/1038939873 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a2c1543a0 msgr2=0x7f8a2c1e3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:27.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.231+0000 7f8a3336c640 1 --2- 172.21.15.5:0/1038939873 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a2c1543a0 0x7f8a2c1e3a40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8a2400efc0 tx=0x7f8a2400c490 comp rx=0 tx=0).stop 2024-03-24T10:45:27.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.231+0000 7f8a3336c640 1 -- 172.21.15.5:0/1038939873 shutdown_connections 2024-03-24T10:45:27.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.231+0000 7f8a3336c640 1 --2- 172.21.15.5:0/1038939873 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8a1006b890 0x7f8a1006dd50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:27.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.231+0000 7f8a3336c640 1 --2- 172.21.15.5:0/1038939873 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8a2c06d4f0 0x7f8a2c1e3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:27.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.231+0000 7f8a3336c640 1 --2- 172.21.15.5:0/1038939873 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8a2c1543a0 0x7f8a2c1e3a40 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:27.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.231+0000 7f8a3336c640 1 --2- 172.21.15.5:0/1038939873 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8a2c06bf50 0x7f8a2c14a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:27.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.231+0000 7f8a3336c640 1 -- 172.21.15.5:0/1038939873 >> 172.21.15.5:0/1038939873 conn(0x7f8a2c072ef0 msgr2=0x7f8a2c1529f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:27.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.231+0000 7f8a3336c640 1 -- 172.21.15.5:0/1038939873 shutdown_connections 2024-03-24T10:45:27.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.231+0000 7f8a3336c640 1 -- 172.21.15.5:0/1038939873 wait complete. 2024-03-24T10:45:27.243 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:45:27.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.355+0000 7fac557e9640 1 Processor -- start 2024-03-24T10:45:27.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.355+0000 7fac557e9640 1 -- start start 2024-03-24T10:45:27.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.356+0000 7fac557e9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fac5015ac00 0x7fac500cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:27.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.356+0000 7fac557e9640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fac500ccaa0 0x7fac5006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:27.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.356+0000 7fac557e9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac500cb2f0 con 0x7fac501543a0 2024-03-24T10:45:27.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.356+0000 7fac557e9640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fac500c6750 con 0x7fac500ccaa0 2024-03-24T10:45:27.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.356+0000 7fac557e9640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fac500cc380 con 0x7fac5015ac00 2024-03-24T10:45:27.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.357+0000 7fac4f7fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fac500ccaa0 0x7fac5006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:27.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.357+0000 7fac4f7fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fac500ccaa0 0x7fac5006bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44812/0 (socket says 172.21.15.5:44812) 2024-03-24T10:45:27.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.357+0000 7fac4f7fe640 1 -- 172.21.15.5:0/118003331 learned_addr learned my addr 172.21.15.5:0/118003331 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:27.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.357+0000 7fac47fff640 1 --2- 172.21.15.5:0/118003331 >> v2:172.21.15.57:3300/0 conn(0x7fac5015ac00 0x7fac500cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:27.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.359+0000 7fac4f7fe640 1 -- 172.21.15.5:0/118003331 >> v2:172.21.15.57:3300/0 conn(0x7fac5015ac00 msgr2=0x7fac500cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:27.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.359+0000 7fac4f7fe640 1 --2- 172.21.15.5:0/118003331 >> v2:172.21.15.57:3300/0 conn(0x7fac5015ac00 0x7fac500cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:27.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.359+0000 7fac4f7fe640 1 -- 172.21.15.5:0/118003331 >> v1:172.21.15.57:6789/0 conn(0x7fac501543a0 legacy=0x7fac5006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:27.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.359+0000 7fac4f7fe640 1 -- 172.21.15.5:0/118003331 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5015b070 con 0x7fac500ccaa0 2024-03-24T10:45:27.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.361+0000 7fac4f7fe640 1 --2- 172.21.15.5:0/118003331 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac500ccaa0 0x7fac5006bf50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fac500c6990 tx=0x7fac4802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0248f9aa0115a9f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:27.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.366+0000 7fac4e7fc640 1 -- 172.21.15.5:0/118003331 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4802cb20 con 0x7fac500ccaa0 2024-03-24T10:45:27.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.366+0000 7fac4e7fc640 1 -- 172.21.15.5:0/118003331 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac4802cc80 con 0x7fac500ccaa0 2024-03-24T10:45:27.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.366+0000 7fac4e7fc640 1 -- 172.21.15.5:0/118003331 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac480343f0 con 0x7fac500ccaa0 2024-03-24T10:45:27.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.367+0000 7fac557e9640 1 -- 172.21.15.5:0/118003331 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac500ccaa0 msgr2=0x7fac5006bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:27.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.367+0000 7fac557e9640 1 --2- 172.21.15.5:0/118003331 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac500ccaa0 0x7fac5006bf50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fac500c6990 tx=0x7fac4802bf80 comp rx=0 tx=0).stop 2024-03-24T10:45:27.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.367+0000 7fac557e9640 1 -- 172.21.15.5:0/118003331 shutdown_connections 2024-03-24T10:45:27.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.367+0000 7fac557e9640 1 --2- 172.21.15.5:0/118003331 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac500ccaa0 0x7fac5006bf50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:27.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.367+0000 7fac557e9640 1 --2- 172.21.15.5:0/118003331 >> v2:172.21.15.57:3300/0 conn(0x7fac5015ac00 0x7fac500cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:27.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.367+0000 7fac557e9640 1 -- 172.21.15.5:0/118003331 >> 172.21.15.5:0/118003331 conn(0x7fac50072ef0 msgr2=0x7fac50061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:27.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.367+0000 7fac557e9640 1 -- 172.21.15.5:0/118003331 shutdown_connections 2024-03-24T10:45:27.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.367+0000 7fac557e9640 1 -- 172.21.15.5:0/118003331 wait complete. 2024-03-24T10:45:27.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.368+0000 7fac557e9640 1 Processor -- start 2024-03-24T10:45:27.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.368+0000 7fac557e9640 1 -- start start 2024-03-24T10:45:27.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.369+0000 7fac557e9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac500ccaa0 0x7fac501f0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:27.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.369+0000 7fac557e9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac501543a0 0x7fac500b4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:27.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.369+0000 7fac557e9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac5015ac00 0x7fac501441c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:27.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.369+0000 7fac557e9640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fac501f0610 con 0x7fac501543a0 2024-03-24T10:45:27.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.369+0000 7fac557e9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fac501f0020 con 0x7fac500ccaa0 2024-03-24T10:45:27.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.369+0000 7fac557e9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fac5014be50 con 0x7fac5015ac00 2024-03-24T10:45:27.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.370+0000 7fac47fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac501543a0 0x7fac500b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:27.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.370+0000 7fac4effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac500ccaa0 0x7fac501f0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:27.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.370+0000 7fac4f7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac5015ac00 0x7fac501441c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:27.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.370+0000 7fac47fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac501543a0 0x7fac500b4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44826/0 (socket says 172.21.15.5:44826) 2024-03-24T10:45:27.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.370+0000 7fac47fff640 1 -- 172.21.15.5:0/2447904269 learned_addr learned my addr 172.21.15.5:0/2447904269 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:27.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.374+0000 7fac47fff640 1 -- 172.21.15.5:0/2447904269 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac5015ac00 msgr2=0x7fac501441c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:27.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.374+0000 7fac47fff640 1 --2- 172.21.15.5:0/2447904269 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac5015ac00 0x7fac501441c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:27.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.374+0000 7fac47fff640 1 -- 172.21.15.5:0/2447904269 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac500ccaa0 msgr2=0x7fac501f0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:27.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.374+0000 7fac47fff640 1 --2- 172.21.15.5:0/2447904269 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac500ccaa0 0x7fac501f0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:27.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.374+0000 7fac47fff640 1 -- 172.21.15.5:0/2447904269 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac48009b30 con 0x7fac501543a0 2024-03-24T10:45:27.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.376+0000 7fac47fff640 1 --2- 172.21.15.5:0/2447904269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac501543a0 0x7fac500b4190 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fac3800e870 tx=0x7fac3800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:27.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.379+0000 7fac477fe640 1 -- 172.21.15.5:0/2447904269 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac380094f0 con 0x7fac501543a0 2024-03-24T10:45:27.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.379+0000 7fac557e9640 1 -- 172.21.15.5:0/2447904269 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac501f0f70 con 0x7fac501543a0 2024-03-24T10:45:27.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.379+0000 7fac557e9640 1 -- 172.21.15.5:0/2447904269 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac5014b890 con 0x7fac501543a0 2024-03-24T10:45:27.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.380+0000 7fac557e9640 1 -- 172.21.15.5:0/2447904269 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac501f0750 con 0x7fac501543a0 2024-03-24T10:45:27.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.594+0000 7fac477fe640 1 -- 172.21.15.5:0/2447904269 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac38005990 con 0x7fac501543a0 2024-03-24T10:45:27.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.650+0000 7fac477fe640 1 -- 172.21.15.5:0/2447904269 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac38005d00 con 0x7fac501543a0 2024-03-24T10:45:27.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.651+0000 7fac477fe640 1 -- 172.21.15.5:0/2447904269 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fac3800c8e0 con 0x7fac501543a0 2024-03-24T10:45:27.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.651+0000 7fac477fe640 1 --2- 172.21.15.5:0/2447904269 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac3006b840 0x7fac3006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:27.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.651+0000 7fac4effd640 1 --2- 172.21.15.5:0/2447904269 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac3006b840 0x7fac3006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:27.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.652+0000 7fac4effd640 1 --2- 172.21.15.5:0/2447904269 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac3006b840 0x7fac3006dd00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fac40001780 tx=0x7fac40009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:27.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.681+0000 7fac477fe640 1 -- 172.21.15.5:0/2447904269 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fac38058fd0 con 0x7fac501543a0 2024-03-24T10:45:27.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.681+0000 7fac477fe640 1 -- 172.21.15.5:0/2447904269 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fac38055270 con 0x7fac501543a0 2024-03-24T10:45:27.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.907+0000 7fac557e9640 1 -- 172.21.15.5:0/2447904269 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fac501f0750 con 0x7fac501543a0 2024-03-24T10:45:27.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.947+0000 7fac477fe640 1 -- 172.21.15.5:0/2447904269 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16485 (secure 0 0 0) 0x7fac501f0750 con 0x7fac501543a0 2024-03-24T10:45:27.948 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:45:27.948 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":21,"fsid":"7d9848db-2fe7-4d95-a6e2-b04f4f639ebc","created":"2024-03-24T10:44:30.435176+0000","modified":"2024-03-24T10:45:02.781257+0000","last_up_change":"2024-03-24T10:44:44.634047+0000","last_in_change":"2024-03-24T10:44:35.986391+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:44:45.829320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:44:58.435543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:45:00.129313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50f590b3-f5ec-4e1b-ac46-25cab307a27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6825","nonce":1991336051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6827","nonce":1991336051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6831","nonce":1991336051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6829","nonce":1991336051}]},"public_addr":"172.21.15.5:6825/1991336051","cluster_addr":"172.21.15.5:6827/1991336051","heartbeat_back_addr":"172.21.15.5:6831/1991336051","heartbeat_front_addr":"172.21.15.5:6829/1991336051","state":["exists","up"]},{"osd":1,"uuid":"ccba9e93-c704-472b-a832-8d4eeb456c47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6801","nonce":2140667501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6803","nonce":2140667501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6807","nonce":2140667501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6805","nonce":2140667501}]},"public_addr":"172.21.15.5:6801/2140667501","cluster_addr":"172.21.15.5:6803/2140667501","heartbeat_back_addr":"172.21.15.5:6807/2140667501","heartbeat_front_addr":"172.21.15.5:6805/2140667501","state":["exists","up"]},{"osd":2,"uuid":"e614c5b1-6504-452e-82e0-95c7ca271737","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6817","nonce":90414679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6819","nonce":90414679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6823","nonce":90414679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6821","nonce":90414679}]},"public_addr":"172.21.15.5:6817/90414679","cluster_addr":"172.21.15.5:6819/90414679","heartbeat_back_addr":"172.21.15.5:6823/90414679","heartbeat_front_addr":"172.21.15.5:6821/90414679","state":["exists","up"]},{"osd":3,"uuid":"c60154d0-85e0-46a1-8273-4863bd8ad2dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6809","nonce":4212473119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6811","nonce":4212473119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6815","nonce":4212473119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6813","nonce":4212473119}]},"public_addr":"172.21.15.5:6809/4212473119","cluster_addr":"172.21.15.5:6811/4212473119","heartbeat_back_addr":"172.21.15.5:6815/4212473119","heartbeat_front_addr":"172.21.15.5:6813/4212473119","state":["exists","up"]},{"osd":4,"uuid":"21d74109-da3a-4fc0-b2d8-93e9ae3ed30c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6817","nonce":2193986505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6819","nonce":2193986505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6823","nonce":2193986505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6821","nonce":2193986505}]},"public_addr":"172.21.15.57:6817/2193986505","cluster_addr":"172.21.15.57:6819/2193986505","heartbeat_back_addr":"172.21.15.57:6823/2193986505","heartbeat_front_addr":"172.21.15.57:6821/2193986505","state":["exists","up"]},{"osd":5,"uuid":"5cf19cc7-215f-4d69-a30a-80543c0293e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6809","nonce":3044987466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6811","nonce":3044987466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6815","nonce":3044987466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6813","nonce":3044987466}]},"public_addr":"172.21.15.57:6809/3044987466","cluster_addr":"172.21.15.57:6811/3044987466","heartbeat_back_addr":"172.21.15.57:6815/3044987466","heartbeat_front_addr":"172.21.15.57:6813/3044987466","state":["exists","up"]},{"osd":6,"uuid":"a5e76c69-c3fc-47b2-8f92-e560cf503e45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6801","nonce":4290897445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6803","nonce":4290897445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6807","nonce":4290897445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6805","nonce":4290897445}]},"public_addr":"172.21.15.57:6801/4290897445","cluster_addr":"172.21.15.57:6803/4290897445","heartbeat_back_addr":"172.21.15.57:6807/4290897445","heartbeat_front_addr":"172.21.15.57:6805/4290897445","state":["exists","up"]},{"osd":7,"uuid":"47d0b873-e3ac-4f40-a372-681436e45d0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6825","nonce":212605698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6827","nonce":212605698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6831","nonce":212605698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6829","nonce":212605698}]},"public_addr":"172.21.15.57:6825/212605698","cluster_addr":"172.21.15.57:6827/212605698","heartbeat_back_addr":"172.21.15.57:6831/212605698","heartbeat_front_addr":"172.21.15.57:6829/212605698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.211237+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.192822+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.273429+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.199212+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.277082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.218768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.190915+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.225294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:45:27.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.949+0000 7fac557e9640 1 -- 172.21.15.5:0/2447904269 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac3006b840 msgr2=0x7fac3006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:27.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.949+0000 7fac557e9640 1 --2- 172.21.15.5:0/2447904269 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac3006b840 0x7fac3006dd00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fac40001780 tx=0x7fac40009040 comp rx=0 tx=0).stop 2024-03-24T10:45:27.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.949+0000 7fac557e9640 1 -- 172.21.15.5:0/2447904269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac501543a0 msgr2=0x7fac500b4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:27.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.949+0000 7fac557e9640 1 --2- 172.21.15.5:0/2447904269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac501543a0 0x7fac500b4190 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fac3800e870 tx=0x7fac3800ed30 comp rx=0 tx=0).stop 2024-03-24T10:45:27.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.949+0000 7fac557e9640 1 -- 172.21.15.5:0/2447904269 shutdown_connections 2024-03-24T10:45:27.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.949+0000 7fac557e9640 1 --2- 172.21.15.5:0/2447904269 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac3006b840 0x7fac3006dd00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:27.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.949+0000 7fac557e9640 1 --2- 172.21.15.5:0/2447904269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac501543a0 0x7fac500b4190 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:27.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.949+0000 7fac557e9640 1 --2- 172.21.15.5:0/2447904269 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac5015ac00 0x7fac501441c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:27.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.949+0000 7fac557e9640 1 --2- 172.21.15.5:0/2447904269 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac500ccaa0 0x7fac501f0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:27.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.949+0000 7fac557e9640 1 -- 172.21.15.5:0/2447904269 >> 172.21.15.5:0/2447904269 conn(0x7fac50072ef0 msgr2=0x7fac50151a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:27.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.950+0000 7fac557e9640 1 -- 172.21.15.5:0/2447904269 shutdown_connections 2024-03-24T10:45:27.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:27.950+0000 7fac557e9640 1 -- 172.21.15.5:0/2447904269 wait complete. 2024-03-24T10:45:27.962 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:45:28.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.064+0000 7fcdde6eb640 1 Processor -- start 2024-03-24T10:45:28.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.064+0000 7fcdde6eb640 1 -- start start 2024-03-24T10:45:28.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.065+0000 7fcdde6eb640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcdd81543a0 0x7fcdd815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:28.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.065+0000 7fcdde6eb640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcdd806d4f0 0x7fcdd80cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:28.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.065+0000 7fcdde6eb640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcdd80ccaa0 0x7fcdd806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:28.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.065+0000 7fcdde6eb640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fcdd80c6750 con 0x7fcdd80ccaa0 2024-03-24T10:45:28.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.065+0000 7fcdde6eb640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fcdd80cc380 con 0x7fcdd81543a0 2024-03-24T10:45:28.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.065+0000 7fcdde6eb640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fcdd80cb2f0 con 0x7fcdd806d4f0 2024-03-24T10:45:28.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.066+0000 7fcdd7fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcdd81543a0 0x7fcdd815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:28.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.066+0000 7fcddcc88640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcdd80ccaa0 0x7fcdd806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:28.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.066+0000 7fcdd7fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcdd81543a0 0x7fcdd815ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:48088/0 (socket says 172.21.15.5:48088) 2024-03-24T10:45:28.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.066+0000 7fcdd7fff640 1 -- 172.21.15.5:0/2911097114 learned_addr learned my addr 172.21.15.5:0/2911097114 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:28.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.066+0000 7fcdd77fe640 1 --2- 172.21.15.5:0/2911097114 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcdd806d4f0 0x7fcdd80cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:28.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.068+0000 7fcdd7fff640 1 -- 172.21.15.5:0/2911097114 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcdd806d4f0 msgr2=0x7fcdd80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:28.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.068+0000 7fcdd7fff640 1 --2- 172.21.15.5:0/2911097114 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcdd806d4f0 0x7fcdd80cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:28.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.068+0000 7fcdd7fff640 1 -- 172.21.15.5:0/2911097114 >> v2:172.21.15.5:3300/0 conn(0x7fcdd80ccaa0 msgr2=0x7fcdd806bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:28.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.068+0000 7fcdd7fff640 1 --2- 172.21.15.5:0/2911097114 >> v2:172.21.15.5:3300/0 conn(0x7fcdd80ccaa0 0x7fcdd806bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:28.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.068+0000 7fcdd7fff640 1 -- 172.21.15.5:0/2911097114 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd80ca7e0 con 0x7fcdd81543a0 2024-03-24T10:45:28.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.070+0000 7fcdd7fff640 1 --2- 172.21.15.5:0/2911097114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdd81543a0 0x7fcdd815ac00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80cd600 tx=0x7fcdc802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dd484b73f5a5eec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:28.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.075+0000 7fcdd6ffd640 1 -- 172.21.15.5:0/2911097114 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8030860 con 0x7fcdd81543a0 2024-03-24T10:45:28.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.075+0000 7fcdd6ffd640 1 -- 172.21.15.5:0/2911097114 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdc80309c0 con 0x7fcdd81543a0 2024-03-24T10:45:28.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.075+0000 7fcdd6ffd640 1 -- 172.21.15.5:0/2911097114 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8030d50 con 0x7fcdd81543a0 2024-03-24T10:45:28.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.076+0000 7fcdde6eb640 1 -- 172.21.15.5:0/2911097114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdd81543a0 msgr2=0x7fcdd815ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:28.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.076+0000 7fcdde6eb640 1 --2- 172.21.15.5:0/2911097114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdd81543a0 0x7fcdd815ac00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80cd600 tx=0x7fcdc802b9a0 comp rx=0 tx=0).stop 2024-03-24T10:45:28.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.076+0000 7fcdde6eb640 1 -- 172.21.15.5:0/2911097114 shutdown_connections 2024-03-24T10:45:28.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.076+0000 7fcdde6eb640 1 --2- 172.21.15.5:0/2911097114 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcdd806d4f0 0x7fcdd80cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:28.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.076+0000 7fcdde6eb640 1 --2- 172.21.15.5:0/2911097114 >> v2:172.21.15.5:3300/0 conn(0x7fcdd80ccaa0 0x7fcdd806bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:28.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.076+0000 7fcdde6eb640 1 --2- 172.21.15.5:0/2911097114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdd81543a0 0x7fcdd815ac00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:28.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.076+0000 7fcdde6eb640 1 -- 172.21.15.5:0/2911097114 >> 172.21.15.5:0/2911097114 conn(0x7fcdd8072ef0 msgr2=0x7fcdd8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:28.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.076+0000 7fcdde6eb640 1 -- 172.21.15.5:0/2911097114 shutdown_connections 2024-03-24T10:45:28.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.076+0000 7fcdde6eb640 1 -- 172.21.15.5:0/2911097114 wait complete. 2024-03-24T10:45:28.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.078+0000 7fcdde6eb640 1 Processor -- start 2024-03-24T10:45:28.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.078+0000 7fcdde6eb640 1 -- start start 2024-03-24T10:45:28.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.079+0000 7fcdde6eb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdd806d4f0 0x7fcdd81f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:28.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.079+0000 7fcdde6eb640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcdd80ccaa0 0x7fcdd81498a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:28.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.079+0000 7fcdde6eb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdd81543a0 0x7fcdd8149de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:28.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.079+0000 7fcdde6eb640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd81556e0 con 0x7fcdd81543a0 2024-03-24T10:45:28.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.079+0000 7fcdde6eb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd80cf340 con 0x7fcdd806d4f0 2024-03-24T10:45:28.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.079+0000 7fcdde6eb640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fcdd80cf620 con 0x7fcdd80ccaa0 2024-03-24T10:45:28.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.080+0000 7fcdd7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdd806d4f0 0x7fcdd81f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:28.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.080+0000 7fcddcc88640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdd81543a0 0x7fcdd8149de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:28.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.080+0000 7fcdd77fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcdd80ccaa0 0x7fcdd81498a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:28.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.080+0000 7fcdd7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdd806d4f0 0x7fcdd81f2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:48094/0 (socket says 172.21.15.5:48094) 2024-03-24T10:45:28.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.080+0000 7fcdd7fff640 1 -- 172.21.15.5:0/925396965 learned_addr learned my addr 172.21.15.5:0/925396965 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:28.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.083+0000 7fcdd7fff640 1 -- 172.21.15.5:0/925396965 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcdd80ccaa0 msgr2=0x7fcdd81498a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:28.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.083+0000 7fcdd7fff640 1 --2- 172.21.15.5:0/925396965 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcdd80ccaa0 0x7fcdd81498a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:28.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.083+0000 7fcdd7fff640 1 -- 172.21.15.5:0/925396965 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdd81543a0 msgr2=0x7fcdd8149de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:28.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.083+0000 7fcdd7fff640 1 --2- 172.21.15.5:0/925396965 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdd81543a0 0x7fcdd8149de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:28.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.083+0000 7fcdd7fff640 1 -- 172.21.15.5:0/925396965 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc80094d0 con 0x7fcdd806d4f0 2024-03-24T10:45:28.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.085+0000 7fcdd7fff640 1 --2- 172.21.15.5:0/925396965 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdd806d4f0 0x7fcdd81f2f50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8000c00 tx=0x7fcdc8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:28.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.090+0000 7fcdd4ff9640 1 -- 172.21.15.5:0/925396965 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc8039a90 con 0x7fcdd806d4f0 2024-03-24T10:45:28.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.090+0000 7fcdd4ff9640 1 -- 172.21.15.5:0/925396965 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdc8039bf0 con 0x7fcdd806d4f0 2024-03-24T10:45:28.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.090+0000 7fcdd4ff9640 1 -- 172.21.15.5:0/925396965 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc80433f0 con 0x7fcdd806d4f0 2024-03-24T10:45:28.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.090+0000 7fcdde6eb640 1 -- 172.21.15.5:0/925396965 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdd80b3360 con 0x7fcdd806d4f0 2024-03-24T10:45:28.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.090+0000 7fcdde6eb640 1 -- 172.21.15.5:0/925396965 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdd814a430 con 0x7fcdd806d4f0 2024-03-24T10:45:28.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.091+0000 7fcdde6eb640 1 -- 172.21.15.5:0/925396965 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdd80b3880 con 0x7fcdd806d4f0 2024-03-24T10:45:28.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.115+0000 7fcdd4ff9640 1 -- 172.21.15.5:0/925396965 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcdc804c3b0 con 0x7fcdd806d4f0 2024-03-24T10:45:28.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.115+0000 7fcdd4ff9640 1 --2- 172.21.15.5:0/925396965 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcdc406b890 0x7fcdc406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:28.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.116+0000 7fcdd77fe640 1 --2- 172.21.15.5:0/925396965 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcdc406b890 0x7fcdc406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:28.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.116+0000 7fcdd77fe640 1 --2- 172.21.15.5:0/925396965 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcdc406b890 0x7fcdc406dd50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8156330 tx=0x7fcdcc009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:28.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.118+0000 7fcdd4ff9640 1 -- 172.21.15.5:0/925396965 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcdc807afd0 con 0x7fcdd806d4f0 2024-03-24T10:45:28.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.396+0000 7fcdd4ff9640 1 -- 172.21.15.5:0/925396965 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcdc8077270 con 0x7fcdd806d4f0 2024-03-24T10:45:28.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.615+0000 7fcdde6eb640 1 -- 172.21.15.5:0/925396965 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcdd80b3880 con 0x7fcdd806d4f0 2024-03-24T10:45:28.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.631+0000 7fcdd4ff9640 1 -- 172.21.15.5:0/925396965 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+6890 (secure 0 0 0) 0x7fcdd80b3880 con 0x7fcdd806d4f0 2024-03-24T10:45:28.631 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:45:28.631 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:45:16.791534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4257,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4257":{"gid":4257,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":5,"addr":"172.21.15.57:6837/1426147484","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":1426147484},{"type":"v1","addr":"172.21.15.57:6837","nonce":1426147484}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4257,4258]},"id":1}]} 2024-03-24T10:45:28.631 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 17 2024-03-24T10:45:28.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.633+0000 7fcdde6eb640 1 -- 172.21.15.5:0/925396965 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcdc406b890 msgr2=0x7fcdc406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:28.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.633+0000 7fcdde6eb640 1 --2- 172.21.15.5:0/925396965 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcdc406b890 0x7fcdc406dd50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8156330 tx=0x7fcdcc009190 comp rx=0 tx=0).stop 2024-03-24T10:45:28.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.633+0000 7fcdde6eb640 1 -- 172.21.15.5:0/925396965 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdd806d4f0 msgr2=0x7fcdd81f2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:28.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.633+0000 7fcdde6eb640 1 --2- 172.21.15.5:0/925396965 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdd806d4f0 0x7fcdd81f2f50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8000c00 tx=0x7fcdc8004730 comp rx=0 tx=0).stop 2024-03-24T10:45:28.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.633+0000 7fcdde6eb640 1 -- 172.21.15.5:0/925396965 shutdown_connections 2024-03-24T10:45:28.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.633+0000 7fcdde6eb640 1 --2- 172.21.15.5:0/925396965 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcdc406b890 0x7fcdc406dd50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:28.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.633+0000 7fcdde6eb640 1 --2- 172.21.15.5:0/925396965 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcdd81543a0 0x7fcdd8149de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:28.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.633+0000 7fcdde6eb640 1 --2- 172.21.15.5:0/925396965 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcdd80ccaa0 0x7fcdd81498a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:28.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.633+0000 7fcdde6eb640 1 --2- 172.21.15.5:0/925396965 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcdd806d4f0 0x7fcdd81f2f50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:28.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.633+0000 7fcdde6eb640 1 -- 172.21.15.5:0/925396965 >> 172.21.15.5:0/925396965 conn(0x7fcdd8072ef0 msgr2=0x7fcdd80cabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:28.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.633+0000 7fcdde6eb640 1 -- 172.21.15.5:0/925396965 shutdown_connections 2024-03-24T10:45:28.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:28.633+0000 7fcdde6eb640 1 -- 172.21.15.5:0/925396965 wait complete. 2024-03-24T10:45:29.249 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:45:16.791534+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4257, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4257': {'gid': 4257, 'name': 'f', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.57:6837/1426147484', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6835', 'nonce': 1426147484}, {'type': 'v1', 'addr': '172.21.15.57:6837', 'nonce': 1426147484}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4257, 4258]} 2024-03-24T10:45:29.249 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-24T10:45:29.249 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T10:45:29.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.352+0000 7f3750def640 1 Processor -- start 2024-03-24T10:45:29.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.352+0000 7f3750def640 1 -- start start 2024-03-24T10:45:29.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.353+0000 7f3750def640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f374c0cc000 0x7f374c15e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:29.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.353+0000 7f3750def640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f374c06bf50 0x7f374c06c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:29.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.353+0000 7f3750def640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f374c0cc3e0 con 0x7f374c0cf340 2024-03-24T10:45:29.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.353+0000 7f3750def640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f374c15a400 con 0x7f374c06bf50 2024-03-24T10:45:29.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.353+0000 7f3750def640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f374c154bd0 con 0x7f374c0cc000 2024-03-24T10:45:29.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.354+0000 7f374ad9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f374c06bf50 0x7f374c06c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:29.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.354+0000 7f3743fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f374c0cf340 0x7f374c0cc640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:58796/0 (socket says 172.21.15.5:58796) 2024-03-24T10:45:29.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.354+0000 7f3743fff640 1 -- 172.21.15.5:0/3909101008 learned_addr learned my addr 172.21.15.5:0/3909101008 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:45:29.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.354+0000 7f374a59c640 1 --2- 172.21.15.5:0/3909101008 >> v2:172.21.15.57:3300/0 conn(0x7f374c0cc000 0x7f374c15e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:29.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.357+0000 7f374ad9d640 1 -- 172.21.15.5:0/3909101008 >> v2:172.21.15.57:3300/0 conn(0x7f374c0cc000 msgr2=0x7f374c15e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:29.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.357+0000 7f374ad9d640 1 --2- 172.21.15.5:0/3909101008 >> v2:172.21.15.57:3300/0 conn(0x7f374c0cc000 0x7f374c15e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:29.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.357+0000 7f374ad9d640 1 -- 172.21.15.5:0/3909101008 >> v1:172.21.15.57:6789/0 conn(0x7f374c0cf340 legacy=0x7f374c0cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:29.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.357+0000 7f374ad9d640 1 -- 172.21.15.5:0/3909101008 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374c06c8e0 con 0x7f374c06bf50 2024-03-24T10:45:29.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.359+0000 7f374ad9d640 1 --2- 172.21.15.5:0/3909101008 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f374c06bf50 0x7f374c06c3a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f374c0b27b0 tx=0x7f374402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=776e6b6206dff314 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:29.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.363+0000 7f3749d9b640 1 -- 172.21.15.5:0/3909101008 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374402cc80 con 0x7f374c06bf50 2024-03-24T10:45:29.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.363+0000 7f3749d9b640 1 -- 172.21.15.5:0/3909101008 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f374402cde0 con 0x7f374c06bf50 2024-03-24T10:45:29.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.363+0000 7f3749d9b640 1 -- 172.21.15.5:0/3909101008 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3744034660 con 0x7f374c06bf50 2024-03-24T10:45:29.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.364+0000 7f3750def640 1 -- 172.21.15.5:0/3909101008 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f374c06bf50 msgr2=0x7f374c06c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:29.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.364+0000 7f3750def640 1 --2- 172.21.15.5:0/3909101008 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f374c06bf50 0x7f374c06c3a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f374c0b27b0 tx=0x7f374402bed0 comp rx=0 tx=0).stop 2024-03-24T10:45:29.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.364+0000 7f3750def640 1 -- 172.21.15.5:0/3909101008 shutdown_connections 2024-03-24T10:45:29.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.364+0000 7f3750def640 1 --2- 172.21.15.5:0/3909101008 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f374c06bf50 0x7f374c06c3a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:29.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.364+0000 7f3750def640 1 --2- 172.21.15.5:0/3909101008 >> v2:172.21.15.57:3300/0 conn(0x7f374c0cc000 0x7f374c15e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:29.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.364+0000 7f3750def640 1 -- 172.21.15.5:0/3909101008 >> 172.21.15.5:0/3909101008 conn(0x7f374c0cfc30 msgr2=0x7f374c0d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:29.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.364+0000 7f3750def640 1 -- 172.21.15.5:0/3909101008 shutdown_connections 2024-03-24T10:45:29.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.364+0000 7f3750def640 1 -- 172.21.15.5:0/3909101008 wait complete. 2024-03-24T10:45:29.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.366+0000 7f3750def640 1 Processor -- start 2024-03-24T10:45:29.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.366+0000 7f3750def640 1 -- start start 2024-03-24T10:45:29.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.366+0000 7f3750def640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f374c06bf50 0x7f374c0c8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:29.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.367+0000 7f3750def640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f374c0cc000 0x7f374c157690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:29.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.367+0000 7f3750def640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f374c0cf340 0x7f374c1f3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:29.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.367+0000 7f3750def640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f374c0c3e10 con 0x7f374c0cf340 2024-03-24T10:45:29.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.367+0000 7f3750def640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f374c0c4230 con 0x7f374c0cc000 2024-03-24T10:45:29.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.367+0000 7f3750def640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f374c0c5450 con 0x7f374c06bf50 2024-03-24T10:45:29.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.367+0000 7f3743fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f374c0cc000 0x7f374c157690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:29.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.367+0000 7f374ad9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f374c0cf340 0x7f374c1f3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:29.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.367+0000 7f3743fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f374c0cc000 0x7f374c157690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:48124/0 (socket says 172.21.15.5:48124) 2024-03-24T10:45:29.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.367+0000 7f3743fff640 1 -- 172.21.15.5:0/3479670570 learned_addr learned my addr 172.21.15.5:0/3479670570 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:29.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.370+0000 7f374a59c640 1 --2- 172.21.15.5:0/3479670570 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f374c06bf50 0x7f374c0c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:29.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.371+0000 7f374ad9d640 1 -- 172.21.15.5:0/3479670570 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f374c06bf50 msgr2=0x7f374c0c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:29.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.371+0000 7f374ad9d640 1 --2- 172.21.15.5:0/3479670570 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f374c06bf50 0x7f374c0c8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:29.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.371+0000 7f374ad9d640 1 -- 172.21.15.5:0/3479670570 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f374c0cc000 msgr2=0x7f374c157690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:29.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.371+0000 7f374ad9d640 1 --2- 172.21.15.5:0/3479670570 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f374c0cc000 0x7f374c157690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:29.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.371+0000 7f374ad9d640 1 -- 172.21.15.5:0/3479670570 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3744009a70 con 0x7f374c0cf340 2024-03-24T10:45:29.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.374+0000 7f374ad9d640 1 --2- 172.21.15.5:0/3479670570 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f374c0cf340 0x7f374c1f3310 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f374c0cc590 tx=0x7f3744004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:29.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.379+0000 7f3742ffd640 1 -- 172.21.15.5:0/3479670570 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374403e600 con 0x7f374c0cf340 2024-03-24T10:45:29.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.379+0000 7f3742ffd640 1 -- 172.21.15.5:0/3479670570 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f374403e760 con 0x7f374c0cf340 2024-03-24T10:45:29.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.379+0000 7f3750def640 1 -- 172.21.15.5:0/3479670570 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f374c0c37c0 con 0x7f374c0cf340 2024-03-24T10:45:29.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.379+0000 7f3742ffd640 1 -- 172.21.15.5:0/3479670570 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374403eb10 con 0x7f374c0cf340 2024-03-24T10:45:29.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.379+0000 7f3750def640 1 -- 172.21.15.5:0/3479670570 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f374c155550 con 0x7f374c0cf340 2024-03-24T10:45:29.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.382+0000 7f37539c5740 1 Processor -- start 2024-03-24T10:45:29.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.382+0000 7f37539c5740 1 -- start start 2024-03-24T10:45:29.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.382+0000 7f37539c5740 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x556e3e477b70 0x556e3e437910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:29.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.382+0000 7f37539c5740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556e3e436610 0x556e3e478530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:29.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.382+0000 7f37539c5740 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x556e3e477f50 0x556e3e476e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:29.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.382+0000 7f37539c5740 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x556e3e3a5180 con 0x556e3e477b70 2024-03-24T10:45:29.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.382+0000 7f37539c5740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x556e3e3a26b0 con 0x556e3e436610 2024-03-24T10:45:29.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.382+0000 7f37539c5740 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x556e3e39de90 con 0x556e3e477f50 2024-03-24T10:45:29.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.383+0000 7f3743fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x556e3e477f50 0x556e3e476e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:29.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.383+0000 7f374a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556e3e436610 0x556e3e478530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:29.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.384+0000 7f3743fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x556e3e477f50 0x556e3e476e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:44158/0 (socket says 172.21.15.5:44158) 2024-03-24T10:45:29.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.384+0000 7f3743fff640 1 -- 172.21.15.5:0/343300715 learned_addr learned my addr 172.21.15.5:0/343300715 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:29.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.386+0000 7f3743fff640 1 -- 172.21.15.5:0/343300715 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556e3e436610 msgr2=0x556e3e478530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:29.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.386+0000 7f3743fff640 1 --2- 172.21.15.5:0/343300715 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556e3e436610 0x556e3e478530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:29.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.387+0000 7f3743fff640 1 -- 172.21.15.5:0/343300715 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x556e3e477b70 msgr2=0x556e3e437910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:29.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.387+0000 7f3743fff640 1 --2- 172.21.15.5:0/343300715 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x556e3e477b70 0x556e3e437910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:29.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.387+0000 7f3743fff640 1 -- 172.21.15.5:0/343300715 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556e3e35ad80 con 0x556e3e477f50 2024-03-24T10:45:29.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.389+0000 7f3743fff640 1 --2- 172.21.15.5:0/343300715 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x556e3e477f50 0x556e3e476e50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3734006d90 tx=0x7f3734007410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcd7fc7d1ca603ad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:29.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.393+0000 7f372bfff640 1 -- 172.21.15.5:0/343300715 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3734001ac0 con 0x556e3e477f50 2024-03-24T10:45:29.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.393+0000 7f372bfff640 1 -- 172.21.15.5:0/343300715 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3734001c20 con 0x556e3e477f50 2024-03-24T10:45:29.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.393+0000 7f372bfff640 1 -- 172.21.15.5:0/343300715 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3734006310 con 0x556e3e477f50 2024-03-24T10:45:29.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.394+0000 7f37539c5740 1 -- 172.21.15.5:0/343300715 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x556e3e477f50 msgr2=0x556e3e476e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:45:29.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.394+0000 7f37539c5740 1 --2- 172.21.15.5:0/343300715 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x556e3e477f50 0x556e3e476e50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3734006d90 tx=0x7f3734007410 comp rx=0 tx=0).stop 2024-03-24T10:45:29.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.394+0000 7f37539c5740 1 -- 172.21.15.5:0/343300715 shutdown_connections 2024-03-24T10:45:29.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.394+0000 7f37539c5740 1 --2- 172.21.15.5:0/343300715 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x556e3e477f50 0x556e3e476e50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:29.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.394+0000 7f37539c5740 1 --2- 172.21.15.5:0/343300715 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556e3e436610 0x556e3e478530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:29.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.394+0000 7f37539c5740 1 --2- 172.21.15.5:0/343300715 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x556e3e477b70 0x556e3e437910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:29.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.394+0000 7f37539c5740 1 -- 172.21.15.5:0/343300715 >> 172.21.15.5:0/343300715 conn(0x556e3e433780 msgr2=0x556e3e436f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:45:29.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.394+0000 7f37539c5740 1 -- 172.21.15.5:0/343300715 shutdown_connections 2024-03-24T10:45:29.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.394+0000 7f37539c5740 1 -- 172.21.15.5:0/343300715 wait complete. 2024-03-24T10:45:29.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.395+0000 7f37539c5740 1 -- start start 2024-03-24T10:45:29.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.395+0000 7f37539c5740 1 Processor -- start 2024-03-24T10:45:29.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.396+0000 7f37539c5740 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556e3e477b70 0x556e3e45ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:29.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.396+0000 7f37539c5740 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x556e3e477f50 0x556e3e45ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:29.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.396+0000 7f37539c5740 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x556e3e432c60 0x556e3e458460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:29.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.396+0000 7f37539c5740 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x556e3e342d50 con 0x556e3e477f50 2024-03-24T10:45:29.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.396+0000 7f37539c5740 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x556e3e071310 con 0x556e3e477b70 2024-03-24T10:45:29.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.396+0000 7f37539c5740 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x556e3e3238f0 con 0x556e3e432c60 2024-03-24T10:45:29.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.396+0000 7f37539c5740 10 client.0 _ll_register_callbacks cb 0x556e3e0aeac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T10:45:29.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.396+0000 7f3743fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x556e3e432c60 0x556e3e458460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:29.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.397+0000 7f3743fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556e3e477b70 0x556e3e45ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:29.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.397+0000 7f3743fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x556e3e432c60 0x556e3e458460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:44166/0 (socket says 172.21.15.5:44166) 2024-03-24T10:45:29.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.397+0000 7f3743fff640 1 -- 172.21.15.5:0/3461217414 learned_addr learned my addr 172.21.15.5:0/3461217414 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:45:29.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.401+0000 7f3743fff640 1 -- 172.21.15.5:0/3461217414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556e3e477b70 msgr2=0x556e3e45ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:29.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.401+0000 7f3743fff640 1 --2- 172.21.15.5:0/3461217414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x556e3e477b70 0x556e3e45ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:29.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.401+0000 7f3743fff640 1 -- 172.21.15.5:0/3461217414 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x556e3e477f50 msgr2=0x556e3e45ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:45:29.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.401+0000 7f3743fff640 1 --2- 172.21.15.5:0/3461217414 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x556e3e477f50 0x556e3e45ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:45:29.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.401+0000 7f3743fff640 1 -- 172.21.15.5:0/3461217414 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f374c0cc3e0 con 0x556e3e432c60 2024-03-24T10:45:29.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.404+0000 7f3742ffd640 1 -- 172.21.15.5:0/3479670570 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f374403ec70 con 0x7f374c0cf340 2024-03-24T10:45:29.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.405+0000 7f3742ffd640 1 --2- 172.21.15.5:0/3479670570 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f372c06b800 0x7f372c06dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:29.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.405+0000 7f374a59c640 1 --2- 172.21.15.5:0/3479670570 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f372c06b800 0x7f372c06dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:45:29.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.405+0000 7f3743fff640 1 --2- 172.21.15.5:0/3461217414 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x556e3e432c60 0x556e3e458460 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f37340016b0 tx=0x7f373400e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6337d5cb408c5bb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:29.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.405+0000 7f37297fa640 10 client.0 ms_handle_connect on v2:172.21.15.57:3301/0 2024-03-24T10:45:29.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.405+0000 7f374a59c640 1 --2- 172.21.15.5:0/3479670570 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f372c06b800 0x7f372c06dcc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x556e3e3e3530 tx=0x7f373c009860 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:45:29.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.411+0000 7f37297fa640 1 -- 172.21.15.5:0/3461217414 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37340060d0 con 0x556e3e432c60 2024-03-24T10:45:29.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.411+0000 7f37297fa640 1 -- 172.21.15.5:0/3461217414 <== mon.2 v2:172.21.15.57:3301/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3734006230 con 0x556e3e432c60 2024-03-24T10:45:29.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.411+0000 7f37539c5740 1 -- 172.21.15.5:0/3461217414 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_get_version(what=fsmap handle=1) -- 0x556e3e09d6c0 con 0x556e3e432c60 2024-03-24T10:45:29.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.413+0000 7f37297fa640 1 -- 172.21.15.5:0/3461217414 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f37340047e0 con 0x556e3e432c60 2024-03-24T10:45:29.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.413+0000 7f37539c5740 10 client.4445 fetch_fsmap learned FSMap version 17 2024-03-24T10:45:29.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.413+0000 7f37539c5740 1 -- 172.21.15.5:0/3461217414 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,fsmap=17,monmap=2+}) -- 0x556e3e39c510 con 0x556e3e432c60 2024-03-24T10:45:29.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.419+0000 7f37297fa640 1 -- 172.21.15.5:0/3461217414 <== mon.2 v2:172.21.15.57:3301/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x556e3e09d6c0 con 0x556e3e432c60 2024-03-24T10:45:29.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.419+0000 7f37297fa640 1 -- 172.21.15.5:0/3461217414 <== mon.2 v2:172.21.15.57:3301/0 5 ==== fsmap(e 17) ==== 4784+0+0 (secure 0 0 0) 0x7f3734004940 con 0x556e3e432c60 2024-03-24T10:45:29.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.419+0000 7f37539c5740 10 client.4445 fetch_fsmap finished waiting for FSMap version 17 2024-03-24T10:45:29.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.419+0000 7f37539c5740 10 client.4445 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T10:45:29.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.419+0000 7f37539c5740 20 client.4445 populate_metadata read hostname 'smithi005' 2024-03-24T10:45:29.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.419+0000 7f37539c5740 1 --2- 172.21.15.5:0/3461217414 >> [v2:172.21.15.57:6834/556535972,v1:172.21.15.57:6836/556535972] conn(0x556e3e455910 0x556e3e455cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:45:29.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.419+0000 7f37539c5740 4 client.4445 mds_command: new command op to 4251 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T10:45:29.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.419+0000 7f37539c5740 1 -- 172.21.15.5:0/3461217414 --> [v2:172.21.15.57:6834/556535972,v1:172.21.15.57:6836/556535972] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556e3e0ee270 con 0x556e3e455910 2024-03-24T10:45:29.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:29.437+0000 7f3742ffd640 1 -- 172.21.15.5:0/3479670570 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f374407b1e0 con 0x7f374c0cf340 2024-03-24T10:45:31.162 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:45:31.165 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:45:39.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:39.426+0000 7f3743fff640 1 -- 172.21.15.5:0/3461217414 >> [v2:172.21.15.57:6834/556535972,v1:172.21.15.57:6836/556535972] conn(0x556e3e455910 msgr2=0x556e3e455cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.57:6834/556535972, fault. 2024-03-24T10:45:39.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:39.426+0000 7f3743fff640 1 --2- 172.21.15.5:0/3461217414 >> [v2:172.21.15.57:6834/556535972,v1:172.21.15.57:6836/556535972] conn(0x556e3e455910 0x556e3e455cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T10:45:49.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:49.633+0000 7f3743fff640 1 -- 172.21.15.5:0/3461217414 >> [v2:172.21.15.57:6834/556535972,v1:172.21.15.57:6836/556535972] conn(0x556e3e455910 msgr2=0x556e3e455cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.57:6834/556535972, fault. 2024-03-24T10:45:49.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:45:49.633+0000 7f3743fff640 1 --2- 172.21.15.5:0/3461217414 >> [v2:172.21.15.57:6834/556535972,v1:172.21.15.57:6836/556535972] conn(0x556e3e455910 0x556e3e455cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T10:46:00.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:46:00.042+0000 7f3743fff640 1 -- 172.21.15.5:0/3461217414 >> [v2:172.21.15.57:6834/556535972,v1:172.21.15.57:6836/556535972] conn(0x556e3e455910 msgr2=0x556e3e455cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.57:6834/556535972, fault. 2024-03-24T10:46:00.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:46:00.042+0000 7f3743fff640 1 --2- 172.21.15.5:0/3461217414 >> [v2:172.21.15.57:6834/556535972,v1:172.21.15.57:6836/556535972] conn(0x556e3e455910 0x556e3e455cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T10:46:01.236 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:46:01.239 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:46:10.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:46:10.852+0000 7f3743fff640 1 -- 172.21.15.5:0/3461217414 >> [v2:172.21.15.57:6834/556535972,v1:172.21.15.57:6836/556535972] conn(0x556e3e455910 msgr2=0x556e3e455cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.57:6834/556535972, fault. 2024-03-24T10:46:10.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:46:10.852+0000 7f3743fff640 1 --2- 172.21.15.5:0/3461217414 >> [v2:172.21.15.57:6834/556535972,v1:172.21.15.57:6836/556535972] conn(0x556e3e455910 0x556e3e455cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T10:46:22.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:46:22.454+0000 7f3743fff640 1 -- 172.21.15.5:0/3461217414 >> [v2:172.21.15.57:6834/556535972,v1:172.21.15.57:6836/556535972] conn(0x556e3e455910 msgr2=0x556e3e455cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.57:6834/556535972, fault. 2024-03-24T10:46:22.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:46:22.454+0000 7f3743fff640 1 --2- 172.21.15.5:0/3461217414 >> [v2:172.21.15.57:6834/556535972,v1:172.21.15.57:6836/556535972] conn(0x556e3e455910 0x556e3e455cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T10:46:31.300 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:46:31.303 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:46:35.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:46:35.660+0000 7f3743fff640 1 -- 172.21.15.5:0/3461217414 >> [v2:172.21.15.57:6834/556535972,v1:172.21.15.57:6836/556535972] conn(0x556e3e455910 msgr2=0x556e3e455cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.57:6834/556535972, fault. 2024-03-24T10:46:35.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:46:35.660+0000 7f3743fff640 1 --2- 172.21.15.5:0/3461217414 >> [v2:172.21.15.57:6834/556535972,v1:172.21.15.57:6836/556535972] conn(0x556e3e455910 0x556e3e455cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T10:46:52.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:46:52.077+0000 7f3743fff640 1 -- 172.21.15.5:0/3461217414 >> [v2:172.21.15.57:6834/556535972,v1:172.21.15.57:6836/556535972] conn(0x556e3e455910 msgr2=0x556e3e455cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.57:6834/556535972, fault. 2024-03-24T10:46:52.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:46:52.077+0000 7f3743fff640 1 --2- 172.21.15.5:0/3461217414 >> [v2:172.21.15.57:6834/556535972,v1:172.21.15.57:6836/556535972] conn(0x556e3e455910 0x556e3e455cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-24T10:47:01.361 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:47:01.363 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:47:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:14.881+0000 7f3743fff640 1 -- 172.21.15.5:0/3461217414 >> [v2:172.21.15.57:6834/556535972,v1:172.21.15.57:6836/556535972] conn(0x556e3e455910 msgr2=0x556e3e455cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.57:6834/556535972, fault. 2024-03-24T10:47:14.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:14.881+0000 7f3743fff640 1 --2- 172.21.15.5:0/3461217414 >> [v2:172.21.15.57:6834/556535972,v1:172.21.15.57:6836/556535972] conn(0x556e3e455910 0x556e3e455cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T10:47:29.280 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-24T10:47:30.282 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:47:30.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.387+0000 7f24219c9640 1 Processor -- start 2024-03-24T10:47:30.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.387+0000 7f24219c9640 1 -- start start 2024-03-24T10:47:30.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.387+0000 7f24219c9640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f241c151090 0x7f241c072e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.387+0000 7f24219c9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f241c073380 0x7f241c1576f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.388+0000 7f24219c9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f241c157c30 0x7f241c0c6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.388+0000 7f24219c9640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f241c156fe0 con 0x7f241c151090 2024-03-24T10:47:30.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.388+0000 7f24219c9640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f241c0c4d10 con 0x7f241c073380 2024-03-24T10:47:30.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.388+0000 7f24219c9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f241c151470 con 0x7f241c157c30 2024-03-24T10:47:30.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.388+0000 7f241affd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f241c151090 0x7f241c072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.388+0000 7f2413fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f241c073380 0x7f241c1576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.389+0000 7f241affd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f241c151090 0x7f241c072e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:58772/0 (socket says 172.21.15.5:58772) 2024-03-24T10:47:30.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.389+0000 7f241affd640 1 -- 172.21.15.5:0/1946515314 learned_addr learned my addr 172.21.15.5:0/1946515314 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:47:30.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.389+0000 7f241b7fe640 1 --2- 172.21.15.5:0/1946515314 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f241c157c30 0x7f241c0c6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.391+0000 7f2413fff640 1 -- 172.21.15.5:0/1946515314 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f241c157c30 msgr2=0x7f241c0c6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:30.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.391+0000 7f2413fff640 1 --2- 172.21.15.5:0/1946515314 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f241c157c30 0x7f241c0c6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.391+0000 7f2413fff640 1 -- 172.21.15.5:0/1946515314 >> v2:172.21.15.5:3300/0 conn(0x7f241c151090 msgr2=0x7f241c072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:30.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.391+0000 7f2413fff640 1 --2- 172.21.15.5:0/1946515314 >> v2:172.21.15.5:3300/0 conn(0x7f241c151090 0x7f241c072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.391+0000 7f2413fff640 1 -- 172.21.15.5:0/1946515314 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f241c0c4190 con 0x7f241c073380 2024-03-24T10:47:30.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.394+0000 7f2413fff640 1 --2- 172.21.15.5:0/1946515314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f241c073380 0x7f241c1576f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f241c071050 tx=0x7f240402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25e7bbeb0a263d1c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:30.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.398+0000 7f241a7fc640 1 -- 172.21.15.5:0/1946515314 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2404031830 con 0x7f241c073380 2024-03-24T10:47:30.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.398+0000 7f241a7fc640 1 -- 172.21.15.5:0/1946515314 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2404031990 con 0x7f241c073380 2024-03-24T10:47:30.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.398+0000 7f241a7fc640 1 -- 172.21.15.5:0/1946515314 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2404031d20 con 0x7f241c073380 2024-03-24T10:47:30.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.399+0000 7f24219c9640 1 -- 172.21.15.5:0/1946515314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f241c073380 msgr2=0x7f241c1576f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:30.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.399+0000 7f24219c9640 1 --2- 172.21.15.5:0/1946515314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f241c073380 0x7f241c1576f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f241c071050 tx=0x7f240402b8d0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.399+0000 7f24219c9640 1 -- 172.21.15.5:0/1946515314 shutdown_connections 2024-03-24T10:47:30.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.399+0000 7f24219c9640 1 --2- 172.21.15.5:0/1946515314 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f241c157c30 0x7f241c0c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.399+0000 7f24219c9640 1 --2- 172.21.15.5:0/1946515314 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f241c073380 0x7f241c1576f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.399+0000 7f24219c9640 1 --2- 172.21.15.5:0/1946515314 >> v2:172.21.15.5:3300/0 conn(0x7f241c151090 0x7f241c072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.399+0000 7f24219c9640 1 -- 172.21.15.5:0/1946515314 >> 172.21.15.5:0/1946515314 conn(0x7f241c06c240 msgr2=0x7f241c06c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:30.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.399+0000 7f24219c9640 1 -- 172.21.15.5:0/1946515314 shutdown_connections 2024-03-24T10:47:30.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.399+0000 7f24219c9640 1 -- 172.21.15.5:0/1946515314 wait complete. 2024-03-24T10:47:30.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.400+0000 7f24219c9640 1 Processor -- start 2024-03-24T10:47:30.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.400+0000 7f24219c9640 1 -- start start 2024-03-24T10:47:30.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.401+0000 7f24219c9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f241c073380 0x7f241c14cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.401+0000 7f24219c9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f241c151090 0x7f241c06ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.401+0000 7f24219c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f241c157c30 0x7f241c1ef190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.401+0000 7f24219c9640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f241c154590 con 0x7f241c151090 2024-03-24T10:47:30.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.401+0000 7f24219c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f241c153a10 con 0x7f241c157c30 2024-03-24T10:47:30.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.401+0000 7f24219c9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f241c153cf0 con 0x7f241c073380 2024-03-24T10:47:30.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.402+0000 7f241affd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f241c073380 0x7f241c14cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.402+0000 7f241b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f241c157c30 0x7f241c1ef190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.402+0000 7f2413fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f241c151090 0x7f241c06ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.402+0000 7f241affd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f241c073380 0x7f241c14cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:43260/0 (socket says 172.21.15.5:43260) 2024-03-24T10:47:30.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.402+0000 7f241affd640 1 -- 172.21.15.5:0/889190874 learned_addr learned my addr 172.21.15.5:0/889190874 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:47:30.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.405+0000 7f241affd640 1 -- 172.21.15.5:0/889190874 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f241c157c30 msgr2=0x7f241c1ef190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:30.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.405+0000 7f241affd640 1 --2- 172.21.15.5:0/889190874 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f241c157c30 0x7f241c1ef190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.405+0000 7f241affd640 1 -- 172.21.15.5:0/889190874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f241c151090 msgr2=0x7f241c06ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:30.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.405+0000 7f241affd640 1 --2- 172.21.15.5:0/889190874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f241c151090 0x7f241c06ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:30.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.405+0000 7f241affd640 1 -- 172.21.15.5:0/889190874 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2404009400 con 0x7f241c073380 2024-03-24T10:47:30.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.408+0000 7f241affd640 1 --2- 172.21.15.5:0/889190874 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f241c073380 0x7f241c14cdb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f240c010d80 tx=0x7f240c00e730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:30.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.412+0000 7f2418ff9640 1 -- 172.21.15.5:0/889190874 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240c011660 con 0x7f241c073380 2024-03-24T10:47:30.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.412+0000 7f2418ff9640 1 -- 172.21.15.5:0/889190874 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f240c0117c0 con 0x7f241c073380 2024-03-24T10:47:30.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.412+0000 7f2418ff9640 1 -- 172.21.15.5:0/889190874 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240c011b30 con 0x7f241c073380 2024-03-24T10:47:30.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.413+0000 7f24219c9640 1 -- 172.21.15.5:0/889190874 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f241c153fd0 con 0x7f241c073380 2024-03-24T10:47:30.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.413+0000 7f24219c9640 1 -- 172.21.15.5:0/889190874 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f241c0c7640 con 0x7f241c073380 2024-03-24T10:47:30.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.413+0000 7f24219c9640 1 -- 172.21.15.5:0/889190874 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f241c0688e0 con 0x7f241c073380 2024-03-24T10:47:30.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.439+0000 7f2418ff9640 1 -- 172.21.15.5:0/889190874 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f240c011c90 con 0x7f241c073380 2024-03-24T10:47:30.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.439+0000 7f2418ff9640 1 --2- 172.21.15.5:0/889190874 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f23fc06b860 0x7f23fc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:30.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.439+0000 7f2413fff640 1 --2- 172.21.15.5:0/889190874 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f23fc06b860 0x7f23fc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:30.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.439+0000 7f2413fff640 1 --2- 172.21.15.5:0/889190874 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f23fc06b860 0x7f23fc06dd20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2404036000 tx=0x7f240403c040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:30.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.741+0000 7f2418ff9640 1 -- 172.21.15.5:0/889190874 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f240c05bbc0 con 0x7f241c073380 2024-03-24T10:47:30.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.761+0000 7f2418ff9640 1 -- 172.21.15.5:0/889190874 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f240c057e60 con 0x7f241c073380 2024-03-24T10:47:30.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:30.986+0000 7f24219c9640 1 -- 172.21.15.5:0/889190874 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f241c153e30 con 0x7f241c073380 2024-03-24T10:47:31.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:31.003+0000 7f2418ff9640 1 -- 172.21.15.5:0/889190874 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f241c153e30 con 0x7f241c073380 2024-03-24T10:47:31.003 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:47:31.003 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:47:31.003 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:47:31.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:31.005+0000 7f24219c9640 1 -- 172.21.15.5:0/889190874 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f23fc06b860 msgr2=0x7f23fc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:31.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:31.005+0000 7f24219c9640 1 --2- 172.21.15.5:0/889190874 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f23fc06b860 0x7f23fc06dd20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2404036000 tx=0x7f240403c040 comp rx=0 tx=0).stop 2024-03-24T10:47:31.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:31.005+0000 7f24219c9640 1 -- 172.21.15.5:0/889190874 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f241c073380 msgr2=0x7f241c14cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:31.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:31.005+0000 7f24219c9640 1 --2- 172.21.15.5:0/889190874 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f241c073380 0x7f241c14cdb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f240c010d80 tx=0x7f240c00e730 comp rx=0 tx=0).stop 2024-03-24T10:47:31.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:31.005+0000 7f24219c9640 1 -- 172.21.15.5:0/889190874 shutdown_connections 2024-03-24T10:47:31.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:31.005+0000 7f24219c9640 1 --2- 172.21.15.5:0/889190874 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f23fc06b860 0x7f23fc06dd20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:31.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:31.005+0000 7f24219c9640 1 --2- 172.21.15.5:0/889190874 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f241c151090 0x7f241c06ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:31.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:31.005+0000 7f24219c9640 1 --2- 172.21.15.5:0/889190874 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f241c157c30 0x7f241c1ef190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:31.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:31.005+0000 7f24219c9640 1 --2- 172.21.15.5:0/889190874 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f241c073380 0x7f241c14cdb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:31.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:31.005+0000 7f24219c9640 1 -- 172.21.15.5:0/889190874 >> 172.21.15.5:0/889190874 conn(0x7f241c06c240 msgr2=0x7f241c154c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:31.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:31.005+0000 7f24219c9640 1 -- 172.21.15.5:0/889190874 shutdown_connections 2024-03-24T10:47:31.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:31.005+0000 7f24219c9640 1 -- 172.21.15.5:0/889190874 wait complete. 2024-03-24T10:47:31.416 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:47:31.418 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:47:31.620 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:47:31.620 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:47:32.621 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:47:32.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.724+0000 7f678135c640 1 Processor -- start 2024-03-24T10:47:32.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.724+0000 7f678135c640 1 -- start start 2024-03-24T10:47:32.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.725+0000 7f678135c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f677c1543a0 0x7f677c15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:32.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.725+0000 7f678135c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f677c06d4f0 0x7f677c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:32.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.725+0000 7f678135c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f677c0cb2f0 con 0x7f677c0ccaa0 2024-03-24T10:47:32.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.725+0000 7f678135c640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f677c0c6750 con 0x7f677c06d4f0 2024-03-24T10:47:32.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.725+0000 7f678135c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f677c0cc380 con 0x7f677c1543a0 2024-03-24T10:47:32.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.726+0000 7f6773fff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f677c06d4f0 0x7f677c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:32.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.726+0000 7f677b7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f677c0ccaa0 0x7f677c06bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:59852/0 (socket says 172.21.15.5:59852) 2024-03-24T10:47:32.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.726+0000 7f677b7fe640 1 -- 172.21.15.5:0/2641114851 learned_addr learned my addr 172.21.15.5:0/2641114851 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:47:32.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.726+0000 7f677affd640 1 --2- 172.21.15.5:0/2641114851 >> v2:172.21.15.57:3300/0 conn(0x7f677c1543a0 0x7f677c15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:32.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.729+0000 7f6773fff640 1 -- 172.21.15.5:0/2641114851 >> v2:172.21.15.57:3300/0 conn(0x7f677c1543a0 msgr2=0x7f677c15ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:32.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.729+0000 7f6773fff640 1 --2- 172.21.15.5:0/2641114851 >> v2:172.21.15.57:3300/0 conn(0x7f677c1543a0 0x7f677c15ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:32.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.729+0000 7f6773fff640 1 -- 172.21.15.5:0/2641114851 >> v1:172.21.15.57:6789/0 conn(0x7f677c0ccaa0 legacy=0x7f677c06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:32.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.729+0000 7f6773fff640 1 -- 172.21.15.5:0/2641114851 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f677c0ca7e0 con 0x7f677c06d4f0 2024-03-24T10:47:32.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.732+0000 7f6773fff640 1 --2- 172.21.15.5:0/2641114851 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f677c06d4f0 0x7f677c0cc560 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f677c0cd600 tx=0x7f676402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=310bfd8750382e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:32.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.736+0000 7f677a7fc640 1 -- 172.21.15.5:0/2641114851 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6764030610 con 0x7f677c06d4f0 2024-03-24T10:47:32.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.736+0000 7f677a7fc640 1 -- 172.21.15.5:0/2641114851 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6764030770 con 0x7f677c06d4f0 2024-03-24T10:47:32.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.736+0000 7f677a7fc640 1 -- 172.21.15.5:0/2641114851 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6764030b00 con 0x7f677c06d4f0 2024-03-24T10:47:32.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.737+0000 7f678135c640 1 -- 172.21.15.5:0/2641114851 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f677c06d4f0 msgr2=0x7f677c0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:32.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.737+0000 7f678135c640 1 --2- 172.21.15.5:0/2641114851 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f677c06d4f0 0x7f677c0cc560 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f677c0cd600 tx=0x7f676402b960 comp rx=0 tx=0).stop 2024-03-24T10:47:32.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.737+0000 7f678135c640 1 -- 172.21.15.5:0/2641114851 shutdown_connections 2024-03-24T10:47:32.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.737+0000 7f678135c640 1 --2- 172.21.15.5:0/2641114851 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f677c06d4f0 0x7f677c0cc560 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:32.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.737+0000 7f678135c640 1 --2- 172.21.15.5:0/2641114851 >> v2:172.21.15.57:3300/0 conn(0x7f677c1543a0 0x7f677c15ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:32.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.737+0000 7f678135c640 1 -- 172.21.15.5:0/2641114851 >> 172.21.15.5:0/2641114851 conn(0x7f677c072ef0 msgr2=0x7f677c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:32.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.737+0000 7f678135c640 1 -- 172.21.15.5:0/2641114851 shutdown_connections 2024-03-24T10:47:32.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.737+0000 7f678135c640 1 -- 172.21.15.5:0/2641114851 wait complete. 2024-03-24T10:47:32.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.738+0000 7f678135c640 1 Processor -- start 2024-03-24T10:47:32.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.739+0000 7f678135c640 1 -- start start 2024-03-24T10:47:32.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.739+0000 7f678135c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f677c06d4f0 0x7f677c0c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:32.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.739+0000 7f678135c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f677c0ccaa0 0x7f677c1f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:32.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.739+0000 7f678135c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f677c1543a0 0x7f677c159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:32.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.739+0000 7f678135c640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f677c1f40f0 con 0x7f677c0ccaa0 2024-03-24T10:47:32.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.739+0000 7f678135c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f677c151690 con 0x7f677c06d4f0 2024-03-24T10:47:32.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.739+0000 7f678135c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f677c150a90 con 0x7f677c1543a0 2024-03-24T10:47:32.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.740+0000 7f6773fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f677c0ccaa0 0x7f677c1f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:32.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.740+0000 7f677affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f677c06d4f0 0x7f677c0c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:32.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.740+0000 7f677b7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f677c1543a0 0x7f677c159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:32.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.740+0000 7f6773fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f677c0ccaa0 0x7f677c1f3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:53414/0 (socket says 172.21.15.5:53414) 2024-03-24T10:47:32.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.740+0000 7f6773fff640 1 -- 172.21.15.5:0/1309669419 learned_addr learned my addr 172.21.15.5:0/1309669419 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:47:32.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.745+0000 7f6773fff640 1 -- 172.21.15.5:0/1309669419 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f677c1543a0 msgr2=0x7f677c159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:32.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.745+0000 7f6773fff640 1 --2- 172.21.15.5:0/1309669419 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f677c1543a0 0x7f677c159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:32.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.745+0000 7f6773fff640 1 -- 172.21.15.5:0/1309669419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f677c06d4f0 msgr2=0x7f677c0c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:32.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.745+0000 7f6773fff640 1 --2- 172.21.15.5:0/1309669419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f677c06d4f0 0x7f677c0c9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:32.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.745+0000 7f6773fff640 1 -- 172.21.15.5:0/1309669419 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6764009400 con 0x7f677c0ccaa0 2024-03-24T10:47:32.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.748+0000 7f6773fff640 1 --2- 172.21.15.5:0/1309669419 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f677c0ccaa0 0x7f677c1f3bb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6764009890 tx=0x7f6764004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:32.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.751+0000 7f67737fe640 1 -- 172.21.15.5:0/1309669419 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676403e040 con 0x7f677c0ccaa0 2024-03-24T10:47:32.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.751+0000 7f678135c640 1 -- 172.21.15.5:0/1309669419 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f677c150d30 con 0x7f677c0ccaa0 2024-03-24T10:47:32.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.751+0000 7f678135c640 1 -- 172.21.15.5:0/1309669419 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f677c1efdd0 con 0x7f677c0ccaa0 2024-03-24T10:47:32.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.752+0000 7f678135c640 1 -- 172.21.15.5:0/1309669419 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f677c0c9ff0 con 0x7f677c0ccaa0 2024-03-24T10:47:32.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:32.963+0000 7f67737fe640 1 -- 172.21.15.5:0/1309669419 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6764039b90 con 0x7f677c0ccaa0 2024-03-24T10:47:33.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.030+0000 7f67737fe640 1 -- 172.21.15.5:0/1309669419 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676404e440 con 0x7f677c0ccaa0 2024-03-24T10:47:33.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.031+0000 7f67737fe640 1 -- 172.21.15.5:0/1309669419 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f67640443f0 con 0x7f677c0ccaa0 2024-03-24T10:47:33.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.031+0000 7f67737fe640 1 --2- 172.21.15.5:0/1309669419 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f675c06b810 0x7f675c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:33.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.031+0000 7f677affd640 1 --2- 172.21.15.5:0/1309669419 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f675c06b810 0x7f675c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:33.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.032+0000 7f677affd640 1 --2- 172.21.15.5:0/1309669419 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f675c06b810 0x7f675c06dcd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f676c009aa0 tx=0x7f676c009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:33.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.055+0000 7f67737fe640 1 -- 172.21.15.5:0/1309669419 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f676407b300 con 0x7f677c0ccaa0 2024-03-24T10:47:33.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.055+0000 7f67737fe640 1 -- 172.21.15.5:0/1309669419 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f67640775a0 con 0x7f677c0ccaa0 2024-03-24T10:47:33.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.282+0000 7f678135c640 1 -- 172.21.15.5:0/1309669419 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f677c1efc30 con 0x7f677c0ccaa0 2024-03-24T10:47:33.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.298+0000 7f67737fe640 1 -- 172.21.15.5:0/1309669419 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f677c1efc30 con 0x7f677c0ccaa0 2024-03-24T10:47:33.298 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:47:33.298 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:47:33.299 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:47:33.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.300+0000 7f678135c640 1 -- 172.21.15.5:0/1309669419 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f675c06b810 msgr2=0x7f675c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:33.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.300+0000 7f678135c640 1 --2- 172.21.15.5:0/1309669419 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f675c06b810 0x7f675c06dcd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f676c009aa0 tx=0x7f676c009190 comp rx=0 tx=0).stop 2024-03-24T10:47:33.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.300+0000 7f678135c640 1 -- 172.21.15.5:0/1309669419 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f677c0ccaa0 msgr2=0x7f677c1f3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:33.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.300+0000 7f678135c640 1 --2- 172.21.15.5:0/1309669419 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f677c0ccaa0 0x7f677c1f3bb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6764009890 tx=0x7f6764004730 comp rx=0 tx=0).stop 2024-03-24T10:47:33.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.300+0000 7f678135c640 1 -- 172.21.15.5:0/1309669419 shutdown_connections 2024-03-24T10:47:33.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.300+0000 7f678135c640 1 --2- 172.21.15.5:0/1309669419 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f675c06b810 0x7f675c06dcd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:33.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.300+0000 7f678135c640 1 --2- 172.21.15.5:0/1309669419 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f677c0ccaa0 0x7f677c1f3bb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:33.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.300+0000 7f678135c640 1 --2- 172.21.15.5:0/1309669419 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f677c1543a0 0x7f677c159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:33.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.300+0000 7f678135c640 1 --2- 172.21.15.5:0/1309669419 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f677c06d4f0 0x7f677c0c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:33.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.300+0000 7f678135c640 1 -- 172.21.15.5:0/1309669419 >> 172.21.15.5:0/1309669419 conn(0x7f677c072ef0 msgr2=0x7f677c06be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:33.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.300+0000 7f678135c640 1 -- 172.21.15.5:0/1309669419 shutdown_connections 2024-03-24T10:47:33.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:33.300+0000 7f678135c640 1 -- 172.21.15.5:0/1309669419 wait complete. 2024-03-24T10:47:33.914 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:47:33.914 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:47:34.915 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:47:35.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.018+0000 7f702fc74640 1 Processor -- start 2024-03-24T10:47:35.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.018+0000 7f702fc74640 1 -- start start 2024-03-24T10:47:35.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.019+0000 7f702fc74640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f702815ac00 0x7f70280cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:35.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.019+0000 7f702fc74640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70280ccaa0 0x7f702806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:35.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.019+0000 7f702fc74640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70280cb2f0 con 0x7f70281543a0 2024-03-24T10:47:35.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.019+0000 7f702fc74640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f70280c6750 con 0x7f702815ac00 2024-03-24T10:47:35.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.019+0000 7f702fc74640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f70280cceb0 con 0x7f70280ccaa0 2024-03-24T10:47:35.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.019+0000 7f702e211640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70280ccaa0 0x7f702806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:35.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.019+0000 7f702d20f640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f702815ac00 0x7f70280cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:35.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.019+0000 7f702da10640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f70281543a0 0x7f702806d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:59862/0 (socket says 172.21.15.5:59862) 2024-03-24T10:47:35.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.019+0000 7f702da10640 1 -- 172.21.15.5:0/4086248088 learned_addr learned my addr 172.21.15.5:0/4086248088 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:47:35.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.022+0000 7f702d20f640 1 -- 172.21.15.5:0/4086248088 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70280ccaa0 msgr2=0x7f702806bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:35.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.022+0000 7f702d20f640 1 --2- 172.21.15.5:0/4086248088 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70280ccaa0 0x7f702806bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:35.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.022+0000 7f702d20f640 1 -- 172.21.15.5:0/4086248088 >> v1:172.21.15.57:6789/0 conn(0x7f70281543a0 legacy=0x7f702806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:35.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.022+0000 7f702d20f640 1 -- 172.21.15.5:0/4086248088 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f702815b070 con 0x7f702815ac00 2024-03-24T10:47:35.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.024+0000 7f702d20f640 1 --2- 172.21.15.5:0/4086248088 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f702815ac00 0x7f70280cc560 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f70180097a0 tx=0x7f701802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1e4eab6304a51fc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:35.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.028+0000 7f702ca0e640 1 -- 172.21.15.5:0/4086248088 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7018031830 con 0x7f702815ac00 2024-03-24T10:47:35.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.028+0000 7f702ca0e640 1 -- 172.21.15.5:0/4086248088 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7018031990 con 0x7f702815ac00 2024-03-24T10:47:35.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.028+0000 7f702ca0e640 1 -- 172.21.15.5:0/4086248088 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7018031d20 con 0x7f702815ac00 2024-03-24T10:47:35.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.029+0000 7f702fc74640 1 -- 172.21.15.5:0/4086248088 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f702815ac00 msgr2=0x7f70280cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:35.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.029+0000 7f702fc74640 1 --2- 172.21.15.5:0/4086248088 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f702815ac00 0x7f70280cc560 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f70180097a0 tx=0x7f701802b8d0 comp rx=0 tx=0).stop 2024-03-24T10:47:35.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.029+0000 7f702fc74640 1 -- 172.21.15.5:0/4086248088 shutdown_connections 2024-03-24T10:47:35.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.029+0000 7f702fc74640 1 --2- 172.21.15.5:0/4086248088 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70280ccaa0 0x7f702806bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:35.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.029+0000 7f702fc74640 1 --2- 172.21.15.5:0/4086248088 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f702815ac00 0x7f70280cc560 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:35.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.029+0000 7f702fc74640 1 -- 172.21.15.5:0/4086248088 >> 172.21.15.5:0/4086248088 conn(0x7f7028072ef0 msgr2=0x7f7028061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:35.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.029+0000 7f702fc74640 1 -- 172.21.15.5:0/4086248088 shutdown_connections 2024-03-24T10:47:35.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.029+0000 7f702fc74640 1 -- 172.21.15.5:0/4086248088 wait complete. 2024-03-24T10:47:35.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.031+0000 7f702fc74640 1 Processor -- start 2024-03-24T10:47:35.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.031+0000 7f702fc74640 1 -- start start 2024-03-24T10:47:35.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.031+0000 7f702fc74640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70280ccaa0 0x7f70280b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:35.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.031+0000 7f702fc74640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70281543a0 0x7f70281f32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:35.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.031+0000 7f702fc74640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f702815ac00 0x7f7028146650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:35.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.031+0000 7f702fc74640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f7028158770 con 0x7f70281543a0 2024-03-24T10:47:35.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.031+0000 7f702fc74640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f70280c77c0 con 0x7f702815ac00 2024-03-24T10:47:35.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.031+0000 7f702fc74640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7028157f60 con 0x7f70280ccaa0 2024-03-24T10:47:35.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.032+0000 7f702d20f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70281543a0 0x7f70281f32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:35.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.032+0000 7f702da10640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70280ccaa0 0x7f70280b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:35.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.032+0000 7f702e211640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f702815ac00 0x7f7028146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:35.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.032+0000 7f702da10640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70280ccaa0 0x7f70280b39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:58074/0 (socket says 172.21.15.5:58074) 2024-03-24T10:47:35.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.032+0000 7f702da10640 1 -- 172.21.15.5:0/2647868780 learned_addr learned my addr 172.21.15.5:0/2647868780 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:47:35.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.036+0000 7f702d20f640 1 -- 172.21.15.5:0/2647868780 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70280ccaa0 msgr2=0x7f70280b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:35.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.036+0000 7f702d20f640 1 --2- 172.21.15.5:0/2647868780 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70280ccaa0 0x7f70280b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:35.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.036+0000 7f702d20f640 1 -- 172.21.15.5:0/2647868780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f702815ac00 msgr2=0x7f7028146650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:35.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.036+0000 7f702d20f640 1 --2- 172.21.15.5:0/2647868780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f702815ac00 0x7f7028146650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:35.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.036+0000 7f702d20f640 1 -- 172.21.15.5:0/2647868780 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7018009400 con 0x7f70281543a0 2024-03-24T10:47:35.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.038+0000 7f702d20f640 1 --2- 172.21.15.5:0/2647868780 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70281543a0 0x7f70281f32a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7018033000 tx=0x7f70180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:35.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.043+0000 7f701e7fc640 1 -- 172.21.15.5:0/2647868780 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701803a9e0 con 0x7f70281543a0 2024-03-24T10:47:35.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.043+0000 7f701e7fc640 1 -- 172.21.15.5:0/2647868780 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f701803ab60 con 0x7f70281543a0 2024-03-24T10:47:35.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.043+0000 7f702fc74640 1 -- 172.21.15.5:0/2647868780 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70281f3c80 con 0x7f70281543a0 2024-03-24T10:47:35.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.043+0000 7f701e7fc640 1 -- 172.21.15.5:0/2647868780 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7018044400 con 0x7f70281543a0 2024-03-24T10:47:35.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.043+0000 7f702fc74640 1 -- 172.21.15.5:0/2647868780 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7028158490 con 0x7f70281543a0 2024-03-24T10:47:35.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.044+0000 7f702fc74640 1 -- 172.21.15.5:0/2647868780 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70280681d0 con 0x7f70281543a0 2024-03-24T10:47:35.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.064+0000 7f701e7fc640 1 -- 172.21.15.5:0/2647868780 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f701804d440 con 0x7f70281543a0 2024-03-24T10:47:35.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.065+0000 7f701e7fc640 1 --2- 172.21.15.5:0/2647868780 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f700c06b810 0x7f700c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:35.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.065+0000 7f702da10640 1 --2- 172.21.15.5:0/2647868780 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f700c06b810 0x7f700c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:35.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.065+0000 7f702da10640 1 --2- 172.21.15.5:0/2647868780 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f700c06b810 0x7f700c06dcd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7024004e40 tx=0x7f7024007f10 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:35.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.328+0000 7f701e7fc640 1 -- 172.21.15.5:0/2647868780 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f701807af20 con 0x7f70281543a0 2024-03-24T10:47:35.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.328+0000 7f701e7fc640 1 -- 172.21.15.5:0/2647868780 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f70180771c0 con 0x7f70281543a0 2024-03-24T10:47:35.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.551+0000 7f702fc74640 1 -- 172.21.15.5:0/2647868780 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7028150310 con 0x7f70281543a0 2024-03-24T10:47:35.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.606+0000 7f701e7fc640 1 -- 172.21.15.5:0/2647868780 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f7018035090 con 0x7f70281543a0 2024-03-24T10:47:35.607 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:47:35.607 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:47:35.607 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:47:35.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.609+0000 7f702fc74640 1 -- 172.21.15.5:0/2647868780 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f700c06b810 msgr2=0x7f700c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:35.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.609+0000 7f702fc74640 1 --2- 172.21.15.5:0/2647868780 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f700c06b810 0x7f700c06dcd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7024004e40 tx=0x7f7024007f10 comp rx=0 tx=0).stop 2024-03-24T10:47:35.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.609+0000 7f702fc74640 1 -- 172.21.15.5:0/2647868780 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70281543a0 msgr2=0x7f70281f32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:35.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.609+0000 7f702fc74640 1 --2- 172.21.15.5:0/2647868780 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70281543a0 0x7f70281f32a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7018033000 tx=0x7f70180046c0 comp rx=0 tx=0).stop 2024-03-24T10:47:35.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.609+0000 7f702fc74640 1 -- 172.21.15.5:0/2647868780 shutdown_connections 2024-03-24T10:47:35.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.609+0000 7f702fc74640 1 --2- 172.21.15.5:0/2647868780 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f700c06b810 0x7f700c06dcd0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:35.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.609+0000 7f702fc74640 1 --2- 172.21.15.5:0/2647868780 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70281543a0 0x7f70281f32a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:35.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.609+0000 7f702fc74640 1 --2- 172.21.15.5:0/2647868780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f702815ac00 0x7f7028146650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:35.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.609+0000 7f702fc74640 1 --2- 172.21.15.5:0/2647868780 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70280ccaa0 0x7f70280b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:35.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.609+0000 7f702fc74640 1 -- 172.21.15.5:0/2647868780 >> 172.21.15.5:0/2647868780 conn(0x7f7028072ef0 msgr2=0x7f7028151200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:35.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.609+0000 7f702fc74640 1 -- 172.21.15.5:0/2647868780 shutdown_connections 2024-03-24T10:47:35.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:35.609+0000 7f702fc74640 1 -- 172.21.15.5:0/2647868780 wait complete. 2024-03-24T10:47:36.224 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:47:36.224 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:47:37.225 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:47:37.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.328+0000 7f3adf59e640 1 Processor -- start 2024-03-24T10:47:37.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.328+0000 7f3adf59e640 1 -- start start 2024-03-24T10:47:37.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.330+0000 7f3adf59e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3ae0154420 0x7f3ae015abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:37.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.330+0000 7f3adf59e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3ae00ca970 0x7f3ae00c4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:37.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.330+0000 7f3adf59e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ae015b0f0 con 0x7f3ae00cbee0 2024-03-24T10:47:37.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.330+0000 7f3adf59e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f3ae015e400 con 0x7f3ae0154420 2024-03-24T10:47:37.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.330+0000 7f3adf59e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f3ae00b35e0 con 0x7f3ae00ca970 2024-03-24T10:47:37.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.331+0000 7f3aded9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3ae00ca970 0x7f3ae00c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:37.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.331+0000 7f3ad7fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f3ae00cbee0 0x7f3ae00cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:59878/0 (socket says 172.21.15.5:59878) 2024-03-24T10:47:37.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.331+0000 7f3ad7fff640 1 -- 172.21.15.5:0/4009023696 learned_addr learned my addr 172.21.15.5:0/4009023696 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:47:37.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.331+0000 7f3ade59c640 1 --2- 172.21.15.5:0/4009023696 >> v2:172.21.15.57:3300/0 conn(0x7f3ae0154420 0x7f3ae015abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:37.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.334+0000 7f3aded9d640 1 -- 172.21.15.5:0/4009023696 >> v2:172.21.15.57:3300/0 conn(0x7f3ae0154420 msgr2=0x7f3ae015abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:37.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.334+0000 7f3aded9d640 1 --2- 172.21.15.5:0/4009023696 >> v2:172.21.15.57:3300/0 conn(0x7f3ae0154420 0x7f3ae015abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:37.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.334+0000 7f3aded9d640 1 -- 172.21.15.5:0/4009023696 >> v1:172.21.15.57:6789/0 conn(0x7f3ae00cbee0 legacy=0x7f3ae00cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:37.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.334+0000 7f3aded9d640 1 -- 172.21.15.5:0/4009023696 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ae015d7c0 con 0x7f3ae00ca970 2024-03-24T10:47:37.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.337+0000 7f3aded9d640 1 --2- 172.21.15.5:0/4009023696 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3ae00ca970 0x7f3ae00c4910 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009d90 tx=0x7f3ad802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4931afe915626607 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:37.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.342+0000 7f3addd9b640 1 -- 172.21.15.5:0/4009023696 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad802cce0 con 0x7f3ae00ca970 2024-03-24T10:47:37.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.342+0000 7f3addd9b640 1 -- 172.21.15.5:0/4009023696 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ad802ce40 con 0x7f3ae00ca970 2024-03-24T10:47:37.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.342+0000 7f3addd9b640 1 -- 172.21.15.5:0/4009023696 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad8034660 con 0x7f3ae00ca970 2024-03-24T10:47:37.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.343+0000 7f3adf59e640 1 -- 172.21.15.5:0/4009023696 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3ae00ca970 msgr2=0x7f3ae00c4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:37.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.343+0000 7f3adf59e640 1 --2- 172.21.15.5:0/4009023696 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3ae00ca970 0x7f3ae00c4910 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009d90 tx=0x7f3ad802bf30 comp rx=0 tx=0).stop 2024-03-24T10:47:37.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.343+0000 7f3adf59e640 1 -- 172.21.15.5:0/4009023696 shutdown_connections 2024-03-24T10:47:37.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.343+0000 7f3adf59e640 1 --2- 172.21.15.5:0/4009023696 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3ae00ca970 0x7f3ae00c4910 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:37.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.343+0000 7f3adf59e640 1 --2- 172.21.15.5:0/4009023696 >> v2:172.21.15.57:3300/0 conn(0x7f3ae0154420 0x7f3ae015abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:37.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.343+0000 7f3adf59e640 1 -- 172.21.15.5:0/4009023696 >> 172.21.15.5:0/4009023696 conn(0x7f3ae006c400 msgr2=0x7f3ae0072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:37.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.343+0000 7f3adf59e640 1 -- 172.21.15.5:0/4009023696 shutdown_connections 2024-03-24T10:47:37.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.343+0000 7f3adf59e640 1 -- 172.21.15.5:0/4009023696 wait complete. 2024-03-24T10:47:37.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.344+0000 7f3adf59e640 1 Processor -- start 2024-03-24T10:47:37.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.344+0000 7f3adf59e640 1 -- start start 2024-03-24T10:47:37.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.345+0000 7f3adf59e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3ae00ca970 0x7f3ae01f0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:37.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.345+0000 7f3adf59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ae00cbee0 0x7f3ae01f3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:37.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.345+0000 7f3adf59e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3ae0154420 0x7f3ae00c9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:37.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.345+0000 7f3adf59e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae0152a90 con 0x7f3ae00ca970 2024-03-24T10:47:37.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.345+0000 7f3adf59e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae01f0f50 con 0x7f3ae00cbee0 2024-03-24T10:47:37.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.345+0000 7f3adf59e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f3ae0151f80 con 0x7f3ae0154420 2024-03-24T10:47:37.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.346+0000 7f3ad7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ae00cbee0 0x7f3ae01f3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:37.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.346+0000 7f3aded9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3ae0154420 0x7f3ae00c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:37.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.346+0000 7f3ade59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3ae00ca970 0x7f3ae01f0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:37.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.346+0000 7f3aded9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3ae0154420 0x7f3ae00c9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:58086/0 (socket says 172.21.15.5:58086) 2024-03-24T10:47:37.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.346+0000 7f3aded9d640 1 -- 172.21.15.5:0/888881684 learned_addr learned my addr 172.21.15.5:0/888881684 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:47:37.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.351+0000 7f3ad7fff640 1 -- 172.21.15.5:0/888881684 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3ae0154420 msgr2=0x7f3ae00c9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:37.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.351+0000 7f3ad7fff640 1 --2- 172.21.15.5:0/888881684 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3ae0154420 0x7f3ae00c9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:37.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.351+0000 7f3ad7fff640 1 -- 172.21.15.5:0/888881684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3ae00ca970 msgr2=0x7f3ae01f0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:37.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.351+0000 7f3ad7fff640 1 --2- 172.21.15.5:0/888881684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3ae00ca970 0x7f3ae01f0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:37.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.351+0000 7f3ad7fff640 1 -- 172.21.15.5:0/888881684 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ad8009a70 con 0x7f3ae00cbee0 2024-03-24T10:47:37.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.355+0000 7f3ad7fff640 1 --2- 172.21.15.5:0/888881684 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ae00cbee0 0x7f3ae01f3ea0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800c370 tx=0x7f3ac800c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:37.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.358+0000 7f3ad6ffd640 1 -- 172.21.15.5:0/888881684 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac8005ce0 con 0x7f3ae00cbee0 2024-03-24T10:47:37.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.358+0000 7f3adf59e640 1 -- 172.21.15.5:0/888881684 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ae01f43e0 con 0x7f3ae00cbee0 2024-03-24T10:47:37.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.358+0000 7f3adf59e640 1 -- 172.21.15.5:0/888881684 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ae01524d0 con 0x7f3ae00cbee0 2024-03-24T10:47:37.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.359+0000 7f3adf59e640 1 -- 172.21.15.5:0/888881684 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ae004f470 con 0x7f3ae00cbee0 2024-03-24T10:47:37.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.579+0000 7f3ad6ffd640 1 -- 172.21.15.5:0/888881684 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ac8005e40 con 0x7f3ae00cbee0 2024-03-24T10:47:37.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.636+0000 7f3ad6ffd640 1 -- 172.21.15.5:0/888881684 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac80072d0 con 0x7f3ae00cbee0 2024-03-24T10:47:37.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.637+0000 7f3ad6ffd640 1 -- 172.21.15.5:0/888881684 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3ac80074f0 con 0x7f3ae00cbee0 2024-03-24T10:47:37.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.637+0000 7f3ad6ffd640 1 --2- 172.21.15.5:0/888881684 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3ac006b840 0x7f3ac006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:37.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.637+0000 7f3ade59c640 1 --2- 172.21.15.5:0/888881684 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3ac006b840 0x7f3ac006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:37.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.638+0000 7f3ade59c640 1 --2- 172.21.15.5:0/888881684 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3ac006b840 0x7f3ac006dd00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0009aa0 tx=0x7f3ad0009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:37.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.640+0000 7f3ad6ffd640 1 -- 172.21.15.5:0/888881684 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f3ac80586f0 con 0x7f3ae00cbee0 2024-03-24T10:47:37.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.652+0000 7f3ad6ffd640 1 -- 172.21.15.5:0/888881684 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3ac8054990 con 0x7f3ae00cbee0 2024-03-24T10:47:37.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.889+0000 7f3adf59e640 1 -- 172.21.15.5:0/888881684 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ae00cb4b0 con 0x7f3ae00cbee0 2024-03-24T10:47:37.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.903+0000 7f3ad6ffd640 1 -- 172.21.15.5:0/888881684 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f3ac8021090 con 0x7f3ae00cbee0 2024-03-24T10:47:37.903 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:47:37.903 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:47:37.903 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:47:37.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.904+0000 7f3adf59e640 1 -- 172.21.15.5:0/888881684 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3ac006b840 msgr2=0x7f3ac006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:37.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.904+0000 7f3adf59e640 1 --2- 172.21.15.5:0/888881684 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3ac006b840 0x7f3ac006dd00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0009aa0 tx=0x7f3ad0009190 comp rx=0 tx=0).stop 2024-03-24T10:47:37.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.904+0000 7f3adf59e640 1 -- 172.21.15.5:0/888881684 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ae00cbee0 msgr2=0x7f3ae01f3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:37.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.905+0000 7f3adf59e640 1 --2- 172.21.15.5:0/888881684 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ae00cbee0 0x7f3ae01f3ea0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800c370 tx=0x7f3ac800c830 comp rx=0 tx=0).stop 2024-03-24T10:47:37.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.905+0000 7f3adf59e640 1 -- 172.21.15.5:0/888881684 shutdown_connections 2024-03-24T10:47:37.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.905+0000 7f3adf59e640 1 --2- 172.21.15.5:0/888881684 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3ac006b840 0x7f3ac006dd00 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:37.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.905+0000 7f3adf59e640 1 --2- 172.21.15.5:0/888881684 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3ae0154420 0x7f3ae00c9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:37.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.905+0000 7f3adf59e640 1 --2- 172.21.15.5:0/888881684 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3ae00cbee0 0x7f3ae01f3ea0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:37.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.905+0000 7f3adf59e640 1 --2- 172.21.15.5:0/888881684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3ae00ca970 0x7f3ae01f0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:37.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.905+0000 7f3adf59e640 1 -- 172.21.15.5:0/888881684 >> 172.21.15.5:0/888881684 conn(0x7f3ae006c400 msgr2=0x7f3ae00cf900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:37.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.905+0000 7f3adf59e640 1 -- 172.21.15.5:0/888881684 shutdown_connections 2024-03-24T10:47:37.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:37.905+0000 7f3adf59e640 1 -- 172.21.15.5:0/888881684 wait complete. 2024-03-24T10:47:38.521 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:47:38.522 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:47:39.522 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:47:39.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.624+0000 7ff34862a640 1 Processor -- start 2024-03-24T10:47:39.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.624+0000 7ff34862a640 1 -- start start 2024-03-24T10:47:39.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.625+0000 7ff34862a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff34015ac00 0x7ff3400cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:39.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.625+0000 7ff34862a640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff3400ccaa0 0x7ff34006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:39.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.625+0000 7ff34862a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3400cb2f0 con 0x7ff3401543a0 2024-03-24T10:47:39.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.625+0000 7ff34862a640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7ff3400c6750 con 0x7ff3400ccaa0 2024-03-24T10:47:39.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.625+0000 7ff34862a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff3400cc380 con 0x7ff34015ac00 2024-03-24T10:47:39.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.626+0000 7ff345bc5640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff34015ac00 0x7ff3400cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:39.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.626+0000 7ff345bc5640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff34015ac00 0x7ff3400cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:58092/0 (socket says 172.21.15.5:58092) 2024-03-24T10:47:39.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.626+0000 7ff345bc5640 1 -- 172.21.15.5:0/2292268225 learned_addr learned my addr 172.21.15.5:0/2292268225 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:47:39.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.626+0000 7ff346bc7640 1 --2- 172.21.15.5:0/2292268225 >> v2:172.21.15.57:3300/0 conn(0x7ff3400ccaa0 0x7ff34006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:39.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.629+0000 7ff345bc5640 1 -- 172.21.15.5:0/2292268225 >> v2:172.21.15.57:3300/0 conn(0x7ff3400ccaa0 msgr2=0x7ff34006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:39.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.629+0000 7ff345bc5640 1 --2- 172.21.15.5:0/2292268225 >> v2:172.21.15.57:3300/0 conn(0x7ff3400ccaa0 0x7ff34006bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:39.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.629+0000 7ff345bc5640 1 -- 172.21.15.5:0/2292268225 >> v1:172.21.15.57:6789/0 conn(0x7ff3401543a0 legacy=0x7ff34006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:39.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.629+0000 7ff345bc5640 1 -- 172.21.15.5:0/2292268225 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff34015b070 con 0x7ff34015ac00 2024-03-24T10:47:39.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.632+0000 7ff345bc5640 1 --2- 172.21.15.5:0/2292268225 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff34015ac00 0x7ff3400cc560 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff330009870 tx=0x7ff33002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6929bf74c338482 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:39.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.636+0000 7ff3453c4640 1 -- 172.21.15.5:0/2292268225 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff330030840 con 0x7ff34015ac00 2024-03-24T10:47:39.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.636+0000 7ff3453c4640 1 -- 172.21.15.5:0/2292268225 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3300309a0 con 0x7ff34015ac00 2024-03-24T10:47:39.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.636+0000 7ff3453c4640 1 -- 172.21.15.5:0/2292268225 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff330030d30 con 0x7ff34015ac00 2024-03-24T10:47:39.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.637+0000 7ff34862a640 1 -- 172.21.15.5:0/2292268225 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff34015ac00 msgr2=0x7ff3400cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:39.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.637+0000 7ff34862a640 1 --2- 172.21.15.5:0/2292268225 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff34015ac00 0x7ff3400cc560 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff330009870 tx=0x7ff33002b980 comp rx=0 tx=0).stop 2024-03-24T10:47:39.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.637+0000 7ff34862a640 1 -- 172.21.15.5:0/2292268225 shutdown_connections 2024-03-24T10:47:39.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.637+0000 7ff34862a640 1 --2- 172.21.15.5:0/2292268225 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff34015ac00 0x7ff3400cc560 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:39.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.637+0000 7ff34862a640 1 --2- 172.21.15.5:0/2292268225 >> v2:172.21.15.57:3300/0 conn(0x7ff3400ccaa0 0x7ff34006bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:39.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.637+0000 7ff34862a640 1 -- 172.21.15.5:0/2292268225 >> 172.21.15.5:0/2292268225 conn(0x7ff340072ef0 msgr2=0x7ff340061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:39.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.637+0000 7ff34862a640 1 -- 172.21.15.5:0/2292268225 shutdown_connections 2024-03-24T10:47:39.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.637+0000 7ff34862a640 1 -- 172.21.15.5:0/2292268225 wait complete. 2024-03-24T10:47:39.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.638+0000 7ff34862a640 1 Processor -- start 2024-03-24T10:47:39.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.638+0000 7ff34862a640 1 -- start start 2024-03-24T10:47:39.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.639+0000 7ff34862a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff3400ccaa0 0x7ff34014a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:39.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.639+0000 7ff34862a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3401543a0 0x7ff34014e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:39.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.639+0000 7ff34862a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff34015ac00 0x7ff34006ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:39.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.639+0000 7ff34862a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff3401f01e0 con 0x7ff3401543a0 2024-03-24T10:47:39.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.639+0000 7ff34862a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff3400c7210 con 0x7ff34015ac00 2024-03-24T10:47:39.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.639+0000 7ff34862a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff340150290 con 0x7ff3400ccaa0 2024-03-24T10:47:39.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.639+0000 7ff3463c6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff3400ccaa0 0x7ff34014a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:39.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.640+0000 7ff346bc7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff34015ac00 0x7ff34006ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:39.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.640+0000 7ff345bc5640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3401543a0 0x7ff34014e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:39.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.640+0000 7ff3463c6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff3400ccaa0 0x7ff34014a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:58104/0 (socket says 172.21.15.5:58104) 2024-03-24T10:47:39.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.640+0000 7ff3463c6640 1 -- 172.21.15.5:0/2474993583 learned_addr learned my addr 172.21.15.5:0/2474993583 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:47:39.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.643+0000 7ff3463c6640 1 -- 172.21.15.5:0/2474993583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff34015ac00 msgr2=0x7ff34006ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:39.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.643+0000 7ff3463c6640 1 --2- 172.21.15.5:0/2474993583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff34015ac00 0x7ff34006ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:39.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.643+0000 7ff3463c6640 1 -- 172.21.15.5:0/2474993583 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3401543a0 msgr2=0x7ff34014e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:39.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.643+0000 7ff3463c6640 1 --2- 172.21.15.5:0/2474993583 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3401543a0 0x7ff34014e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:39.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.644+0000 7ff3463c6640 1 -- 172.21.15.5:0/2474993583 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3300094d0 con 0x7ff3400ccaa0 2024-03-24T10:47:39.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.646+0000 7ff3463c6640 1 --2- 172.21.15.5:0/2474993583 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff3400ccaa0 0x7ff34014a770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00c430 tx=0x7ff33c00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:39.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.652+0000 7ff336ffd640 1 -- 172.21.15.5:0/2474993583 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33c0052d0 con 0x7ff3400ccaa0 2024-03-24T10:47:39.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.652+0000 7ff336ffd640 1 -- 172.21.15.5:0/2474993583 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff33c005450 con 0x7ff3400ccaa0 2024-03-24T10:47:39.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.652+0000 7ff34862a640 1 -- 172.21.15.5:0/2474993583 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff340150570 con 0x7ff3400ccaa0 2024-03-24T10:47:39.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.652+0000 7ff336ffd640 1 -- 172.21.15.5:0/2474993583 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33c007260 con 0x7ff3400ccaa0 2024-03-24T10:47:39.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.652+0000 7ff34862a640 1 -- 172.21.15.5:0/2474993583 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3401efe00 con 0x7ff3400ccaa0 2024-03-24T10:47:39.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.653+0000 7ff34862a640 1 -- 172.21.15.5:0/2474993583 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3400b2860 con 0x7ff3400ccaa0 2024-03-24T10:47:39.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.925+0000 7ff336ffd640 1 -- 172.21.15.5:0/2474993583 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff33c0073c0 con 0x7ff3400ccaa0 2024-03-24T10:47:39.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.926+0000 7ff336ffd640 1 --2- 172.21.15.5:0/2474993583 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff32406b840 0x7ff32406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:39.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.926+0000 7ff345bc5640 1 --2- 172.21.15.5:0/2474993583 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff32406b840 0x7ff32406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:39.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.926+0000 7ff345bc5640 1 --2- 172.21.15.5:0/2474993583 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff32406b840 0x7ff32406dd00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff330036000 tx=0x7ff330008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:39.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.927+0000 7ff336ffd640 1 -- 172.21.15.5:0/2474993583 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7ff33c058310 con 0x7ff3400ccaa0 2024-03-24T10:47:39.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:39.947+0000 7ff336ffd640 1 -- 172.21.15.5:0/2474993583 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff33c0545b0 con 0x7ff3400ccaa0 2024-03-24T10:47:40.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:40.173+0000 7ff34862a640 1 -- 172.21.15.5:0/2474993583 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3400681d0 con 0x7ff3400ccaa0 2024-03-24T10:47:40.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:40.196+0000 7ff336ffd640 1 -- 172.21.15.5:0/2474993583 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7ff33c020020 con 0x7ff3400ccaa0 2024-03-24T10:47:40.196 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:47:40.197 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:47:40.197 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:47:40.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:40.198+0000 7ff34862a640 1 -- 172.21.15.5:0/2474993583 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff32406b840 msgr2=0x7ff32406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:40.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:40.198+0000 7ff34862a640 1 --2- 172.21.15.5:0/2474993583 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff32406b840 0x7ff32406dd00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff330036000 tx=0x7ff330008030 comp rx=0 tx=0).stop 2024-03-24T10:47:40.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:40.198+0000 7ff34862a640 1 -- 172.21.15.5:0/2474993583 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff3400ccaa0 msgr2=0x7ff34014a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:40.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:40.198+0000 7ff34862a640 1 --2- 172.21.15.5:0/2474993583 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff3400ccaa0 0x7ff34014a770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff33c00c430 tx=0x7ff33c00c8f0 comp rx=0 tx=0).stop 2024-03-24T10:47:40.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:40.199+0000 7ff34862a640 1 -- 172.21.15.5:0/2474993583 shutdown_connections 2024-03-24T10:47:40.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:40.199+0000 7ff34862a640 1 --2- 172.21.15.5:0/2474993583 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff32406b840 0x7ff32406dd00 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:40.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:40.199+0000 7ff34862a640 1 --2- 172.21.15.5:0/2474993583 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff3401543a0 0x7ff34014e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:40.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:40.199+0000 7ff34862a640 1 --2- 172.21.15.5:0/2474993583 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff34015ac00 0x7ff34006ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:40.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:40.199+0000 7ff34862a640 1 --2- 172.21.15.5:0/2474993583 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff3400ccaa0 0x7ff34014a770 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:40.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:40.199+0000 7ff34862a640 1 -- 172.21.15.5:0/2474993583 >> 172.21.15.5:0/2474993583 conn(0x7ff340072ef0 msgr2=0x7ff340158b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:40.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:40.199+0000 7ff34862a640 1 -- 172.21.15.5:0/2474993583 shutdown_connections 2024-03-24T10:47:40.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:40.199+0000 7ff34862a640 1 -- 172.21.15.5:0/2474993583 wait complete. 2024-03-24T10:47:40.814 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:47:40.814 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:47:41.815 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:47:41.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.920+0000 7f8219756640 1 Processor -- start 2024-03-24T10:47:41.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.920+0000 7f8219756640 1 -- start start 2024-03-24T10:47:41.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.921+0000 7f8219756640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f82141543a0 0x7f821415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:41.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.921+0000 7f8219756640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f821406bf50 0x7f821406c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:41.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.921+0000 7f8219756640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82140cb2f0 con 0x7f821406d4f0 2024-03-24T10:47:41.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.921+0000 7f8219756640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f82140c6750 con 0x7f821406bf50 2024-03-24T10:47:41.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.921+0000 7f8219756640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f82140cc380 con 0x7f82141543a0 2024-03-24T10:47:41.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.921+0000 7f82137fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f821406bf50 0x7f821406c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:41.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.921+0000 7f8212ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f82141543a0 0x7f821415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:41.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.922+0000 7f8212ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f82141543a0 0x7f821415ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:39996/0 (socket says 172.21.15.5:39996) 2024-03-24T10:47:41.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.922+0000 7f8212ffd640 1 -- 172.21.15.5:0/4006865062 learned_addr learned my addr 172.21.15.5:0/4006865062 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:47:41.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.924+0000 7f8212ffd640 1 -- 172.21.15.5:0/4006865062 >> v2:172.21.15.5:3300/0 conn(0x7f821406bf50 msgr2=0x7f821406c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:41.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.924+0000 7f8212ffd640 1 --2- 172.21.15.5:0/4006865062 >> v2:172.21.15.5:3300/0 conn(0x7f821406bf50 0x7f821406c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:41.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.924+0000 7f8212ffd640 1 -- 172.21.15.5:0/4006865062 >> v1:172.21.15.5:6789/0 conn(0x7f821406d4f0 legacy=0x7f82140cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:41.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.924+0000 7f8212ffd640 1 -- 172.21.15.5:0/4006865062 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82140ca7e0 con 0x7f82141543a0 2024-03-24T10:47:41.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.926+0000 7f8212ffd640 1 --2- 172.21.15.5:0/4006865062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82141543a0 0x7f821415ac00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f82140cd600 tx=0x7f820402ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6960f81694fcee57 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:41.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.930+0000 7f8211ffb640 1 -- 172.21.15.5:0/4006865062 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820402f850 con 0x7f82141543a0 2024-03-24T10:47:41.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.930+0000 7f8211ffb640 1 -- 172.21.15.5:0/4006865062 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f820402f9b0 con 0x7f82141543a0 2024-03-24T10:47:41.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.930+0000 7f8211ffb640 1 -- 172.21.15.5:0/4006865062 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820402fd40 con 0x7f82141543a0 2024-03-24T10:47:41.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.931+0000 7f8219756640 1 -- 172.21.15.5:0/4006865062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82141543a0 msgr2=0x7f821415ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:41.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.931+0000 7f8219756640 1 --2- 172.21.15.5:0/4006865062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82141543a0 0x7f821415ac00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f82140cd600 tx=0x7f820402ba30 comp rx=0 tx=0).stop 2024-03-24T10:47:41.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.931+0000 7f8219756640 1 -- 172.21.15.5:0/4006865062 shutdown_connections 2024-03-24T10:47:41.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.931+0000 7f8219756640 1 --2- 172.21.15.5:0/4006865062 >> v2:172.21.15.5:3300/0 conn(0x7f821406bf50 0x7f821406c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:41.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.931+0000 7f8219756640 1 --2- 172.21.15.5:0/4006865062 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82141543a0 0x7f821415ac00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:41.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.931+0000 7f8219756640 1 -- 172.21.15.5:0/4006865062 >> 172.21.15.5:0/4006865062 conn(0x7f8214072ef0 msgr2=0x7f8214061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:41.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.931+0000 7f8219756640 1 -- 172.21.15.5:0/4006865062 shutdown_connections 2024-03-24T10:47:41.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.931+0000 7f8219756640 1 -- 172.21.15.5:0/4006865062 wait complete. 2024-03-24T10:47:41.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.932+0000 7f8219756640 1 Processor -- start 2024-03-24T10:47:41.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.933+0000 7f8219756640 1 -- start start 2024-03-24T10:47:41.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.933+0000 7f8219756640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f821406bf50 0x7f8214156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:41.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.933+0000 7f8219756640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f821406d4f0 0x7f821414ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:41.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.933+0000 7f8219756640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82141543a0 0x7f82141f3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:41.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.933+0000 7f8219756640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f8214154d40 con 0x7f821406d4f0 2024-03-24T10:47:41.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.933+0000 7f8219756640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8214156560 con 0x7f82141543a0 2024-03-24T10:47:41.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.933+0000 7f8219756640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f821414f390 con 0x7f821406bf50 2024-03-24T10:47:41.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.934+0000 7f82127fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f821406d4f0 0x7f821414ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:41.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.934+0000 7f82137fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82141543a0 0x7f82141f3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:41.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.934+0000 7f8212ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f821406bf50 0x7f8214156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:41.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.934+0000 7f82127fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f821406d4f0 0x7f821414ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:53476/0 (socket says 172.21.15.5:53476) 2024-03-24T10:47:41.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.934+0000 7f82127fc640 1 -- 172.21.15.5:0/160731930 learned_addr learned my addr 172.21.15.5:0/160731930 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:47:41.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.938+0000 7f82127fc640 1 -- 172.21.15.5:0/160731930 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f821406bf50 msgr2=0x7f8214156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:41.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.938+0000 7f82127fc640 1 --2- 172.21.15.5:0/160731930 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f821406bf50 0x7f8214156ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:41.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.938+0000 7f82127fc640 1 -- 172.21.15.5:0/160731930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82141543a0 msgr2=0x7f82141f3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:41.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.938+0000 7f82127fc640 1 --2- 172.21.15.5:0/160731930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82141543a0 0x7f82141f3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:41.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.938+0000 7f82127fc640 1 -- 172.21.15.5:0/160731930 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82040094d0 con 0x7f821406d4f0 2024-03-24T10:47:41.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.940+0000 7f82127fc640 1 --2- 172.21.15.5:0/160731930 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f821406d4f0 0x7f821414ff60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f820800c400 tx=0x7f820800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:41.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.945+0000 7f81fb7fe640 1 -- 172.21.15.5:0/160731930 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82080052d0 con 0x7f821406d4f0 2024-03-24T10:47:41.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.945+0000 7f81fb7fe640 1 -- 172.21.15.5:0/160731930 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8208005450 con 0x7f821406d4f0 2024-03-24T10:47:41.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.945+0000 7f8219756640 1 -- 172.21.15.5:0/160731930 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8214154a60 con 0x7f821406d4f0 2024-03-24T10:47:41.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.945+0000 7f81fb7fe640 1 -- 172.21.15.5:0/160731930 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8208007260 con 0x7f821406d4f0 2024-03-24T10:47:41.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.945+0000 7f8219756640 1 -- 172.21.15.5:0/160731930 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8214157890 con 0x7f821406d4f0 2024-03-24T10:47:41.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.947+0000 7f8219756640 1 -- 172.21.15.5:0/160731930 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82140681d0 con 0x7f821406d4f0 2024-03-24T10:47:41.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.971+0000 7f81fb7fe640 1 -- 172.21.15.5:0/160731930 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f82080073c0 con 0x7f821406d4f0 2024-03-24T10:47:41.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.971+0000 7f81fb7fe640 1 --2- 172.21.15.5:0/160731930 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f820006b810 0x7f820006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:41.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.972+0000 7f8212ffd640 1 --2- 172.21.15.5:0/160731930 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f820006b810 0x7f820006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:41.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:41.972+0000 7f8212ffd640 1 --2- 172.21.15.5:0/160731930 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f820006b810 0x7f820006dcd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8204036000 tx=0x7f8204008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:42.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:42.264+0000 7f81fb7fe640 1 -- 172.21.15.5:0/160731930 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f8208007670 con 0x7f821406d4f0 2024-03-24T10:47:42.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:42.264+0000 7f81fb7fe640 1 -- 172.21.15.5:0/160731930 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f82080545c0 con 0x7f821406d4f0 2024-03-24T10:47:42.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:42.508+0000 7f8219756640 1 -- 172.21.15.5:0/160731930 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8214152850 con 0x7f821406d4f0 2024-03-24T10:47:42.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:42.521+0000 7f81fb7fe640 1 -- 172.21.15.5:0/160731930 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f8208020020 con 0x7f821406d4f0 2024-03-24T10:47:42.521 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:47:42.521 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:47:42.521 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:47:42.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:42.523+0000 7f8219756640 1 -- 172.21.15.5:0/160731930 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f820006b810 msgr2=0x7f820006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:42.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:42.523+0000 7f8219756640 1 --2- 172.21.15.5:0/160731930 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f820006b810 0x7f820006dcd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8204036000 tx=0x7f8204008030 comp rx=0 tx=0).stop 2024-03-24T10:47:42.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:42.523+0000 7f8219756640 1 -- 172.21.15.5:0/160731930 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f821406d4f0 msgr2=0x7f821414ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:42.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:42.523+0000 7f8219756640 1 --2- 172.21.15.5:0/160731930 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f821406d4f0 0x7f821414ff60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f820800c400 tx=0x7f820800c8c0 comp rx=0 tx=0).stop 2024-03-24T10:47:42.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:42.524+0000 7f8219756640 1 -- 172.21.15.5:0/160731930 shutdown_connections 2024-03-24T10:47:42.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:42.524+0000 7f8219756640 1 --2- 172.21.15.5:0/160731930 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f820006b810 0x7f820006dcd0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:42.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:42.524+0000 7f8219756640 1 --2- 172.21.15.5:0/160731930 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f821406d4f0 0x7f821414ff60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:42.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:42.524+0000 7f8219756640 1 --2- 172.21.15.5:0/160731930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82141543a0 0x7f82141f3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:42.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:42.524+0000 7f8219756640 1 --2- 172.21.15.5:0/160731930 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f821406bf50 0x7f8214156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:42.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:42.524+0000 7f8219756640 1 -- 172.21.15.5:0/160731930 >> 172.21.15.5:0/160731930 conn(0x7f8214072ef0 msgr2=0x7f82141598d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:42.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:42.524+0000 7f8219756640 1 -- 172.21.15.5:0/160731930 shutdown_connections 2024-03-24T10:47:42.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:42.524+0000 7f8219756640 1 -- 172.21.15.5:0/160731930 wait complete. 2024-03-24T10:47:43.139 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:47:43.139 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:47:44.141 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:47:44.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.244+0000 7fea4e984640 1 Processor -- start 2024-03-24T10:47:44.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.244+0000 7fea4e984640 1 -- start start 2024-03-24T10:47:44.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.245+0000 7fea4e984640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fea481543a0 0x7fea4815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:44.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.245+0000 7fea4e984640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea480cb2f0 con 0x7fea4806d4f0 2024-03-24T10:47:44.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.245+0000 7fea4e984640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea480ca7e0 con 0x7fea4806bf50 2024-03-24T10:47:44.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.245+0000 7fea4e984640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fea480c6750 con 0x7fea481543a0 2024-03-24T10:47:44.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.245+0000 7fea47fff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fea481543a0 0x7fea4815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:44.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.245+0000 7fea4cf21640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fea4806bf50 0x7fea4806c330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:44826/0 (socket says 172.21.15.5:44826) 2024-03-24T10:47:44.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.245+0000 7fea4cf21640 1 -- 172.21.15.5:0/3186939892 learned_addr learned my addr 172.21.15.5:0/3186939892 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:47:44.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.249+0000 7fea46ffd640 1 -- 172.21.15.5:0/3186939892 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 638297269 0 0) 0x7fea480ca7e0 con 0x7fea4806bf50 2024-03-24T10:47:44.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.249+0000 7fea46ffd640 1 -- 172.21.15.5:0/3186939892 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea30003520 con 0x7fea4806bf50 2024-03-24T10:47:44.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.249+0000 7fea47fff640 1 -- 172.21.15.5:0/3186939892 >> v1:172.21.15.57:6789/0 conn(0x7fea4806bf50 legacy=0x7fea4806c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:44.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.249+0000 7fea47fff640 1 -- 172.21.15.5:0/3186939892 >> v1:172.21.15.5:6789/0 conn(0x7fea4806d4f0 legacy=0x7fea480cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:44.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.249+0000 7fea47fff640 1 -- 172.21.15.5:0/3186939892 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea480cb520 con 0x7fea481543a0 2024-03-24T10:47:44.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.252+0000 7fea47fff640 1 --2- 172.21.15.5:0/3186939892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea481543a0 0x7fea4815ac00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fea380097a0 tx=0x7fea3802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfe7b89d0c76c811 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:44.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.258+0000 7fea46ffd640 1 -- 172.21.15.5:0/3186939892 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea38030610 con 0x7fea481543a0 2024-03-24T10:47:44.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.258+0000 7fea46ffd640 1 -- 172.21.15.5:0/3186939892 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea38030770 con 0x7fea481543a0 2024-03-24T10:47:44.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.258+0000 7fea46ffd640 1 -- 172.21.15.5:0/3186939892 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea38030b00 con 0x7fea481543a0 2024-03-24T10:47:44.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.259+0000 7fea4e984640 1 -- 172.21.15.5:0/3186939892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea481543a0 msgr2=0x7fea4815ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:44.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.259+0000 7fea4e984640 1 --2- 172.21.15.5:0/3186939892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea481543a0 0x7fea4815ac00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fea380097a0 tx=0x7fea3802b960 comp rx=0 tx=0).stop 2024-03-24T10:47:44.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.259+0000 7fea4e984640 1 -- 172.21.15.5:0/3186939892 shutdown_connections 2024-03-24T10:47:44.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.259+0000 7fea4e984640 1 --2- 172.21.15.5:0/3186939892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea481543a0 0x7fea4815ac00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:44.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.259+0000 7fea4e984640 1 -- 172.21.15.5:0/3186939892 >> 172.21.15.5:0/3186939892 conn(0x7fea48072ef0 msgr2=0x7fea48061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:44.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.259+0000 7fea4e984640 1 -- 172.21.15.5:0/3186939892 shutdown_connections 2024-03-24T10:47:44.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.259+0000 7fea4e984640 1 -- 172.21.15.5:0/3186939892 wait complete. 2024-03-24T10:47:44.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.260+0000 7fea4e984640 1 Processor -- start 2024-03-24T10:47:44.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.260+0000 7fea4e984640 1 -- start start 2024-03-24T10:47:44.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.261+0000 7fea4e984640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fea4806bf50 0x7fea480b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:44.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.261+0000 7fea4e984640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea4806d4f0 0x7fea48158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:44.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.261+0000 7fea4e984640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea481543a0 0x7fea48159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:44.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.261+0000 7fea4e984640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fea480cf210 con 0x7fea4806d4f0 2024-03-24T10:47:44.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.261+0000 7fea4e984640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fea480b37b0 con 0x7fea481543a0 2024-03-24T10:47:44.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.261+0000 7fea4e984640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fea480b4050 con 0x7fea4806bf50 2024-03-24T10:47:44.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.262+0000 7fea477fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea4806d4f0 0x7fea48158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:44.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.262+0000 7fea4cf21640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea481543a0 0x7fea48159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:44.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.262+0000 7fea47fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fea4806bf50 0x7fea480b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:44.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.262+0000 7fea477fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea4806d4f0 0x7fea48158cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35700/0 (socket says 172.21.15.5:35700) 2024-03-24T10:47:44.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.262+0000 7fea477fe640 1 -- 172.21.15.5:0/1286784777 learned_addr learned my addr 172.21.15.5:0/1286784777 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:47:44.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.266+0000 7fea477fe640 1 -- 172.21.15.5:0/1286784777 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fea4806bf50 msgr2=0x7fea480b47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:44.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.266+0000 7fea477fe640 1 --2- 172.21.15.5:0/1286784777 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fea4806bf50 0x7fea480b47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:44.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.266+0000 7fea477fe640 1 -- 172.21.15.5:0/1286784777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea481543a0 msgr2=0x7fea48159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:44.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.266+0000 7fea477fe640 1 --2- 172.21.15.5:0/1286784777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea481543a0 0x7fea48159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:44.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.266+0000 7fea477fe640 1 -- 172.21.15.5:0/1286784777 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea38009400 con 0x7fea4806d4f0 2024-03-24T10:47:44.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.268+0000 7fea477fe640 1 --2- 172.21.15.5:0/1286784777 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea4806d4f0 0x7fea48158cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fea3c00c460 tx=0x7fea3c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:44.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.272+0000 7fea44ff9640 1 -- 172.21.15.5:0/1286784777 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3c012830 con 0x7fea4806d4f0 2024-03-24T10:47:44.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.272+0000 7fea44ff9640 1 -- 172.21.15.5:0/1286784777 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea3c0129d0 con 0x7fea4806d4f0 2024-03-24T10:47:44.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.272+0000 7fea44ff9640 1 -- 172.21.15.5:0/1286784777 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3c012d80 con 0x7fea4806d4f0 2024-03-24T10:47:44.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.272+0000 7fea4e984640 1 -- 172.21.15.5:0/1286784777 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea481580f0 con 0x7fea4806d4f0 2024-03-24T10:47:44.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.272+0000 7fea4e984640 1 -- 172.21.15.5:0/1286784777 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea481f08c0 con 0x7fea4806d4f0 2024-03-24T10:47:44.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.274+0000 7fea4e984640 1 -- 172.21.15.5:0/1286784777 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea480681d0 con 0x7fea4806d4f0 2024-03-24T10:47:44.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.298+0000 7fea44ff9640 1 -- 172.21.15.5:0/1286784777 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fea3c01b400 con 0x7fea4806d4f0 2024-03-24T10:47:44.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.299+0000 7fea44ff9640 1 --2- 172.21.15.5:0/1286784777 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fea3406b860 0x7fea3406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:44.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.299+0000 7fea47fff640 1 --2- 172.21.15.5:0/1286784777 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fea3406b860 0x7fea3406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:44.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.299+0000 7fea47fff640 1 --2- 172.21.15.5:0/1286784777 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fea3406b860 0x7fea3406dd20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fea38036000 tx=0x7fea3803b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:44.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.570+0000 7fea44ff9640 1 -- 172.21.15.5:0/1286784777 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fea3c059160 con 0x7fea4806d4f0 2024-03-24T10:47:44.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.570+0000 7fea44ff9640 1 -- 172.21.15.5:0/1286784777 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fea3c055400 con 0x7fea4806d4f0 2024-03-24T10:47:44.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.793+0000 7fea4e984640 1 -- 172.21.15.5:0/1286784777 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea480cae80 con 0x7fea4806d4f0 2024-03-24T10:47:44.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.807+0000 7fea44ff9640 1 -- 172.21.15.5:0/1286784777 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fea3c058ac0 con 0x7fea4806d4f0 2024-03-24T10:47:44.807 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:47:44.807 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:47:44.808 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:47:44.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.810+0000 7fea4e984640 1 -- 172.21.15.5:0/1286784777 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fea3406b860 msgr2=0x7fea3406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.810+0000 7fea4e984640 1 --2- 172.21.15.5:0/1286784777 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fea3406b860 0x7fea3406dd20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fea38036000 tx=0x7fea3803b040 comp rx=0 tx=0).stop 2024-03-24T10:47:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.810+0000 7fea4e984640 1 -- 172.21.15.5:0/1286784777 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea4806d4f0 msgr2=0x7fea48158cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.810+0000 7fea4e984640 1 --2- 172.21.15.5:0/1286784777 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea4806d4f0 0x7fea48158cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fea3c00c460 tx=0x7fea3c00c920 comp rx=0 tx=0).stop 2024-03-24T10:47:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.810+0000 7fea4e984640 1 -- 172.21.15.5:0/1286784777 shutdown_connections 2024-03-24T10:47:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.810+0000 7fea4e984640 1 --2- 172.21.15.5:0/1286784777 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fea3406b860 0x7fea3406dd20 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.810+0000 7fea4e984640 1 --2- 172.21.15.5:0/1286784777 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea4806d4f0 0x7fea48158cc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.810+0000 7fea4e984640 1 --2- 172.21.15.5:0/1286784777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea481543a0 0x7fea48159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.810+0000 7fea4e984640 1 --2- 172.21.15.5:0/1286784777 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fea4806bf50 0x7fea480b47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.810+0000 7fea4e984640 1 -- 172.21.15.5:0/1286784777 >> 172.21.15.5:0/1286784777 conn(0x7fea48072ef0 msgr2=0x7fea481533e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.810+0000 7fea4e984640 1 -- 172.21.15.5:0/1286784777 shutdown_connections 2024-03-24T10:47:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:44.810+0000 7fea4e984640 1 -- 172.21.15.5:0/1286784777 wait complete. 2024-03-24T10:47:45.427 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:47:45.427 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:47:46.429 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:47:46.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.530+0000 7f6e1492e640 1 Processor -- start 2024-03-24T10:47:46.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.530+0000 7f6e1492e640 1 -- start start 2024-03-24T10:47:46.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.532+0000 7f6e1492e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e101543a0 0x7f6e1015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.532+0000 7f6e1492e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6e1006d4f0 0x7f6e100cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.532+0000 7f6e1492e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6e100ccaa0 0x7f6e1006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.532+0000 7f6e1492e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f6e100c6750 con 0x7f6e100ccaa0 2024-03-24T10:47:46.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.532+0000 7f6e1492e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f6e100cc380 con 0x7f6e1006d4f0 2024-03-24T10:47:46.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.532+0000 7f6e1492e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f6e100cb2f0 con 0x7f6e101543a0 2024-03-24T10:47:46.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.532+0000 7f6e0ed9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6e100ccaa0 0x7f6e1006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.532+0000 7f6e0e59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e101543a0 0x7f6e1015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.533+0000 7f6e0dd9b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6e1006d4f0 0x7f6e100cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.533+0000 7f6e0ed9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6e100ccaa0 0x7f6e1006bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35722/0 (socket says 172.21.15.5:35722) 2024-03-24T10:47:46.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.533+0000 7f6e0ed9d640 1 -- 172.21.15.5:0/3614328901 learned_addr learned my addr 172.21.15.5:0/3614328901 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:47:46.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.535+0000 7f6e0ed9d640 1 -- 172.21.15.5:0/3614328901 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e101543a0 msgr2=0x7f6e1015ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:46.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.535+0000 7f6e0ed9d640 1 --2- 172.21.15.5:0/3614328901 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e101543a0 0x7f6e1015ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.535+0000 7f6e0ed9d640 1 -- 172.21.15.5:0/3614328901 >> v2:172.21.15.57:3300/0 conn(0x7f6e1006d4f0 msgr2=0x7f6e100cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:46.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.535+0000 7f6e0ed9d640 1 --2- 172.21.15.5:0/3614328901 >> v2:172.21.15.57:3300/0 conn(0x7f6e1006d4f0 0x7f6e100cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.535+0000 7f6e0ed9d640 1 -- 172.21.15.5:0/3614328901 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e100ca7e0 con 0x7f6e100ccaa0 2024-03-24T10:47:46.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.537+0000 7f6e0ed9d640 1 --2- 172.21.15.5:0/3614328901 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e100ccaa0 0x7f6e1006bf50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6e10062a50 tx=0x7f6e0802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=741678bf8e4aa52d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:46.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.541+0000 7f6e0d59a640 1 -- 172.21.15.5:0/3614328901 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e0802cb00 con 0x7f6e100ccaa0 2024-03-24T10:47:46.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.541+0000 7f6e0d59a640 1 -- 172.21.15.5:0/3614328901 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e0802cc60 con 0x7f6e100ccaa0 2024-03-24T10:47:46.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.541+0000 7f6e0d59a640 1 -- 172.21.15.5:0/3614328901 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e080343f0 con 0x7f6e100ccaa0 2024-03-24T10:47:46.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.542+0000 7f6e1492e640 1 -- 172.21.15.5:0/3614328901 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e100ccaa0 msgr2=0x7f6e1006bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:46.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.542+0000 7f6e1492e640 1 --2- 172.21.15.5:0/3614328901 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e100ccaa0 0x7f6e1006bf50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6e10062a50 tx=0x7f6e0802bf60 comp rx=0 tx=0).stop 2024-03-24T10:47:46.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.542+0000 7f6e1492e640 1 -- 172.21.15.5:0/3614328901 shutdown_connections 2024-03-24T10:47:46.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.542+0000 7f6e1492e640 1 --2- 172.21.15.5:0/3614328901 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e100ccaa0 0x7f6e1006bf50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.542+0000 7f6e1492e640 1 --2- 172.21.15.5:0/3614328901 >> v2:172.21.15.57:3300/0 conn(0x7f6e1006d4f0 0x7f6e100cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.542+0000 7f6e1492e640 1 --2- 172.21.15.5:0/3614328901 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e101543a0 0x7f6e1015ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.542+0000 7f6e1492e640 1 -- 172.21.15.5:0/3614328901 >> 172.21.15.5:0/3614328901 conn(0x7f6e10072ef0 msgr2=0x7f6e10061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:46.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.542+0000 7f6e1492e640 1 -- 172.21.15.5:0/3614328901 shutdown_connections 2024-03-24T10:47:46.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.542+0000 7f6e1492e640 1 -- 172.21.15.5:0/3614328901 wait complete. 2024-03-24T10:47:46.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.543+0000 7f6e1492e640 1 Processor -- start 2024-03-24T10:47:46.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.544+0000 7f6e1492e640 1 -- start start 2024-03-24T10:47:46.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.544+0000 7f6e1492e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e1006d4f0 0x7f6e1014d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.544+0000 7f6e1492e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e100ccaa0 0x7f6e100b4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.544+0000 7f6e1492e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e101543a0 0x7f6e101442e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.544+0000 7f6e1492e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f6e100cf260 con 0x7f6e1006d4f0 2024-03-24T10:47:46.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.544+0000 7f6e1492e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6e1014b4d0 con 0x7f6e100ccaa0 2024-03-24T10:47:46.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.544+0000 7f6e1492e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f6e100cece0 con 0x7f6e101543a0 2024-03-24T10:47:46.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.545+0000 7f6e0e59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e1006d4f0 0x7f6e1014d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.545+0000 7f6e0dd9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e100ccaa0 0x7f6e100b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.545+0000 7f6e0ed9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e101543a0 0x7f6e101442e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.545+0000 7f6e0e59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e1006d4f0 0x7f6e1014d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35732/0 (socket says 172.21.15.5:35732) 2024-03-24T10:47:46.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.545+0000 7f6e0e59c640 1 -- 172.21.15.5:0/3680964743 learned_addr learned my addr 172.21.15.5:0/3680964743 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:47:46.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.549+0000 7f6e0e59c640 1 -- 172.21.15.5:0/3680964743 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e101543a0 msgr2=0x7f6e101442e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:46.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.549+0000 7f6e0e59c640 1 --2- 172.21.15.5:0/3680964743 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e101543a0 0x7f6e101442e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.549+0000 7f6e0e59c640 1 -- 172.21.15.5:0/3680964743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e100ccaa0 msgr2=0x7f6e100b4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:46.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.549+0000 7f6e0e59c640 1 --2- 172.21.15.5:0/3680964743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e100ccaa0 0x7f6e100b4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:46.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.549+0000 7f6e0e59c640 1 -- 172.21.15.5:0/3680964743 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e08009b30 con 0x7f6e1006d4f0 2024-03-24T10:47:46.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.552+0000 7f6e0e59c640 1 --2- 172.21.15.5:0/3680964743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e1006d4f0 0x7f6e1014d030 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000e8a0 tx=0x7f6e0000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:46.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.556+0000 7f6dfeffd640 1 -- 172.21.15.5:0/3680964743 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e000094f0 con 0x7f6e1006d4f0 2024-03-24T10:47:46.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.556+0000 7f6dfeffd640 1 -- 172.21.15.5:0/3680964743 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e0000c940 con 0x7f6e1006d4f0 2024-03-24T10:47:46.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.556+0000 7f6dfeffd640 1 -- 172.21.15.5:0/3680964743 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e0000ccf0 con 0x7f6e1006d4f0 2024-03-24T10:47:46.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.556+0000 7f6e1492e640 1 -- 172.21.15.5:0/3680964743 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e10152840 con 0x7f6e1006d4f0 2024-03-24T10:47:46.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.557+0000 7f6e1492e640 1 -- 172.21.15.5:0/3680964743 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e1014dd10 con 0x7f6e1006d4f0 2024-03-24T10:47:46.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.558+0000 7f6e1492e640 1 -- 172.21.15.5:0/3680964743 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e10153370 con 0x7f6e1006d4f0 2024-03-24T10:47:46.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.844+0000 7f6dfeffd640 1 -- 172.21.15.5:0/3680964743 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6e00013470 con 0x7f6e1006d4f0 2024-03-24T10:47:46.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.845+0000 7f6dfeffd640 1 --2- 172.21.15.5:0/3680964743 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6df006b890 0x7f6df006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:46.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.845+0000 7f6e0dd9b640 1 --2- 172.21.15.5:0/3680964743 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6df006b890 0x7f6df006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:46.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.845+0000 7f6e0dd9b640 1 --2- 172.21.15.5:0/3680964743 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6df006b890 0x7f6df006dd50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6e1015a000 tx=0x7f6e04009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:46.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.867+0000 7f6dfeffd640 1 -- 172.21.15.5:0/3680964743 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f6e00059110 con 0x7f6e1006d4f0 2024-03-24T10:47:46.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:46.867+0000 7f6dfeffd640 1 -- 172.21.15.5:0/3680964743 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6e000553b0 con 0x7f6e1006d4f0 2024-03-24T10:47:47.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:47.107+0000 7f6e1492e640 1 -- 172.21.15.5:0/3680964743 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e100681d0 con 0x7f6e1006d4f0 2024-03-24T10:47:47.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:47.122+0000 7f6dfeffd640 1 -- 172.21.15.5:0/3680964743 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f6e00021090 con 0x7f6e1006d4f0 2024-03-24T10:47:47.122 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:47:47.123 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:47:47.123 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:47:47.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:47.124+0000 7f6e1492e640 1 -- 172.21.15.5:0/3680964743 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6df006b890 msgr2=0x7f6df006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:47.124+0000 7f6e1492e640 1 --2- 172.21.15.5:0/3680964743 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6df006b890 0x7f6df006dd50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6e1015a000 tx=0x7f6e04009190 comp rx=0 tx=0).stop 2024-03-24T10:47:47.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:47.124+0000 7f6e1492e640 1 -- 172.21.15.5:0/3680964743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e1006d4f0 msgr2=0x7f6e1014d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:47.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:47.124+0000 7f6e1492e640 1 --2- 172.21.15.5:0/3680964743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e1006d4f0 0x7f6e1014d030 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000e8a0 tx=0x7f6e0000ed60 comp rx=0 tx=0).stop 2024-03-24T10:47:47.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:47.124+0000 7f6e1492e640 1 -- 172.21.15.5:0/3680964743 shutdown_connections 2024-03-24T10:47:47.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:47.124+0000 7f6e1492e640 1 --2- 172.21.15.5:0/3680964743 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6df006b890 0x7f6df006dd50 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:47.124+0000 7f6e1492e640 1 --2- 172.21.15.5:0/3680964743 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e101543a0 0x7f6e101442e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:47.124+0000 7f6e1492e640 1 --2- 172.21.15.5:0/3680964743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e100ccaa0 0x7f6e100b4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:47.124+0000 7f6e1492e640 1 --2- 172.21.15.5:0/3680964743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e1006d4f0 0x7f6e1014d030 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:47.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:47.124+0000 7f6e1492e640 1 -- 172.21.15.5:0/3680964743 >> 172.21.15.5:0/3680964743 conn(0x7f6e10072ef0 msgr2=0x7f6e100c4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:47.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:47.124+0000 7f6e1492e640 1 -- 172.21.15.5:0/3680964743 shutdown_connections 2024-03-24T10:47:47.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:47.124+0000 7f6e1492e640 1 -- 172.21.15.5:0/3680964743 wait complete. 2024-03-24T10:47:47.738 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:47:47.738 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:47:48.739 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:47:48.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.841+0000 7f398f36c640 1 Processor -- start 2024-03-24T10:47:48.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.841+0000 7f398f36c640 1 -- start start 2024-03-24T10:47:48.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.842+0000 7f398f36c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f39881543a0 0x7f398815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.842+0000 7f398f36c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f398806bf50 0x7f398806c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.842+0000 7f398f36c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39880cb2f0 con 0x7f398806d4f0 2024-03-24T10:47:48.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.842+0000 7f398f36c640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f39880c6750 con 0x7f39881543a0 2024-03-24T10:47:48.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.842+0000 7f398f36c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f39880cc380 con 0x7f398806bf50 2024-03-24T10:47:48.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.843+0000 7f398d108640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f39881543a0 0x7f398815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.843+0000 7f398d909640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f398806bf50 0x7f398806c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.843+0000 7f398d108640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f39881543a0 0x7f398815ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35746/0 (socket says 172.21.15.5:35746) 2024-03-24T10:47:48.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.843+0000 7f398d108640 1 -- 172.21.15.5:0/2572409439 learned_addr learned my addr 172.21.15.5:0/2572409439 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:47:48.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.846+0000 7f398d108640 1 -- 172.21.15.5:0/2572409439 >> v2:172.21.15.57:3300/0 conn(0x7f398806bf50 msgr2=0x7f398806c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:48.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.846+0000 7f398d108640 1 --2- 172.21.15.5:0/2572409439 >> v2:172.21.15.57:3300/0 conn(0x7f398806bf50 0x7f398806c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.846+0000 7f398d108640 1 -- 172.21.15.5:0/2572409439 >> v1:172.21.15.57:6789/0 conn(0x7f398806d4f0 legacy=0x7f39880cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.846+0000 7f398d108640 1 -- 172.21.15.5:0/2572409439 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39880ca7e0 con 0x7f39881543a0 2024-03-24T10:47:48.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.848+0000 7f398d108640 1 --2- 172.21.15.5:0/2572409439 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39881543a0 0x7f398815ac00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f39880b51c0 tx=0x7f398402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e00922316f26ad1a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:48.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.853+0000 7f397ffff640 1 -- 172.21.15.5:0/2572409439 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398402f850 con 0x7f39881543a0 2024-03-24T10:47:48.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.853+0000 7f397ffff640 1 -- 172.21.15.5:0/2572409439 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f398402f9b0 con 0x7f39881543a0 2024-03-24T10:47:48.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.853+0000 7f397ffff640 1 -- 172.21.15.5:0/2572409439 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398402fd40 con 0x7f39881543a0 2024-03-24T10:47:48.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.853+0000 7f398f36c640 1 -- 172.21.15.5:0/2572409439 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39881543a0 msgr2=0x7f398815ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.853+0000 7f398f36c640 1 --2- 172.21.15.5:0/2572409439 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39881543a0 0x7f398815ac00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f39880b51c0 tx=0x7f398402ba30 comp rx=0 tx=0).stop 2024-03-24T10:47:48.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.853+0000 7f398f36c640 1 -- 172.21.15.5:0/2572409439 shutdown_connections 2024-03-24T10:47:48.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.853+0000 7f398f36c640 1 --2- 172.21.15.5:0/2572409439 >> v2:172.21.15.57:3300/0 conn(0x7f398806bf50 0x7f398806c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.853+0000 7f398f36c640 1 --2- 172.21.15.5:0/2572409439 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39881543a0 0x7f398815ac00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.853+0000 7f398f36c640 1 -- 172.21.15.5:0/2572409439 >> 172.21.15.5:0/2572409439 conn(0x7f3988072ef0 msgr2=0x7f3988061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:48.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.854+0000 7f398f36c640 1 -- 172.21.15.5:0/2572409439 shutdown_connections 2024-03-24T10:47:48.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.854+0000 7f398f36c640 1 -- 172.21.15.5:0/2572409439 wait complete. 2024-03-24T10:47:48.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.855+0000 7f398f36c640 1 Processor -- start 2024-03-24T10:47:48.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.856+0000 7f398f36c640 1 -- start start 2024-03-24T10:47:48.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.857+0000 7f398f36c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f398806bf50 0x7f39880cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.857+0000 7f398f36c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f398806d4f0 0x7f39881492a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.857+0000 7f398f36c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39881543a0 0x7f39881497e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:48.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.857+0000 7f398f36c640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f3988148e80 con 0x7f398806bf50 2024-03-24T10:47:48.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.857+0000 7f398f36c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f39881e3b80 con 0x7f39881543a0 2024-03-24T10:47:48.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.857+0000 7f398f36c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f39880ce160 con 0x7f398806d4f0 2024-03-24T10:47:48.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.857+0000 7f398d108640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f398806bf50 0x7f39880cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.857+0000 7f398d909640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39881543a0 0x7f39881497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.857+0000 7f398d108640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f398806bf50 0x7f39880cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35762/0 (socket says 172.21.15.5:35762) 2024-03-24T10:47:48.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.857+0000 7f398d108640 1 -- 172.21.15.5:0/3091319342 learned_addr learned my addr 172.21.15.5:0/3091319342 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:47:48.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.857+0000 7f398c907640 1 --2- 172.21.15.5:0/3091319342 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f398806d4f0 0x7f39881492a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:48.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.861+0000 7f398d108640 1 -- 172.21.15.5:0/3091319342 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f398806d4f0 msgr2=0x7f39881492a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.861+0000 7f398d108640 1 --2- 172.21.15.5:0/3091319342 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f398806d4f0 0x7f39881492a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.861+0000 7f398d108640 1 -- 172.21.15.5:0/3091319342 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39881543a0 msgr2=0x7f39881497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:48.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.861+0000 7f398d108640 1 --2- 172.21.15.5:0/3091319342 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39881543a0 0x7f39881497e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:48.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.861+0000 7f398d108640 1 -- 172.21.15.5:0/3091319342 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39840094d0 con 0x7f398806bf50 2024-03-24T10:47:48.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.865+0000 7f398d108640 1 --2- 172.21.15.5:0/3091319342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f398806bf50 0x7f39880cdb40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3984000c00 tx=0x7f3984004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:48.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.868+0000 7f397dffb640 1 -- 172.21.15.5:0/3091319342 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3984037c40 con 0x7f398806bf50 2024-03-24T10:47:48.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.868+0000 7f398f36c640 1 -- 172.21.15.5:0/3091319342 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39880ce440 con 0x7f398806bf50 2024-03-24T10:47:48.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.868+0000 7f398f36c640 1 -- 172.21.15.5:0/3091319342 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3988148aa0 con 0x7f398806bf50 2024-03-24T10:47:48.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.869+0000 7f398f36c640 1 -- 172.21.15.5:0/3091319342 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3988148900 con 0x7f398806bf50 2024-03-24T10:47:48.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.870+0000 7f397dffb640 1 -- 172.21.15.5:0/3091319342 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3984037da0 con 0x7f398806bf50 2024-03-24T10:47:48.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:48.870+0000 7f397dffb640 1 -- 172.21.15.5:0/3091319342 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39840426a0 con 0x7f398806bf50 2024-03-24T10:47:49.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:49.141+0000 7f397dffb640 1 -- 172.21.15.5:0/3091319342 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3984042800 con 0x7f398806bf50 2024-03-24T10:47:49.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:49.141+0000 7f397dffb640 1 --2- 172.21.15.5:0/3091319342 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f396c06b840 0x7f396c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:49.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:49.141+0000 7f398c907640 1 --2- 172.21.15.5:0/3091319342 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f396c06b840 0x7f396c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:49.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:49.142+0000 7f398c907640 1 --2- 172.21.15.5:0/3091319342 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f396c06b840 0x7f396c06dd00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f39880c5a60 tx=0x7f3978009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:49.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:49.156+0000 7f397dffb640 1 -- 172.21.15.5:0/3091319342 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f398403c070 con 0x7f398806bf50 2024-03-24T10:47:49.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:49.157+0000 7f397dffb640 1 -- 172.21.15.5:0/3091319342 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3984077730 con 0x7f398806bf50 2024-03-24T10:47:49.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:49.377+0000 7f398f36c640 1 -- 172.21.15.5:0/3091319342 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3988148900 con 0x7f398806bf50 2024-03-24T10:47:49.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:49.402+0000 7f397dffb640 1 -- 172.21.15.5:0/3091319342 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f3988148900 con 0x7f398806bf50 2024-03-24T10:47:49.402 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:47:49.402 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:47:49.402 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:47:49.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:49.404+0000 7f398f36c640 1 -- 172.21.15.5:0/3091319342 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f396c06b840 msgr2=0x7f396c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:49.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:49.404+0000 7f398f36c640 1 --2- 172.21.15.5:0/3091319342 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f396c06b840 0x7f396c06dd00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f39880c5a60 tx=0x7f3978009040 comp rx=0 tx=0).stop 2024-03-24T10:47:49.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:49.404+0000 7f398f36c640 1 -- 172.21.15.5:0/3091319342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f398806bf50 msgr2=0x7f39880cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:49.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:49.404+0000 7f398f36c640 1 --2- 172.21.15.5:0/3091319342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f398806bf50 0x7f39880cdb40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3984000c00 tx=0x7f3984004730 comp rx=0 tx=0).stop 2024-03-24T10:47:49.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:49.404+0000 7f398f36c640 1 -- 172.21.15.5:0/3091319342 shutdown_connections 2024-03-24T10:47:49.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:49.404+0000 7f398f36c640 1 --2- 172.21.15.5:0/3091319342 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f396c06b840 0x7f396c06dd00 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:49.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:49.404+0000 7f398f36c640 1 --2- 172.21.15.5:0/3091319342 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39881543a0 0x7f39881497e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:49.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:49.404+0000 7f398f36c640 1 --2- 172.21.15.5:0/3091319342 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f398806d4f0 0x7f39881492a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:49.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:49.404+0000 7f398f36c640 1 --2- 172.21.15.5:0/3091319342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f398806bf50 0x7f39880cdb40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:49.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:49.404+0000 7f398f36c640 1 -- 172.21.15.5:0/3091319342 >> 172.21.15.5:0/3091319342 conn(0x7f3988072ef0 msgr2=0x7f39880cc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:49.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:49.404+0000 7f398f36c640 1 -- 172.21.15.5:0/3091319342 shutdown_connections 2024-03-24T10:47:49.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:49.404+0000 7f398f36c640 1 -- 172.21.15.5:0/3091319342 wait complete. 2024-03-24T10:47:50.022 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:47:50.023 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:47:51.024 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:47:51.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.126+0000 7f215c823640 1 Processor -- start 2024-03-24T10:47:51.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.126+0000 7f215c823640 1 -- start start 2024-03-24T10:47:51.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.127+0000 7f215c823640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f215815ac00 0x7f21580cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:51.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.127+0000 7f215c823640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21580ccaa0 0x7f215806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:51.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.127+0000 7f215c823640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21580cb2f0 con 0x7f21581543a0 2024-03-24T10:47:51.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.127+0000 7f215c823640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f21580c6750 con 0x7f215815ac00 2024-03-24T10:47:51.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.127+0000 7f215c823640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f21580cceb0 con 0x7f21580ccaa0 2024-03-24T10:47:51.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.128+0000 7f2156d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21580ccaa0 0x7f215806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:51.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.128+0000 7f2155d9b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f215815ac00 0x7f21580cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:51.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.129+0000 7f2156d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21580ccaa0 0x7f215806bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:51388/0 (socket says 172.21.15.5:51388) 2024-03-24T10:47:51.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.129+0000 7f2156d9d640 1 -- 172.21.15.5:0/2571253594 learned_addr learned my addr 172.21.15.5:0/2571253594 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:47:51.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.131+0000 7f2156d9d640 1 -- 172.21.15.5:0/2571253594 >> v2:172.21.15.57:3300/0 conn(0x7f215815ac00 msgr2=0x7f21580cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:51.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.131+0000 7f2156d9d640 1 --2- 172.21.15.5:0/2571253594 >> v2:172.21.15.57:3300/0 conn(0x7f215815ac00 0x7f21580cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:51.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.131+0000 7f2156d9d640 1 -- 172.21.15.5:0/2571253594 >> v1:172.21.15.57:6789/0 conn(0x7f21581543a0 legacy=0x7f215806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:51.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.131+0000 7f2156d9d640 1 -- 172.21.15.5:0/2571253594 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f215815b070 con 0x7f21580ccaa0 2024-03-24T10:47:51.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.134+0000 7f2156d9d640 1 --2- 172.21.15.5:0/2571253594 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21580ccaa0 0x7f215806bf50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2150009e50 tx=0x7f215002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0ae2862b104d664 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:51.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.138+0000 7f215559a640 1 -- 172.21.15.5:0/2571253594 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215002ca70 con 0x7f21580ccaa0 2024-03-24T10:47:51.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.138+0000 7f215559a640 1 -- 172.21.15.5:0/2571253594 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f215002cbd0 con 0x7f21580ccaa0 2024-03-24T10:47:51.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.138+0000 7f215559a640 1 -- 172.21.15.5:0/2571253594 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21500343f0 con 0x7f21580ccaa0 2024-03-24T10:47:51.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.138+0000 7f215c823640 1 -- 172.21.15.5:0/2571253594 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21580ccaa0 msgr2=0x7f215806bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:51.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.138+0000 7f215c823640 1 --2- 172.21.15.5:0/2571253594 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21580ccaa0 0x7f215806bf50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2150009e50 tx=0x7f215002bed0 comp rx=0 tx=0).stop 2024-03-24T10:47:51.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.138+0000 7f215c823640 1 -- 172.21.15.5:0/2571253594 shutdown_connections 2024-03-24T10:47:51.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.138+0000 7f215c823640 1 --2- 172.21.15.5:0/2571253594 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21580ccaa0 0x7f215806bf50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:51.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.138+0000 7f215c823640 1 --2- 172.21.15.5:0/2571253594 >> v2:172.21.15.57:3300/0 conn(0x7f215815ac00 0x7f21580cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:51.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.138+0000 7f215c823640 1 -- 172.21.15.5:0/2571253594 >> 172.21.15.5:0/2571253594 conn(0x7f2158072ef0 msgr2=0x7f2158061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:51.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.138+0000 7f215c823640 1 -- 172.21.15.5:0/2571253594 shutdown_connections 2024-03-24T10:47:51.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.138+0000 7f215c823640 1 -- 172.21.15.5:0/2571253594 wait complete. 2024-03-24T10:47:51.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.140+0000 7f215c823640 1 Processor -- start 2024-03-24T10:47:51.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.140+0000 7f215c823640 1 -- start start 2024-03-24T10:47:51.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.140+0000 7f215c823640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21580ccaa0 0x7f21580c8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:51.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.140+0000 7f215c823640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21581543a0 0x7f21581f2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:51.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.140+0000 7f215c823640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f215815ac00 0x7f2158147af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:51.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.140+0000 7f215c823640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f21580cf820 con 0x7f21580ccaa0 2024-03-24T10:47:51.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.140+0000 7f215c823640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2158152890 con 0x7f215815ac00 2024-03-24T10:47:51.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.140+0000 7f215c823640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f21581497a0 con 0x7f21581543a0 2024-03-24T10:47:51.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.141+0000 7f215659c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21580ccaa0 0x7f21580c8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:51.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.141+0000 7f2155d9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21581543a0 0x7f21581f2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:51.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.141+0000 7f2156d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f215815ac00 0x7f2158147af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:51.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.141+0000 7f215659c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21580ccaa0 0x7f21580c8f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35768/0 (socket says 172.21.15.5:35768) 2024-03-24T10:47:51.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.141+0000 7f215659c640 1 -- 172.21.15.5:0/59804444 learned_addr learned my addr 172.21.15.5:0/59804444 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:47:51.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.145+0000 7f215659c640 1 -- 172.21.15.5:0/59804444 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21581543a0 msgr2=0x7f21581f2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:51.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.145+0000 7f215659c640 1 --2- 172.21.15.5:0/59804444 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21581543a0 0x7f21581f2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:51.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.145+0000 7f215659c640 1 -- 172.21.15.5:0/59804444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f215815ac00 msgr2=0x7f2158147af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:51.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.145+0000 7f215659c640 1 --2- 172.21.15.5:0/59804444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f215815ac00 0x7f2158147af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:51.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.145+0000 7f215659c640 1 -- 172.21.15.5:0/59804444 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2150009b30 con 0x7f21580ccaa0 2024-03-24T10:47:51.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.147+0000 7f215659c640 1 --2- 172.21.15.5:0/59804444 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21580ccaa0 0x7f21580c8f40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f214800c430 tx=0x7f214800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:51.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.152+0000 7f2146ffd640 1 -- 172.21.15.5:0/59804444 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2148011790 con 0x7f21580ccaa0 2024-03-24T10:47:51.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.152+0000 7f2146ffd640 1 -- 172.21.15.5:0/59804444 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21480118f0 con 0x7f21580ccaa0 2024-03-24T10:47:51.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.152+0000 7f215c823640 1 -- 172.21.15.5:0/59804444 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21580c84a0 con 0x7f21580ccaa0 2024-03-24T10:47:51.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.152+0000 7f2146ffd640 1 -- 172.21.15.5:0/59804444 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2148011ca0 con 0x7f21580ccaa0 2024-03-24T10:47:51.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.152+0000 7f215c823640 1 -- 172.21.15.5:0/59804444 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21580c8a20 con 0x7f21580ccaa0 2024-03-24T10:47:51.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.153+0000 7f215c823640 1 -- 172.21.15.5:0/59804444 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f215806cfd0 con 0x7f21580ccaa0 2024-03-24T10:47:51.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.443+0000 7f2146ffd640 1 -- 172.21.15.5:0/59804444 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f214801c460 con 0x7f21580ccaa0 2024-03-24T10:47:51.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.443+0000 7f2146ffd640 1 --2- 172.21.15.5:0/59804444 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f213806b890 0x7f213806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:51.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.444+0000 7f2155d9b640 1 --2- 172.21.15.5:0/59804444 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f213806b890 0x7f213806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:51.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.444+0000 7f2155d9b640 1 --2- 172.21.15.5:0/59804444 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f213806b890 0x7f213806dd50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f21580c96f0 tx=0x7f214c009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:51.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.458+0000 7f2146ffd640 1 -- 172.21.15.5:0/59804444 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f21480592b0 con 0x7f21580ccaa0 2024-03-24T10:47:51.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.459+0000 7f2146ffd640 1 -- 172.21.15.5:0/59804444 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2148055550 con 0x7f21580ccaa0 2024-03-24T10:47:51.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.682+0000 7f215c823640 1 -- 172.21.15.5:0/59804444 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21580c8c10 con 0x7f21580ccaa0 2024-03-24T10:47:51.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.697+0000 7f2146ffd640 1 -- 172.21.15.5:0/59804444 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f2148058c10 con 0x7f21580ccaa0 2024-03-24T10:47:51.698 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:47:51.698 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:47:51.698 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:47:51.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.699+0000 7f215c823640 1 -- 172.21.15.5:0/59804444 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f213806b890 msgr2=0x7f213806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:51.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.699+0000 7f215c823640 1 --2- 172.21.15.5:0/59804444 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f213806b890 0x7f213806dd50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f21580c96f0 tx=0x7f214c009190 comp rx=0 tx=0).stop 2024-03-24T10:47:51.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.699+0000 7f215c823640 1 -- 172.21.15.5:0/59804444 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21580ccaa0 msgr2=0x7f21580c8f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:51.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.699+0000 7f215c823640 1 --2- 172.21.15.5:0/59804444 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21580ccaa0 0x7f21580c8f40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f214800c430 tx=0x7f214800c8f0 comp rx=0 tx=0).stop 2024-03-24T10:47:51.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.699+0000 7f215c823640 1 -- 172.21.15.5:0/59804444 shutdown_connections 2024-03-24T10:47:51.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.700+0000 7f215c823640 1 --2- 172.21.15.5:0/59804444 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f213806b890 0x7f213806dd50 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:51.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.700+0000 7f215c823640 1 --2- 172.21.15.5:0/59804444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f215815ac00 0x7f2158147af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:51.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.700+0000 7f215c823640 1 --2- 172.21.15.5:0/59804444 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21581543a0 0x7f21581f2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:51.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.700+0000 7f215c823640 1 --2- 172.21.15.5:0/59804444 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21580ccaa0 0x7f21580c8f40 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:51.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.700+0000 7f215c823640 1 -- 172.21.15.5:0/59804444 >> 172.21.15.5:0/59804444 conn(0x7f2158072ef0 msgr2=0x7f2158061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:51.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.700+0000 7f215c823640 1 -- 172.21.15.5:0/59804444 shutdown_connections 2024-03-24T10:47:51.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:51.700+0000 7f215c823640 1 -- 172.21.15.5:0/59804444 wait complete. 2024-03-24T10:47:52.315 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:47:52.315 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:47:53.316 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:47:53.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.419+0000 7f4ea42c9640 1 Processor -- start 2024-03-24T10:47:53.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.419+0000 7f4ea42c9640 1 -- start start 2024-03-24T10:47:53.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.420+0000 7f4ea42c9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e9c1543a0 0x7f4e9c15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:53.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.420+0000 7f4ea42c9640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4e9c06bf50 0x7f4e9c06c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:53.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.420+0000 7f4ea42c9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e9c0cb2f0 con 0x7f4e9c06d4f0 2024-03-24T10:47:53.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.420+0000 7f4ea42c9640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f4e9c0c6750 con 0x7f4e9c06bf50 2024-03-24T10:47:53.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.420+0000 7f4ea42c9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4e9c0cc380 con 0x7f4e9c1543a0 2024-03-24T10:47:53.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.421+0000 7f4ea2065640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e9c1543a0 0x7f4e9c15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:53.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.421+0000 7f4ea1864640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f4e9c06d4f0 0x7f4e9c0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:35028/0 (socket says 172.21.15.5:35028) 2024-03-24T10:47:53.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.421+0000 7f4ea1864640 1 -- 172.21.15.5:0/883935812 learned_addr learned my addr 172.21.15.5:0/883935812 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:47:53.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.424+0000 7f4ea2065640 1 -- 172.21.15.5:0/883935812 >> v2:172.21.15.5:3300/0 conn(0x7f4e9c06bf50 msgr2=0x7f4e9c06c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:53.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.424+0000 7f4ea2065640 1 --2- 172.21.15.5:0/883935812 >> v2:172.21.15.5:3300/0 conn(0x7f4e9c06bf50 0x7f4e9c06c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:53.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.424+0000 7f4ea2065640 1 -- 172.21.15.5:0/883935812 >> v1:172.21.15.57:6789/0 conn(0x7f4e9c06d4f0 legacy=0x7f4e9c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:53.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.424+0000 7f4ea2065640 1 -- 172.21.15.5:0/883935812 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9c0ca7e0 con 0x7f4e9c1543a0 2024-03-24T10:47:53.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.427+0000 7f4ea2065640 1 --2- 172.21.15.5:0/883935812 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e9c1543a0 0x7f4e9c15ac00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4e980097a0 tx=0x7f4e9802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e83ca5b42a5b1e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:53.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.431+0000 7f4ea1063640 1 -- 172.21.15.5:0/883935812 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e98030610 con 0x7f4e9c1543a0 2024-03-24T10:47:53.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.431+0000 7f4ea1063640 1 -- 172.21.15.5:0/883935812 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e98030770 con 0x7f4e9c1543a0 2024-03-24T10:47:53.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.431+0000 7f4ea1063640 1 -- 172.21.15.5:0/883935812 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e98030b00 con 0x7f4e9c1543a0 2024-03-24T10:47:53.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.432+0000 7f4ea42c9640 1 -- 172.21.15.5:0/883935812 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e9c1543a0 msgr2=0x7f4e9c15ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:53.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.432+0000 7f4ea42c9640 1 --2- 172.21.15.5:0/883935812 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e9c1543a0 0x7f4e9c15ac00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4e980097a0 tx=0x7f4e9802b960 comp rx=0 tx=0).stop 2024-03-24T10:47:53.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.432+0000 7f4ea42c9640 1 -- 172.21.15.5:0/883935812 shutdown_connections 2024-03-24T10:47:53.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.432+0000 7f4ea42c9640 1 --2- 172.21.15.5:0/883935812 >> v2:172.21.15.5:3300/0 conn(0x7f4e9c06bf50 0x7f4e9c06c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:53.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.432+0000 7f4ea42c9640 1 --2- 172.21.15.5:0/883935812 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e9c1543a0 0x7f4e9c15ac00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:53.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.432+0000 7f4ea42c9640 1 -- 172.21.15.5:0/883935812 >> 172.21.15.5:0/883935812 conn(0x7f4e9c072ef0 msgr2=0x7f4e9c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:53.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.432+0000 7f4ea42c9640 1 -- 172.21.15.5:0/883935812 shutdown_connections 2024-03-24T10:47:53.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.432+0000 7f4ea42c9640 1 -- 172.21.15.5:0/883935812 wait complete. 2024-03-24T10:47:53.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.434+0000 7f4ea42c9640 1 Processor -- start 2024-03-24T10:47:53.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.434+0000 7f4ea42c9640 1 -- start start 2024-03-24T10:47:53.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.435+0000 7f4ea42c9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e9c06bf50 0x7f4e9c157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:53.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.435+0000 7f4ea42c9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e9c06d4f0 0x7f4e9c0c8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:53.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.436+0000 7f4ea42c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e9c1543a0 0x7f4e9c0c8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:53.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.436+0000 7f4ea42c9640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9c1f0250 con 0x7f4e9c06bf50 2024-03-24T10:47:53.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.436+0000 7f4ea42c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9c1f0780 con 0x7f4e9c1543a0 2024-03-24T10:47:53.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.436+0000 7f4ea42c9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4e9c154c20 con 0x7f4e9c06d4f0 2024-03-24T10:47:53.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.436+0000 7f4ea2065640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e9c06bf50 0x7f4e9c157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:53.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.436+0000 7f4ea1864640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e9c06d4f0 0x7f4e9c0c8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:53.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.436+0000 7f4ea2866640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e9c1543a0 0x7f4e9c0c8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:53.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.436+0000 7f4ea2065640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e9c06bf50 0x7f4e9c157f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42246/0 (socket says 172.21.15.5:42246) 2024-03-24T10:47:53.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.436+0000 7f4ea2065640 1 -- 172.21.15.5:0/1057327296 learned_addr learned my addr 172.21.15.5:0/1057327296 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:47:53.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.440+0000 7f4ea2065640 1 -- 172.21.15.5:0/1057327296 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e9c06d4f0 msgr2=0x7f4e9c0c8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:53.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.440+0000 7f4ea2065640 1 --2- 172.21.15.5:0/1057327296 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e9c06d4f0 0x7f4e9c0c8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:53.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.440+0000 7f4ea2065640 1 -- 172.21.15.5:0/1057327296 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e9c1543a0 msgr2=0x7f4e9c0c8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:53.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.440+0000 7f4ea2065640 1 --2- 172.21.15.5:0/1057327296 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e9c1543a0 0x7f4e9c0c8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:53.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.440+0000 7f4ea2065640 1 -- 172.21.15.5:0/1057327296 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e98009400 con 0x7f4e9c06bf50 2024-03-24T10:47:53.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.443+0000 7f4ea2065640 1 --2- 172.21.15.5:0/1057327296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e9c06bf50 0x7f4e9c157f60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4e9802bf70 tx=0x7f4e98004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:53.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.447+0000 7f4e92ffd640 1 -- 172.21.15.5:0/1057327296 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9803e040 con 0x7f4e9c06bf50 2024-03-24T10:47:53.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.447+0000 7f4ea42c9640 1 -- 172.21.15.5:0/1057327296 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e9c0c9190 con 0x7f4e9c06bf50 2024-03-24T10:47:53.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.447+0000 7f4ea42c9640 1 -- 172.21.15.5:0/1057327296 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e9c159e10 con 0x7f4e9c06bf50 2024-03-24T10:47:53.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.448+0000 7f4e92ffd640 1 -- 172.21.15.5:0/1057327296 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e98039c50 con 0x7f4e9c06bf50 2024-03-24T10:47:53.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.448+0000 7f4e92ffd640 1 -- 172.21.15.5:0/1057327296 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9804d440 con 0x7f4e9c06bf50 2024-03-24T10:47:53.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.449+0000 7f4ea42c9640 1 -- 172.21.15.5:0/1057327296 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e84005290 con 0x7f4e9c06bf50 2024-03-24T10:47:53.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.733+0000 7f4e92ffd640 1 -- 172.21.15.5:0/1057327296 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4e9804d5a0 con 0x7f4e9c06bf50 2024-03-24T10:47:53.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.734+0000 7f4e92ffd640 1 --2- 172.21.15.5:0/1057327296 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e8006b810 0x7f4e8006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:53.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.734+0000 7f4ea1864640 1 --2- 172.21.15.5:0/1057327296 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e8006b810 0x7f4e8006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:53.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.734+0000 7f4ea1864640 1 --2- 172.21.15.5:0/1057327296 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e8006b810 0x7f4e8006dcd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c0b23d0 tx=0x7f4e8c009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:53.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.753+0000 7f4e92ffd640 1 -- 172.21.15.5:0/1057327296 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f4e9807b560 con 0x7f4e9c06bf50 2024-03-24T10:47:53.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.753+0000 7f4e92ffd640 1 -- 172.21.15.5:0/1057327296 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4e98077800 con 0x7f4e9c06bf50 2024-03-24T10:47:53.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:53.986+0000 7f4ea42c9640 1 -- 172.21.15.5:0/1057327296 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e840058f0 con 0x7f4e9c06bf50 2024-03-24T10:47:54.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:54.000+0000 7f4e92ffd640 1 -- 172.21.15.5:0/1057327296 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f4e840058f0 con 0x7f4e9c06bf50 2024-03-24T10:47:54.001 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:47:54.001 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:47:54.001 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:47:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:54.003+0000 7f4ea42c9640 1 -- 172.21.15.5:0/1057327296 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e8006b810 msgr2=0x7f4e8006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:54.003+0000 7f4ea42c9640 1 --2- 172.21.15.5:0/1057327296 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e8006b810 0x7f4e8006dcd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c0b23d0 tx=0x7f4e8c009040 comp rx=0 tx=0).stop 2024-03-24T10:47:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:54.003+0000 7f4ea42c9640 1 -- 172.21.15.5:0/1057327296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e9c06bf50 msgr2=0x7f4e9c157f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:54.003+0000 7f4ea42c9640 1 --2- 172.21.15.5:0/1057327296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e9c06bf50 0x7f4e9c157f60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4e9802bf70 tx=0x7f4e98004730 comp rx=0 tx=0).stop 2024-03-24T10:47:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:54.003+0000 7f4ea42c9640 1 -- 172.21.15.5:0/1057327296 shutdown_connections 2024-03-24T10:47:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:54.003+0000 7f4ea42c9640 1 --2- 172.21.15.5:0/1057327296 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e8006b810 0x7f4e8006dcd0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:54.003+0000 7f4ea42c9640 1 --2- 172.21.15.5:0/1057327296 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e9c1543a0 0x7f4e9c0c8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:54.003+0000 7f4ea42c9640 1 --2- 172.21.15.5:0/1057327296 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e9c06d4f0 0x7f4e9c0c8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:54.003+0000 7f4ea42c9640 1 --2- 172.21.15.5:0/1057327296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e9c06bf50 0x7f4e9c157f60 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:54.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:54.003+0000 7f4ea42c9640 1 -- 172.21.15.5:0/1057327296 >> 172.21.15.5:0/1057327296 conn(0x7f4e9c072ef0 msgr2=0x7f4e9c061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:54.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:54.003+0000 7f4ea42c9640 1 -- 172.21.15.5:0/1057327296 shutdown_connections 2024-03-24T10:47:54.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:54.003+0000 7f4ea42c9640 1 -- 172.21.15.5:0/1057327296 wait complete. 2024-03-24T10:47:54.617 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:47:54.618 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:47:55.619 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:47:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.724+0000 7f74b09af640 1 Processor -- start 2024-03-24T10:47:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.724+0000 7f74b09af640 1 -- start start 2024-03-24T10:47:55.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.725+0000 7f74b09af640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f74ac1543a0 0x7f74ac15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:55.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.726+0000 7f74b09af640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f74ac06bf50 0x7f74ac06c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:55.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.726+0000 7f74b09af640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74ac0cb2f0 con 0x7f74ac06d4f0 2024-03-24T10:47:55.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.726+0000 7f74b09af640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f74ac0c6750 con 0x7f74ac1543a0 2024-03-24T10:47:55.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.726+0000 7f74b09af640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f74ac0cc380 con 0x7f74ac06bf50 2024-03-24T10:47:55.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.726+0000 7f74aad9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f74ac06bf50 0x7f74ac06c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:55.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.727+0000 7f74a9d9b640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f74ac06d4f0 0x7f74ac0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:49012/0 (socket says 172.21.15.5:49012) 2024-03-24T10:47:55.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.727+0000 7f74a9d9b640 1 -- 172.21.15.5:0/3751750176 learned_addr learned my addr 172.21.15.5:0/3751750176 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:47:55.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.727+0000 7f74aa59c640 1 --2- 172.21.15.5:0/3751750176 >> v2:172.21.15.5:3300/0 conn(0x7f74ac1543a0 0x7f74ac15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.730+0000 7f74aad9d640 1 -- 172.21.15.5:0/3751750176 >> v2:172.21.15.5:3300/0 conn(0x7f74ac1543a0 msgr2=0x7f74ac15ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:47:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.730+0000 7f74aad9d640 1 --2- 172.21.15.5:0/3751750176 >> v2:172.21.15.5:3300/0 conn(0x7f74ac1543a0 0x7f74ac15ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.730+0000 7f74aad9d640 1 -- 172.21.15.5:0/3751750176 >> v1:172.21.15.5:6789/0 conn(0x7f74ac06d4f0 legacy=0x7f74ac0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.730+0000 7f74aad9d640 1 -- 172.21.15.5:0/3751750176 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74ac0ca7e0 con 0x7f74ac06bf50 2024-03-24T10:47:55.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.733+0000 7f74aad9d640 1 --2- 172.21.15.5:0/3751750176 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f74ac06bf50 0x7f74ac06c360 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f74a4009d90 tx=0x7f74a402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e342018377de75c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:55.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.737+0000 7f74a959a640 1 -- 172.21.15.5:0/3751750176 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a402cce0 con 0x7f74ac06bf50 2024-03-24T10:47:55.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.737+0000 7f74a959a640 1 -- 172.21.15.5:0/3751750176 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74a402ce40 con 0x7f74ac06bf50 2024-03-24T10:47:55.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.737+0000 7f74a959a640 1 -- 172.21.15.5:0/3751750176 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a4034660 con 0x7f74ac06bf50 2024-03-24T10:47:55.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.738+0000 7f74b09af640 1 -- 172.21.15.5:0/3751750176 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f74ac06bf50 msgr2=0x7f74ac06c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:55.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.738+0000 7f74b09af640 1 --2- 172.21.15.5:0/3751750176 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f74ac06bf50 0x7f74ac06c360 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f74a4009d90 tx=0x7f74a402bf30 comp rx=0 tx=0).stop 2024-03-24T10:47:55.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.738+0000 7f74b09af640 1 -- 172.21.15.5:0/3751750176 shutdown_connections 2024-03-24T10:47:55.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.738+0000 7f74b09af640 1 --2- 172.21.15.5:0/3751750176 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f74ac06bf50 0x7f74ac06c360 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:55.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.738+0000 7f74b09af640 1 --2- 172.21.15.5:0/3751750176 >> v2:172.21.15.5:3300/0 conn(0x7f74ac1543a0 0x7f74ac15ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:55.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.738+0000 7f74b09af640 1 -- 172.21.15.5:0/3751750176 >> 172.21.15.5:0/3751750176 conn(0x7f74ac072ef0 msgr2=0x7f74ac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:55.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.738+0000 7f74b09af640 1 -- 172.21.15.5:0/3751750176 shutdown_connections 2024-03-24T10:47:55.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.739+0000 7f74b09af640 1 -- 172.21.15.5:0/3751750176 wait complete. 2024-03-24T10:47:55.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.740+0000 7f74b09af640 1 Processor -- start 2024-03-24T10:47:55.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.741+0000 7f74b09af640 1 -- start start 2024-03-24T10:47:55.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.742+0000 7f74b09af640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74ac06bf50 0x7f74ac1ef840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:55.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.742+0000 7f74b09af640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74ac06d4f0 0x7f74ac1581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:55.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.742+0000 7f74b09af640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f74ac1543a0 0x7f74ac0ced00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:55.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.742+0000 7f74b09af640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f74ac0ce5d0 con 0x7f74ac06d4f0 2024-03-24T10:47:55.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.742+0000 7f74b09af640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f74ac15a030 con 0x7f74ac06bf50 2024-03-24T10:47:55.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.742+0000 7f74b09af640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f74ac0ce870 con 0x7f74ac1543a0 2024-03-24T10:47:55.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.742+0000 7f74a9d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74ac06d4f0 0x7f74ac1581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:55.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.742+0000 7f74aad9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f74ac1543a0 0x7f74ac0ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:55.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.742+0000 7f74aa59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74ac06bf50 0x7f74ac1ef840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:55.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.743+0000 7f74aad9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f74ac1543a0 0x7f74ac0ced00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:46918/0 (socket says 172.21.15.5:46918) 2024-03-24T10:47:55.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.743+0000 7f74a9d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74ac06d4f0 0x7f74ac1581d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42262/0 (socket says 172.21.15.5:42262) 2024-03-24T10:47:55.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.743+0000 7f74aad9d640 1 -- 172.21.15.5:0/1045631347 learned_addr learned my addr 172.21.15.5:0/1045631347 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:47:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.746+0000 7f74a9d9b640 1 -- 172.21.15.5:0/1045631347 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f74ac1543a0 msgr2=0x7f74ac0ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.746+0000 7f74a9d9b640 1 --2- 172.21.15.5:0/1045631347 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f74ac1543a0 0x7f74ac0ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.746+0000 7f74a9d9b640 1 -- 172.21.15.5:0/1045631347 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74ac06bf50 msgr2=0x7f74ac1ef840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.746+0000 7f74a9d9b640 1 --2- 172.21.15.5:0/1045631347 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74ac06bf50 0x7f74ac1ef840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.746+0000 7f74a9d9b640 1 -- 172.21.15.5:0/1045631347 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a4009a70 con 0x7f74ac06d4f0 2024-03-24T10:47:55.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.749+0000 7f74a9d9b640 1 --2- 172.21.15.5:0/1045631347 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74ac06d4f0 0x7f74ac1581d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f74a000c370 tx=0x7f74a000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:55.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.752+0000 7f749affd640 1 -- 172.21.15.5:0/1045631347 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0005ce0 con 0x7f74ac06d4f0 2024-03-24T10:47:55.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.752+0000 7f749affd640 1 -- 172.21.15.5:0/1045631347 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74a0005e40 con 0x7f74ac06d4f0 2024-03-24T10:47:55.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.752+0000 7f74b09af640 1 -- 172.21.15.5:0/1045631347 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74ac152a00 con 0x7f74ac06d4f0 2024-03-24T10:47:55.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.752+0000 7f749affd640 1 -- 172.21.15.5:0/1045631347 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0007310 con 0x7f74ac06d4f0 2024-03-24T10:47:55.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.752+0000 7f74b09af640 1 -- 172.21.15.5:0/1045631347 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74ac1594b0 con 0x7f74ac06d4f0 2024-03-24T10:47:55.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:55.754+0000 7f74b09af640 1 -- 172.21.15.5:0/1045631347 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74ac0cea20 con 0x7f74ac06d4f0 2024-03-24T10:47:56.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:56.034+0000 7f749affd640 1 -- 172.21.15.5:0/1045631347 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f74a0007470 con 0x7f74ac06d4f0 2024-03-24T10:47:56.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:56.035+0000 7f749affd640 1 --2- 172.21.15.5:0/1045631347 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f748c06b860 0x7f748c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:56.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:56.035+0000 7f74aa59c640 1 --2- 172.21.15.5:0/1045631347 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f748c06b860 0x7f748c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:56.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:56.035+0000 7f74aa59c640 1 --2- 172.21.15.5:0/1045631347 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f748c06b860 0x7f748c06dd20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f749c0079a0 tx=0x7f749c008040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:56.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:56.054+0000 7f749affd640 1 -- 172.21.15.5:0/1045631347 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f74a0007720 con 0x7f74ac06d4f0 2024-03-24T10:47:56.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:56.054+0000 7f749affd640 1 -- 172.21.15.5:0/1045631347 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f74a0054420 con 0x7f74ac06d4f0 2024-03-24T10:47:56.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:56.285+0000 7f74b09af640 1 -- 172.21.15.5:0/1045631347 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74ac0ce9e0 con 0x7f74ac06d4f0 2024-03-24T10:47:56.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:56.379+0000 7f749affd640 1 -- 172.21.15.5:0/1045631347 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f74a0017020 con 0x7f74ac06d4f0 2024-03-24T10:47:56.379 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:47:56.379 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:47:56.379 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:47:56.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:56.382+0000 7f74b09af640 1 -- 172.21.15.5:0/1045631347 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f748c06b860 msgr2=0x7f748c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:56.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:56.382+0000 7f74b09af640 1 --2- 172.21.15.5:0/1045631347 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f748c06b860 0x7f748c06dd20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f749c0079a0 tx=0x7f749c008040 comp rx=0 tx=0).stop 2024-03-24T10:47:56.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:56.382+0000 7f74b09af640 1 -- 172.21.15.5:0/1045631347 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74ac06d4f0 msgr2=0x7f74ac1581d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:56.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:56.382+0000 7f74b09af640 1 --2- 172.21.15.5:0/1045631347 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74ac06d4f0 0x7f74ac1581d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f74a000c370 tx=0x7f74a000c830 comp rx=0 tx=0).stop 2024-03-24T10:47:56.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:56.382+0000 7f74b09af640 1 -- 172.21.15.5:0/1045631347 shutdown_connections 2024-03-24T10:47:56.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:56.382+0000 7f74b09af640 1 --2- 172.21.15.5:0/1045631347 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f748c06b860 0x7f748c06dd20 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:56.382+0000 7f74b09af640 1 --2- 172.21.15.5:0/1045631347 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f74ac06d4f0 0x7f74ac1581d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:56.382+0000 7f74b09af640 1 --2- 172.21.15.5:0/1045631347 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f74ac1543a0 0x7f74ac0ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:56.382+0000 7f74b09af640 1 --2- 172.21.15.5:0/1045631347 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f74ac06bf50 0x7f74ac1ef840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:56.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:56.382+0000 7f74b09af640 1 -- 172.21.15.5:0/1045631347 >> 172.21.15.5:0/1045631347 conn(0x7f74ac072ef0 msgr2=0x7f74ac149430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:56.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:56.382+0000 7f74b09af640 1 -- 172.21.15.5:0/1045631347 shutdown_connections 2024-03-24T10:47:56.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:56.382+0000 7f74b09af640 1 -- 172.21.15.5:0/1045631347 wait complete. 2024-03-24T10:47:57.000 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:47:57.000 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:47:58.002 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:47:58.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.104+0000 7fbf91525640 1 Processor -- start 2024-03-24T10:47:58.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.104+0000 7fbf91525640 1 -- start start 2024-03-24T10:47:58.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.105+0000 7fbf91525640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf8c06bf50 0x7fbf8c06c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:58.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.105+0000 7fbf91525640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf8c0cb2f0 con 0x7fbf8c15ac00 2024-03-24T10:47:58.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.105+0000 7fbf91525640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf8c15b070 con 0x7fbf8c1543a0 2024-03-24T10:47:58.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.105+0000 7fbf91525640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fbf8c0c6750 con 0x7fbf8c06bf50 2024-03-24T10:47:58.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.105+0000 7fbf8a7fc640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fbf8c15ac00 0x7fbf8c0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:49014/0 (socket says 172.21.15.5:49014) 2024-03-24T10:47:58.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.105+0000 7fbf8a7fc640 1 -- 172.21.15.5:0/300053527 learned_addr learned my addr 172.21.15.5:0/300053527 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:47:58.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.105+0000 7fbf8b7fe640 1 --2- 172.21.15.5:0/300053527 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf8c06bf50 0x7fbf8c06c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:58.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.108+0000 7fbf89ffb640 1 -- 172.21.15.5:0/300053527 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1398232811 0 0) 0x7fbf8c0cb2f0 con 0x7fbf8c15ac00 2024-03-24T10:47:58.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.108+0000 7fbf89ffb640 1 -- 172.21.15.5:0/300053527 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf74003520 con 0x7fbf8c15ac00 2024-03-24T10:47:58.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.108+0000 7fbf89ffb640 1 -- 172.21.15.5:0/300053527 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2484640229 0 0) 0x7fbf8c15b070 con 0x7fbf8c1543a0 2024-03-24T10:47:58.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.108+0000 7fbf89ffb640 1 -- 172.21.15.5:0/300053527 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf8c0cb2f0 con 0x7fbf8c1543a0 2024-03-24T10:47:58.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.109+0000 7fbf8b7fe640 1 -- 172.21.15.5:0/300053527 >> v1:172.21.15.57:6789/0 conn(0x7fbf8c1543a0 legacy=0x7fbf8c06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:58.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.109+0000 7fbf8b7fe640 1 -- 172.21.15.5:0/300053527 >> v1:172.21.15.5:6789/0 conn(0x7fbf8c15ac00 legacy=0x7fbf8c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:58.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.109+0000 7fbf8b7fe640 1 -- 172.21.15.5:0/300053527 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8c0ca7e0 con 0x7fbf8c06bf50 2024-03-24T10:47:58.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.113+0000 7fbf8b7fe640 1 --2- 172.21.15.5:0/300053527 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf8c06bf50 0x7fbf8c06c360 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009dc0 tx=0x7fbf8402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=144a199c4886d334 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:58.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.118+0000 7fbf89ffb640 1 -- 172.21.15.5:0/300053527 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8402ccd0 con 0x7fbf8c06bf50 2024-03-24T10:47:58.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.118+0000 7fbf89ffb640 1 -- 172.21.15.5:0/300053527 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf8402ce30 con 0x7fbf8c06bf50 2024-03-24T10:47:58.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.118+0000 7fbf89ffb640 1 -- 172.21.15.5:0/300053527 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf84034660 con 0x7fbf8c06bf50 2024-03-24T10:47:58.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.119+0000 7fbf91525640 1 -- 172.21.15.5:0/300053527 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf8c06bf50 msgr2=0x7fbf8c06c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:58.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.119+0000 7fbf91525640 1 --2- 172.21.15.5:0/300053527 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf8c06bf50 0x7fbf8c06c360 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009dc0 tx=0x7fbf8402bf20 comp rx=0 tx=0).stop 2024-03-24T10:47:58.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.119+0000 7fbf91525640 1 -- 172.21.15.5:0/300053527 shutdown_connections 2024-03-24T10:47:58.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.119+0000 7fbf91525640 1 --2- 172.21.15.5:0/300053527 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf8c06bf50 0x7fbf8c06c360 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:58.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.119+0000 7fbf91525640 1 -- 172.21.15.5:0/300053527 >> 172.21.15.5:0/300053527 conn(0x7fbf8c072ef0 msgr2=0x7fbf8c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:58.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.119+0000 7fbf91525640 1 -- 172.21.15.5:0/300053527 shutdown_connections 2024-03-24T10:47:58.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.119+0000 7fbf91525640 1 -- 172.21.15.5:0/300053527 wait complete. 2024-03-24T10:47:58.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.120+0000 7fbf91525640 1 Processor -- start 2024-03-24T10:47:58.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.120+0000 7fbf91525640 1 -- start start 2024-03-24T10:47:58.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.121+0000 7fbf91525640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf8c06bf50 0x7fbf8c14a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:58.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.121+0000 7fbf91525640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf8c1543a0 0x7fbf8c1551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:58.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.121+0000 7fbf91525640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf8c15ac00 0x7fbf8c155710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:58.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.121+0000 7fbf91525640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8c14adb0 con 0x7fbf8c06bf50 2024-03-24T10:47:58.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.121+0000 7fbf91525640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8c0cf890 con 0x7fbf8c1543a0 2024-03-24T10:47:58.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.121+0000 7fbf91525640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fbf8c154a60 con 0x7fbf8c15ac00 2024-03-24T10:47:58.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.122+0000 7fbf8affd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf8c06bf50 0x7fbf8c14a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:58.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.122+0000 7fbf8a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf8c1543a0 0x7fbf8c1551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:58.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.122+0000 7fbf8b7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf8c15ac00 0x7fbf8c155710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:58.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.122+0000 7fbf8affd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf8c06bf50 0x7fbf8c14a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42270/0 (socket says 172.21.15.5:42270) 2024-03-24T10:47:58.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.122+0000 7fbf8affd640 1 -- 172.21.15.5:0/3458546623 learned_addr learned my addr 172.21.15.5:0/3458546623 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:47:58.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.125+0000 7fbf8affd640 1 -- 172.21.15.5:0/3458546623 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf8c15ac00 msgr2=0x7fbf8c155710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:58.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.125+0000 7fbf8affd640 1 --2- 172.21.15.5:0/3458546623 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf8c15ac00 0x7fbf8c155710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:58.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.125+0000 7fbf8affd640 1 -- 172.21.15.5:0/3458546623 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf8c1543a0 msgr2=0x7fbf8c1551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:58.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.125+0000 7fbf8affd640 1 --2- 172.21.15.5:0/3458546623 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf8c1543a0 0x7fbf8c1551d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:58.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.126+0000 7fbf8affd640 1 -- 172.21.15.5:0/3458546623 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf84009a70 con 0x7fbf8c06bf50 2024-03-24T10:47:58.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.128+0000 7fbf8affd640 1 --2- 172.21.15.5:0/3458546623 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf8c06bf50 0x7fbf8c14a870 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00c490 tx=0x7fbf7c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:58.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.131+0000 7fbf737fe640 1 -- 172.21.15.5:0/3458546623 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c0052d0 con 0x7fbf8c06bf50 2024-03-24T10:47:58.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.132+0000 7fbf737fe640 1 -- 172.21.15.5:0/3458546623 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf7c005430 con 0x7fbf8c06bf50 2024-03-24T10:47:58.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.132+0000 7fbf737fe640 1 -- 172.21.15.5:0/3458546623 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c003830 con 0x7fbf8c06bf50 2024-03-24T10:47:58.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.132+0000 7fbf91525640 1 -- 172.21.15.5:0/3458546623 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf8c1f0610 con 0x7fbf8c06bf50 2024-03-24T10:47:58.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.132+0000 7fbf91525640 1 -- 172.21.15.5:0/3458546623 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf8c1f0bb0 con 0x7fbf8c06bf50 2024-03-24T10:47:58.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.133+0000 7fbf91525640 1 -- 172.21.15.5:0/3458546623 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf8c0681d0 con 0x7fbf8c06bf50 2024-03-24T10:47:58.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.399+0000 7fbf737fe640 1 -- 172.21.15.5:0/3458546623 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbf7c003990 con 0x7fbf8c06bf50 2024-03-24T10:47:58.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.399+0000 7fbf737fe640 1 --2- 172.21.15.5:0/3458546623 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbf7806b810 0x7fbf7806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:47:58.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.399+0000 7fbf8a7fc640 1 --2- 172.21.15.5:0/3458546623 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbf7806b810 0x7fbf7806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:47:58.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.400+0000 7fbf8a7fc640 1 --2- 172.21.15.5:0/3458546623 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbf7806b810 0x7fbf7806dcd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c14c850 tx=0x7fbf80007cf0 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:47:58.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.429+0000 7fbf737fe640 1 -- 172.21.15.5:0/3458546623 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fbf7c058420 con 0x7fbf8c06bf50 2024-03-24T10:47:58.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.430+0000 7fbf737fe640 1 -- 172.21.15.5:0/3458546623 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbf7c0546c0 con 0x7fbf8c06bf50 2024-03-24T10:47:58.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.659+0000 7fbf91525640 1 -- 172.21.15.5:0/3458546623 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf8c1514a0 con 0x7fbf8c06bf50 2024-03-24T10:47:58.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.680+0000 7fbf737fe640 1 -- 172.21.15.5:0/3458546623 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fbf7c017020 con 0x7fbf8c06bf50 2024-03-24T10:47:58.681 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:47:58.681 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:47:58.681 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:47:58.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.682+0000 7fbf91525640 1 -- 172.21.15.5:0/3458546623 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbf7806b810 msgr2=0x7fbf7806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:58.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.682+0000 7fbf91525640 1 --2- 172.21.15.5:0/3458546623 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbf7806b810 0x7fbf7806dcd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c14c850 tx=0x7fbf80007cf0 comp rx=0 tx=0).stop 2024-03-24T10:47:58.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.682+0000 7fbf91525640 1 -- 172.21.15.5:0/3458546623 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf8c06bf50 msgr2=0x7fbf8c14a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:47:58.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.682+0000 7fbf91525640 1 --2- 172.21.15.5:0/3458546623 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf8c06bf50 0x7fbf8c14a870 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00c490 tx=0x7fbf7c00c950 comp rx=0 tx=0).stop 2024-03-24T10:47:58.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.682+0000 7fbf91525640 1 -- 172.21.15.5:0/3458546623 shutdown_connections 2024-03-24T10:47:58.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.682+0000 7fbf91525640 1 --2- 172.21.15.5:0/3458546623 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbf7806b810 0x7fbf7806dcd0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:58.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.682+0000 7fbf91525640 1 --2- 172.21.15.5:0/3458546623 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf8c15ac00 0x7fbf8c155710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:58.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.682+0000 7fbf91525640 1 --2- 172.21.15.5:0/3458546623 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf8c1543a0 0x7fbf8c1551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:58.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.682+0000 7fbf91525640 1 --2- 172.21.15.5:0/3458546623 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf8c06bf50 0x7fbf8c14a870 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:47:58.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.682+0000 7fbf91525640 1 -- 172.21.15.5:0/3458546623 >> 172.21.15.5:0/3458546623 conn(0x7fbf8c072ef0 msgr2=0x7fbf8c0b48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:47:58.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.683+0000 7fbf91525640 1 -- 172.21.15.5:0/3458546623 shutdown_connections 2024-03-24T10:47:58.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:47:58.683+0000 7fbf91525640 1 -- 172.21.15.5:0/3458546623 wait complete. 2024-03-24T10:47:59.299 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:47:59.299 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:00.300 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:00.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.402+0000 7fe8a5f3a640 1 Processor -- start 2024-03-24T10:48:00.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.402+0000 7fe8a5f3a640 1 -- start start 2024-03-24T10:48:00.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.402+0000 7fe8a5f3a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8a006bf50 0x7fe8a006c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:00.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.402+0000 7fe8a5f3a640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8a00cb2f0 con 0x7fe8a015ac00 2024-03-24T10:48:00.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.402+0000 7fe8a5f3a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8a015b070 con 0x7fe8a01543a0 2024-03-24T10:48:00.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.402+0000 7fe8a5f3a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe8a00c6750 con 0x7fe8a006bf50 2024-03-24T10:48:00.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.403+0000 7fe89effd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fe8a015ac00 0x7fe8a00cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:49020/0 (socket says 172.21.15.5:49020) 2024-03-24T10:48:00.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.403+0000 7fe89effd640 1 -- 172.21.15.5:0/2903708572 learned_addr learned my addr 172.21.15.5:0/2903708572 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:48:00.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.403+0000 7fe89ffff640 1 --2- 172.21.15.5:0/2903708572 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8a006bf50 0x7fe8a006c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:00.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.406+0000 7fe89e7fc640 1 -- 172.21.15.5:0/2903708572 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1872420001 0 0) 0x7fe8a00cb2f0 con 0x7fe8a015ac00 2024-03-24T10:48:00.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.406+0000 7fe89e7fc640 1 -- 172.21.15.5:0/2903708572 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe888003520 con 0x7fe8a015ac00 2024-03-24T10:48:00.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.406+0000 7fe89e7fc640 1 -- 172.21.15.5:0/2903708572 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 368895592 0 0) 0x7fe8a015b070 con 0x7fe8a01543a0 2024-03-24T10:48:00.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.406+0000 7fe89e7fc640 1 -- 172.21.15.5:0/2903708572 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8a00cb2f0 con 0x7fe8a01543a0 2024-03-24T10:48:00.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.407+0000 7fe89ffff640 1 -- 172.21.15.5:0/2903708572 >> v1:172.21.15.57:6789/0 conn(0x7fe8a01543a0 legacy=0x7fe8a006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:00.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.407+0000 7fe89ffff640 1 -- 172.21.15.5:0/2903708572 >> v1:172.21.15.5:6789/0 conn(0x7fe8a015ac00 legacy=0x7fe8a00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:00.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.407+0000 7fe89ffff640 1 -- 172.21.15.5:0/2903708572 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a00ca7e0 con 0x7fe8a006bf50 2024-03-24T10:48:00.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.410+0000 7fe89ffff640 1 --2- 172.21.15.5:0/2903708572 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8a006bf50 0x7fe8a006c360 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe898009dc0 tx=0x7fe89802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93b0224452755856 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:00.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.413+0000 7fe89e7fc640 1 -- 172.21.15.5:0/2903708572 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe89802ccd0 con 0x7fe8a006bf50 2024-03-24T10:48:00.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.415+0000 7fe89e7fc640 1 -- 172.21.15.5:0/2903708572 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe89802ce30 con 0x7fe8a006bf50 2024-03-24T10:48:00.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.416+0000 7fe89e7fc640 1 -- 172.21.15.5:0/2903708572 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe898034640 con 0x7fe8a006bf50 2024-03-24T10:48:00.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.416+0000 7fe8a5f3a640 1 -- 172.21.15.5:0/2903708572 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8a006bf50 msgr2=0x7fe8a006c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:00.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.416+0000 7fe8a5f3a640 1 --2- 172.21.15.5:0/2903708572 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8a006bf50 0x7fe8a006c360 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe898009dc0 tx=0x7fe89802bf20 comp rx=0 tx=0).stop 2024-03-24T10:48:00.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.417+0000 7fe8a5f3a640 1 -- 172.21.15.5:0/2903708572 shutdown_connections 2024-03-24T10:48:00.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.417+0000 7fe8a5f3a640 1 --2- 172.21.15.5:0/2903708572 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8a006bf50 0x7fe8a006c360 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:00.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.417+0000 7fe8a5f3a640 1 -- 172.21.15.5:0/2903708572 >> 172.21.15.5:0/2903708572 conn(0x7fe8a0072ef0 msgr2=0x7fe8a0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:00.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.417+0000 7fe8a5f3a640 1 -- 172.21.15.5:0/2903708572 shutdown_connections 2024-03-24T10:48:00.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.417+0000 7fe8a5f3a640 1 -- 172.21.15.5:0/2903708572 wait complete. 2024-03-24T10:48:00.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.418+0000 7fe8a5f3a640 1 Processor -- start 2024-03-24T10:48:00.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.419+0000 7fe8a5f3a640 1 -- start start 2024-03-24T10:48:00.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.420+0000 7fe8a5f3a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8a006bf50 0x7fe8a00c6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:00.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.420+0000 7fe8a5f3a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8a01543a0 0x7fe8a01f07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:00.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.420+0000 7fe8a5f3a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8a015ac00 0x7fe8a0156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:00.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.420+0000 7fe8a5f3a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a00c8900 con 0x7fe8a015ac00 2024-03-24T10:48:00.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.420+0000 7fe8a5f3a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a0154a60 con 0x7fe8a006bf50 2024-03-24T10:48:00.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.420+0000 7fe8a5f3a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe8a00c8b50 con 0x7fe8a01543a0 2024-03-24T10:48:00.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.420+0000 7fe89ffff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8a015ac00 0x7fe8a0156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:00.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.421+0000 7fe89f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8a006bf50 0x7fe8a00c6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:00.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.421+0000 7fe89ffff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8a015ac00 0x7fe8a0156a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42286/0 (socket says 172.21.15.5:42286) 2024-03-24T10:48:00.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.421+0000 7fe89ffff640 1 -- 172.21.15.5:0/1182161015 learned_addr learned my addr 172.21.15.5:0/1182161015 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:00.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.422+0000 7fe89effd640 1 --2- 172.21.15.5:0/1182161015 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8a01543a0 0x7fe8a01f07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:00.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.424+0000 7fe89ffff640 1 -- 172.21.15.5:0/1182161015 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8a01543a0 msgr2=0x7fe8a01f07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:00.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.424+0000 7fe89ffff640 1 --2- 172.21.15.5:0/1182161015 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8a01543a0 0x7fe8a01f07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:00.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.424+0000 7fe89ffff640 1 -- 172.21.15.5:0/1182161015 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8a006bf50 msgr2=0x7fe8a00c6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:00.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.424+0000 7fe89ffff640 1 --2- 172.21.15.5:0/1182161015 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8a006bf50 0x7fe8a00c6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:00.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.424+0000 7fe89ffff640 1 -- 172.21.15.5:0/1182161015 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe898009a70 con 0x7fe8a015ac00 2024-03-24T10:48:00.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.427+0000 7fe89ffff640 1 --2- 172.21.15.5:0/1182161015 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8a015ac00 0x7fe8a0156a00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe898037040 tx=0x7fe898004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:00.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.431+0000 7fe887fff640 1 -- 172.21.15.5:0/1182161015 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe898004800 con 0x7fe8a015ac00 2024-03-24T10:48:00.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.431+0000 7fe887fff640 1 -- 172.21.15.5:0/1182161015 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe898009450 con 0x7fe8a015ac00 2024-03-24T10:48:00.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.431+0000 7fe887fff640 1 -- 172.21.15.5:0/1182161015 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe898040690 con 0x7fe8a015ac00 2024-03-24T10:48:00.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.431+0000 7fe8a5f3a640 1 -- 172.21.15.5:0/1182161015 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8a00c8e30 con 0x7fe8a015ac00 2024-03-24T10:48:00.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.431+0000 7fe8a5f3a640 1 -- 172.21.15.5:0/1182161015 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8a0154f90 con 0x7fe8a015ac00 2024-03-24T10:48:00.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.432+0000 7fe8a5f3a640 1 -- 172.21.15.5:0/1182161015 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8a00cceb0 con 0x7fe8a015ac00 2024-03-24T10:48:00.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.455+0000 7fe887fff640 1 -- 172.21.15.5:0/1182161015 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe8980407f0 con 0x7fe8a015ac00 2024-03-24T10:48:00.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.455+0000 7fe887fff640 1 --2- 172.21.15.5:0/1182161015 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe88c06b860 0x7fe88c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:00.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.455+0000 7fe89f7fe640 1 --2- 172.21.15.5:0/1182161015 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe88c06b860 0x7fe88c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:00.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.456+0000 7fe89f7fe640 1 --2- 172.21.15.5:0/1182161015 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe88c06b860 0x7fe88c06dd20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe8900011e0 tx=0x7fe890009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:00.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.459+0000 7fe887fff640 1 -- 172.21.15.5:0/1182161015 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fe89807bbf0 con 0x7fe8a015ac00 2024-03-24T10:48:00.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.715+0000 7fe887fff640 1 -- 172.21.15.5:0/1182161015 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe898077e90 con 0x7fe8a015ac00 2024-03-24T10:48:00.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.939+0000 7fe8a5f3a640 1 -- 172.21.15.5:0/1182161015 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8a00681d0 con 0x7fe8a015ac00 2024-03-24T10:48:00.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.953+0000 7fe887fff640 1 -- 172.21.15.5:0/1182161015 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fe898044090 con 0x7fe8a015ac00 2024-03-24T10:48:00.953 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:00.953 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:00.954 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:00.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.955+0000 7fe8a5f3a640 1 -- 172.21.15.5:0/1182161015 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe88c06b860 msgr2=0x7fe88c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:00.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.955+0000 7fe8a5f3a640 1 --2- 172.21.15.5:0/1182161015 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe88c06b860 0x7fe88c06dd20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe8900011e0 tx=0x7fe890009040 comp rx=0 tx=0).stop 2024-03-24T10:48:00.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.955+0000 7fe8a5f3a640 1 -- 172.21.15.5:0/1182161015 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8a015ac00 msgr2=0x7fe8a0156a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:00.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.955+0000 7fe8a5f3a640 1 --2- 172.21.15.5:0/1182161015 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8a015ac00 0x7fe8a0156a00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe898037040 tx=0x7fe898004a40 comp rx=0 tx=0).stop 2024-03-24T10:48:00.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.955+0000 7fe8a5f3a640 1 -- 172.21.15.5:0/1182161015 shutdown_connections 2024-03-24T10:48:00.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.955+0000 7fe8a5f3a640 1 --2- 172.21.15.5:0/1182161015 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe88c06b860 0x7fe88c06dd20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:00.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.955+0000 7fe8a5f3a640 1 --2- 172.21.15.5:0/1182161015 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8a015ac00 0x7fe8a0156a00 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:00.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.955+0000 7fe8a5f3a640 1 --2- 172.21.15.5:0/1182161015 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8a01543a0 0x7fe8a01f07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:00.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.955+0000 7fe8a5f3a640 1 --2- 172.21.15.5:0/1182161015 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe8a006bf50 0x7fe8a00c6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:00.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.955+0000 7fe8a5f3a640 1 -- 172.21.15.5:0/1182161015 >> 172.21.15.5:0/1182161015 conn(0x7fe8a0072ef0 msgr2=0x7fe8a0152620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:00.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.955+0000 7fe8a5f3a640 1 -- 172.21.15.5:0/1182161015 shutdown_connections 2024-03-24T10:48:00.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:00.955+0000 7fe8a5f3a640 1 -- 172.21.15.5:0/1182161015 wait complete. 2024-03-24T10:48:01.488 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:48:01.490 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:48:01.569 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:01.569 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:02.571 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:02.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.673+0000 7f7d14047640 1 Processor -- start 2024-03-24T10:48:02.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.673+0000 7f7d14047640 1 -- start start 2024-03-24T10:48:02.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.674+0000 7f7d14047640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7d0c1543a0 0x7f7d0c15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:02.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.674+0000 7f7d14047640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7d0c06bf50 0x7f7d0c06c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:02.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.674+0000 7f7d14047640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d0c0cb2f0 con 0x7f7d0c06d4f0 2024-03-24T10:48:02.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.674+0000 7f7d14047640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f7d0c0c6750 con 0x7f7d0c1543a0 2024-03-24T10:48:02.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.674+0000 7f7d14047640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f7d0c0cc380 con 0x7f7d0c06bf50 2024-03-24T10:48:02.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.674+0000 7f7d11de3640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7d0c1543a0 0x7f7d0c15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:02.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.674+0000 7f7d125e4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7d0c06bf50 0x7f7d0c06c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:02.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.675+0000 7f7d11de3640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7d0c1543a0 0x7f7d0c15ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42260/0 (socket says 172.21.15.5:42260) 2024-03-24T10:48:02.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.675+0000 7f7d11de3640 1 -- 172.21.15.5:0/1826459869 learned_addr learned my addr 172.21.15.5:0/1826459869 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:02.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.678+0000 7f7d11de3640 1 -- 172.21.15.5:0/1826459869 >> v2:172.21.15.57:3300/0 conn(0x7f7d0c06bf50 msgr2=0x7f7d0c06c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:02.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.678+0000 7f7d11de3640 1 --2- 172.21.15.5:0/1826459869 >> v2:172.21.15.57:3300/0 conn(0x7f7d0c06bf50 0x7f7d0c06c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:02.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.678+0000 7f7d11de3640 1 -- 172.21.15.5:0/1826459869 >> v1:172.21.15.57:6789/0 conn(0x7f7d0c06d4f0 legacy=0x7f7d0c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:02.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.678+0000 7f7d11de3640 1 -- 172.21.15.5:0/1826459869 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d0c0ca7e0 con 0x7f7d0c1543a0 2024-03-24T10:48:02.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.681+0000 7f7d11de3640 1 --2- 172.21.15.5:0/1826459869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d0c1543a0 0x7f7d0c15ac00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0b51c0 tx=0x7f7d0802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab595b6a41a48f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:02.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.686+0000 7f7d10de1640 1 -- 172.21.15.5:0/1826459869 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08030860 con 0x7f7d0c1543a0 2024-03-24T10:48:02.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.686+0000 7f7d10de1640 1 -- 172.21.15.5:0/1826459869 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d080309c0 con 0x7f7d0c1543a0 2024-03-24T10:48:02.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.686+0000 7f7d10de1640 1 -- 172.21.15.5:0/1826459869 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08030d50 con 0x7f7d0c1543a0 2024-03-24T10:48:02.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.687+0000 7f7d14047640 1 -- 172.21.15.5:0/1826459869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d0c1543a0 msgr2=0x7f7d0c15ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:02.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.687+0000 7f7d14047640 1 --2- 172.21.15.5:0/1826459869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d0c1543a0 0x7f7d0c15ac00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0b51c0 tx=0x7f7d0802b9a0 comp rx=0 tx=0).stop 2024-03-24T10:48:02.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.687+0000 7f7d14047640 1 -- 172.21.15.5:0/1826459869 shutdown_connections 2024-03-24T10:48:02.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.687+0000 7f7d14047640 1 --2- 172.21.15.5:0/1826459869 >> v2:172.21.15.57:3300/0 conn(0x7f7d0c06bf50 0x7f7d0c06c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:02.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.687+0000 7f7d14047640 1 --2- 172.21.15.5:0/1826459869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d0c1543a0 0x7f7d0c15ac00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:02.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.687+0000 7f7d14047640 1 -- 172.21.15.5:0/1826459869 >> 172.21.15.5:0/1826459869 conn(0x7f7d0c072ef0 msgr2=0x7f7d0c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:02.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.687+0000 7f7d14047640 1 -- 172.21.15.5:0/1826459869 shutdown_connections 2024-03-24T10:48:02.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.687+0000 7f7d14047640 1 -- 172.21.15.5:0/1826459869 wait complete. 2024-03-24T10:48:02.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.688+0000 7f7d14047640 1 Processor -- start 2024-03-24T10:48:02.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.688+0000 7f7d14047640 1 -- start start 2024-03-24T10:48:02.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.689+0000 7f7d14047640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d0c06bf50 0x7f7d0c1559a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:02.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.689+0000 7f7d14047640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7d0c06d4f0 0x7f7d0c1573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:02.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.689+0000 7f7d14047640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d0c1543a0 0x7f7d0c154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:02.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.689+0000 7f7d14047640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c1590e0 con 0x7f7d0c1543a0 2024-03-24T10:48:02.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.689+0000 7f7d14047640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c06dab0 con 0x7f7d0c06bf50 2024-03-24T10:48:02.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.689+0000 7f7d14047640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7d0c15bb10 con 0x7f7d0c06d4f0 2024-03-24T10:48:02.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.690+0000 7f7d125e4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d0c1543a0 0x7f7d0c154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:02.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.690+0000 7f7d11de3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d0c06bf50 0x7f7d0c1559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:02.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.690+0000 7f7d115e2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7d0c06d4f0 0x7f7d0c1573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:02.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.690+0000 7f7d125e4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d0c1543a0 0x7f7d0c154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42272/0 (socket says 172.21.15.5:42272) 2024-03-24T10:48:02.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.690+0000 7f7d125e4640 1 -- 172.21.15.5:0/2761659484 learned_addr learned my addr 172.21.15.5:0/2761659484 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:02.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.694+0000 7f7d11de3640 1 -- 172.21.15.5:0/2761659484 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7d0c06d4f0 msgr2=0x7f7d0c1573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:02.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.694+0000 7f7d11de3640 1 --2- 172.21.15.5:0/2761659484 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7d0c06d4f0 0x7f7d0c1573b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:02.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.694+0000 7f7d11de3640 1 -- 172.21.15.5:0/2761659484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d0c1543a0 msgr2=0x7f7d0c154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:02.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.694+0000 7f7d11de3640 1 --2- 172.21.15.5:0/2761659484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d0c1543a0 0x7f7d0c154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:02.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.694+0000 7f7d11de3640 1 -- 172.21.15.5:0/2761659484 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d080094d0 con 0x7f7d0c06bf50 2024-03-24T10:48:02.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.696+0000 7f7d11de3640 1 --2- 172.21.15.5:0/2761659484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d0c06bf50 0x7f7d0c1559a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7d08033000 tx=0x7f7d08004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:02.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.701+0000 7f7d027fc640 1 -- 172.21.15.5:0/2761659484 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08039a10 con 0x7f7d0c06bf50 2024-03-24T10:48:02.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.701+0000 7f7d027fc640 1 -- 172.21.15.5:0/2761659484 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d08039b90 con 0x7f7d0c06bf50 2024-03-24T10:48:02.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.701+0000 7f7d14047640 1 -- 172.21.15.5:0/2761659484 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d0c15b2b0 con 0x7f7d0c06bf50 2024-03-24T10:48:02.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.701+0000 7f7d027fc640 1 -- 172.21.15.5:0/2761659484 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d080433f0 con 0x7f7d0c06bf50 2024-03-24T10:48:02.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.701+0000 7f7d14047640 1 -- 172.21.15.5:0/2761659484 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d0c15b830 con 0x7f7d0c06bf50 2024-03-24T10:48:02.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.702+0000 7f7d14047640 1 -- 172.21.15.5:0/2761659484 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d0c0681d0 con 0x7f7d0c06bf50 2024-03-24T10:48:02.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.971+0000 7f7d027fc640 1 -- 172.21.15.5:0/2761659484 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7d0804c3b0 con 0x7f7d0c06bf50 2024-03-24T10:48:02.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.972+0000 7f7d027fc640 1 --2- 172.21.15.5:0/2761659484 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7cf006b810 0x7f7cf006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:02.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.972+0000 7f7d115e2640 1 --2- 172.21.15.5:0/2761659484 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7cf006b810 0x7f7cf006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:02.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.972+0000 7f7d115e2640 1 --2- 172.21.15.5:0/2761659484 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7cf006b810 0x7f7cf006dcd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0cc280 tx=0x7f7cfc007e30 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:02.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.978+0000 7f7d027fc640 1 -- 172.21.15.5:0/2761659484 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f7d0807aeb0 con 0x7f7d0c06bf50 2024-03-24T10:48:02.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:02.989+0000 7f7d027fc640 1 -- 172.21.15.5:0/2761659484 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7d08077150 con 0x7f7d0c06bf50 2024-03-24T10:48:03.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:03.227+0000 7f7d14047640 1 -- 172.21.15.5:0/2761659484 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d0c153860 con 0x7f7d0c06bf50 2024-03-24T10:48:03.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:03.249+0000 7f7d027fc640 1 -- 172.21.15.5:0/2761659484 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f7d08034090 con 0x7f7d0c06bf50 2024-03-24T10:48:03.250 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:03.250 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:03.250 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:03.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:03.251+0000 7f7d14047640 1 -- 172.21.15.5:0/2761659484 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7cf006b810 msgr2=0x7f7cf006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:03.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:03.251+0000 7f7d14047640 1 --2- 172.21.15.5:0/2761659484 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7cf006b810 0x7f7cf006dcd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0cc280 tx=0x7f7cfc007e30 comp rx=0 tx=0).stop 2024-03-24T10:48:03.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:03.251+0000 7f7d14047640 1 -- 172.21.15.5:0/2761659484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d0c06bf50 msgr2=0x7f7d0c1559a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:03.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:03.251+0000 7f7d14047640 1 --2- 172.21.15.5:0/2761659484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d0c06bf50 0x7f7d0c1559a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7d08033000 tx=0x7f7d08004730 comp rx=0 tx=0).stop 2024-03-24T10:48:03.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:03.251+0000 7f7d14047640 1 -- 172.21.15.5:0/2761659484 shutdown_connections 2024-03-24T10:48:03.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:03.251+0000 7f7d14047640 1 --2- 172.21.15.5:0/2761659484 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7cf006b810 0x7f7cf006dcd0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:03.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:03.252+0000 7f7d14047640 1 --2- 172.21.15.5:0/2761659484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d0c1543a0 0x7f7d0c154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:03.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:03.252+0000 7f7d14047640 1 --2- 172.21.15.5:0/2761659484 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7d0c06d4f0 0x7f7d0c1573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:03.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:03.252+0000 7f7d14047640 1 --2- 172.21.15.5:0/2761659484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d0c06bf50 0x7f7d0c1559a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:03.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:03.252+0000 7f7d14047640 1 -- 172.21.15.5:0/2761659484 >> 172.21.15.5:0/2761659484 conn(0x7f7d0c072ef0 msgr2=0x7f7d0c152a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:03.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:03.252+0000 7f7d14047640 1 -- 172.21.15.5:0/2761659484 shutdown_connections 2024-03-24T10:48:03.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:03.252+0000 7f7d14047640 1 -- 172.21.15.5:0/2761659484 wait complete. 2024-03-24T10:48:03.866 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:03.866 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:04.867 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:04.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.968+0000 7fcbdad43640 1 Processor -- start 2024-03-24T10:48:04.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.968+0000 7fcbdad43640 1 -- start start 2024-03-24T10:48:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.969+0000 7fcbdad43640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcbd40cae90 0x7fcbd40cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.969+0000 7fcbdad43640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcbd40cb270 0x7fcbd40cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.969+0000 7fcbdad43640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcbd40cd960 0x7fcbd40cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.969+0000 7fcbdad43640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fcbd40ccba0 con 0x7fcbd40cd960 2024-03-24T10:48:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.969+0000 7fcbdad43640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fcbd40cce40 con 0x7fcbd40cae90 2024-03-24T10:48:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.969+0000 7fcbdad43640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fcbd415a590 con 0x7fcbd40cb270 2024-03-24T10:48:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.970+0000 7fcbd92e0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcbd40cd960 0x7fcbd40cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:04.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.970+0000 7fcbd3fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcbd40cb270 0x7fcbd40cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:04.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.970+0000 7fcbd92e0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcbd40cd960 0x7fcbd40cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42284/0 (socket says 172.21.15.5:42284) 2024-03-24T10:48:04.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.970+0000 7fcbd92e0640 1 -- 172.21.15.5:0/2140443141 learned_addr learned my addr 172.21.15.5:0/2140443141 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:04.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.970+0000 7fcbd8adf640 1 --2- 172.21.15.5:0/2140443141 >> v2:172.21.15.57:3300/0 conn(0x7fcbd40cae90 0x7fcbd40cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:04.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.972+0000 7fcbd92e0640 1 -- 172.21.15.5:0/2140443141 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcbd40cb270 msgr2=0x7fcbd40cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:04.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.972+0000 7fcbd92e0640 1 --2- 172.21.15.5:0/2140443141 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcbd40cb270 0x7fcbd40cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:04.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.972+0000 7fcbd92e0640 1 -- 172.21.15.5:0/2140443141 >> v2:172.21.15.57:3300/0 conn(0x7fcbd40cae90 msgr2=0x7fcbd40cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:04.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.972+0000 7fcbd92e0640 1 --2- 172.21.15.5:0/2140443141 >> v2:172.21.15.57:3300/0 conn(0x7fcbd40cae90 0x7fcbd40cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:04.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.972+0000 7fcbd92e0640 1 -- 172.21.15.5:0/2140443141 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbd415a870 con 0x7fcbd40cd960 2024-03-24T10:48:04.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.975+0000 7fcbd92e0640 1 --2- 172.21.15.5:0/2140443141 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbd40cd960 0x7fcbd40cb6f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcbd40b26a0 tx=0x7fcbcc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80e4cfd7588168e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:04.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.979+0000 7fcbd37fe640 1 -- 172.21.15.5:0/2140443141 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc02cb00 con 0x7fcbd40cd960 2024-03-24T10:48:04.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.979+0000 7fcbd37fe640 1 -- 172.21.15.5:0/2140443141 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbcc02cc60 con 0x7fcbd40cd960 2024-03-24T10:48:04.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.979+0000 7fcbd37fe640 1 -- 172.21.15.5:0/2140443141 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc0343f0 con 0x7fcbd40cd960 2024-03-24T10:48:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.980+0000 7fcbdad43640 1 -- 172.21.15.5:0/2140443141 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbd40cd960 msgr2=0x7fcbd40cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.980+0000 7fcbdad43640 1 --2- 172.21.15.5:0/2140443141 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbd40cd960 0x7fcbd40cb6f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcbd40b26a0 tx=0x7fcbcc02bf60 comp rx=0 tx=0).stop 2024-03-24T10:48:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.980+0000 7fcbdad43640 1 -- 172.21.15.5:0/2140443141 shutdown_connections 2024-03-24T10:48:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.980+0000 7fcbdad43640 1 --2- 172.21.15.5:0/2140443141 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcbd40cb270 0x7fcbd40cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.980+0000 7fcbdad43640 1 --2- 172.21.15.5:0/2140443141 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbd40cd960 0x7fcbd40cb6f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:04.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.980+0000 7fcbdad43640 1 --2- 172.21.15.5:0/2140443141 >> v2:172.21.15.57:3300/0 conn(0x7fcbd40cae90 0x7fcbd40cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:04.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.980+0000 7fcbdad43640 1 -- 172.21.15.5:0/2140443141 >> 172.21.15.5:0/2140443141 conn(0x7fcbd40cfb80 msgr2=0x7fcbd40cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:04.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.980+0000 7fcbdad43640 1 -- 172.21.15.5:0/2140443141 shutdown_connections 2024-03-24T10:48:04.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.980+0000 7fcbdad43640 1 -- 172.21.15.5:0/2140443141 wait complete. 2024-03-24T10:48:04.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.982+0000 7fcbdad43640 1 Processor -- start 2024-03-24T10:48:04.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.982+0000 7fcbdad43640 1 -- start start 2024-03-24T10:48:04.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.983+0000 7fcbdad43640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbd40cae90 0x7fcbd40aee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:04.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.983+0000 7fcbdad43640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbd40cb270 0x7fcbd41ef7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:04.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.983+0000 7fcbdad43640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcbd40cd960 0x7fcbd4153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:04.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.983+0000 7fcbdad43640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd40c56d0 con 0x7fcbd40cae90 2024-03-24T10:48:04.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.983+0000 7fcbdad43640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd40aeb80 con 0x7fcbd40cb270 2024-03-24T10:48:04.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.983+0000 7fcbdad43640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fcbd41f0210 con 0x7fcbd40cd960 2024-03-24T10:48:04.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.983+0000 7fcbd3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbd40cb270 0x7fcbd41ef7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:04.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.983+0000 7fcbd8adf640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbd40cae90 0x7fcbd40aee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:04.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.984+0000 7fcbd92e0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcbd40cd960 0x7fcbd4153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:04.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.984+0000 7fcbd8adf640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbd40cae90 0x7fcbd40aee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42298/0 (socket says 172.21.15.5:42298) 2024-03-24T10:48:04.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.984+0000 7fcbd8adf640 1 -- 172.21.15.5:0/4141718510 learned_addr learned my addr 172.21.15.5:0/4141718510 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:04.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.987+0000 7fcbd8adf640 1 -- 172.21.15.5:0/4141718510 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcbd40cd960 msgr2=0x7fcbd4153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:04.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.987+0000 7fcbd8adf640 1 --2- 172.21.15.5:0/4141718510 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcbd40cd960 0x7fcbd4153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:04.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.987+0000 7fcbd8adf640 1 -- 172.21.15.5:0/4141718510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbd40cb270 msgr2=0x7fcbd41ef7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:04.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.987+0000 7fcbd8adf640 1 --2- 172.21.15.5:0/4141718510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbd40cb270 0x7fcbd41ef7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:04.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.987+0000 7fcbd8adf640 1 -- 172.21.15.5:0/4141718510 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbcc009b30 con 0x7fcbd40cae90 2024-03-24T10:48:04.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.990+0000 7fcbd8adf640 1 --2- 172.21.15.5:0/4141718510 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbd40cae90 0x7fcbd40aee60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800e890 tx=0x7fcbc800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:04.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.994+0000 7fcbd17fa640 1 -- 172.21.15.5:0/4141718510 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc800c940 con 0x7fcbd40cae90 2024-03-24T10:48:04.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.994+0000 7fcbd17fa640 1 -- 172.21.15.5:0/4141718510 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbc800caa0 con 0x7fcbd40cae90 2024-03-24T10:48:04.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.994+0000 7fcbdad43640 1 -- 172.21.15.5:0/4141718510 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbd40ab300 con 0x7fcbd40cae90 2024-03-24T10:48:04.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.994+0000 7fcbd17fa640 1 -- 172.21.15.5:0/4141718510 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc800ce30 con 0x7fcbd40cae90 2024-03-24T10:48:04.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.994+0000 7fcbdad43640 1 -- 172.21.15.5:0/4141718510 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbd41efce0 con 0x7fcbd40cae90 2024-03-24T10:48:04.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:04.995+0000 7fcbdad43640 1 -- 172.21.15.5:0/4141718510 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbd40c5940 con 0x7fcbd40cae90 2024-03-24T10:48:05.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:05.014+0000 7fcbd17fa640 1 -- 172.21.15.5:0/4141718510 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcbc8013470 con 0x7fcbd40cae90 2024-03-24T10:48:05.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:05.015+0000 7fcbd17fa640 1 --2- 172.21.15.5:0/4141718510 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcbb806b810 0x7fcbb806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:05.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:05.015+0000 7fcbd3fff640 1 --2- 172.21.15.5:0/4141718510 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcbb806b810 0x7fcbb806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:05.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:05.015+0000 7fcbd3fff640 1 --2- 172.21.15.5:0/4141718510 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcbb806b810 0x7fcbb806dcd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcbd406c4e0 tx=0x7fcbc0009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:05.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:05.278+0000 7fcbd17fa640 1 -- 172.21.15.5:0/4141718510 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fcbc8058f30 con 0x7fcbd40cae90 2024-03-24T10:48:05.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:05.278+0000 7fcbd17fa640 1 -- 172.21.15.5:0/4141718510 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcbc80551d0 con 0x7fcbd40cae90 2024-03-24T10:48:05.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:05.560+0000 7fcbdad43640 1 -- 172.21.15.5:0/4141718510 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcbd40ab820 con 0x7fcbd40cae90 2024-03-24T10:48:05.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:05.576+0000 7fcbd17fa640 1 -- 172.21.15.5:0/4141718510 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fcbd40ab820 con 0x7fcbd40cae90 2024-03-24T10:48:05.577 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:05.577 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:05.577 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:05.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:05.578+0000 7fcbdad43640 1 -- 172.21.15.5:0/4141718510 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcbb806b810 msgr2=0x7fcbb806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:05.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:05.578+0000 7fcbdad43640 1 --2- 172.21.15.5:0/4141718510 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcbb806b810 0x7fcbb806dcd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcbd406c4e0 tx=0x7fcbc0009190 comp rx=0 tx=0).stop 2024-03-24T10:48:05.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:05.578+0000 7fcbdad43640 1 -- 172.21.15.5:0/4141718510 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbd40cae90 msgr2=0x7fcbd40aee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:05.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:05.579+0000 7fcbdad43640 1 --2- 172.21.15.5:0/4141718510 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbd40cae90 0x7fcbd40aee60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcbc800e890 tx=0x7fcbc800ed50 comp rx=0 tx=0).stop 2024-03-24T10:48:05.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:05.579+0000 7fcbdad43640 1 -- 172.21.15.5:0/4141718510 shutdown_connections 2024-03-24T10:48:05.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:05.579+0000 7fcbdad43640 1 --2- 172.21.15.5:0/4141718510 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcbb806b810 0x7fcbb806dcd0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:05.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:05.579+0000 7fcbdad43640 1 --2- 172.21.15.5:0/4141718510 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcbd40cd960 0x7fcbd4153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:05.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:05.579+0000 7fcbdad43640 1 --2- 172.21.15.5:0/4141718510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcbd40cb270 0x7fcbd41ef7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:05.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:05.579+0000 7fcbdad43640 1 --2- 172.21.15.5:0/4141718510 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcbd40cae90 0x7fcbd40aee60 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:05.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:05.579+0000 7fcbdad43640 1 -- 172.21.15.5:0/4141718510 >> 172.21.15.5:0/4141718510 conn(0x7fcbd40cfb80 msgr2=0x7fcbd40c79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:05.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:05.579+0000 7fcbdad43640 1 -- 172.21.15.5:0/4141718510 shutdown_connections 2024-03-24T10:48:05.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:05.579+0000 7fcbdad43640 1 -- 172.21.15.5:0/4141718510 wait complete. 2024-03-24T10:48:06.195 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:06.195 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:07.195 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:07.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.297+0000 7fc5ccfe4640 1 Processor -- start 2024-03-24T10:48:07.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.297+0000 7fc5ccfe4640 1 -- start start 2024-03-24T10:48:07.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.298+0000 7fc5ccfe4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5c81543a0 0x7fc5c815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:07.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.298+0000 7fc5ccfe4640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc5c806bf50 0x7fc5c806c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:07.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.298+0000 7fc5ccfe4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c80cb2f0 con 0x7fc5c806d4f0 2024-03-24T10:48:07.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.298+0000 7fc5ccfe4640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fc5c80c6750 con 0x7fc5c806bf50 2024-03-24T10:48:07.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.298+0000 7fc5ccfe4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc5c80cc380 con 0x7fc5c81543a0 2024-03-24T10:48:07.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.299+0000 7fc5c659c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5c81543a0 0x7fc5c815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:07.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.299+0000 7fc5c6d9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc5c806bf50 0x7fc5c806c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:07.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.299+0000 7fc5c659c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5c81543a0 0x7fc5c815ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:41228/0 (socket says 172.21.15.5:41228) 2024-03-24T10:48:07.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.299+0000 7fc5c659c640 1 -- 172.21.15.5:0/1400325845 learned_addr learned my addr 172.21.15.5:0/1400325845 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:07.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.299+0000 7fc5c6d9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc5c806bf50 0x7fc5c806c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42304/0 (socket says 172.21.15.5:42304) 2024-03-24T10:48:07.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.301+0000 7fc5c6d9d640 1 -- 172.21.15.5:0/1400325845 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5c81543a0 msgr2=0x7fc5c815ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:07.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.301+0000 7fc5c6d9d640 1 --2- 172.21.15.5:0/1400325845 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5c81543a0 0x7fc5c815ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:07.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.301+0000 7fc5c6d9d640 1 -- 172.21.15.5:0/1400325845 >> v1:172.21.15.57:6789/0 conn(0x7fc5c806d4f0 legacy=0x7fc5c80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:07.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.301+0000 7fc5c6d9d640 1 -- 172.21.15.5:0/1400325845 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c80ca7e0 con 0x7fc5c806bf50 2024-03-24T10:48:07.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.304+0000 7fc5c6d9d640 1 --2- 172.21.15.5:0/1400325845 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc5c806bf50 0x7fc5c806c380 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009d90 tx=0x7fc5c002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e44a6cbc155cb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:07.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.308+0000 7fc5c5d9b640 1 -- 172.21.15.5:0/1400325845 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c002cce0 con 0x7fc5c806bf50 2024-03-24T10:48:07.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.308+0000 7fc5c5d9b640 1 -- 172.21.15.5:0/1400325845 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5c002ce40 con 0x7fc5c806bf50 2024-03-24T10:48:07.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.308+0000 7fc5c5d9b640 1 -- 172.21.15.5:0/1400325845 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c0034660 con 0x7fc5c806bf50 2024-03-24T10:48:07.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.308+0000 7fc5ccfe4640 1 -- 172.21.15.5:0/1400325845 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc5c806bf50 msgr2=0x7fc5c806c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:07.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.308+0000 7fc5ccfe4640 1 --2- 172.21.15.5:0/1400325845 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc5c806bf50 0x7fc5c806c380 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009d90 tx=0x7fc5c002bf30 comp rx=0 tx=0).stop 2024-03-24T10:48:07.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.308+0000 7fc5ccfe4640 1 -- 172.21.15.5:0/1400325845 shutdown_connections 2024-03-24T10:48:07.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.308+0000 7fc5ccfe4640 1 --2- 172.21.15.5:0/1400325845 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc5c806bf50 0x7fc5c806c380 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:07.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.308+0000 7fc5ccfe4640 1 --2- 172.21.15.5:0/1400325845 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5c81543a0 0x7fc5c815ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:07.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.308+0000 7fc5ccfe4640 1 -- 172.21.15.5:0/1400325845 >> 172.21.15.5:0/1400325845 conn(0x7fc5c8072ef0 msgr2=0x7fc5c8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:07.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.309+0000 7fc5ccfe4640 1 -- 172.21.15.5:0/1400325845 shutdown_connections 2024-03-24T10:48:07.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.309+0000 7fc5ccfe4640 1 -- 172.21.15.5:0/1400325845 wait complete. 2024-03-24T10:48:07.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.310+0000 7fc5ccfe4640 1 Processor -- start 2024-03-24T10:48:07.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.310+0000 7fc5ccfe4640 1 -- start start 2024-03-24T10:48:07.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.310+0000 7fc5ccfe4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5c806bf50 0x7fc5c80b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:07.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.311+0000 7fc5ccfe4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5c806d4f0 0x7fc5c80cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:07.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.311+0000 7fc5ccfe4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc5c81543a0 0x7fc5c80cf200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:07.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.311+0000 7fc5ccfe4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c81efb00 con 0x7fc5c81543a0 2024-03-24T10:48:07.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.311+0000 7fc5ccfe4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c80b45e0 con 0x7fc5c806bf50 2024-03-24T10:48:07.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.311+0000 7fc5ccfe4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc5c814e170 con 0x7fc5c806d4f0 2024-03-24T10:48:07.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.311+0000 7fc5c6d9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc5c81543a0 0x7fc5c80cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:07.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.311+0000 7fc5bffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5c806d4f0 0x7fc5c80cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:07.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.311+0000 7fc5c659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5c806bf50 0x7fc5c80b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:07.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.311+0000 7fc5c6d9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc5c81543a0 0x7fc5c80cf200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42318/0 (socket says 172.21.15.5:42318) 2024-03-24T10:48:07.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.311+0000 7fc5c6d9d640 1 -- 172.21.15.5:0/3977090338 learned_addr learned my addr 172.21.15.5:0/3977090338 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:07.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.315+0000 7fc5c6d9d640 1 -- 172.21.15.5:0/3977090338 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5c806d4f0 msgr2=0x7fc5c80cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:07.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.315+0000 7fc5c6d9d640 1 --2- 172.21.15.5:0/3977090338 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5c806d4f0 0x7fc5c80cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:07.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.315+0000 7fc5c6d9d640 1 -- 172.21.15.5:0/3977090338 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5c806bf50 msgr2=0x7fc5c80b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:07.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.315+0000 7fc5c6d9d640 1 --2- 172.21.15.5:0/3977090338 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5c806bf50 0x7fc5c80b40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:07.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.315+0000 7fc5c6d9d640 1 -- 172.21.15.5:0/3977090338 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c0009a70 con 0x7fc5c81543a0 2024-03-24T10:48:07.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.318+0000 7fc5c6d9d640 1 --2- 172.21.15.5:0/3977090338 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc5c81543a0 0x7fc5c80cf200 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009d90 tx=0x7fc5c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:07.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.321+0000 7fc5beffd640 1 -- 172.21.15.5:0/3977090338 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c003d600 con 0x7fc5c81543a0 2024-03-24T10:48:07.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.321+0000 7fc5beffd640 1 -- 172.21.15.5:0/3977090338 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5c003d780 con 0x7fc5c81543a0 2024-03-24T10:48:07.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.321+0000 7fc5beffd640 1 -- 172.21.15.5:0/3977090338 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c003db30 con 0x7fc5c81543a0 2024-03-24T10:48:07.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.321+0000 7fc5ccfe4640 1 -- 172.21.15.5:0/3977090338 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c80b3170 con 0x7fc5c81543a0 2024-03-24T10:48:07.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.321+0000 7fc5ccfe4640 1 -- 172.21.15.5:0/3977090338 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5c81effd0 con 0x7fc5c81543a0 2024-03-24T10:48:07.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.323+0000 7fc5ccfe4640 1 -- 172.21.15.5:0/3977090338 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5c80681d0 con 0x7fc5c81543a0 2024-03-24T10:48:07.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.598+0000 7fc5beffd640 1 -- 172.21.15.5:0/3977090338 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc5c003b040 con 0x7fc5c81543a0 2024-03-24T10:48:07.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.598+0000 7fc5beffd640 1 --2- 172.21.15.5:0/3977090338 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc5a806b860 0x7fc5a806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:07.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.599+0000 7fc5c659c640 1 --2- 172.21.15.5:0/3977090338 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc5a806b860 0x7fc5a806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:07.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.599+0000 7fc5c659c640 1 --2- 172.21.15.5:0/3977090338 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc5a806b860 0x7fc5a806dd20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8009bd0 tx=0x7fc5b8009240 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:07.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.613+0000 7fc5beffd640 1 -- 172.21.15.5:0/3977090338 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fc5c007b420 con 0x7fc5c81543a0 2024-03-24T10:48:07.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.613+0000 7fc5beffd640 1 -- 172.21.15.5:0/3977090338 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc5c00776c0 con 0x7fc5c81543a0 2024-03-24T10:48:07.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.853+0000 7fc5ccfe4640 1 -- 172.21.15.5:0/3977090338 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5c80cf820 con 0x7fc5c81543a0 2024-03-24T10:48:07.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.918+0000 7fc5beffd640 1 -- 172.21.15.5:0/3977090338 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fc5c004e020 con 0x7fc5c81543a0 2024-03-24T10:48:07.919 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:07.919 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:07.919 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:07.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.921+0000 7fc5ccfe4640 1 -- 172.21.15.5:0/3977090338 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc5a806b860 msgr2=0x7fc5a806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:07.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.921+0000 7fc5ccfe4640 1 --2- 172.21.15.5:0/3977090338 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc5a806b860 0x7fc5a806dd20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc5b8009bd0 tx=0x7fc5b8009240 comp rx=0 tx=0).stop 2024-03-24T10:48:07.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.921+0000 7fc5ccfe4640 1 -- 172.21.15.5:0/3977090338 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc5c81543a0 msgr2=0x7fc5c80cf200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:07.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.921+0000 7fc5ccfe4640 1 --2- 172.21.15.5:0/3977090338 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc5c81543a0 0x7fc5c80cf200 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0009d90 tx=0x7fc5c0004a40 comp rx=0 tx=0).stop 2024-03-24T10:48:07.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.921+0000 7fc5ccfe4640 1 -- 172.21.15.5:0/3977090338 shutdown_connections 2024-03-24T10:48:07.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.921+0000 7fc5ccfe4640 1 --2- 172.21.15.5:0/3977090338 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc5a806b860 0x7fc5a806dd20 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:07.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.921+0000 7fc5ccfe4640 1 --2- 172.21.15.5:0/3977090338 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc5c81543a0 0x7fc5c80cf200 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:07.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.921+0000 7fc5ccfe4640 1 --2- 172.21.15.5:0/3977090338 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5c806d4f0 0x7fc5c80cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:07.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.921+0000 7fc5ccfe4640 1 --2- 172.21.15.5:0/3977090338 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5c806bf50 0x7fc5c80b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:07.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.921+0000 7fc5ccfe4640 1 -- 172.21.15.5:0/3977090338 >> 172.21.15.5:0/3977090338 conn(0x7fc5c8072ef0 msgr2=0x7fc5c8151bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:07.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.921+0000 7fc5ccfe4640 1 -- 172.21.15.5:0/3977090338 shutdown_connections 2024-03-24T10:48:07.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:07.921+0000 7fc5ccfe4640 1 -- 172.21.15.5:0/3977090338 wait complete. 2024-03-24T10:48:08.536 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:08.536 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:09.537 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:09.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.639+0000 7f9a0eb1e640 1 Processor -- start 2024-03-24T10:48:09.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.639+0000 7f9a0eb1e640 1 -- start start 2024-03-24T10:48:09.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.640+0000 7f9a0eb1e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9a0815ac00 0x7f9a080cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:09.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.640+0000 7f9a0eb1e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a080cb2f0 con 0x7f9a081543a0 2024-03-24T10:48:09.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.640+0000 7f9a0eb1e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a0815b070 con 0x7f9a080ccaa0 2024-03-24T10:48:09.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.640+0000 7f9a0eb1e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f9a080c6750 con 0x7f9a0815ac00 2024-03-24T10:48:09.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.641+0000 7f9a07fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9a0815ac00 0x7f9a080cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:09.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.641+0000 7f9a0d0bb640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f9a080ccaa0 0x7f9a0806bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:60444/0 (socket says 172.21.15.5:60444) 2024-03-24T10:48:09.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.641+0000 7f9a0d0bb640 1 -- 172.21.15.5:0/2178153690 learned_addr learned my addr 172.21.15.5:0/2178153690 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:48:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.645+0000 7f9a077fe640 1 -- 172.21.15.5:0/2178153690 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2780152458 0 0) 0x7f9a080cb2f0 con 0x7f9a081543a0 2024-03-24T10:48:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.645+0000 7f9a077fe640 1 -- 172.21.15.5:0/2178153690 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99f8003520 con 0x7f9a081543a0 2024-03-24T10:48:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.645+0000 7f9a07fff640 1 -- 172.21.15.5:0/2178153690 >> v1:172.21.15.57:6789/0 conn(0x7f9a080ccaa0 legacy=0x7f9a0806bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.645+0000 7f9a07fff640 1 -- 172.21.15.5:0/2178153690 >> v1:172.21.15.5:6789/0 conn(0x7f9a081543a0 legacy=0x7f9a0806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:09.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.645+0000 7f9a07fff640 1 -- 172.21.15.5:0/2178153690 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a080ca7e0 con 0x7f9a0815ac00 2024-03-24T10:48:09.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.651+0000 7f9a07fff640 1 --2- 172.21.15.5:0/2178153690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a0815ac00 0x7f9a080cc560 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f99f40097a0 tx=0x7f99f402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0356340e13c07ca server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.656+0000 7f9a077fe640 1 -- 172.21.15.5:0/2178153690 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f4030610 con 0x7f9a0815ac00 2024-03-24T10:48:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.656+0000 7f9a077fe640 1 -- 172.21.15.5:0/2178153690 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f4030770 con 0x7f9a0815ac00 2024-03-24T10:48:09.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.656+0000 7f9a077fe640 1 -- 172.21.15.5:0/2178153690 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f4030b00 con 0x7f9a0815ac00 2024-03-24T10:48:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.656+0000 7f9a0eb1e640 1 -- 172.21.15.5:0/2178153690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a0815ac00 msgr2=0x7f9a080cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.656+0000 7f9a0eb1e640 1 --2- 172.21.15.5:0/2178153690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a0815ac00 0x7f9a080cc560 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f99f40097a0 tx=0x7f99f402b960 comp rx=0 tx=0).stop 2024-03-24T10:48:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.656+0000 7f9a0eb1e640 1 -- 172.21.15.5:0/2178153690 shutdown_connections 2024-03-24T10:48:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.656+0000 7f9a0eb1e640 1 --2- 172.21.15.5:0/2178153690 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a0815ac00 0x7f9a080cc560 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.656+0000 7f9a0eb1e640 1 -- 172.21.15.5:0/2178153690 >> 172.21.15.5:0/2178153690 conn(0x7f9a08072ef0 msgr2=0x7f9a08061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.656+0000 7f9a0eb1e640 1 -- 172.21.15.5:0/2178153690 shutdown_connections 2024-03-24T10:48:09.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.656+0000 7f9a0eb1e640 1 -- 172.21.15.5:0/2178153690 wait complete. 2024-03-24T10:48:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.658+0000 7f9a0eb1e640 1 Processor -- start 2024-03-24T10:48:09.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.658+0000 7f9a0eb1e640 1 -- start start 2024-03-24T10:48:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.658+0000 7f9a0eb1e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a080ccaa0 0x7f9a0815c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.658+0000 7f9a0eb1e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9a081543a0 0x7f9a0815d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.659+0000 7f9a0eb1e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a0815ac00 0x7f9a08157de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.659+0000 7f9a0eb1e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f9a081f0640 con 0x7f9a080ccaa0 2024-03-24T10:48:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.659+0000 7f9a0eb1e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0815cca0 con 0x7f9a0815ac00 2024-03-24T10:48:09.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.659+0000 7f9a0eb1e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f9a0815b9d0 con 0x7f9a081543a0 2024-03-24T10:48:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.659+0000 7f9a0c8ba640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a080ccaa0 0x7f9a0815c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.659+0000 7f9a0d0bb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a0815ac00 0x7f9a08157de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.660+0000 7f9a0c8ba640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a080ccaa0 0x7f9a0815c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42332/0 (socket says 172.21.15.5:42332) 2024-03-24T10:48:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.660+0000 7f9a0d0bb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a0815ac00 0x7f9a08157de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:35568/0 (socket says 172.21.15.5:35568) 2024-03-24T10:48:09.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.660+0000 7f9a0c8ba640 1 -- 172.21.15.5:0/882813309 learned_addr learned my addr 172.21.15.5:0/882813309 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:09.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.662+0000 7f9a07fff640 1 --2- 172.21.15.5:0/882813309 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9a081543a0 0x7f9a0815d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:09.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.663+0000 7f9a0c8ba640 1 -- 172.21.15.5:0/882813309 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9a081543a0 msgr2=0x7f9a0815d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:09.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.663+0000 7f9a0c8ba640 1 --2- 172.21.15.5:0/882813309 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9a081543a0 0x7f9a0815d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:09.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.663+0000 7f9a0c8ba640 1 -- 172.21.15.5:0/882813309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a0815ac00 msgr2=0x7f9a08157de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:09.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.663+0000 7f9a0c8ba640 1 --2- 172.21.15.5:0/882813309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a0815ac00 0x7f9a08157de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:09.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.663+0000 7f9a0c8ba640 1 -- 172.21.15.5:0/882813309 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99f4009400 con 0x7f9a080ccaa0 2024-03-24T10:48:09.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.666+0000 7f9a0c8ba640 1 --2- 172.21.15.5:0/882813309 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a080ccaa0 0x7f9a0815c760 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f99fc00c490 tx=0x7f99fc00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:09.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.675+0000 7f9a057fa640 1 -- 172.21.15.5:0/882813309 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99fc0052d0 con 0x7f9a080ccaa0 2024-03-24T10:48:09.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.675+0000 7f9a057fa640 1 -- 172.21.15.5:0/882813309 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99fc005430 con 0x7f9a080ccaa0 2024-03-24T10:48:09.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.675+0000 7f9a0eb1e640 1 -- 172.21.15.5:0/882813309 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a0815b6f0 con 0x7f9a080ccaa0 2024-03-24T10:48:09.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.675+0000 7f9a057fa640 1 -- 172.21.15.5:0/882813309 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99fc012870 con 0x7f9a080ccaa0 2024-03-24T10:48:09.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.675+0000 7f9a0eb1e640 1 -- 172.21.15.5:0/882813309 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a081f0260 con 0x7f9a080ccaa0 2024-03-24T10:48:09.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.676+0000 7f9a0eb1e640 1 -- 172.21.15.5:0/882813309 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a080681d0 con 0x7f9a080ccaa0 2024-03-24T10:48:09.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.696+0000 7f9a057fa640 1 -- 172.21.15.5:0/882813309 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f99fc0129d0 con 0x7f9a080ccaa0 2024-03-24T10:48:09.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.697+0000 7f9a057fa640 1 --2- 172.21.15.5:0/882813309 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f99ec06b860 0x7f99ec06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:09.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.697+0000 7f9a07fff640 1 --2- 172.21.15.5:0/882813309 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f99ec06b860 0x7f99ec06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:09.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.697+0000 7f9a07fff640 1 --2- 172.21.15.5:0/882813309 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f99ec06b860 0x7f99ec06dd20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f99f4036000 tx=0x7f99f403b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:09.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.959+0000 7f9a057fa640 1 -- 172.21.15.5:0/882813309 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f99fc058f70 con 0x7f9a080ccaa0 2024-03-24T10:48:09.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:09.959+0000 7f9a057fa640 1 -- 172.21.15.5:0/882813309 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f99fc055210 con 0x7f9a080ccaa0 2024-03-24T10:48:10.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:10.189+0000 7f9a0eb1e640 1 -- 172.21.15.5:0/882813309 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a080b2620 con 0x7f9a080ccaa0 2024-03-24T10:48:10.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:10.201+0000 7f9a057fa640 1 -- 172.21.15.5:0/882813309 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f99fc0588d0 con 0x7f9a080ccaa0 2024-03-24T10:48:10.202 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:10.202 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:10.202 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:10.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:10.203+0000 7f9a0eb1e640 1 -- 172.21.15.5:0/882813309 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f99ec06b860 msgr2=0x7f99ec06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:10.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:10.203+0000 7f9a0eb1e640 1 --2- 172.21.15.5:0/882813309 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f99ec06b860 0x7f99ec06dd20 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f99f4036000 tx=0x7f99f403b040 comp rx=0 tx=0).stop 2024-03-24T10:48:10.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:10.203+0000 7f9a0eb1e640 1 -- 172.21.15.5:0/882813309 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a080ccaa0 msgr2=0x7f9a0815c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:10.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:10.203+0000 7f9a0eb1e640 1 --2- 172.21.15.5:0/882813309 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a080ccaa0 0x7f9a0815c760 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f99fc00c490 tx=0x7f99fc00c950 comp rx=0 tx=0).stop 2024-03-24T10:48:10.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:10.203+0000 7f9a0eb1e640 1 -- 172.21.15.5:0/882813309 shutdown_connections 2024-03-24T10:48:10.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:10.203+0000 7f9a0eb1e640 1 --2- 172.21.15.5:0/882813309 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f99ec06b860 0x7f99ec06dd20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:10.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:10.204+0000 7f9a0eb1e640 1 --2- 172.21.15.5:0/882813309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a0815ac00 0x7f9a08157de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:10.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:10.204+0000 7f9a0eb1e640 1 --2- 172.21.15.5:0/882813309 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9a081543a0 0x7f9a0815d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:10.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:10.204+0000 7f9a0eb1e640 1 --2- 172.21.15.5:0/882813309 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a080ccaa0 0x7f9a0815c760 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:10.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:10.204+0000 7f9a0eb1e640 1 -- 172.21.15.5:0/882813309 >> 172.21.15.5:0/882813309 conn(0x7f9a08072ef0 msgr2=0x7f9a081530f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:10.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:10.204+0000 7f9a0eb1e640 1 -- 172.21.15.5:0/882813309 shutdown_connections 2024-03-24T10:48:10.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:10.204+0000 7f9a0eb1e640 1 -- 172.21.15.5:0/882813309 wait complete. 2024-03-24T10:48:10.818 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:10.818 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:11.819 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:11.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.921+0000 7ff590ac3640 1 Processor -- start 2024-03-24T10:48:11.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.921+0000 7ff590ac3640 1 -- start start 2024-03-24T10:48:11.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.922+0000 7ff590ac3640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff58c0cae90 0x7ff58c0cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:11.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.922+0000 7ff590ac3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff58c0cb270 0x7ff58c0cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:11.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.922+0000 7ff590ac3640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff58c0ccba0 con 0x7ff58c0cd960 2024-03-24T10:48:11.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.922+0000 7ff590ac3640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7ff58c0cce40 con 0x7ff58c0cb270 2024-03-24T10:48:11.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.922+0000 7ff590ac3640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff58c15a590 con 0x7ff58c0cae90 2024-03-24T10:48:11.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.923+0000 7ff58a59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff58c0cae90 0x7ff58c0cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:11.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.923+0000 7ff58ad9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff58c0cd960 0x7ff58c0cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:60456/0 (socket says 172.21.15.5:60456) 2024-03-24T10:48:11.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.923+0000 7ff58ad9d640 1 -- 172.21.15.5:0/2989064919 learned_addr learned my addr 172.21.15.5:0/2989064919 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:48:11.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.923+0000 7ff589d9b640 1 --2- 172.21.15.5:0/2989064919 >> v2:172.21.15.57:3300/0 conn(0x7ff58c0cb270 0x7ff58c0cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:11.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.926+0000 7ff58a59c640 1 -- 172.21.15.5:0/2989064919 >> v2:172.21.15.57:3300/0 conn(0x7ff58c0cb270 msgr2=0x7ff58c0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:11.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.926+0000 7ff58a59c640 1 --2- 172.21.15.5:0/2989064919 >> v2:172.21.15.57:3300/0 conn(0x7ff58c0cb270 0x7ff58c0cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:11.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.926+0000 7ff58a59c640 1 -- 172.21.15.5:0/2989064919 >> v1:172.21.15.57:6789/0 conn(0x7ff58c0cd960 legacy=0x7ff58c0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:11.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.926+0000 7ff58a59c640 1 -- 172.21.15.5:0/2989064919 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff58c15a870 con 0x7ff58c0cae90 2024-03-24T10:48:11.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.929+0000 7ff58a59c640 1 --2- 172.21.15.5:0/2989064919 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff58c0cae90 0x7ff58c0cf530 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff57c0097a0 tx=0x7ff57c02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=beeead7e53c8b31b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:11.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.934+0000 7ff58959a640 1 -- 172.21.15.5:0/2989064919 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57c030610 con 0x7ff58c0cae90 2024-03-24T10:48:11.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.934+0000 7ff58959a640 1 -- 172.21.15.5:0/2989064919 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff57c030770 con 0x7ff58c0cae90 2024-03-24T10:48:11.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.934+0000 7ff58959a640 1 -- 172.21.15.5:0/2989064919 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57c030b00 con 0x7ff58c0cae90 2024-03-24T10:48:11.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.935+0000 7ff590ac3640 1 -- 172.21.15.5:0/2989064919 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff58c0cae90 msgr2=0x7ff58c0cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:11.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.935+0000 7ff590ac3640 1 --2- 172.21.15.5:0/2989064919 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff58c0cae90 0x7ff58c0cf530 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff57c0097a0 tx=0x7ff57c02b960 comp rx=0 tx=0).stop 2024-03-24T10:48:11.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.935+0000 7ff590ac3640 1 -- 172.21.15.5:0/2989064919 shutdown_connections 2024-03-24T10:48:11.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.935+0000 7ff590ac3640 1 --2- 172.21.15.5:0/2989064919 >> v2:172.21.15.57:3300/0 conn(0x7ff58c0cb270 0x7ff58c0cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:11.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.935+0000 7ff590ac3640 1 --2- 172.21.15.5:0/2989064919 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff58c0cae90 0x7ff58c0cf530 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:11.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.935+0000 7ff590ac3640 1 -- 172.21.15.5:0/2989064919 >> 172.21.15.5:0/2989064919 conn(0x7ff58c0cfb80 msgr2=0x7ff58c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:11.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.935+0000 7ff590ac3640 1 -- 172.21.15.5:0/2989064919 shutdown_connections 2024-03-24T10:48:11.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.935+0000 7ff590ac3640 1 -- 172.21.15.5:0/2989064919 wait complete. 2024-03-24T10:48:11.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.936+0000 7ff590ac3640 1 Processor -- start 2024-03-24T10:48:11.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.936+0000 7ff590ac3640 1 -- start start 2024-03-24T10:48:11.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.937+0000 7ff590ac3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff58c0cae90 0x7ff58c0ab160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:11.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.937+0000 7ff590ac3640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff58c0cb270 0x7ff58c0ae150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:11.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.937+0000 7ff590ac3640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff58c0cd960 0x7ff58c15dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:11.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.937+0000 7ff590ac3640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff58c0acca0 con 0x7ff58c0cb270 2024-03-24T10:48:11.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.937+0000 7ff590ac3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff58c1f16e0 con 0x7ff58c0cae90 2024-03-24T10:48:11.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.937+0000 7ff590ac3640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff58c0ad0b0 con 0x7ff58c0cd960 2024-03-24T10:48:11.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.938+0000 7ff58a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff58c0cae90 0x7ff58c0ab160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:11.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.938+0000 7ff589d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff58c0cb270 0x7ff58c0ae150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:11.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.938+0000 7ff58ad9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff58c0cd960 0x7ff58c15dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:11.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.938+0000 7ff58a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff58c0cae90 0x7ff58c0ab160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:35586/0 (socket says 172.21.15.5:35586) 2024-03-24T10:48:11.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.938+0000 7ff58a59c640 1 -- 172.21.15.5:0/2395737117 learned_addr learned my addr 172.21.15.5:0/2395737117 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:11.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.942+0000 7ff589d9b640 1 -- 172.21.15.5:0/2395737117 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff58c0cd960 msgr2=0x7ff58c15dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:11.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.942+0000 7ff589d9b640 1 --2- 172.21.15.5:0/2395737117 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff58c0cd960 0x7ff58c15dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:11.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.942+0000 7ff589d9b640 1 -- 172.21.15.5:0/2395737117 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff58c0cae90 msgr2=0x7ff58c0ab160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:11.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.942+0000 7ff589d9b640 1 --2- 172.21.15.5:0/2395737117 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff58c0cae90 0x7ff58c0ab160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:11.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.942+0000 7ff589d9b640 1 -- 172.21.15.5:0/2395737117 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff57c009400 con 0x7ff58c0cb270 2024-03-24T10:48:11.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.944+0000 7ff589d9b640 1 --2- 172.21.15.5:0/2395737117 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff58c0cb270 0x7ff58c0ae150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff58000e850 tx=0x7ff58000ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:11.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.949+0000 7ff57affd640 1 -- 172.21.15.5:0/2395737117 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5800095b0 con 0x7ff58c0cb270 2024-03-24T10:48:11.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.949+0000 7ff57affd640 1 -- 172.21.15.5:0/2395737117 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff58000c940 con 0x7ff58c0cb270 2024-03-24T10:48:11.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.949+0000 7ff590ac3640 1 -- 172.21.15.5:0/2395737117 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff58c0ad390 con 0x7ff58c0cb270 2024-03-24T10:48:11.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.949+0000 7ff57affd640 1 -- 172.21.15.5:0/2395737117 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58000ccd0 con 0x7ff58c0cb270 2024-03-24T10:48:11.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.949+0000 7ff590ac3640 1 -- 172.21.15.5:0/2395737117 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff58c0ad950 con 0x7ff58c0cb270 2024-03-24T10:48:11.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:11.950+0000 7ff590ac3640 1 -- 172.21.15.5:0/2395737117 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff58c06d5a0 con 0x7ff58c0cb270 2024-03-24T10:48:12.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:12.236+0000 7ff57affd640 1 -- 172.21.15.5:0/2395737117 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff580013470 con 0x7ff58c0cb270 2024-03-24T10:48:12.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:12.237+0000 7ff57affd640 1 --2- 172.21.15.5:0/2395737117 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff56c06b810 0x7ff56c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:12.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:12.237+0000 7ff58a59c640 1 --2- 172.21.15.5:0/2395737117 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff56c06b810 0x7ff56c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:12.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:12.237+0000 7ff58a59c640 1 --2- 172.21.15.5:0/2395737117 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff56c06b810 0x7ff56c06dcd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff57c02b9a0 tx=0x7ff57c008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:12.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:12.255+0000 7ff57affd640 1 -- 172.21.15.5:0/2395737117 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7ff58000ce30 con 0x7ff58c0cb270 2024-03-24T10:48:12.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:12.255+0000 7ff57affd640 1 -- 172.21.15.5:0/2395737117 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff580055590 con 0x7ff58c0cb270 2024-03-24T10:48:12.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:12.496+0000 7ff590ac3640 1 -- 172.21.15.5:0/2395737117 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff58c0ac8c0 con 0x7ff58c0cb270 2024-03-24T10:48:12.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:12.517+0000 7ff57affd640 1 -- 172.21.15.5:0/2395737117 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7ff58c0ac8c0 con 0x7ff58c0cb270 2024-03-24T10:48:12.518 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:12.518 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:12.518 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:12.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:12.519+0000 7ff590ac3640 1 -- 172.21.15.5:0/2395737117 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff56c06b810 msgr2=0x7ff56c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:12.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:12.520+0000 7ff590ac3640 1 --2- 172.21.15.5:0/2395737117 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff56c06b810 0x7ff56c06dcd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff57c02b9a0 tx=0x7ff57c008030 comp rx=0 tx=0).stop 2024-03-24T10:48:12.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:12.520+0000 7ff590ac3640 1 -- 172.21.15.5:0/2395737117 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff58c0cb270 msgr2=0x7ff58c0ae150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:12.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:12.520+0000 7ff590ac3640 1 --2- 172.21.15.5:0/2395737117 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff58c0cb270 0x7ff58c0ae150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff58000e850 tx=0x7ff58000ed10 comp rx=0 tx=0).stop 2024-03-24T10:48:12.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:12.520+0000 7ff590ac3640 1 -- 172.21.15.5:0/2395737117 shutdown_connections 2024-03-24T10:48:12.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:12.520+0000 7ff590ac3640 1 --2- 172.21.15.5:0/2395737117 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff56c06b810 0x7ff56c06dcd0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:12.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:12.520+0000 7ff590ac3640 1 --2- 172.21.15.5:0/2395737117 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff58c0cb270 0x7ff58c0ae150 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:12.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:12.520+0000 7ff590ac3640 1 --2- 172.21.15.5:0/2395737117 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff58c0cd960 0x7ff58c15dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:12.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:12.520+0000 7ff590ac3640 1 --2- 172.21.15.5:0/2395737117 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff58c0cae90 0x7ff58c0ab160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:12.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:12.520+0000 7ff590ac3640 1 -- 172.21.15.5:0/2395737117 >> 172.21.15.5:0/2395737117 conn(0x7ff58c0cfb80 msgr2=0x7ff58c0c7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:12.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:12.520+0000 7ff590ac3640 1 -- 172.21.15.5:0/2395737117 shutdown_connections 2024-03-24T10:48:12.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:12.520+0000 7ff590ac3640 1 -- 172.21.15.5:0/2395737117 wait complete. 2024-03-24T10:48:13.136 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:13.136 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:14.137 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.239+0000 7f293ca07640 1 Processor -- start 2024-03-24T10:48:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.239+0000 7f293ca07640 1 -- start start 2024-03-24T10:48:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.240+0000 7f293ca07640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f293815ac00 0x7f29380cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.240+0000 7f293ca07640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f29380ccaa0 0x7f293806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.240+0000 7f293ca07640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29380cb2f0 con 0x7f29381543a0 2024-03-24T10:48:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.240+0000 7f293ca07640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f29380c6750 con 0x7f29380ccaa0 2024-03-24T10:48:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.240+0000 7f293ca07640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f29380cc380 con 0x7f293815ac00 2024-03-24T10:48:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.241+0000 7f293659c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f29381543a0 0x7f293806d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:60442/0 (socket says 172.21.15.5:60442) 2024-03-24T10:48:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.241+0000 7f293659c640 1 -- 172.21.15.5:0/1435860835 learned_addr learned my addr 172.21.15.5:0/1435860835 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:48:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.241+0000 7f2936d9d640 1 --2- 172.21.15.5:0/1435860835 >> v2:172.21.15.5:3300/0 conn(0x7f29380ccaa0 0x7f293806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.241+0000 7f292ffff640 1 --2- 172.21.15.5:0/1435860835 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f293815ac00 0x7f29380cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.243+0000 7f2936d9d640 1 -- 172.21.15.5:0/1435860835 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f293815ac00 msgr2=0x7f29380cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.243+0000 7f2936d9d640 1 --2- 172.21.15.5:0/1435860835 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f293815ac00 0x7f29380cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:14.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.243+0000 7f2936d9d640 1 -- 172.21.15.5:0/1435860835 >> v1:172.21.15.57:6789/0 conn(0x7f29381543a0 legacy=0x7f293806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:14.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.243+0000 7f2936d9d640 1 -- 172.21.15.5:0/1435860835 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f293815b070 con 0x7f29380ccaa0 2024-03-24T10:48:14.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.246+0000 7f2936d9d640 1 --2- 172.21.15.5:0/1435860835 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f29380ccaa0 0x7f293806bf50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2930009d90 tx=0x7f293002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af2e2c6c44171dbf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:14.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.249+0000 7f2935d9b640 1 -- 172.21.15.5:0/1435860835 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f293002cc50 con 0x7f29380ccaa0 2024-03-24T10:48:14.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.249+0000 7f2935d9b640 1 -- 172.21.15.5:0/1435860835 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f293002cdb0 con 0x7f29380ccaa0 2024-03-24T10:48:14.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.249+0000 7f2935d9b640 1 -- 172.21.15.5:0/1435860835 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2930034660 con 0x7f29380ccaa0 2024-03-24T10:48:14.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.250+0000 7f293ca07640 1 -- 172.21.15.5:0/1435860835 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f29380ccaa0 msgr2=0x7f293806bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:14.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.250+0000 7f293ca07640 1 --2- 172.21.15.5:0/1435860835 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f29380ccaa0 0x7f293806bf50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2930009d90 tx=0x7f293002bea0 comp rx=0 tx=0).stop 2024-03-24T10:48:14.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.250+0000 7f293ca07640 1 -- 172.21.15.5:0/1435860835 shutdown_connections 2024-03-24T10:48:14.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.250+0000 7f293ca07640 1 --2- 172.21.15.5:0/1435860835 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f293815ac00 0x7f29380cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:14.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.250+0000 7f293ca07640 1 --2- 172.21.15.5:0/1435860835 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f29380ccaa0 0x7f293806bf50 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:14.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.250+0000 7f293ca07640 1 -- 172.21.15.5:0/1435860835 >> 172.21.15.5:0/1435860835 conn(0x7f2938072ef0 msgr2=0x7f2938061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:14.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.250+0000 7f293ca07640 1 -- 172.21.15.5:0/1435860835 shutdown_connections 2024-03-24T10:48:14.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.250+0000 7f293ca07640 1 -- 172.21.15.5:0/1435860835 wait complete. 2024-03-24T10:48:14.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.252+0000 7f293ca07640 1 Processor -- start 2024-03-24T10:48:14.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.252+0000 7f293ca07640 1 -- start start 2024-03-24T10:48:14.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.252+0000 7f293ca07640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f29380ccaa0 0x7f293814a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:14.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.252+0000 7f293ca07640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29381543a0 0x7f2938154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:14.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.252+0000 7f293ca07640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f293815ac00 0x7f293814c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:14.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.252+0000 7f293ca07640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f293814b7e0 con 0x7f29380ccaa0 2024-03-24T10:48:14.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.252+0000 7f293ca07640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f293814c680 con 0x7f29381543a0 2024-03-24T10:48:14.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.252+0000 7f293ca07640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f2938156730 con 0x7f293815ac00 2024-03-24T10:48:14.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.253+0000 7f293659c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f29380ccaa0 0x7f293814a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:14.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.253+0000 7f2936d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f293815ac00 0x7f293814c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:14.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.253+0000 7f292ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29381543a0 0x7f2938154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:14.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.253+0000 7f293659c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f29380ccaa0 0x7f293814a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42504/0 (socket says 172.21.15.5:42504) 2024-03-24T10:48:14.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.253+0000 7f293659c640 1 -- 172.21.15.5:0/695296174 learned_addr learned my addr 172.21.15.5:0/695296174 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:14.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.257+0000 7f293659c640 1 -- 172.21.15.5:0/695296174 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f293815ac00 msgr2=0x7f293814c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:14.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.257+0000 7f293659c640 1 --2- 172.21.15.5:0/695296174 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f293815ac00 0x7f293814c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:14.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.257+0000 7f293659c640 1 -- 172.21.15.5:0/695296174 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29381543a0 msgr2=0x7f2938154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:14.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.257+0000 7f293659c640 1 --2- 172.21.15.5:0/695296174 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29381543a0 0x7f2938154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:14.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.257+0000 7f293659c640 1 -- 172.21.15.5:0/695296174 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2930009a70 con 0x7f29380ccaa0 2024-03-24T10:48:14.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.259+0000 7f293659c640 1 --2- 172.21.15.5:0/695296174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f29380ccaa0 0x7f293814a770 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f292800c400 tx=0x7f292800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:14.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.261+0000 7f292effd640 1 -- 172.21.15.5:0/695296174 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2928005ce0 con 0x7f29380ccaa0 2024-03-24T10:48:14.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.261+0000 7f293ca07640 1 -- 172.21.15.5:0/695296174 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2938155310 con 0x7f29380ccaa0 2024-03-24T10:48:14.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.261+0000 7f293ca07640 1 -- 172.21.15.5:0/695296174 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2938155e00 con 0x7f29380ccaa0 2024-03-24T10:48:14.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.262+0000 7f293ca07640 1 -- 172.21.15.5:0/695296174 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29380681d0 con 0x7f29380ccaa0 2024-03-24T10:48:14.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.263+0000 7f292effd640 1 -- 172.21.15.5:0/695296174 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2928005e40 con 0x7f29380ccaa0 2024-03-24T10:48:14.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.263+0000 7f292effd640 1 -- 172.21.15.5:0/695296174 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2928007270 con 0x7f29380ccaa0 2024-03-24T10:48:14.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.284+0000 7f292effd640 1 -- 172.21.15.5:0/695296174 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2928007490 con 0x7f29380ccaa0 2024-03-24T10:48:14.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.284+0000 7f292effd640 1 --2- 172.21.15.5:0/695296174 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f291806b810 0x7f291806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:14.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.284+0000 7f292ffff640 1 --2- 172.21.15.5:0/695296174 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f291806b810 0x7f291806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:14.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.285+0000 7f292ffff640 1 --2- 172.21.15.5:0/695296174 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f291806b810 0x7f291806dcd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f29380b24f0 tx=0x7f2920009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:14.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.547+0000 7f292effd640 1 -- 172.21.15.5:0/695296174 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f29280587b0 con 0x7f29380ccaa0 2024-03-24T10:48:14.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.547+0000 7f292effd640 1 -- 172.21.15.5:0/695296174 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2928054a50 con 0x7f29380ccaa0 2024-03-24T10:48:14.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.774+0000 7f293ca07640 1 -- 172.21.15.5:0/695296174 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29380b27d0 con 0x7f29380ccaa0 2024-03-24T10:48:14.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.796+0000 7f292effd640 1 -- 172.21.15.5:0/695296174 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f2928017020 con 0x7f29380ccaa0 2024-03-24T10:48:14.797 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:14.797 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:14.797 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:14.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.798+0000 7f293ca07640 1 -- 172.21.15.5:0/695296174 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f291806b810 msgr2=0x7f291806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:14.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.798+0000 7f293ca07640 1 --2- 172.21.15.5:0/695296174 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f291806b810 0x7f291806dcd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f29380b24f0 tx=0x7f2920009190 comp rx=0 tx=0).stop 2024-03-24T10:48:14.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.798+0000 7f293ca07640 1 -- 172.21.15.5:0/695296174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f29380ccaa0 msgr2=0x7f293814a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:14.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.798+0000 7f293ca07640 1 --2- 172.21.15.5:0/695296174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f29380ccaa0 0x7f293814a770 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f292800c400 tx=0x7f292800c8c0 comp rx=0 tx=0).stop 2024-03-24T10:48:14.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.798+0000 7f293ca07640 1 -- 172.21.15.5:0/695296174 shutdown_connections 2024-03-24T10:48:14.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.798+0000 7f293ca07640 1 --2- 172.21.15.5:0/695296174 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f291806b810 0x7f291806dcd0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:14.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.798+0000 7f293ca07640 1 --2- 172.21.15.5:0/695296174 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f293815ac00 0x7f293814c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:14.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.798+0000 7f293ca07640 1 --2- 172.21.15.5:0/695296174 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f29381543a0 0x7f2938154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:14.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.798+0000 7f293ca07640 1 --2- 172.21.15.5:0/695296174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f29380ccaa0 0x7f293814a770 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:14.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.798+0000 7f293ca07640 1 -- 172.21.15.5:0/695296174 >> 172.21.15.5:0/695296174 conn(0x7f2938072ef0 msgr2=0x7f293806be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:14.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.798+0000 7f293ca07640 1 -- 172.21.15.5:0/695296174 shutdown_connections 2024-03-24T10:48:14.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:14.798+0000 7f293ca07640 1 -- 172.21.15.5:0/695296174 wait complete. 2024-03-24T10:48:15.413 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:15.413 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:16.415 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:16.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.517+0000 7ff86282a640 1 Processor -- start 2024-03-24T10:48:16.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.517+0000 7ff86282a640 1 -- start start 2024-03-24T10:48:16.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.517+0000 7ff86282a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff85c15ac00 0x7ff85c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:16.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.517+0000 7ff86282a640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff85c0cb2f0 con 0x7ff85c1543a0 2024-03-24T10:48:16.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.517+0000 7ff86282a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff85c15b070 con 0x7ff85c0ccaa0 2024-03-24T10:48:16.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.517+0000 7ff86282a640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7ff85c0c6750 con 0x7ff85c15ac00 2024-03-24T10:48:16.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.518+0000 7ff860dc7640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff85c0ccaa0 0x7ff85c06bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:60452/0 (socket says 172.21.15.5:60452) 2024-03-24T10:48:16.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.518+0000 7ff860dc7640 1 -- 172.21.15.5:0/1275535684 learned_addr learned my addr 172.21.15.5:0/1275535684 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:48:16.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.518+0000 7ff85b7fe640 1 --2- 172.21.15.5:0/1275535684 >> v2:172.21.15.5:3300/0 conn(0x7ff85c15ac00 0x7ff85c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:16.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.521+0000 7ff85affd640 1 -- 172.21.15.5:0/1275535684 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2011198723 0 0) 0x7ff85c15b070 con 0x7ff85c0ccaa0 2024-03-24T10:48:16.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.521+0000 7ff85affd640 1 -- 172.21.15.5:0/1275535684 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff844003520 con 0x7ff85c0ccaa0 2024-03-24T10:48:16.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.522+0000 7ff85b7fe640 1 -- 172.21.15.5:0/1275535684 >> v1:172.21.15.57:6789/0 conn(0x7ff85c0ccaa0 legacy=0x7ff85c06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:16.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.522+0000 7ff85b7fe640 1 -- 172.21.15.5:0/1275535684 >> v1:172.21.15.5:6789/0 conn(0x7ff85c1543a0 legacy=0x7ff85c06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:16.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.522+0000 7ff85b7fe640 1 -- 172.21.15.5:0/1275535684 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff85c0ca7e0 con 0x7ff85c15ac00 2024-03-24T10:48:16.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.527+0000 7ff85b7fe640 1 --2- 172.21.15.5:0/1275535684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff85c15ac00 0x7ff85c0cc560 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff8500097a0 tx=0x7ff85002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e31a904dbd8919fd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:16.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.533+0000 7ff85affd640 1 -- 172.21.15.5:0/1275535684 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff850030610 con 0x7ff85c15ac00 2024-03-24T10:48:16.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.533+0000 7ff85affd640 1 -- 172.21.15.5:0/1275535684 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff850030770 con 0x7ff85c15ac00 2024-03-24T10:48:16.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.533+0000 7ff85affd640 1 -- 172.21.15.5:0/1275535684 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff850030b00 con 0x7ff85c15ac00 2024-03-24T10:48:16.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.533+0000 7ff86282a640 1 -- 172.21.15.5:0/1275535684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff85c15ac00 msgr2=0x7ff85c0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:16.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.533+0000 7ff86282a640 1 --2- 172.21.15.5:0/1275535684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff85c15ac00 0x7ff85c0cc560 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff8500097a0 tx=0x7ff85002b960 comp rx=0 tx=0).stop 2024-03-24T10:48:16.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.534+0000 7ff86282a640 1 -- 172.21.15.5:0/1275535684 shutdown_connections 2024-03-24T10:48:16.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.534+0000 7ff86282a640 1 --2- 172.21.15.5:0/1275535684 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff85c15ac00 0x7ff85c0cc560 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:16.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.534+0000 7ff86282a640 1 -- 172.21.15.5:0/1275535684 >> 172.21.15.5:0/1275535684 conn(0x7ff85c072ef0 msgr2=0x7ff85c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:16.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.534+0000 7ff86282a640 1 -- 172.21.15.5:0/1275535684 shutdown_connections 2024-03-24T10:48:16.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.534+0000 7ff86282a640 1 -- 172.21.15.5:0/1275535684 wait complete. 2024-03-24T10:48:16.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.535+0000 7ff86282a640 1 Processor -- start 2024-03-24T10:48:16.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.535+0000 7ff86282a640 1 -- start start 2024-03-24T10:48:16.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.536+0000 7ff86282a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff85c0ccaa0 0x7ff85c1f0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:16.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.536+0000 7ff86282a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff85c1543a0 0x7ff85c0b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:16.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.536+0000 7ff86282a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff85c15ac00 0x7ff85c157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:16.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.536+0000 7ff86282a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff85c0b4f40 con 0x7ff85c1543a0 2024-03-24T10:48:16.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.536+0000 7ff86282a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff85c1f0ac0 con 0x7ff85c15ac00 2024-03-24T10:48:16.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.536+0000 7ff86282a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff85c0b4090 con 0x7ff85c0ccaa0 2024-03-24T10:48:16.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.536+0000 7ff85b7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff85c1543a0 0x7ff85c0b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:16.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.537+0000 7ff85bfff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff85c0ccaa0 0x7ff85c1f0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:16.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.537+0000 7ff860dc7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff85c15ac00 0x7ff85c157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:16.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.537+0000 7ff85b7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff85c1543a0 0x7ff85c0b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42526/0 (socket says 172.21.15.5:42526) 2024-03-24T10:48:16.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.537+0000 7ff85b7fe640 1 -- 172.21.15.5:0/1507652669 learned_addr learned my addr 172.21.15.5:0/1507652669 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:16.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.540+0000 7ff85b7fe640 1 -- 172.21.15.5:0/1507652669 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff85c0ccaa0 msgr2=0x7ff85c1f0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:16.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.540+0000 7ff85b7fe640 1 --2- 172.21.15.5:0/1507652669 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff85c0ccaa0 0x7ff85c1f0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:16.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.540+0000 7ff85b7fe640 1 -- 172.21.15.5:0/1507652669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff85c15ac00 msgr2=0x7ff85c157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:16.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.540+0000 7ff85b7fe640 1 --2- 172.21.15.5:0/1507652669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff85c15ac00 0x7ff85c157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:16.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.540+0000 7ff85b7fe640 1 -- 172.21.15.5:0/1507652669 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff850009400 con 0x7ff85c1543a0 2024-03-24T10:48:16.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.543+0000 7ff85b7fe640 1 --2- 172.21.15.5:0/1507652669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff85c1543a0 0x7ff85c0b37d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff85002bf70 tx=0x7ff8500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:16.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.547+0000 7ff858ff9640 1 -- 172.21.15.5:0/1507652669 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85003e070 con 0x7ff85c1543a0 2024-03-24T10:48:16.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.547+0000 7ff858ff9640 1 -- 172.21.15.5:0/1507652669 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8500397a0 con 0x7ff85c1543a0 2024-03-24T10:48:16.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.547+0000 7ff86282a640 1 -- 172.21.15.5:0/1507652669 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff85c0b4330 con 0x7ff85c1543a0 2024-03-24T10:48:16.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.548+0000 7ff858ff9640 1 -- 172.21.15.5:0/1507652669 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff850039b30 con 0x7ff85c1543a0 2024-03-24T10:48:16.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.548+0000 7ff86282a640 1 -- 172.21.15.5:0/1507652669 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff85c0b48f0 con 0x7ff85c1543a0 2024-03-24T10:48:16.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.548+0000 7ff86282a640 1 -- 172.21.15.5:0/1507652669 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff85c0681d0 con 0x7ff85c1543a0 2024-03-24T10:48:16.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.567+0000 7ff858ff9640 1 -- 172.21.15.5:0/1507652669 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff850039c90 con 0x7ff85c1543a0 2024-03-24T10:48:16.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.567+0000 7ff858ff9640 1 --2- 172.21.15.5:0/1507652669 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff84806b890 0x7ff84806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:16.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.568+0000 7ff85bfff640 1 --2- 172.21.15.5:0/1507652669 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff84806b890 0x7ff84806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:16.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.568+0000 7ff85bfff640 1 --2- 172.21.15.5:0/1507652669 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff84806b890 0x7ff84806dd50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff84c001780 tx=0x7ff84c009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:16.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.832+0000 7ff858ff9640 1 -- 172.21.15.5:0/1507652669 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7ff85007b6f0 con 0x7ff85c1543a0 2024-03-24T10:48:16.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:16.832+0000 7ff858ff9640 1 -- 172.21.15.5:0/1507652669 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff850077990 con 0x7ff85c1543a0 2024-03-24T10:48:17.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:17.056+0000 7ff86282a640 1 -- 172.21.15.5:0/1507652669 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff85c0caf40 con 0x7ff85c1543a0 2024-03-24T10:48:17.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:17.069+0000 7ff858ff9640 1 -- 172.21.15.5:0/1507652669 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7ff850034070 con 0x7ff85c1543a0 2024-03-24T10:48:17.070 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:17.070 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:17.071 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:17.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:17.071+0000 7ff86282a640 1 -- 172.21.15.5:0/1507652669 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff84806b890 msgr2=0x7ff84806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:17.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:17.071+0000 7ff86282a640 1 --2- 172.21.15.5:0/1507652669 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff84806b890 0x7ff84806dd50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff84c001780 tx=0x7ff84c009040 comp rx=0 tx=0).stop 2024-03-24T10:48:17.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:17.071+0000 7ff86282a640 1 -- 172.21.15.5:0/1507652669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff85c1543a0 msgr2=0x7ff85c0b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:17.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:17.071+0000 7ff86282a640 1 --2- 172.21.15.5:0/1507652669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff85c1543a0 0x7ff85c0b37d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff85002bf70 tx=0x7ff8500046c0 comp rx=0 tx=0).stop 2024-03-24T10:48:17.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:17.072+0000 7ff86282a640 1 -- 172.21.15.5:0/1507652669 shutdown_connections 2024-03-24T10:48:17.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:17.072+0000 7ff86282a640 1 --2- 172.21.15.5:0/1507652669 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff84806b890 0x7ff84806dd50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:17.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:17.072+0000 7ff86282a640 1 --2- 172.21.15.5:0/1507652669 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff85c1543a0 0x7ff85c0b37d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:17.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:17.072+0000 7ff86282a640 1 --2- 172.21.15.5:0/1507652669 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff85c15ac00 0x7ff85c157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:17.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:17.072+0000 7ff86282a640 1 --2- 172.21.15.5:0/1507652669 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff85c0ccaa0 0x7ff85c1f0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:17.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:17.072+0000 7ff86282a640 1 -- 172.21.15.5:0/1507652669 >> 172.21.15.5:0/1507652669 conn(0x7ff85c072ef0 msgr2=0x7ff85c153080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:17.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:17.072+0000 7ff86282a640 1 -- 172.21.15.5:0/1507652669 shutdown_connections 2024-03-24T10:48:17.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:17.072+0000 7ff86282a640 1 -- 172.21.15.5:0/1507652669 wait complete. 2024-03-24T10:48:17.687 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:17.687 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:18.688 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:18.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.789+0000 7f1bbb809640 1 Processor -- start 2024-03-24T10:48:18.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.790+0000 7f1bbb809640 1 -- start start 2024-03-24T10:48:18.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.790+0000 7f1bbb809640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1bb40cae90 0x7f1bb40cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.790+0000 7f1bbb809640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1bb40cd960 0x7f1bb40cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.790+0000 7f1bbb809640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bb40ccba0 con 0x7f1bb40cb270 2024-03-24T10:48:18.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.790+0000 7f1bbb809640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f1bb40cc9b0 con 0x7f1bb40cae90 2024-03-24T10:48:18.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.790+0000 7f1bbb809640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1bb40cce40 con 0x7f1bb40cd960 2024-03-24T10:48:18.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.791+0000 7f1bb95a5640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1bb40cae90 0x7f1bb40cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.791+0000 7f1bb9da6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1bb40cd960 0x7f1bb40cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.791+0000 7f1bb8da4640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1bb40cb270 0x7f1bb40cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:60458/0 (socket says 172.21.15.5:60458) 2024-03-24T10:48:18.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.791+0000 7f1bb8da4640 1 -- 172.21.15.5:0/1916884025 learned_addr learned my addr 172.21.15.5:0/1916884025 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:48:18.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.794+0000 7f1bb95a5640 1 -- 172.21.15.5:0/1916884025 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1bb40cd960 msgr2=0x7f1bb40cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:18.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.794+0000 7f1bb95a5640 1 --2- 172.21.15.5:0/1916884025 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1bb40cd960 0x7f1bb40cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.794+0000 7f1bb95a5640 1 -- 172.21.15.5:0/1916884025 >> v1:172.21.15.57:6789/0 conn(0x7f1bb40cb270 legacy=0x7f1bb40cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:18.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.794+0000 7f1bb95a5640 1 -- 172.21.15.5:0/1916884025 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb415a590 con 0x7f1bb40cae90 2024-03-24T10:48:18.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.796+0000 7f1bb95a5640 1 --2- 172.21.15.5:0/1916884025 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bb40cae90 0x7f1bb40cf530 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1bb00097a0 tx=0x7f1bb002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d22e8d4d93c6e8fc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:18.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.800+0000 7f1babfff640 1 -- 172.21.15.5:0/1916884025 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0030610 con 0x7f1bb40cae90 2024-03-24T10:48:18.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.800+0000 7f1babfff640 1 -- 172.21.15.5:0/1916884025 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bb0030770 con 0x7f1bb40cae90 2024-03-24T10:48:18.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.800+0000 7f1babfff640 1 -- 172.21.15.5:0/1916884025 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0030b00 con 0x7f1bb40cae90 2024-03-24T10:48:18.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.801+0000 7f1bbb809640 1 -- 172.21.15.5:0/1916884025 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bb40cae90 msgr2=0x7f1bb40cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:18.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.801+0000 7f1bbb809640 1 --2- 172.21.15.5:0/1916884025 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bb40cae90 0x7f1bb40cf530 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1bb00097a0 tx=0x7f1bb002b960 comp rx=0 tx=0).stop 2024-03-24T10:48:18.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.801+0000 7f1bbb809640 1 -- 172.21.15.5:0/1916884025 shutdown_connections 2024-03-24T10:48:18.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.801+0000 7f1bbb809640 1 --2- 172.21.15.5:0/1916884025 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1bb40cd960 0x7f1bb40cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.801+0000 7f1bbb809640 1 --2- 172.21.15.5:0/1916884025 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bb40cae90 0x7f1bb40cf530 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.801+0000 7f1bbb809640 1 -- 172.21.15.5:0/1916884025 >> 172.21.15.5:0/1916884025 conn(0x7f1bb40cfb80 msgr2=0x7f1bb40cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:18.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.801+0000 7f1bbb809640 1 -- 172.21.15.5:0/1916884025 shutdown_connections 2024-03-24T10:48:18.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.801+0000 7f1bbb809640 1 -- 172.21.15.5:0/1916884025 wait complete. 2024-03-24T10:48:18.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.802+0000 7f1bbb809640 1 Processor -- start 2024-03-24T10:48:18.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.802+0000 7f1bbb809640 1 -- start start 2024-03-24T10:48:18.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.803+0000 7f1bbb809640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1bb40cae90 0x7f1bb40b0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.803+0000 7f1bbb809640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bb40cb270 0x7f1bb4155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.803+0000 7f1bbb809640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1bb40cd960 0x7f1bb4155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:18.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.803+0000 7f1bbb809640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb4157cd0 con 0x7f1bb40cb270 2024-03-24T10:48:18.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.803+0000 7f1bbb809640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb415b9b0 con 0x7f1bb40cae90 2024-03-24T10:48:18.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.803+0000 7f1bbb809640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1bb415bc90 con 0x7f1bb40cd960 2024-03-24T10:48:18.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.803+0000 7f1bb8da4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bb40cb270 0x7f1bb4155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.803+0000 7f1bb95a5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1bb40cae90 0x7f1bb40b0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.803+0000 7f1bb9da6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1bb40cd960 0x7f1bb4155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:18.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.804+0000 7f1bb8da4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bb40cb270 0x7f1bb4155010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42536/0 (socket says 172.21.15.5:42536) 2024-03-24T10:48:18.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.804+0000 7f1bb8da4640 1 -- 172.21.15.5:0/3837564439 learned_addr learned my addr 172.21.15.5:0/3837564439 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:18.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.807+0000 7f1bb8da4640 1 -- 172.21.15.5:0/3837564439 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1bb40cd960 msgr2=0x7f1bb4155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:18.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.807+0000 7f1bb8da4640 1 --2- 172.21.15.5:0/3837564439 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1bb40cd960 0x7f1bb4155550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.807+0000 7f1bb8da4640 1 -- 172.21.15.5:0/3837564439 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1bb40cae90 msgr2=0x7f1bb40b0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:18.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.807+0000 7f1bb8da4640 1 --2- 172.21.15.5:0/3837564439 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1bb40cae90 0x7f1bb40b0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:18.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.807+0000 7f1bb8da4640 1 -- 172.21.15.5:0/3837564439 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb0009400 con 0x7f1bb40cb270 2024-03-24T10:48:18.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.809+0000 7f1bb8da4640 1 --2- 172.21.15.5:0/3837564439 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bb40cb270 0x7f1bb4155010 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400c400 tx=0x7f1ba400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:18.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.811+0000 7f1ba9ffb640 1 -- 172.21.15.5:0/3837564439 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4005ce0 con 0x7f1bb40cb270 2024-03-24T10:48:18.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.811+0000 7f1bbb809640 1 -- 172.21.15.5:0/3837564439 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bb415bf70 con 0x7f1bb40cb270 2024-03-24T10:48:18.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.812+0000 7f1bbb809640 1 -- 172.21.15.5:0/3837564439 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bb415c530 con 0x7f1bb40cb270 2024-03-24T10:48:18.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:18.813+0000 7f1bbb809640 1 -- 172.21.15.5:0/3837564439 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bb40681d0 con 0x7f1bb40cb270 2024-03-24T10:48:19.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.099+0000 7f1ba9ffb640 1 -- 172.21.15.5:0/3837564439 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ba4005e40 con 0x7f1bb40cb270 2024-03-24T10:48:19.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.099+0000 7f1ba9ffb640 1 -- 172.21.15.5:0/3837564439 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4007270 con 0x7f1bb40cb270 2024-03-24T10:48:19.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.100+0000 7f1ba9ffb640 1 -- 172.21.15.5:0/3837564439 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1ba4007490 con 0x7f1bb40cb270 2024-03-24T10:48:19.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.101+0000 7f1ba9ffb640 1 --2- 172.21.15.5:0/3837564439 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1b9806b810 0x7f1b9806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:19.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.101+0000 7f1bb95a5640 1 --2- 172.21.15.5:0/3837564439 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1b9806b810 0x7f1b9806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:19.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.101+0000 7f1bb95a5640 1 --2- 172.21.15.5:0/3837564439 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1b9806b810 0x7f1b9806dcd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0036000 tx=0x7f1bb0008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:19.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.122+0000 7f1ba9ffb640 1 -- 172.21.15.5:0/3837564439 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f1ba4058750 con 0x7f1bb40cb270 2024-03-24T10:48:19.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.122+0000 7f1ba9ffb640 1 -- 172.21.15.5:0/3837564439 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1ba40549f0 con 0x7f1bb40cb270 2024-03-24T10:48:19.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.342+0000 7f1bbb809640 1 -- 172.21.15.5:0/3837564439 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1bb40c8e40 con 0x7f1bb40cb270 2024-03-24T10:48:19.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.356+0000 7f1ba9ffb640 1 -- 172.21.15.5:0/3837564439 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f1ba4017090 con 0x7f1bb40cb270 2024-03-24T10:48:19.357 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:19.357 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:19.357 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:19.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.358+0000 7f1bbb809640 1 -- 172.21.15.5:0/3837564439 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1b9806b810 msgr2=0x7f1b9806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.358+0000 7f1bbb809640 1 --2- 172.21.15.5:0/3837564439 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1b9806b810 0x7f1b9806dcd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0036000 tx=0x7f1bb0008030 comp rx=0 tx=0).stop 2024-03-24T10:48:19.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.358+0000 7f1bbb809640 1 -- 172.21.15.5:0/3837564439 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bb40cb270 msgr2=0x7f1bb4155010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:19.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.358+0000 7f1bbb809640 1 --2- 172.21.15.5:0/3837564439 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bb40cb270 0x7f1bb4155010 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400c400 tx=0x7f1ba400c8c0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.358+0000 7f1bbb809640 1 -- 172.21.15.5:0/3837564439 shutdown_connections 2024-03-24T10:48:19.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.358+0000 7f1bbb809640 1 --2- 172.21.15.5:0/3837564439 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1b9806b810 0x7f1b9806dcd0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.358+0000 7f1bbb809640 1 --2- 172.21.15.5:0/3837564439 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1bb40cb270 0x7f1bb4155010 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.358+0000 7f1bbb809640 1 --2- 172.21.15.5:0/3837564439 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1bb40cd960 0x7f1bb4155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.358+0000 7f1bbb809640 1 --2- 172.21.15.5:0/3837564439 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1bb40cae90 0x7f1bb40b0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:19.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.359+0000 7f1bbb809640 1 -- 172.21.15.5:0/3837564439 >> 172.21.15.5:0/3837564439 conn(0x7f1bb40cfb80 msgr2=0x7f1bb4158170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:19.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.359+0000 7f1bbb809640 1 -- 172.21.15.5:0/3837564439 shutdown_connections 2024-03-24T10:48:19.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:19.359+0000 7f1bbb809640 1 -- 172.21.15.5:0/3837564439 wait complete. 2024-03-24T10:48:19.975 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:19.975 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:20.976 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:21.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.078+0000 7fd129932640 1 Processor -- start 2024-03-24T10:48:21.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.078+0000 7fd129932640 1 -- start start 2024-03-24T10:48:21.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.079+0000 7fd129932640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd124072640 0x7fd1240731d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.079+0000 7fd129932640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd1241544c0 0x7fd12415d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.079+0000 7fd129932640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd124073710 con 0x7fd1240c9650 2024-03-24T10:48:21.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.079+0000 7fd129932640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fd124070e00 con 0x7fd1241544c0 2024-03-24T10:48:21.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.079+0000 7fd129932640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fd12415a590 con 0x7fd124072640 2024-03-24T10:48:21.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.080+0000 7fd122ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fd1240c9650 0x7fd124153db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:60474/0 (socket says 172.21.15.5:60474) 2024-03-24T10:48:21.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.080+0000 7fd122ffd640 1 -- 172.21.15.5:0/3323326056 learned_addr learned my addr 172.21.15.5:0/3323326056 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:48:21.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.080+0000 7fd11bfff640 1 --2- 172.21.15.5:0/3323326056 >> v2:172.21.15.57:3300/0 conn(0x7fd124072640 0x7fd1240731d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.081+0000 7fd1237fe640 1 --2- 172.21.15.5:0/3323326056 >> v2:172.21.15.5:3300/0 conn(0x7fd1241544c0 0x7fd12415d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.084+0000 7fd1237fe640 1 -- 172.21.15.5:0/3323326056 >> v2:172.21.15.57:3300/0 conn(0x7fd124072640 msgr2=0x7fd1240731d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:21.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.084+0000 7fd1237fe640 1 --2- 172.21.15.5:0/3323326056 >> v2:172.21.15.57:3300/0 conn(0x7fd124072640 0x7fd1240731d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.084+0000 7fd1237fe640 1 -- 172.21.15.5:0/3323326056 >> v1:172.21.15.57:6789/0 conn(0x7fd1240c9650 legacy=0x7fd124153db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:21.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.084+0000 7fd1237fe640 1 -- 172.21.15.5:0/3323326056 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd12415a870 con 0x7fd1241544c0 2024-03-24T10:48:21.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.086+0000 7fd1237fe640 1 --2- 172.21.15.5:0/3323326056 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1241544c0 0x7fd12415d830 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd1240710e0 tx=0x7fd11c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6964795ee15dff6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:21.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.090+0000 7fd1227fc640 1 -- 172.21.15.5:0/3323326056 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11c02cd00 con 0x7fd1241544c0 2024-03-24T10:48:21.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.090+0000 7fd1227fc640 1 -- 172.21.15.5:0/3323326056 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd11c02ce60 con 0x7fd1241544c0 2024-03-24T10:48:21.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.090+0000 7fd1227fc640 1 -- 172.21.15.5:0/3323326056 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11c034660 con 0x7fd1241544c0 2024-03-24T10:48:21.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.091+0000 7fd129932640 1 -- 172.21.15.5:0/3323326056 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1241544c0 msgr2=0x7fd12415d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:21.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.091+0000 7fd129932640 1 --2- 172.21.15.5:0/3323326056 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1241544c0 0x7fd12415d830 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd1240710e0 tx=0x7fd11c02bf50 comp rx=0 tx=0).stop 2024-03-24T10:48:21.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.091+0000 7fd129932640 1 -- 172.21.15.5:0/3323326056 shutdown_connections 2024-03-24T10:48:21.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.091+0000 7fd129932640 1 --2- 172.21.15.5:0/3323326056 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1241544c0 0x7fd12415d830 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.091+0000 7fd129932640 1 --2- 172.21.15.5:0/3323326056 >> v2:172.21.15.57:3300/0 conn(0x7fd124072640 0x7fd1240731d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.091+0000 7fd129932640 1 -- 172.21.15.5:0/3323326056 >> 172.21.15.5:0/3323326056 conn(0x7fd12406c240 msgr2=0x7fd12406c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:21.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.091+0000 7fd129932640 1 -- 172.21.15.5:0/3323326056 shutdown_connections 2024-03-24T10:48:21.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.091+0000 7fd129932640 1 -- 172.21.15.5:0/3323326056 wait complete. 2024-03-24T10:48:21.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.092+0000 7fd129932640 1 Processor -- start 2024-03-24T10:48:21.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.092+0000 7fd129932640 1 -- start start 2024-03-24T10:48:21.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.093+0000 7fd129932640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd124072640 0x7fd124157650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.093+0000 7fd129932640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd1240c9650 0x7fd12414bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.093+0000 7fd129932640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1241544c0 0x7fd12414c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.093+0000 7fd129932640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd1241f25b0 con 0x7fd1241544c0 2024-03-24T10:48:21.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.093+0000 7fd129932640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd1241f2aa0 con 0x7fd124072640 2024-03-24T10:48:21.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.093+0000 7fd129932640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd124158050 con 0x7fd1240c9650 2024-03-24T10:48:21.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.093+0000 7fd1237fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1241544c0 0x7fd12414c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.093+0000 7fd122ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd124072640 0x7fd124157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.093+0000 7fd1237fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1241544c0 0x7fd12414c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42552/0 (socket says 172.21.15.5:42552) 2024-03-24T10:48:21.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.093+0000 7fd1237fe640 1 -- 172.21.15.5:0/3847653454 learned_addr learned my addr 172.21.15.5:0/3847653454 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:21.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.093+0000 7fd11bfff640 1 --2- 172.21.15.5:0/3847653454 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd1240c9650 0x7fd12414bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.097+0000 7fd1237fe640 1 -- 172.21.15.5:0/3847653454 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd1240c9650 msgr2=0x7fd12414bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:21.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.097+0000 7fd1237fe640 1 --2- 172.21.15.5:0/3847653454 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd1240c9650 0x7fd12414bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.097+0000 7fd1237fe640 1 -- 172.21.15.5:0/3847653454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd124072640 msgr2=0x7fd124157650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:21.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.097+0000 7fd1237fe640 1 --2- 172.21.15.5:0/3847653454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd124072640 0x7fd124157650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.097+0000 7fd1237fe640 1 -- 172.21.15.5:0/3847653454 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd11c009a70 con 0x7fd1241544c0 2024-03-24T10:48:21.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.100+0000 7fd1237fe640 1 --2- 172.21.15.5:0/3847653454 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1241544c0 0x7fd12414c270 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd11c02c560 tx=0x7fd11c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:21.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.104+0000 7fd120ff9640 1 -- 172.21.15.5:0/3847653454 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11c03d600 con 0x7fd1241544c0 2024-03-24T10:48:21.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.104+0000 7fd120ff9640 1 -- 172.21.15.5:0/3847653454 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd11c03d780 con 0x7fd1241544c0 2024-03-24T10:48:21.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.104+0000 7fd120ff9640 1 -- 172.21.15.5:0/3847653454 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd11c03db30 con 0x7fd1241544c0 2024-03-24T10:48:21.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.104+0000 7fd129932640 1 -- 172.21.15.5:0/3847653454 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1241555b0 con 0x7fd1241544c0 2024-03-24T10:48:21.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.104+0000 7fd129932640 1 -- 172.21.15.5:0/3847653454 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1241f38a0 con 0x7fd1241544c0 2024-03-24T10:48:21.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.105+0000 7fd129932640 1 -- 172.21.15.5:0/3847653454 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd124154e00 con 0x7fd1241544c0 2024-03-24T10:48:21.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.125+0000 7fd120ff9640 1 -- 172.21.15.5:0/3847653454 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd11c03b040 con 0x7fd1241544c0 2024-03-24T10:48:21.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.125+0000 7fd120ff9640 1 --2- 172.21.15.5:0/3847653454 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd10406b810 0x7fd10406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:21.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.125+0000 7fd122ffd640 1 --2- 172.21.15.5:0/3847653454 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd10406b810 0x7fd10406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:21.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.126+0000 7fd122ffd640 1 --2- 172.21.15.5:0/3847653454 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd10406b810 0x7fd10406dcd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd114007fd0 tx=0x7fd114007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:21.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.129+0000 7fd120ff9640 1 -- 172.21.15.5:0/3847653454 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fd11c07b1c0 con 0x7fd1241544c0 2024-03-24T10:48:21.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.389+0000 7fd120ff9640 1 -- 172.21.15.5:0/3847653454 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd11c077460 con 0x7fd1241544c0 2024-03-24T10:48:21.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.613+0000 7fd129932640 1 -- 172.21.15.5:0/3847653454 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd124154e00 con 0x7fd1241544c0 2024-03-24T10:48:21.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.655+0000 7fd120ff9640 1 -- 172.21.15.5:0/3847653454 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fd124154e00 con 0x7fd1241544c0 2024-03-24T10:48:21.655 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:21.655 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:21.656 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:21.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.658+0000 7fd129932640 1 -- 172.21.15.5:0/3847653454 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd10406b810 msgr2=0x7fd10406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:21.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.658+0000 7fd129932640 1 --2- 172.21.15.5:0/3847653454 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd10406b810 0x7fd10406dcd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd114007fd0 tx=0x7fd114007e80 comp rx=0 tx=0).stop 2024-03-24T10:48:21.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.658+0000 7fd129932640 1 -- 172.21.15.5:0/3847653454 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1241544c0 msgr2=0x7fd12414c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:21.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.658+0000 7fd129932640 1 --2- 172.21.15.5:0/3847653454 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1241544c0 0x7fd12414c270 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd11c02c560 tx=0x7fd11c004a40 comp rx=0 tx=0).stop 2024-03-24T10:48:21.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.658+0000 7fd129932640 1 -- 172.21.15.5:0/3847653454 shutdown_connections 2024-03-24T10:48:21.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.658+0000 7fd129932640 1 --2- 172.21.15.5:0/3847653454 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd10406b810 0x7fd10406dcd0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.658+0000 7fd129932640 1 --2- 172.21.15.5:0/3847653454 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1241544c0 0x7fd12414c270 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.658+0000 7fd129932640 1 --2- 172.21.15.5:0/3847653454 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd1240c9650 0x7fd12414bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.658+0000 7fd129932640 1 --2- 172.21.15.5:0/3847653454 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd124072640 0x7fd124157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:21.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.658+0000 7fd129932640 1 -- 172.21.15.5:0/3847653454 >> 172.21.15.5:0/3847653454 conn(0x7fd12406c240 msgr2=0x7fd1240c49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:21.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.658+0000 7fd129932640 1 -- 172.21.15.5:0/3847653454 shutdown_connections 2024-03-24T10:48:21.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:21.658+0000 7fd129932640 1 -- 172.21.15.5:0/3847653454 wait complete. 2024-03-24T10:48:22.272 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:22.273 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:23.273 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:23.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.375+0000 7f94308e1640 1 Processor -- start 2024-03-24T10:48:23.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.375+0000 7f94308e1640 1 -- start start 2024-03-24T10:48:23.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.376+0000 7f94308e1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f942c0cae90 0x7f942c0cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:23.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.376+0000 7f94308e1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f942c0cd960 0x7f942c0cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:23.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.376+0000 7f94308e1640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f942c0ccba0 con 0x7f942c0cb270 2024-03-24T10:48:23.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.376+0000 7f94308e1640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f942c0cc9b0 con 0x7f942c0cae90 2024-03-24T10:48:23.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.376+0000 7f94308e1640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f942c0cce40 con 0x7f942c0cd960 2024-03-24T10:48:23.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.376+0000 7f942ad9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f942c0cd960 0x7f942c0cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:23.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.377+0000 7f942ad9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f942c0cd960 0x7f942c0cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:36130/0 (socket says 172.21.15.5:36130) 2024-03-24T10:48:23.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.377+0000 7f9423fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f942c0cb270 0x7f942c0cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:57732/0 (socket says 172.21.15.5:57732) 2024-03-24T10:48:23.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.377+0000 7f942ad9d640 1 -- 172.21.15.5:0/1987071402 learned_addr learned my addr 172.21.15.5:0/1987071402 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:23.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.377+0000 7f942a59c640 1 --2- 172.21.15.5:0/1987071402 >> v2:172.21.15.57:3300/0 conn(0x7f942c0cae90 0x7f942c0cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:23.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.380+0000 7f942ad9d640 1 -- 172.21.15.5:0/1987071402 >> v2:172.21.15.57:3300/0 conn(0x7f942c0cae90 msgr2=0x7f942c0cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:23.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.380+0000 7f942ad9d640 1 --2- 172.21.15.5:0/1987071402 >> v2:172.21.15.57:3300/0 conn(0x7f942c0cae90 0x7f942c0cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:23.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.380+0000 7f942ad9d640 1 -- 172.21.15.5:0/1987071402 >> v1:172.21.15.57:6789/0 conn(0x7f942c0cb270 legacy=0x7f942c0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:23.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.380+0000 7f942ad9d640 1 -- 172.21.15.5:0/1987071402 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f942c15a590 con 0x7f942c0cd960 2024-03-24T10:48:23.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.383+0000 7f942ad9d640 1 --2- 172.21.15.5:0/1987071402 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f942c0cd960 0x7f942c0cb6f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9424009e50 tx=0x7f942402bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5369a981ae49a017 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:23.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.388+0000 7f9429d9b640 1 -- 172.21.15.5:0/1987071402 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f942402cb90 con 0x7f942c0cd960 2024-03-24T10:48:23.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.388+0000 7f9429d9b640 1 -- 172.21.15.5:0/1987071402 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f942402ccf0 con 0x7f942c0cd960 2024-03-24T10:48:23.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.388+0000 7f9429d9b640 1 -- 172.21.15.5:0/1987071402 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9424034610 con 0x7f942c0cd960 2024-03-24T10:48:23.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.388+0000 7f94308e1640 1 -- 172.21.15.5:0/1987071402 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f942c0cd960 msgr2=0x7f942c0cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:23.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.388+0000 7f94308e1640 1 --2- 172.21.15.5:0/1987071402 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f942c0cd960 0x7f942c0cb6f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9424009e50 tx=0x7f942402bff0 comp rx=0 tx=0).stop 2024-03-24T10:48:23.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.388+0000 7f94308e1640 1 -- 172.21.15.5:0/1987071402 shutdown_connections 2024-03-24T10:48:23.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.388+0000 7f94308e1640 1 --2- 172.21.15.5:0/1987071402 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f942c0cd960 0x7f942c0cb6f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:23.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.388+0000 7f94308e1640 1 --2- 172.21.15.5:0/1987071402 >> v2:172.21.15.57:3300/0 conn(0x7f942c0cae90 0x7f942c0cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:23.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.388+0000 7f94308e1640 1 -- 172.21.15.5:0/1987071402 >> 172.21.15.5:0/1987071402 conn(0x7f942c0cfb80 msgr2=0x7f942c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:23.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.388+0000 7f94308e1640 1 -- 172.21.15.5:0/1987071402 shutdown_connections 2024-03-24T10:48:23.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.389+0000 7f94308e1640 1 -- 172.21.15.5:0/1987071402 wait complete. 2024-03-24T10:48:23.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.390+0000 7f94308e1640 1 Processor -- start 2024-03-24T10:48:23.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.390+0000 7f94308e1640 1 -- start start 2024-03-24T10:48:23.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.390+0000 7f94308e1640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f942c0cae90 0x7f942c0b2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:23.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.390+0000 7f94308e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f942c0cb270 0x7f942c0b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:23.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.390+0000 7f94308e1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f942c0cd960 0x7f942c0b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:23.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.390+0000 7f94308e1640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f942c1efb30 con 0x7f942c0cae90 2024-03-24T10:48:23.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.390+0000 7f94308e1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f942c15e830 con 0x7f942c0cb270 2024-03-24T10:48:23.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.390+0000 7f94308e1640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f942c159680 con 0x7f942c0cd960 2024-03-24T10:48:23.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.391+0000 7f942ad9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f942c0cd960 0x7f942c0b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:23.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.391+0000 7f942a59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f942c0cae90 0x7f942c0b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:23.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.391+0000 7f9423fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f942c0cb270 0x7f942c0b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:23.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.392+0000 7f942ad9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f942c0cd960 0x7f942c0b4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:36140/0 (socket says 172.21.15.5:36140) 2024-03-24T10:48:23.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.392+0000 7f942ad9d640 1 -- 172.21.15.5:0/1430074702 learned_addr learned my addr 172.21.15.5:0/1430074702 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:23.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.397+0000 7f9423fff640 1 -- 172.21.15.5:0/1430074702 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f942c0cd960 msgr2=0x7f942c0b4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:23.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.397+0000 7f9423fff640 1 --2- 172.21.15.5:0/1430074702 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f942c0cd960 0x7f942c0b4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:23.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.397+0000 7f9423fff640 1 -- 172.21.15.5:0/1430074702 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f942c0cae90 msgr2=0x7f942c0b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:23.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.397+0000 7f9423fff640 1 --2- 172.21.15.5:0/1430074702 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f942c0cae90 0x7f942c0b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:23.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.397+0000 7f9423fff640 1 -- 172.21.15.5:0/1430074702 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9424009b30 con 0x7f942c0cb270 2024-03-24T10:48:23.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.400+0000 7f9423fff640 1 --2- 172.21.15.5:0/1430074702 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f942c0cb270 0x7f942c0b2c90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f942c0b5cf0 tx=0x7f941400c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:23.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.406+0000 7f9422ffd640 1 -- 172.21.15.5:0/1430074702 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94140074e0 con 0x7f942c0cb270 2024-03-24T10:48:23.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.406+0000 7f9422ffd640 1 -- 172.21.15.5:0/1430074702 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9414007640 con 0x7f942c0cb270 2024-03-24T10:48:23.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.406+0000 7f9422ffd640 1 -- 172.21.15.5:0/1430074702 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9414004a00 con 0x7f942c0cb270 2024-03-24T10:48:23.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.406+0000 7f94308e1640 1 -- 172.21.15.5:0/1430074702 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f942c1efdb0 con 0x7f942c0cb270 2024-03-24T10:48:23.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.406+0000 7f94308e1640 1 -- 172.21.15.5:0/1430074702 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f942c1f0370 con 0x7f942c0cb270 2024-03-24T10:48:23.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.407+0000 7f94308e1640 1 -- 172.21.15.5:0/1430074702 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f942c0681d0 con 0x7f942c0cb270 2024-03-24T10:48:23.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.435+0000 7f9422ffd640 1 -- 172.21.15.5:0/1430074702 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9414004b60 con 0x7f942c0cb270 2024-03-24T10:48:23.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.435+0000 7f9422ffd640 1 --2- 172.21.15.5:0/1430074702 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f940c06b860 0x7f940c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:23.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.435+0000 7f942a59c640 1 --2- 172.21.15.5:0/1430074702 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f940c06b860 0x7f940c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:23.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.436+0000 7f942a59c640 1 --2- 172.21.15.5:0/1430074702 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f940c06b860 0x7f940c06dd20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f941c009b40 tx=0x7f941c009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:23.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.438+0000 7f9422ffd640 1 -- 172.21.15.5:0/1430074702 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f94140589c0 con 0x7f942c0cb270 2024-03-24T10:48:23.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.701+0000 7f9422ffd640 1 -- 172.21.15.5:0/1430074702 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9414054c60 con 0x7f942c0cb270 2024-03-24T10:48:23.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.924+0000 7f94308e1640 1 -- 172.21.15.5:0/1430074702 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f942c0c8d90 con 0x7f942c0cb270 2024-03-24T10:48:23.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.937+0000 7f9422ffd640 1 -- 172.21.15.5:0/1430074702 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f9414058320 con 0x7f942c0cb270 2024-03-24T10:48:23.937 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:23.938 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:23.938 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:23.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.940+0000 7f94308e1640 1 -- 172.21.15.5:0/1430074702 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f940c06b860 msgr2=0x7f940c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:23.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.940+0000 7f94308e1640 1 --2- 172.21.15.5:0/1430074702 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f940c06b860 0x7f940c06dd20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f941c009b40 tx=0x7f941c009190 comp rx=0 tx=0).stop 2024-03-24T10:48:23.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.940+0000 7f94308e1640 1 -- 172.21.15.5:0/1430074702 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f942c0cb270 msgr2=0x7f942c0b2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:23.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.940+0000 7f94308e1640 1 --2- 172.21.15.5:0/1430074702 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f942c0cb270 0x7f942c0b2c90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f942c0b5cf0 tx=0x7f941400c800 comp rx=0 tx=0).stop 2024-03-24T10:48:23.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.940+0000 7f94308e1640 1 -- 172.21.15.5:0/1430074702 shutdown_connections 2024-03-24T10:48:23.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.940+0000 7f94308e1640 1 --2- 172.21.15.5:0/1430074702 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f940c06b860 0x7f940c06dd20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:23.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.940+0000 7f94308e1640 1 --2- 172.21.15.5:0/1430074702 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f942c0cd960 0x7f942c0b4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:23.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.940+0000 7f94308e1640 1 --2- 172.21.15.5:0/1430074702 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f942c0cb270 0x7f942c0b2c90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:23.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.940+0000 7f94308e1640 1 --2- 172.21.15.5:0/1430074702 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f942c0cae90 0x7f942c0b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:23.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.940+0000 7f94308e1640 1 -- 172.21.15.5:0/1430074702 >> 172.21.15.5:0/1430074702 conn(0x7f942c0cfb80 msgr2=0x7f942c159d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:23.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.940+0000 7f94308e1640 1 -- 172.21.15.5:0/1430074702 shutdown_connections 2024-03-24T10:48:23.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:23.940+0000 7f94308e1640 1 -- 172.21.15.5:0/1430074702 wait complete. 2024-03-24T10:48:24.554 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:24.554 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:25.555 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:25.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.657+0000 7f6e4b5c9640 1 Processor -- start 2024-03-24T10:48:25.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.657+0000 7f6e4b5c9640 1 -- start start 2024-03-24T10:48:25.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.658+0000 7f6e4b5c9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6e441543a0 0x7f6e4415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:25.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.658+0000 7f6e4b5c9640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6e4406bf50 0x7f6e4406c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:25.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.658+0000 7f6e4b5c9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e440cb2f0 con 0x7f6e4406d4f0 2024-03-24T10:48:25.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.658+0000 7f6e4b5c9640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f6e440c6750 con 0x7f6e4406bf50 2024-03-24T10:48:25.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.658+0000 7f6e4b5c9640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f6e440cc380 con 0x7f6e441543a0 2024-03-24T10:48:25.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.659+0000 7f6e49b66640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6e4406bf50 0x7f6e4406c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:25.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.659+0000 7f6e49365640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6e441543a0 0x7f6e4415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:25.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.660+0000 7f6e49b66640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6e4406bf50 0x7f6e4406c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:41992/0 (socket says 172.21.15.5:41992) 2024-03-24T10:48:25.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.660+0000 7f6e49b66640 1 -- 172.21.15.5:0/3290955469 learned_addr learned my addr 172.21.15.5:0/3290955469 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:25.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.663+0000 7f6e49b66640 1 -- 172.21.15.5:0/3290955469 >> v2:172.21.15.57:3300/0 conn(0x7f6e441543a0 msgr2=0x7f6e4415ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:25.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.663+0000 7f6e49b66640 1 --2- 172.21.15.5:0/3290955469 >> v2:172.21.15.57:3300/0 conn(0x7f6e441543a0 0x7f6e4415ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:25.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.663+0000 7f6e49b66640 1 -- 172.21.15.5:0/3290955469 >> v1:172.21.15.57:6789/0 conn(0x7f6e4406d4f0 legacy=0x7f6e440cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:25.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.663+0000 7f6e49b66640 1 -- 172.21.15.5:0/3290955469 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e440ca7e0 con 0x7f6e4406bf50 2024-03-24T10:48:25.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.665+0000 7f6e49b66640 1 --2- 172.21.15.5:0/3290955469 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e4406bf50 0x7f6e4406c330 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6e440cd600 tx=0x7f6e3c02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2917dfad185cdc75 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:25.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.674+0000 7f6e3bfff640 1 -- 172.21.15.5:0/3290955469 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c02cbb0 con 0x7f6e4406bf50 2024-03-24T10:48:25.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.674+0000 7f6e3bfff640 1 -- 172.21.15.5:0/3290955469 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e3c02cd10 con 0x7f6e4406bf50 2024-03-24T10:48:25.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.674+0000 7f6e3bfff640 1 -- 172.21.15.5:0/3290955469 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c034610 con 0x7f6e4406bf50 2024-03-24T10:48:25.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.674+0000 7f6e4b5c9640 1 -- 172.21.15.5:0/3290955469 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e4406bf50 msgr2=0x7f6e4406c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:25.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.674+0000 7f6e4b5c9640 1 --2- 172.21.15.5:0/3290955469 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e4406bf50 0x7f6e4406c330 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6e440cd600 tx=0x7f6e3c02c010 comp rx=0 tx=0).stop 2024-03-24T10:48:25.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.675+0000 7f6e4b5c9640 1 -- 172.21.15.5:0/3290955469 shutdown_connections 2024-03-24T10:48:25.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.675+0000 7f6e4b5c9640 1 --2- 172.21.15.5:0/3290955469 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e4406bf50 0x7f6e4406c330 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:25.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.675+0000 7f6e4b5c9640 1 --2- 172.21.15.5:0/3290955469 >> v2:172.21.15.57:3300/0 conn(0x7f6e441543a0 0x7f6e4415ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:25.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.675+0000 7f6e4b5c9640 1 -- 172.21.15.5:0/3290955469 >> 172.21.15.5:0/3290955469 conn(0x7f6e44072ef0 msgr2=0x7f6e44061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:25.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.675+0000 7f6e4b5c9640 1 -- 172.21.15.5:0/3290955469 shutdown_connections 2024-03-24T10:48:25.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.675+0000 7f6e4b5c9640 1 -- 172.21.15.5:0/3290955469 wait complete. 2024-03-24T10:48:25.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.676+0000 7f6e4b5c9640 1 Processor -- start 2024-03-24T10:48:25.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.676+0000 7f6e4b5c9640 1 -- start start 2024-03-24T10:48:25.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.677+0000 7f6e4b5c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e4406bf50 0x7f6e44148fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:25.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.677+0000 7f6e4b5c9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e4406d4f0 0x7f6e441edac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:25.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.677+0000 7f6e4b5c9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e441543a0 0x7f6e440ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:25.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.677+0000 7f6e4b5c9640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f6e44150660 con 0x7f6e4406d4f0 2024-03-24T10:48:25.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.677+0000 7f6e4b5c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4414f200 con 0x7f6e4406bf50 2024-03-24T10:48:25.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.677+0000 7f6e4b5c9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f6e4414e480 con 0x7f6e441543a0 2024-03-24T10:48:25.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.677+0000 7f6e48b64640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e4406d4f0 0x7f6e441edac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:25.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.678+0000 7f6e49365640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e4406bf50 0x7f6e44148fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:25.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.678+0000 7f6e49b66640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e441543a0 0x7f6e440ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:25.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.678+0000 7f6e48b64640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e4406d4f0 0x7f6e441edac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42006/0 (socket says 172.21.15.5:42006) 2024-03-24T10:48:25.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.678+0000 7f6e48b64640 1 -- 172.21.15.5:0/2992661342 learned_addr learned my addr 172.21.15.5:0/2992661342 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:25.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.682+0000 7f6e49b66640 1 -- 172.21.15.5:0/2992661342 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e4406bf50 msgr2=0x7f6e44148fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:25.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.682+0000 7f6e49b66640 1 --2- 172.21.15.5:0/2992661342 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e4406bf50 0x7f6e44148fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:25.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.682+0000 7f6e49b66640 1 -- 172.21.15.5:0/2992661342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e4406d4f0 msgr2=0x7f6e441edac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:25.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.682+0000 7f6e49b66640 1 --2- 172.21.15.5:0/2992661342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e4406d4f0 0x7f6e441edac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:25.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.682+0000 7f6e49b66640 1 -- 172.21.15.5:0/2992661342 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e3c009b30 con 0x7f6e441543a0 2024-03-24T10:48:25.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.685+0000 7f6e49b66640 1 --2- 172.21.15.5:0/2992661342 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e441543a0 0x7f6e440ceb90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c032000 tx=0x7f6e3c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:25.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.690+0000 7f6e39ffb640 1 -- 172.21.15.5:0/2992661342 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c009510 con 0x7f6e441543a0 2024-03-24T10:48:25.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.690+0000 7f6e39ffb640 1 -- 172.21.15.5:0/2992661342 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e3c03e450 con 0x7f6e441543a0 2024-03-24T10:48:25.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.690+0000 7f6e39ffb640 1 -- 172.21.15.5:0/2992661342 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c03e800 con 0x7f6e441543a0 2024-03-24T10:48:25.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.690+0000 7f6e4b5c9640 1 -- 172.21.15.5:0/2992661342 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e441f06e0 con 0x7f6e441543a0 2024-03-24T10:48:25.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.690+0000 7f6e4b5c9640 1 -- 172.21.15.5:0/2992661342 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e4414f780 con 0x7f6e441543a0 2024-03-24T10:48:25.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.691+0000 7f6e4b5c9640 1 -- 172.21.15.5:0/2992661342 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e440681d0 con 0x7f6e441543a0 2024-03-24T10:48:25.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.716+0000 7f6e39ffb640 1 -- 172.21.15.5:0/2992661342 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6e3c03e960 con 0x7f6e441543a0 2024-03-24T10:48:25.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.716+0000 7f6e39ffb640 1 --2- 172.21.15.5:0/2992661342 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6e2806b890 0x7f6e2806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:25.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.716+0000 7f6e49365640 1 --2- 172.21.15.5:0/2992661342 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6e2806b890 0x7f6e2806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:25.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.717+0000 7f6e49365640 1 --2- 172.21.15.5:0/2992661342 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6e2806b890 0x7f6e2806dd50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6e40009aa0 tx=0x7f6e40009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:25.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:25.719+0000 7f6e39ffb640 1 -- 172.21.15.5:0/2992661342 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f6e3c07bcc0 con 0x7f6e441543a0 2024-03-24T10:48:26.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:26.010+0000 7f6e39ffb640 1 -- 172.21.15.5:0/2992661342 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6e3c077f60 con 0x7f6e441543a0 2024-03-24T10:48:26.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:26.237+0000 7f6e4b5c9640 1 -- 172.21.15.5:0/2992661342 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e441ee000 con 0x7f6e441543a0 2024-03-24T10:48:26.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:26.254+0000 7f6e39ffb640 1 -- 172.21.15.5:0/2992661342 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f6e3c07b620 con 0x7f6e441543a0 2024-03-24T10:48:26.255 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:26.255 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:26.255 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:26.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:26.256+0000 7f6e4b5c9640 1 -- 172.21.15.5:0/2992661342 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6e2806b890 msgr2=0x7f6e2806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:26.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:26.256+0000 7f6e4b5c9640 1 --2- 172.21.15.5:0/2992661342 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6e2806b890 0x7f6e2806dd50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6e40009aa0 tx=0x7f6e40009190 comp rx=0 tx=0).stop 2024-03-24T10:48:26.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:26.256+0000 7f6e4b5c9640 1 -- 172.21.15.5:0/2992661342 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e441543a0 msgr2=0x7f6e440ceb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:26.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:26.256+0000 7f6e4b5c9640 1 --2- 172.21.15.5:0/2992661342 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e441543a0 0x7f6e440ceb90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c032000 tx=0x7f6e3c004a40 comp rx=0 tx=0).stop 2024-03-24T10:48:26.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:26.256+0000 7f6e4b5c9640 1 -- 172.21.15.5:0/2992661342 shutdown_connections 2024-03-24T10:48:26.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:26.256+0000 7f6e4b5c9640 1 --2- 172.21.15.5:0/2992661342 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6e2806b890 0x7f6e2806dd50 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:26.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:26.256+0000 7f6e4b5c9640 1 --2- 172.21.15.5:0/2992661342 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e4406d4f0 0x7f6e441edac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:26.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:26.256+0000 7f6e4b5c9640 1 --2- 172.21.15.5:0/2992661342 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e441543a0 0x7f6e440ceb90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:26.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:26.256+0000 7f6e4b5c9640 1 --2- 172.21.15.5:0/2992661342 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e4406bf50 0x7f6e44148fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:26.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:26.256+0000 7f6e4b5c9640 1 -- 172.21.15.5:0/2992661342 >> 172.21.15.5:0/2992661342 conn(0x7f6e44072ef0 msgr2=0x7f6e440c52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:26.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:26.256+0000 7f6e4b5c9640 1 -- 172.21.15.5:0/2992661342 shutdown_connections 2024-03-24T10:48:26.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:26.256+0000 7f6e4b5c9640 1 -- 172.21.15.5:0/2992661342 wait complete. 2024-03-24T10:48:26.872 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:26.872 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:27.873 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:27.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.977+0000 7fea9c1a5640 1 Processor -- start 2024-03-24T10:48:27.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.977+0000 7fea9c1a5640 1 -- start start 2024-03-24T10:48:27.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.978+0000 7fea9c1a5640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fea9415ac00 0x7fea940cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:27.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.978+0000 7fea9c1a5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fea940ccaa0 0x7fea9406bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:27.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.978+0000 7fea9c1a5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea940cb2f0 con 0x7fea941543a0 2024-03-24T10:48:27.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.978+0000 7fea9c1a5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fea940c6750 con 0x7fea940ccaa0 2024-03-24T10:48:27.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.978+0000 7fea9c1a5640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fea940cc380 con 0x7fea9415ac00 2024-03-24T10:48:27.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.979+0000 7fea99740640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fea9415ac00 0x7fea940cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:27.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.979+0000 7fea99f41640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fea941543a0 0x7fea9406d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:57754/0 (socket says 172.21.15.5:57754) 2024-03-24T10:48:27.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.979+0000 7fea99f41640 1 -- 172.21.15.5:0/4026353288 learned_addr learned my addr 172.21.15.5:0/4026353288 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:48:27.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.979+0000 7fea9a742640 1 --2- 172.21.15.5:0/4026353288 >> v2:172.21.15.57:3300/0 conn(0x7fea940ccaa0 0x7fea9406bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:27.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.983+0000 7fea99740640 1 -- 172.21.15.5:0/4026353288 >> v2:172.21.15.57:3300/0 conn(0x7fea940ccaa0 msgr2=0x7fea9406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:27.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.983+0000 7fea99740640 1 --2- 172.21.15.5:0/4026353288 >> v2:172.21.15.57:3300/0 conn(0x7fea940ccaa0 0x7fea9406bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:27.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.983+0000 7fea99740640 1 -- 172.21.15.5:0/4026353288 >> v1:172.21.15.57:6789/0 conn(0x7fea941543a0 legacy=0x7fea9406d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:27.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.983+0000 7fea99740640 1 -- 172.21.15.5:0/4026353288 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea9415b070 con 0x7fea9415ac00 2024-03-24T10:48:27.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.986+0000 7fea99740640 1 --2- 172.21.15.5:0/4026353288 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fea9415ac00 0x7fea940cc560 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fea840097a0 tx=0x7fea8402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=449dd267984b1566 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:27.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.990+0000 7fea98f3f640 1 -- 172.21.15.5:0/4026353288 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea84031830 con 0x7fea9415ac00 2024-03-24T10:48:27.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.991+0000 7fea98f3f640 1 -- 172.21.15.5:0/4026353288 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea84031990 con 0x7fea9415ac00 2024-03-24T10:48:27.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.991+0000 7fea98f3f640 1 -- 172.21.15.5:0/4026353288 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea84031d20 con 0x7fea9415ac00 2024-03-24T10:48:27.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.991+0000 7fea9c1a5640 1 -- 172.21.15.5:0/4026353288 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fea9415ac00 msgr2=0x7fea940cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:27.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.991+0000 7fea9c1a5640 1 --2- 172.21.15.5:0/4026353288 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fea9415ac00 0x7fea940cc560 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fea840097a0 tx=0x7fea8402b8d0 comp rx=0 tx=0).stop 2024-03-24T10:48:27.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.991+0000 7fea9c1a5640 1 -- 172.21.15.5:0/4026353288 shutdown_connections 2024-03-24T10:48:27.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.991+0000 7fea9c1a5640 1 --2- 172.21.15.5:0/4026353288 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fea9415ac00 0x7fea940cc560 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:27.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.991+0000 7fea9c1a5640 1 --2- 172.21.15.5:0/4026353288 >> v2:172.21.15.57:3300/0 conn(0x7fea940ccaa0 0x7fea9406bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:27.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.991+0000 7fea9c1a5640 1 -- 172.21.15.5:0/4026353288 >> 172.21.15.5:0/4026353288 conn(0x7fea94072ef0 msgr2=0x7fea94061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:27.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.991+0000 7fea9c1a5640 1 -- 172.21.15.5:0/4026353288 shutdown_connections 2024-03-24T10:48:27.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.991+0000 7fea9c1a5640 1 -- 172.21.15.5:0/4026353288 wait complete. 2024-03-24T10:48:27.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.992+0000 7fea9c1a5640 1 Processor -- start 2024-03-24T10:48:27.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.993+0000 7fea9c1a5640 1 -- start start 2024-03-24T10:48:27.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.993+0000 7fea9c1a5640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fea940ccaa0 0x7fea9414a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:27.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.993+0000 7fea9c1a5640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea941543a0 0x7fea9414e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:27.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.993+0000 7fea9c1a5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea9415ac00 0x7fea9406ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:27.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.993+0000 7fea9c1a5640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fea941f01e0 con 0x7fea941543a0 2024-03-24T10:48:27.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.993+0000 7fea9c1a5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fea940c7210 con 0x7fea9415ac00 2024-03-24T10:48:27.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.993+0000 7fea9c1a5640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fea94150290 con 0x7fea940ccaa0 2024-03-24T10:48:27.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.994+0000 7fea99f41640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fea940ccaa0 0x7fea9414a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:27.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.994+0000 7fea9a742640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea9415ac00 0x7fea9406ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:27.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.994+0000 7fea99740640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea941543a0 0x7fea9414e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:27.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.994+0000 7fea9a742640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea9415ac00 0x7fea9406ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:35350/0 (socket says 172.21.15.5:35350) 2024-03-24T10:48:27.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.994+0000 7fea9a742640 1 -- 172.21.15.5:0/3018135452 learned_addr learned my addr 172.21.15.5:0/3018135452 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:27.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.998+0000 7fea9a742640 1 -- 172.21.15.5:0/3018135452 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fea940ccaa0 msgr2=0x7fea9414a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:27.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.998+0000 7fea9a742640 1 --2- 172.21.15.5:0/3018135452 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fea940ccaa0 0x7fea9414a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:27.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.998+0000 7fea9a742640 1 -- 172.21.15.5:0/3018135452 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea941543a0 msgr2=0x7fea9414e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:27.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.998+0000 7fea9a742640 1 --2- 172.21.15.5:0/3018135452 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea941543a0 0x7fea9414e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:27.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:27.998+0000 7fea9a742640 1 -- 172.21.15.5:0/3018135452 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea84009400 con 0x7fea9415ac00 2024-03-24T10:48:28.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.001+0000 7fea9a742640 1 --2- 172.21.15.5:0/3018135452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea9415ac00 0x7fea9406ceb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fea8c00efc0 tx=0x7fea8c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:28.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.007+0000 7fea8a7fc640 1 -- 172.21.15.5:0/3018135452 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c014480 con 0x7fea9415ac00 2024-03-24T10:48:28.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.007+0000 7fea8a7fc640 1 -- 172.21.15.5:0/3018135452 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea8c00f040 con 0x7fea9415ac00 2024-03-24T10:48:28.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.007+0000 7fea8a7fc640 1 -- 172.21.15.5:0/3018135452 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c014820 con 0x7fea9415ac00 2024-03-24T10:48:28.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.007+0000 7fea9c1a5640 1 -- 172.21.15.5:0/3018135452 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea94150570 con 0x7fea9415ac00 2024-03-24T10:48:28.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.007+0000 7fea9c1a5640 1 -- 172.21.15.5:0/3018135452 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea941efe00 con 0x7fea9415ac00 2024-03-24T10:48:28.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.008+0000 7fea9c1a5640 1 -- 172.21.15.5:0/3018135452 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea940b2860 con 0x7fea9415ac00 2024-03-24T10:48:28.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.280+0000 7fea8a7fc640 1 -- 172.21.15.5:0/3018135452 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fea8c014980 con 0x7fea9415ac00 2024-03-24T10:48:28.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.281+0000 7fea8a7fc640 1 --2- 172.21.15.5:0/3018135452 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fea7806b890 0x7fea7806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:28.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.281+0000 7fea99f41640 1 --2- 172.21.15.5:0/3018135452 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fea7806b890 0x7fea7806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:28.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.281+0000 7fea99f41640 1 --2- 172.21.15.5:0/3018135452 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fea7806b890 0x7fea7806dd50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fea90007fd0 tx=0x7fea90007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:28.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.284+0000 7fea8a7fc640 1 -- 172.21.15.5:0/3018135452 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fea8c059cd0 con 0x7fea9415ac00 2024-03-24T10:48:28.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.300+0000 7fea8a7fc640 1 -- 172.21.15.5:0/3018135452 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fea8c055f70 con 0x7fea9415ac00 2024-03-24T10:48:28.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.550+0000 7fea9c1a5640 1 -- 172.21.15.5:0/3018135452 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea940681d0 con 0x7fea9415ac00 2024-03-24T10:48:28.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.600+0000 7fea8a7fc640 1 -- 172.21.15.5:0/3018135452 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fea8c018090 con 0x7fea9415ac00 2024-03-24T10:48:28.601 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:28.601 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:28.601 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:28.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.603+0000 7fea9c1a5640 1 -- 172.21.15.5:0/3018135452 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fea7806b890 msgr2=0x7fea7806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:28.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.603+0000 7fea9c1a5640 1 --2- 172.21.15.5:0/3018135452 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fea7806b890 0x7fea7806dd50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fea90007fd0 tx=0x7fea90007e80 comp rx=0 tx=0).stop 2024-03-24T10:48:28.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.603+0000 7fea9c1a5640 1 -- 172.21.15.5:0/3018135452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea9415ac00 msgr2=0x7fea9406ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:28.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.603+0000 7fea9c1a5640 1 --2- 172.21.15.5:0/3018135452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea9415ac00 0x7fea9406ceb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fea8c00efc0 tx=0x7fea8c00c490 comp rx=0 tx=0).stop 2024-03-24T10:48:28.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.603+0000 7fea9c1a5640 1 -- 172.21.15.5:0/3018135452 shutdown_connections 2024-03-24T10:48:28.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.603+0000 7fea9c1a5640 1 --2- 172.21.15.5:0/3018135452 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fea7806b890 0x7fea7806dd50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:28.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.603+0000 7fea9c1a5640 1 --2- 172.21.15.5:0/3018135452 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fea941543a0 0x7fea9414e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:28.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.603+0000 7fea9c1a5640 1 --2- 172.21.15.5:0/3018135452 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fea9415ac00 0x7fea9406ceb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:28.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.603+0000 7fea9c1a5640 1 --2- 172.21.15.5:0/3018135452 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fea940ccaa0 0x7fea9414a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:28.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.603+0000 7fea9c1a5640 1 -- 172.21.15.5:0/3018135452 >> 172.21.15.5:0/3018135452 conn(0x7fea94072ef0 msgr2=0x7fea94158b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:28.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.603+0000 7fea9c1a5640 1 -- 172.21.15.5:0/3018135452 shutdown_connections 2024-03-24T10:48:28.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:28.603+0000 7fea9c1a5640 1 -- 172.21.15.5:0/3018135452 wait complete. 2024-03-24T10:48:29.218 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:29.219 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:30.220 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:30.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.322+0000 7f38b1d08640 1 Processor -- start 2024-03-24T10:48:30.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.322+0000 7f38b1d08640 1 -- start start 2024-03-24T10:48:30.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.323+0000 7f38b1d08640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38ac06bf50 0x7f38ac06c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:30.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.323+0000 7f38b1d08640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38ac0cb2f0 con 0x7f38ac1543a0 2024-03-24T10:48:30.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.323+0000 7f38b1d08640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38ac15b070 con 0x7f38ac15ac00 2024-03-24T10:48:30.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.323+0000 7f38b1d08640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f38ac0c6750 con 0x7f38ac06bf50 2024-03-24T10:48:30.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.323+0000 7f38ab7fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f38ac1543a0 0x7f38ac06d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36658/0 (socket says 172.21.15.5:36658) 2024-03-24T10:48:30.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.324+0000 7f38ab7fe640 1 -- 172.21.15.5:0/3998850800 learned_addr learned my addr 172.21.15.5:0/3998850800 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:48:30.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.324+0000 7f38abfff640 1 --2- 172.21.15.5:0/3998850800 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38ac06bf50 0x7f38ac06c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:30.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.326+0000 7f38aa7fc640 1 -- 172.21.15.5:0/3998850800 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3945973140 0 0) 0x7f38ac15b070 con 0x7f38ac15ac00 2024-03-24T10:48:30.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.326+0000 7f38aa7fc640 1 -- 172.21.15.5:0/3998850800 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3894003520 con 0x7f38ac15ac00 2024-03-24T10:48:30.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.327+0000 7f38aa7fc640 1 -- 172.21.15.5:0/3998850800 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2783385512 0 0) 0x7f38ac0cb2f0 con 0x7f38ac1543a0 2024-03-24T10:48:30.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.327+0000 7f38aa7fc640 1 -- 172.21.15.5:0/3998850800 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38ac15b070 con 0x7f38ac1543a0 2024-03-24T10:48:30.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.328+0000 7f38abfff640 1 -- 172.21.15.5:0/3998850800 >> v1:172.21.15.57:6789/0 conn(0x7f38ac15ac00 legacy=0x7f38ac0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:30.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.328+0000 7f38abfff640 1 -- 172.21.15.5:0/3998850800 >> v1:172.21.15.5:6789/0 conn(0x7f38ac1543a0 legacy=0x7f38ac06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:30.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.328+0000 7f38abfff640 1 -- 172.21.15.5:0/3998850800 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38ac0ca7e0 con 0x7f38ac06bf50 2024-03-24T10:48:30.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.331+0000 7f38abfff640 1 --2- 172.21.15.5:0/3998850800 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38ac06bf50 0x7f38ac06c360 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f38a4009dc0 tx=0x7f38a402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0a1fc7e330adcea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:30.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.334+0000 7f38aa7fc640 1 -- 172.21.15.5:0/3998850800 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a402ccd0 con 0x7f38ac06bf50 2024-03-24T10:48:30.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.336+0000 7f38aa7fc640 1 -- 172.21.15.5:0/3998850800 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38a402ce30 con 0x7f38ac06bf50 2024-03-24T10:48:30.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.337+0000 7f38b1d08640 1 -- 172.21.15.5:0/3998850800 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38ac06bf50 msgr2=0x7f38ac06c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:30.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.337+0000 7f38b1d08640 1 --2- 172.21.15.5:0/3998850800 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38ac06bf50 0x7f38ac06c360 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f38a4009dc0 tx=0x7f38a402bf20 comp rx=0 tx=0).stop 2024-03-24T10:48:30.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.337+0000 7f38b1d08640 1 -- 172.21.15.5:0/3998850800 shutdown_connections 2024-03-24T10:48:30.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.337+0000 7f38b1d08640 1 --2- 172.21.15.5:0/3998850800 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38ac06bf50 0x7f38ac06c360 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:30.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.337+0000 7f38b1d08640 1 -- 172.21.15.5:0/3998850800 >> 172.21.15.5:0/3998850800 conn(0x7f38ac072ef0 msgr2=0x7f38ac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:30.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.337+0000 7f38b1d08640 1 -- 172.21.15.5:0/3998850800 shutdown_connections 2024-03-24T10:48:30.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.337+0000 7f38b1d08640 1 -- 172.21.15.5:0/3998850800 wait complete. 2024-03-24T10:48:30.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.338+0000 7f38b1d08640 1 Processor -- start 2024-03-24T10:48:30.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.338+0000 7f38b1d08640 1 -- start start 2024-03-24T10:48:30.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.339+0000 7f38b1d08640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38ac06bf50 0x7f38ac0b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:30.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.339+0000 7f38b1d08640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38ac1543a0 0x7f38ac1f37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:30.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.339+0000 7f38b1d08640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38ac15ac00 0x7f38ac0cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:30.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.339+0000 7f38b1d08640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f38ac1f3d20 con 0x7f38ac1543a0 2024-03-24T10:48:30.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.339+0000 7f38b1d08640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f38ac147920 con 0x7f38ac06bf50 2024-03-24T10:48:30.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.339+0000 7f38b1d08640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f38ac147c00 con 0x7f38ac15ac00 2024-03-24T10:48:30.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.339+0000 7f38aaffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38ac1543a0 0x7f38ac1f37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:30.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.339+0000 7f38ab7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38ac06bf50 0x7f38ac0b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:30.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.340+0000 7f38aaffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38ac1543a0 0x7f38ac1f37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42016/0 (socket says 172.21.15.5:42016) 2024-03-24T10:48:30.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.340+0000 7f38aaffd640 1 -- 172.21.15.5:0/595507784 learned_addr learned my addr 172.21.15.5:0/595507784 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:30.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.340+0000 7f38abfff640 1 --2- 172.21.15.5:0/595507784 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38ac15ac00 0x7f38ac0cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:30.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.343+0000 7f38aaffd640 1 -- 172.21.15.5:0/595507784 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38ac15ac00 msgr2=0x7f38ac0cf1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:30.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.343+0000 7f38aaffd640 1 --2- 172.21.15.5:0/595507784 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38ac15ac00 0x7f38ac0cf1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:30.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.343+0000 7f38aaffd640 1 -- 172.21.15.5:0/595507784 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38ac06bf50 msgr2=0x7f38ac0b23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:30.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.343+0000 7f38aaffd640 1 --2- 172.21.15.5:0/595507784 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38ac06bf50 0x7f38ac0b23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:30.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.343+0000 7f38aaffd640 1 -- 172.21.15.5:0/595507784 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a4009a70 con 0x7f38ac1543a0 2024-03-24T10:48:30.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.345+0000 7f38aaffd640 1 --2- 172.21.15.5:0/595507784 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38ac1543a0 0x7f38ac1f37e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f38a000c390 tx=0x7f38a000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:30.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.350+0000 7f3893fff640 1 -- 172.21.15.5:0/595507784 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a00052d0 con 0x7f38ac1543a0 2024-03-24T10:48:30.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.350+0000 7f3893fff640 1 -- 172.21.15.5:0/595507784 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38a0005430 con 0x7f38ac1543a0 2024-03-24T10:48:30.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.350+0000 7f38b1d08640 1 -- 172.21.15.5:0/595507784 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38ac147ee0 con 0x7f38ac1543a0 2024-03-24T10:48:30.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.350+0000 7f3893fff640 1 -- 172.21.15.5:0/595507784 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a0003830 con 0x7f38ac1543a0 2024-03-24T10:48:30.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.350+0000 7f38b1d08640 1 -- 172.21.15.5:0/595507784 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38ac1f32c0 con 0x7f38ac1543a0 2024-03-24T10:48:30.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.351+0000 7f38b1d08640 1 -- 172.21.15.5:0/595507784 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38ac1f2e40 con 0x7f38ac1543a0 2024-03-24T10:48:30.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.385+0000 7f3893fff640 1 -- 172.21.15.5:0/595507784 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f38a0003990 con 0x7f38ac1543a0 2024-03-24T10:48:30.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.386+0000 7f3893fff640 1 --2- 172.21.15.5:0/595507784 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f389806b810 0x7f389806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:30.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.386+0000 7f38ab7fe640 1 --2- 172.21.15.5:0/595507784 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f389806b810 0x7f389806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:30.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.386+0000 7f38ab7fe640 1 --2- 172.21.15.5:0/595507784 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f389806b810 0x7f389806dcd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f389c005880 tx=0x7f389c009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:30.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.390+0000 7f3893fff640 1 -- 172.21.15.5:0/595507784 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f38a0058280 con 0x7f38ac1543a0 2024-03-24T10:48:30.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.660+0000 7f3893fff640 1 -- 172.21.15.5:0/595507784 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f38a0054520 con 0x7f38ac1543a0 2024-03-24T10:48:30.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.885+0000 7f38b1d08640 1 -- 172.21.15.5:0/595507784 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38ac1f2e40 con 0x7f38ac1543a0 2024-03-24T10:48:30.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.898+0000 7f3893fff640 1 -- 172.21.15.5:0/595507784 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f38ac1f2e40 con 0x7f38ac1543a0 2024-03-24T10:48:30.898 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:30.898 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:30.898 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:30.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.900+0000 7f38b1d08640 1 -- 172.21.15.5:0/595507784 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f389806b810 msgr2=0x7f389806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:30.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.900+0000 7f38b1d08640 1 --2- 172.21.15.5:0/595507784 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f389806b810 0x7f389806dcd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f389c005880 tx=0x7f389c009040 comp rx=0 tx=0).stop 2024-03-24T10:48:30.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.900+0000 7f38b1d08640 1 -- 172.21.15.5:0/595507784 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38ac1543a0 msgr2=0x7f38ac1f37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:30.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.900+0000 7f38b1d08640 1 --2- 172.21.15.5:0/595507784 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38ac1543a0 0x7f38ac1f37e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f38a000c390 tx=0x7f38a000c850 comp rx=0 tx=0).stop 2024-03-24T10:48:30.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.900+0000 7f38b1d08640 1 -- 172.21.15.5:0/595507784 shutdown_connections 2024-03-24T10:48:30.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.900+0000 7f38b1d08640 1 --2- 172.21.15.5:0/595507784 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f389806b810 0x7f389806dcd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:30.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.900+0000 7f38b1d08640 1 --2- 172.21.15.5:0/595507784 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38ac1543a0 0x7f38ac1f37e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:30.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.900+0000 7f38b1d08640 1 --2- 172.21.15.5:0/595507784 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38ac15ac00 0x7f38ac0cf1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:30.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.900+0000 7f38b1d08640 1 --2- 172.21.15.5:0/595507784 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38ac06bf50 0x7f38ac0b23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:30.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.900+0000 7f38b1d08640 1 -- 172.21.15.5:0/595507784 >> 172.21.15.5:0/595507784 conn(0x7f38ac072ef0 msgr2=0x7f38ac151b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:30.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.900+0000 7f38b1d08640 1 -- 172.21.15.5:0/595507784 shutdown_connections 2024-03-24T10:48:30.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:30.901+0000 7f38b1d08640 1 -- 172.21.15.5:0/595507784 wait complete. 2024-03-24T10:48:31.515 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:31.516 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:31.567 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:48:31.570 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:48:32.517 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:32.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.619+0000 7f4a4a781640 1 Processor -- start 2024-03-24T10:48:32.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.619+0000 7f4a4a781640 1 -- start start 2024-03-24T10:48:32.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.619+0000 7f4a4a781640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4a441543a0 0x7f4a4415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:32.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.619+0000 7f4a4a781640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4a4406d4f0 0x7f4a440cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:32.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.619+0000 7f4a4a781640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a440cb2f0 con 0x7f4a440ccaa0 2024-03-24T10:48:32.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.619+0000 7f4a4a781640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f4a440c6750 con 0x7f4a4406d4f0 2024-03-24T10:48:32.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.619+0000 7f4a4a781640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f4a440cc380 con 0x7f4a441543a0 2024-03-24T10:48:32.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.620+0000 7f4a437fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4a4406d4f0 0x7f4a440cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:32.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.621+0000 7f4a48d1e640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f4a440ccaa0 0x7f4a4406bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:57270/0 (socket says 172.21.15.5:57270) 2024-03-24T10:48:32.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.621+0000 7f4a48d1e640 1 -- 172.21.15.5:0/458884549 learned_addr learned my addr 172.21.15.5:0/458884549 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:48:32.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.621+0000 7f4a43fff640 1 --2- 172.21.15.5:0/458884549 >> v2:172.21.15.57:3300/0 conn(0x7f4a441543a0 0x7f4a4415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:32.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.623+0000 7f4a437fe640 1 -- 172.21.15.5:0/458884549 >> v2:172.21.15.57:3300/0 conn(0x7f4a441543a0 msgr2=0x7f4a4415ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:32.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.623+0000 7f4a437fe640 1 --2- 172.21.15.5:0/458884549 >> v2:172.21.15.57:3300/0 conn(0x7f4a441543a0 0x7f4a4415ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:32.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.623+0000 7f4a437fe640 1 -- 172.21.15.5:0/458884549 >> v1:172.21.15.57:6789/0 conn(0x7f4a440ccaa0 legacy=0x7f4a4406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:32.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.623+0000 7f4a437fe640 1 -- 172.21.15.5:0/458884549 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a440ca7e0 con 0x7f4a4406d4f0 2024-03-24T10:48:32.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.625+0000 7f4a437fe640 1 --2- 172.21.15.5:0/458884549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a4406d4f0 0x7f4a440cc560 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4a440cd600 tx=0x7f4a3802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c6611f830deae4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:32.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.629+0000 7f4a42ffd640 1 -- 172.21.15.5:0/458884549 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a38030610 con 0x7f4a4406d4f0 2024-03-24T10:48:32.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.629+0000 7f4a42ffd640 1 -- 172.21.15.5:0/458884549 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a38030770 con 0x7f4a4406d4f0 2024-03-24T10:48:32.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.629+0000 7f4a42ffd640 1 -- 172.21.15.5:0/458884549 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a38030b00 con 0x7f4a4406d4f0 2024-03-24T10:48:32.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.630+0000 7f4a4a781640 1 -- 172.21.15.5:0/458884549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a4406d4f0 msgr2=0x7f4a440cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:32.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.630+0000 7f4a4a781640 1 --2- 172.21.15.5:0/458884549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a4406d4f0 0x7f4a440cc560 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4a440cd600 tx=0x7f4a3802b960 comp rx=0 tx=0).stop 2024-03-24T10:48:32.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.630+0000 7f4a4a781640 1 -- 172.21.15.5:0/458884549 shutdown_connections 2024-03-24T10:48:32.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.630+0000 7f4a4a781640 1 --2- 172.21.15.5:0/458884549 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a4406d4f0 0x7f4a440cc560 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:32.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.630+0000 7f4a4a781640 1 --2- 172.21.15.5:0/458884549 >> v2:172.21.15.57:3300/0 conn(0x7f4a441543a0 0x7f4a4415ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:32.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.630+0000 7f4a4a781640 1 -- 172.21.15.5:0/458884549 >> 172.21.15.5:0/458884549 conn(0x7f4a44072ef0 msgr2=0x7f4a44061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:32.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.630+0000 7f4a4a781640 1 -- 172.21.15.5:0/458884549 shutdown_connections 2024-03-24T10:48:32.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.630+0000 7f4a4a781640 1 -- 172.21.15.5:0/458884549 wait complete. 2024-03-24T10:48:32.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.631+0000 7f4a4a781640 1 Processor -- start 2024-03-24T10:48:32.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.632+0000 7f4a4a781640 1 -- start start 2024-03-24T10:48:32.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.632+0000 7f4a4a781640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a4406d4f0 0x7f4a440c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:32.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.632+0000 7f4a4a781640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a440ccaa0 0x7f4a441f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:32.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.632+0000 7f4a4a781640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a441543a0 0x7f4a44159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:32.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.632+0000 7f4a4a781640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f4a441f40f0 con 0x7f4a441543a0 2024-03-24T10:48:32.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.632+0000 7f4a4a781640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4a44151690 con 0x7f4a4406d4f0 2024-03-24T10:48:32.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.632+0000 7f4a4a781640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4a44150a90 con 0x7f4a440ccaa0 2024-03-24T10:48:32.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.633+0000 7f4a437fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a440ccaa0 0x7f4a441f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:32.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.633+0000 7f4a48d1e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a441543a0 0x7f4a44159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:32.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.633+0000 7f4a43fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a4406d4f0 0x7f4a440c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:32.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.633+0000 7f4a48d1e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a441543a0 0x7f4a44159350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:53966/0 (socket says 172.21.15.5:53966) 2024-03-24T10:48:32.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.633+0000 7f4a48d1e640 1 -- 172.21.15.5:0/495320432 learned_addr learned my addr 172.21.15.5:0/495320432 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:32.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.638+0000 7f4a48d1e640 1 -- 172.21.15.5:0/495320432 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a440ccaa0 msgr2=0x7f4a441f3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:32.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.638+0000 7f4a48d1e640 1 --2- 172.21.15.5:0/495320432 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a440ccaa0 0x7f4a441f3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:32.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.638+0000 7f4a48d1e640 1 -- 172.21.15.5:0/495320432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a4406d4f0 msgr2=0x7f4a440c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:32.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.638+0000 7f4a48d1e640 1 --2- 172.21.15.5:0/495320432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a4406d4f0 0x7f4a440c9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:32.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.638+0000 7f4a48d1e640 1 -- 172.21.15.5:0/495320432 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a38009400 con 0x7f4a441543a0 2024-03-24T10:48:32.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.642+0000 7f4a48d1e640 1 --2- 172.21.15.5:0/495320432 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a441543a0 0x7f4a44159350 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c00cbc0 tx=0x7f4a3c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:32.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.646+0000 7f4a40ff9640 1 -- 172.21.15.5:0/495320432 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c00ad40 con 0x7f4a441543a0 2024-03-24T10:48:32.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.646+0000 7f4a40ff9640 1 -- 172.21.15.5:0/495320432 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a3c012400 con 0x7f4a441543a0 2024-03-24T10:48:32.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.646+0000 7f4a40ff9640 1 -- 172.21.15.5:0/495320432 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c00aea0 con 0x7f4a441543a0 2024-03-24T10:48:32.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.646+0000 7f4a4a781640 1 -- 172.21.15.5:0/495320432 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a44150d30 con 0x7f4a441543a0 2024-03-24T10:48:32.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.646+0000 7f4a4a781640 1 -- 172.21.15.5:0/495320432 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a441efdd0 con 0x7f4a441543a0 2024-03-24T10:48:32.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.648+0000 7f4a4a781640 1 -- 172.21.15.5:0/495320432 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a440c9ff0 con 0x7f4a441543a0 2024-03-24T10:48:32.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.679+0000 7f4a40ff9640 1 -- 172.21.15.5:0/495320432 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4a3c007070 con 0x7f4a441543a0 2024-03-24T10:48:32.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.679+0000 7f4a40ff9640 1 --2- 172.21.15.5:0/495320432 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4a3006b860 0x7f4a3006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:32.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.679+0000 7f4a43fff640 1 --2- 172.21.15.5:0/495320432 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4a3006b860 0x7f4a3006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:32.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.680+0000 7f4a43fff640 1 --2- 172.21.15.5:0/495320432 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4a3006b860 0x7f4a3006dd20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4a340079a0 tx=0x7f4a34008040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:32.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.685+0000 7f4a40ff9640 1 -- 172.21.15.5:0/495320432 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f4a3c058dc0 con 0x7f4a441543a0 2024-03-24T10:48:32.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:32.952+0000 7f4a40ff9640 1 -- 172.21.15.5:0/495320432 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4a3c055060 con 0x7f4a441543a0 2024-03-24T10:48:33.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:33.174+0000 7f4a4a781640 1 -- 172.21.15.5:0/495320432 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a441efc30 con 0x7f4a441543a0 2024-03-24T10:48:33.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:33.189+0000 7f4a40ff9640 1 -- 172.21.15.5:0/495320432 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f4a441efc30 con 0x7f4a441543a0 2024-03-24T10:48:33.190 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:33.190 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:33.190 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:33.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:33.192+0000 7f4a4a781640 1 -- 172.21.15.5:0/495320432 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4a3006b860 msgr2=0x7f4a3006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:33.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:33.192+0000 7f4a4a781640 1 --2- 172.21.15.5:0/495320432 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4a3006b860 0x7f4a3006dd20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4a340079a0 tx=0x7f4a34008040 comp rx=0 tx=0).stop 2024-03-24T10:48:33.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:33.192+0000 7f4a4a781640 1 -- 172.21.15.5:0/495320432 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a441543a0 msgr2=0x7f4a44159350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:33.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:33.192+0000 7f4a4a781640 1 --2- 172.21.15.5:0/495320432 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a441543a0 0x7f4a44159350 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c00cbc0 tx=0x7f4a3c00a430 comp rx=0 tx=0).stop 2024-03-24T10:48:33.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:33.192+0000 7f4a4a781640 1 -- 172.21.15.5:0/495320432 shutdown_connections 2024-03-24T10:48:33.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:33.192+0000 7f4a4a781640 1 --2- 172.21.15.5:0/495320432 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4a3006b860 0x7f4a3006dd20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:33.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:33.192+0000 7f4a4a781640 1 --2- 172.21.15.5:0/495320432 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a441543a0 0x7f4a44159350 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:33.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:33.192+0000 7f4a4a781640 1 --2- 172.21.15.5:0/495320432 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a440ccaa0 0x7f4a441f3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:33.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:33.192+0000 7f4a4a781640 1 --2- 172.21.15.5:0/495320432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a4406d4f0 0x7f4a440c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:33.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:33.192+0000 7f4a4a781640 1 -- 172.21.15.5:0/495320432 >> 172.21.15.5:0/495320432 conn(0x7f4a44072ef0 msgr2=0x7f4a4406be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:33.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:33.192+0000 7f4a4a781640 1 -- 172.21.15.5:0/495320432 shutdown_connections 2024-03-24T10:48:33.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:33.192+0000 7f4a4a781640 1 -- 172.21.15.5:0/495320432 wait complete. 2024-03-24T10:48:33.807 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:33.807 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:34.808 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:34.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.910+0000 7fcf52756640 1 Processor -- start 2024-03-24T10:48:34.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.910+0000 7fcf52756640 1 -- start start 2024-03-24T10:48:34.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.911+0000 7fcf52756640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcf4c06bf50 0x7fcf4c06c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:34.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.911+0000 7fcf52756640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf4c0cb2f0 con 0x7fcf4c15ac00 2024-03-24T10:48:34.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.911+0000 7fcf52756640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf4c15b070 con 0x7fcf4c1543a0 2024-03-24T10:48:34.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.911+0000 7fcf52756640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fcf4c0c6750 con 0x7fcf4c06bf50 2024-03-24T10:48:34.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.912+0000 7fcf4b7fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fcf4c15ac00 0x7fcf4c0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:58420/0 (socket says 172.21.15.5:58420) 2024-03-24T10:48:34.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.912+0000 7fcf4b7fe640 1 -- 172.21.15.5:0/1607731551 learned_addr learned my addr 172.21.15.5:0/1607731551 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:48:34.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.912+0000 7fcf50cf3640 1 --2- 172.21.15.5:0/1607731551 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcf4c06bf50 0x7fcf4c06c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:34.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.914+0000 7fcf4affd640 1 -- 172.21.15.5:0/1607731551 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3358859131 0 0) 0x7fcf4c0cb2f0 con 0x7fcf4c15ac00 2024-03-24T10:48:34.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.914+0000 7fcf4affd640 1 -- 172.21.15.5:0/1607731551 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf34003520 con 0x7fcf4c15ac00 2024-03-24T10:48:34.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.915+0000 7fcf4affd640 1 -- 172.21.15.5:0/1607731551 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2929905220 0 0) 0x7fcf4c15b070 con 0x7fcf4c1543a0 2024-03-24T10:48:34.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.915+0000 7fcf4affd640 1 -- 172.21.15.5:0/1607731551 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf4c0cb2f0 con 0x7fcf4c1543a0 2024-03-24T10:48:34.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.915+0000 7fcf50cf3640 1 -- 172.21.15.5:0/1607731551 >> v1:172.21.15.57:6789/0 conn(0x7fcf4c1543a0 legacy=0x7fcf4c06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:34.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.915+0000 7fcf50cf3640 1 -- 172.21.15.5:0/1607731551 >> v1:172.21.15.5:6789/0 conn(0x7fcf4c15ac00 legacy=0x7fcf4c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:34.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.916+0000 7fcf50cf3640 1 -- 172.21.15.5:0/1607731551 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf4c0ca7e0 con 0x7fcf4c06bf50 2024-03-24T10:48:34.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.918+0000 7fcf50cf3640 1 --2- 172.21.15.5:0/1607731551 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcf4c06bf50 0x7fcf4c06c360 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcf44009dc0 tx=0x7fcf4402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=422f20073f042182 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:34.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.924+0000 7fcf4affd640 1 -- 172.21.15.5:0/1607731551 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4402ccd0 con 0x7fcf4c06bf50 2024-03-24T10:48:34.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.924+0000 7fcf4affd640 1 -- 172.21.15.5:0/1607731551 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf4402ce30 con 0x7fcf4c06bf50 2024-03-24T10:48:34.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.924+0000 7fcf4affd640 1 -- 172.21.15.5:0/1607731551 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf44034660 con 0x7fcf4c06bf50 2024-03-24T10:48:34.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.925+0000 7fcf52756640 1 -- 172.21.15.5:0/1607731551 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcf4c06bf50 msgr2=0x7fcf4c06c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:34.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.925+0000 7fcf52756640 1 --2- 172.21.15.5:0/1607731551 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcf4c06bf50 0x7fcf4c06c360 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcf44009dc0 tx=0x7fcf4402bf20 comp rx=0 tx=0).stop 2024-03-24T10:48:34.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.925+0000 7fcf52756640 1 -- 172.21.15.5:0/1607731551 shutdown_connections 2024-03-24T10:48:34.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.925+0000 7fcf52756640 1 --2- 172.21.15.5:0/1607731551 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcf4c06bf50 0x7fcf4c06c360 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:34.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.925+0000 7fcf52756640 1 -- 172.21.15.5:0/1607731551 >> 172.21.15.5:0/1607731551 conn(0x7fcf4c072ef0 msgr2=0x7fcf4c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:34.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.925+0000 7fcf52756640 1 -- 172.21.15.5:0/1607731551 shutdown_connections 2024-03-24T10:48:34.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.925+0000 7fcf52756640 1 -- 172.21.15.5:0/1607731551 wait complete. 2024-03-24T10:48:34.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.927+0000 7fcf52756640 1 Processor -- start 2024-03-24T10:48:34.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.927+0000 7fcf52756640 1 -- start start 2024-03-24T10:48:34.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.928+0000 7fcf52756640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf4c06bf50 0x7fcf4c159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:34.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.928+0000 7fcf52756640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcf4c1543a0 0x7fcf4c0c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:34.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.928+0000 7fcf52756640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf4c15ac00 0x7fcf4c1f0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:34.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.928+0000 7fcf52756640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4c15b910 con 0x7fcf4c15ac00 2024-03-24T10:48:34.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.928+0000 7fcf52756640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4c1f0750 con 0x7fcf4c06bf50 2024-03-24T10:48:34.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.928+0000 7fcf52756640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fcf4c15b350 con 0x7fcf4c1543a0 2024-03-24T10:48:34.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.929+0000 7fcf50cf3640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf4c15ac00 0x7fcf4c1f0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:34.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.929+0000 7fcf4bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf4c06bf50 0x7fcf4c159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:34.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.929+0000 7fcf4b7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcf4c1543a0 0x7fcf4c0c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:34.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.929+0000 7fcf50cf3640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf4c15ac00 0x7fcf4c1f0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:53978/0 (socket says 172.21.15.5:53978) 2024-03-24T10:48:34.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.929+0000 7fcf50cf3640 1 -- 172.21.15.5:0/2571142207 learned_addr learned my addr 172.21.15.5:0/2571142207 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:34.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.932+0000 7fcf50cf3640 1 -- 172.21.15.5:0/2571142207 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcf4c1543a0 msgr2=0x7fcf4c0c83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:34.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.932+0000 7fcf50cf3640 1 --2- 172.21.15.5:0/2571142207 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcf4c1543a0 0x7fcf4c0c83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:34.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.932+0000 7fcf50cf3640 1 -- 172.21.15.5:0/2571142207 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf4c06bf50 msgr2=0x7fcf4c159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:34.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.932+0000 7fcf50cf3640 1 --2- 172.21.15.5:0/2571142207 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf4c06bf50 0x7fcf4c159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:34.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.932+0000 7fcf50cf3640 1 -- 172.21.15.5:0/2571142207 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf44009a70 con 0x7fcf4c15ac00 2024-03-24T10:48:34.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.935+0000 7fcf50cf3640 1 --2- 172.21.15.5:0/2571142207 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf4c15ac00 0x7fcf4c1f0210 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcf4402dfd0 tx=0x7fcf44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:34.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.938+0000 7fcf48ff9640 1 -- 172.21.15.5:0/2571142207 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4403d600 con 0x7fcf4c15ac00 2024-03-24T10:48:34.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.938+0000 7fcf48ff9640 1 -- 172.21.15.5:0/2571142207 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf4403d760 con 0x7fcf4c15ac00 2024-03-24T10:48:34.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.938+0000 7fcf52756640 1 -- 172.21.15.5:0/2571142207 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf4c15b630 con 0x7fcf4c15ac00 2024-03-24T10:48:34.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.938+0000 7fcf48ff9640 1 -- 172.21.15.5:0/2571142207 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4403db10 con 0x7fcf4c15ac00 2024-03-24T10:48:34.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.938+0000 7fcf52756640 1 -- 172.21.15.5:0/2571142207 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf4c159430 con 0x7fcf4c15ac00 2024-03-24T10:48:34.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:34.940+0000 7fcf52756640 1 -- 172.21.15.5:0/2571142207 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf4c158fb0 con 0x7fcf4c15ac00 2024-03-24T10:48:35.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:35.209+0000 7fcf48ff9640 1 -- 172.21.15.5:0/2571142207 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcf4403b040 con 0x7fcf4c15ac00 2024-03-24T10:48:35.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:35.210+0000 7fcf48ff9640 1 --2- 172.21.15.5:0/2571142207 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcf3806b890 0x7fcf3806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:35.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:35.210+0000 7fcf4bfff640 1 --2- 172.21.15.5:0/2571142207 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcf3806b890 0x7fcf3806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:35.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:35.210+0000 7fcf4bfff640 1 --2- 172.21.15.5:0/2571142207 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcf3806b890 0x7fcf3806dd50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c0011e0 tx=0x7fcf3c009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:35.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:35.224+0000 7fcf48ff9640 1 -- 172.21.15.5:0/2571142207 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fcf4407b2d0 con 0x7fcf4c15ac00 2024-03-24T10:48:35.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:35.225+0000 7fcf48ff9640 1 -- 172.21.15.5:0/2571142207 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcf44077570 con 0x7fcf4c15ac00 2024-03-24T10:48:35.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:35.488+0000 7fcf52756640 1 -- 172.21.15.5:0/2571142207 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf4c158fb0 con 0x7fcf4c15ac00 2024-03-24T10:48:35.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:35.501+0000 7fcf48ff9640 1 -- 172.21.15.5:0/2571142207 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fcf44041090 con 0x7fcf4c15ac00 2024-03-24T10:48:35.501 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:35.501 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:35.502 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:35.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:35.503+0000 7fcf52756640 1 -- 172.21.15.5:0/2571142207 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcf3806b890 msgr2=0x7fcf3806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:35.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:35.503+0000 7fcf52756640 1 --2- 172.21.15.5:0/2571142207 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcf3806b890 0x7fcf3806dd50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c0011e0 tx=0x7fcf3c009040 comp rx=0 tx=0).stop 2024-03-24T10:48:35.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:35.503+0000 7fcf52756640 1 -- 172.21.15.5:0/2571142207 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf4c15ac00 msgr2=0x7fcf4c1f0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:35.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:35.503+0000 7fcf52756640 1 --2- 172.21.15.5:0/2571142207 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf4c15ac00 0x7fcf4c1f0210 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcf4402dfd0 tx=0x7fcf44004a40 comp rx=0 tx=0).stop 2024-03-24T10:48:35.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:35.503+0000 7fcf52756640 1 -- 172.21.15.5:0/2571142207 shutdown_connections 2024-03-24T10:48:35.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:35.503+0000 7fcf52756640 1 --2- 172.21.15.5:0/2571142207 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcf3806b890 0x7fcf3806dd50 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:35.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:35.503+0000 7fcf52756640 1 --2- 172.21.15.5:0/2571142207 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcf4c15ac00 0x7fcf4c1f0210 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:35.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:35.503+0000 7fcf52756640 1 --2- 172.21.15.5:0/2571142207 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcf4c1543a0 0x7fcf4c0c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:35.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:35.503+0000 7fcf52756640 1 --2- 172.21.15.5:0/2571142207 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcf4c06bf50 0x7fcf4c159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:35.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:35.503+0000 7fcf52756640 1 -- 172.21.15.5:0/2571142207 >> 172.21.15.5:0/2571142207 conn(0x7fcf4c072ef0 msgr2=0x7fcf4c06d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:35.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:35.504+0000 7fcf52756640 1 -- 172.21.15.5:0/2571142207 shutdown_connections 2024-03-24T10:48:35.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:35.504+0000 7fcf52756640 1 -- 172.21.15.5:0/2571142207 wait complete. 2024-03-24T10:48:36.120 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:36.120 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:37.120 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:37.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.222+0000 7fc668db5640 1 Processor -- start 2024-03-24T10:48:37.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.222+0000 7fc668db5640 1 -- start start 2024-03-24T10:48:37.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.223+0000 7fc668db5640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc6641543a0 0x7fc66415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:37.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.223+0000 7fc668db5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc66406d4f0 0x7fc6640cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:37.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.223+0000 7fc668db5640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6640cb2f0 con 0x7fc6640ccaa0 2024-03-24T10:48:37.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.223+0000 7fc668db5640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fc6640c6750 con 0x7fc6641543a0 2024-03-24T10:48:37.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.223+0000 7fc668db5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fc6640cc380 con 0x7fc66406d4f0 2024-03-24T10:48:37.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.224+0000 7fc661d9b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc66406d4f0 0x7fc6640cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:37.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.224+0000 7fc661d9b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc66406d4f0 0x7fc6640cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:40324/0 (socket says 172.21.15.5:40324) 2024-03-24T10:48:37.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.224+0000 7fc661d9b640 1 -- 172.21.15.5:0/2025641784 learned_addr learned my addr 172.21.15.5:0/2025641784 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:37.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.224+0000 7fc66259c640 1 --2- 172.21.15.5:0/2025641784 >> v2:172.21.15.5:3300/0 conn(0x7fc6641543a0 0x7fc66415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:37.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.226+0000 7fc661d9b640 1 -- 172.21.15.5:0/2025641784 >> v2:172.21.15.5:3300/0 conn(0x7fc6641543a0 msgr2=0x7fc66415ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:37.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.226+0000 7fc661d9b640 1 --2- 172.21.15.5:0/2025641784 >> v2:172.21.15.5:3300/0 conn(0x7fc6641543a0 0x7fc66415ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:37.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.226+0000 7fc661d9b640 1 -- 172.21.15.5:0/2025641784 >> v1:172.21.15.5:6789/0 conn(0x7fc6640ccaa0 legacy=0x7fc66406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:37.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.226+0000 7fc661d9b640 1 -- 172.21.15.5:0/2025641784 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6640ca7e0 con 0x7fc66406d4f0 2024-03-24T10:48:37.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.229+0000 7fc661d9b640 1 --2- 172.21.15.5:0/2025641784 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc66406d4f0 0x7fc6640cc560 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc6640c6990 tx=0x7fc65802ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d83843ad4b0d72b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:37.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.234+0000 7fc66159a640 1 -- 172.21.15.5:0/2025641784 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65802f850 con 0x7fc66406d4f0 2024-03-24T10:48:37.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.234+0000 7fc66159a640 1 -- 172.21.15.5:0/2025641784 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc65802f9b0 con 0x7fc66406d4f0 2024-03-24T10:48:37.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.234+0000 7fc66159a640 1 -- 172.21.15.5:0/2025641784 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65802fd40 con 0x7fc66406d4f0 2024-03-24T10:48:37.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.234+0000 7fc668db5640 1 -- 172.21.15.5:0/2025641784 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc66406d4f0 msgr2=0x7fc6640cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:37.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.234+0000 7fc668db5640 1 --2- 172.21.15.5:0/2025641784 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc66406d4f0 0x7fc6640cc560 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc6640c6990 tx=0x7fc65802ba30 comp rx=0 tx=0).stop 2024-03-24T10:48:37.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.235+0000 7fc668db5640 1 -- 172.21.15.5:0/2025641784 shutdown_connections 2024-03-24T10:48:37.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.235+0000 7fc668db5640 1 --2- 172.21.15.5:0/2025641784 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc66406d4f0 0x7fc6640cc560 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:37.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.235+0000 7fc668db5640 1 --2- 172.21.15.5:0/2025641784 >> v2:172.21.15.5:3300/0 conn(0x7fc6641543a0 0x7fc66415ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:37.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.235+0000 7fc668db5640 1 -- 172.21.15.5:0/2025641784 >> 172.21.15.5:0/2025641784 conn(0x7fc664072ef0 msgr2=0x7fc664061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:37.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.235+0000 7fc668db5640 1 -- 172.21.15.5:0/2025641784 shutdown_connections 2024-03-24T10:48:37.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.235+0000 7fc668db5640 1 -- 172.21.15.5:0/2025641784 wait complete. 2024-03-24T10:48:37.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.236+0000 7fc668db5640 1 Processor -- start 2024-03-24T10:48:37.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.236+0000 7fc668db5640 1 -- start start 2024-03-24T10:48:37.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.237+0000 7fc668db5640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc66406d4f0 0x7fc66414e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:37.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.237+0000 7fc668db5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6640ccaa0 0x7fc664150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:37.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.237+0000 7fc668db5640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6641543a0 0x7fc6641506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:37.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.237+0000 7fc668db5640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fc66414fd40 con 0x7fc6641543a0 2024-03-24T10:48:37.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.237+0000 7fc668db5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc6641f06e0 con 0x7fc6640ccaa0 2024-03-24T10:48:37.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.237+0000 7fc668db5640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc664159e50 con 0x7fc66406d4f0 2024-03-24T10:48:37.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.238+0000 7fc66259c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc66406d4f0 0x7fc66414e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:37.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.238+0000 7fc662d9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6641543a0 0x7fc6641506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:37.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.238+0000 7fc662d9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6641543a0 0x7fc6641506a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54006/0 (socket says 172.21.15.5:54006) 2024-03-24T10:48:37.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.238+0000 7fc662d9d640 1 -- 172.21.15.5:0/2877265719 learned_addr learned my addr 172.21.15.5:0/2877265719 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:37.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.239+0000 7fc661d9b640 1 --2- 172.21.15.5:0/2877265719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6640ccaa0 0x7fc664150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:37.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.242+0000 7fc662d9d640 1 -- 172.21.15.5:0/2877265719 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc66406d4f0 msgr2=0x7fc66414e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:37.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.242+0000 7fc662d9d640 1 --2- 172.21.15.5:0/2877265719 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc66406d4f0 0x7fc66414e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:37.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.242+0000 7fc662d9d640 1 -- 172.21.15.5:0/2877265719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6640ccaa0 msgr2=0x7fc664150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:37.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.242+0000 7fc662d9d640 1 --2- 172.21.15.5:0/2877265719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6640ccaa0 0x7fc664150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:37.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.242+0000 7fc662d9d640 1 -- 172.21.15.5:0/2877265719 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6580094d0 con 0x7fc6641543a0 2024-03-24T10:48:37.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.246+0000 7fc662d9d640 1 --2- 172.21.15.5:0/2877265719 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6641543a0 0x7fc6641506a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00c7a0 tx=0x7fc65c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:37.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.251+0000 7fc652ffd640 1 -- 172.21.15.5:0/2877265719 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c003b70 con 0x7fc6641543a0 2024-03-24T10:48:37.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.251+0000 7fc652ffd640 1 -- 172.21.15.5:0/2877265719 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc65c003cf0 con 0x7fc6641543a0 2024-03-24T10:48:37.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.251+0000 7fc652ffd640 1 -- 172.21.15.5:0/2877265719 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc65c00aa90 con 0x7fc6641543a0 2024-03-24T10:48:37.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.251+0000 7fc668db5640 1 -- 172.21.15.5:0/2877265719 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6641f0950 con 0x7fc6641543a0 2024-03-24T10:48:37.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.251+0000 7fc668db5640 1 -- 172.21.15.5:0/2877265719 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc66414f600 con 0x7fc6641543a0 2024-03-24T10:48:37.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.252+0000 7fc668db5640 1 -- 172.21.15.5:0/2877265719 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc66414fe80 con 0x7fc6641543a0 2024-03-24T10:48:37.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.277+0000 7fc652ffd640 1 -- 172.21.15.5:0/2877265719 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc65c00abf0 con 0x7fc6641543a0 2024-03-24T10:48:37.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.277+0000 7fc652ffd640 1 --2- 172.21.15.5:0/2877265719 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc64406b810 0x7fc64406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:37.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.278+0000 7fc66259c640 1 --2- 172.21.15.5:0/2877265719 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc64406b810 0x7fc64406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:37.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.278+0000 7fc66259c640 1 --2- 172.21.15.5:0/2877265719 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc64406b810 0x7fc64406dcd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc654009aa0 tx=0x7fc654009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:37.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.550+0000 7fc652ffd640 1 -- 172.21.15.5:0/2877265719 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fc65c059080 con 0x7fc6641543a0 2024-03-24T10:48:37.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.550+0000 7fc652ffd640 1 -- 172.21.15.5:0/2877265719 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc65c055320 con 0x7fc6641543a0 2024-03-24T10:48:37.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.807+0000 7fc668db5640 1 -- 172.21.15.5:0/2877265719 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc66414fe80 con 0x7fc6641543a0 2024-03-24T10:48:37.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.820+0000 7fc652ffd640 1 -- 172.21.15.5:0/2877265719 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fc66414fe80 con 0x7fc6641543a0 2024-03-24T10:48:37.820 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:37.820 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:37.820 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:37.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.822+0000 7fc668db5640 1 -- 172.21.15.5:0/2877265719 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc64406b810 msgr2=0x7fc64406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:37.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.822+0000 7fc668db5640 1 --2- 172.21.15.5:0/2877265719 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc64406b810 0x7fc64406dcd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc654009aa0 tx=0x7fc654009190 comp rx=0 tx=0).stop 2024-03-24T10:48:37.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.822+0000 7fc668db5640 1 -- 172.21.15.5:0/2877265719 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6641543a0 msgr2=0x7fc6641506a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:37.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.822+0000 7fc668db5640 1 --2- 172.21.15.5:0/2877265719 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6641543a0 0x7fc6641506a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc65c00c7a0 tx=0x7fc65c00cc60 comp rx=0 tx=0).stop 2024-03-24T10:48:37.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.822+0000 7fc668db5640 1 -- 172.21.15.5:0/2877265719 shutdown_connections 2024-03-24T10:48:37.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.822+0000 7fc668db5640 1 --2- 172.21.15.5:0/2877265719 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc64406b810 0x7fc64406dcd0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:37.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.822+0000 7fc668db5640 1 --2- 172.21.15.5:0/2877265719 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6641543a0 0x7fc6641506a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:37.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.822+0000 7fc668db5640 1 --2- 172.21.15.5:0/2877265719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6640ccaa0 0x7fc664150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:37.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.822+0000 7fc668db5640 1 --2- 172.21.15.5:0/2877265719 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc66406d4f0 0x7fc66414e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:37.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.822+0000 7fc668db5640 1 -- 172.21.15.5:0/2877265719 >> 172.21.15.5:0/2877265719 conn(0x7fc664072ef0 msgr2=0x7fc664157f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:37.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.822+0000 7fc668db5640 1 -- 172.21.15.5:0/2877265719 shutdown_connections 2024-03-24T10:48:37.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:37.822+0000 7fc668db5640 1 -- 172.21.15.5:0/2877265719 wait complete. 2024-03-24T10:48:38.436 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:38.436 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:39.438 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:39.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.539+0000 7f655c655640 1 Processor -- start 2024-03-24T10:48:39.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.539+0000 7f655c655640 1 -- start start 2024-03-24T10:48:39.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.541+0000 7f655c655640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f65541543a0 0x7f655415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:39.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.541+0000 7f655c655640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f655406bf50 0x7f655406c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:39.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.541+0000 7f655c655640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65540cb2f0 con 0x7f655406d4f0 2024-03-24T10:48:39.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.541+0000 7f655c655640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f65540c6750 con 0x7f655406bf50 2024-03-24T10:48:39.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.541+0000 7f655c655640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f65540cc380 con 0x7f65541543a0 2024-03-24T10:48:39.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.542+0000 7f655abf2640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f655406bf50 0x7f655406c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:39.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.542+0000 7f655a3f1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f65541543a0 0x7f655415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:39.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.542+0000 7f6559bf0640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f655406d4f0 0x7f65540cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:57286/0 (socket says 172.21.15.5:57286) 2024-03-24T10:48:39.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.542+0000 7f6559bf0640 1 -- 172.21.15.5:0/2713293668 learned_addr learned my addr 172.21.15.5:0/2713293668 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:48:39.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.545+0000 7f655a3f1640 1 -- 172.21.15.5:0/2713293668 >> v2:172.21.15.57:3300/0 conn(0x7f655406bf50 msgr2=0x7f655406c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:39.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.545+0000 7f655a3f1640 1 --2- 172.21.15.5:0/2713293668 >> v2:172.21.15.57:3300/0 conn(0x7f655406bf50 0x7f655406c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:39.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.545+0000 7f655a3f1640 1 -- 172.21.15.5:0/2713293668 >> v1:172.21.15.57:6789/0 conn(0x7f655406d4f0 legacy=0x7f65540cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:39.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.545+0000 7f655a3f1640 1 -- 172.21.15.5:0/2713293668 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65540ca7e0 con 0x7f65541543a0 2024-03-24T10:48:39.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.548+0000 7f655a3f1640 1 --2- 172.21.15.5:0/2713293668 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f65541543a0 0x7f655415ac00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f65500097a0 tx=0x7f655002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c360f95319a0e3e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:39.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.553+0000 7f65593ef640 1 -- 172.21.15.5:0/2713293668 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6550030610 con 0x7f65541543a0 2024-03-24T10:48:39.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.553+0000 7f65593ef640 1 -- 172.21.15.5:0/2713293668 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6550030770 con 0x7f65541543a0 2024-03-24T10:48:39.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.553+0000 7f65593ef640 1 -- 172.21.15.5:0/2713293668 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6550030b00 con 0x7f65541543a0 2024-03-24T10:48:39.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.554+0000 7f655c655640 1 -- 172.21.15.5:0/2713293668 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f65541543a0 msgr2=0x7f655415ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:39.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.554+0000 7f655c655640 1 --2- 172.21.15.5:0/2713293668 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f65541543a0 0x7f655415ac00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f65500097a0 tx=0x7f655002b960 comp rx=0 tx=0).stop 2024-03-24T10:48:39.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.554+0000 7f655c655640 1 -- 172.21.15.5:0/2713293668 shutdown_connections 2024-03-24T10:48:39.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.554+0000 7f655c655640 1 --2- 172.21.15.5:0/2713293668 >> v2:172.21.15.57:3300/0 conn(0x7f655406bf50 0x7f655406c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:39.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.554+0000 7f655c655640 1 --2- 172.21.15.5:0/2713293668 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f65541543a0 0x7f655415ac00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:39.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.555+0000 7f655c655640 1 -- 172.21.15.5:0/2713293668 >> 172.21.15.5:0/2713293668 conn(0x7f6554072ef0 msgr2=0x7f6554061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:39.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.555+0000 7f655c655640 1 -- 172.21.15.5:0/2713293668 shutdown_connections 2024-03-24T10:48:39.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.555+0000 7f655c655640 1 -- 172.21.15.5:0/2713293668 wait complete. 2024-03-24T10:48:39.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.556+0000 7f655c655640 1 Processor -- start 2024-03-24T10:48:39.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.556+0000 7f655c655640 1 -- start start 2024-03-24T10:48:39.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.558+0000 7f655c655640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f655406bf50 0x7f6554159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:39.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.558+0000 7f655c655640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f655406d4f0 0x7f65541efa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:39.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.558+0000 7f655c655640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f65541543a0 0x7f65541effd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:39.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.558+0000 7f655c655640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f655415c560 con 0x7f655406bf50 2024-03-24T10:48:39.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.558+0000 7f655c655640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f655415bd80 con 0x7f655406d4f0 2024-03-24T10:48:39.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.558+0000 7f655c655640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f655415c1a0 con 0x7f65541543a0 2024-03-24T10:48:39.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.558+0000 7f6559bf0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f655406d4f0 0x7f65541efa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:39.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.558+0000 7f655a3f1640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f655406bf50 0x7f6554159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:39.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.558+0000 7f655abf2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f65541543a0 0x7f65541effd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:39.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.559+0000 7f655a3f1640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f655406bf50 0x7f6554159220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54008/0 (socket says 172.21.15.5:54008) 2024-03-24T10:48:39.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.559+0000 7f655a3f1640 1 -- 172.21.15.5:0/1423417220 learned_addr learned my addr 172.21.15.5:0/1423417220 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:39.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.562+0000 7f655a3f1640 1 -- 172.21.15.5:0/1423417220 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f65541543a0 msgr2=0x7f65541effd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:39.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.562+0000 7f655a3f1640 1 --2- 172.21.15.5:0/1423417220 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f65541543a0 0x7f65541effd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:39.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.562+0000 7f655a3f1640 1 -- 172.21.15.5:0/1423417220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f655406d4f0 msgr2=0x7f65541efa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:39.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.562+0000 7f655a3f1640 1 --2- 172.21.15.5:0/1423417220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f655406d4f0 0x7f65541efa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:39.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.562+0000 7f655a3f1640 1 -- 172.21.15.5:0/1423417220 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6550009400 con 0x7f655406bf50 2024-03-24T10:48:39.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.565+0000 7f655a3f1640 1 --2- 172.21.15.5:0/1423417220 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f655406bf50 0x7f6554159220 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f655002bf70 tx=0x7f6550004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:39.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.570+0000 7f654affd640 1 -- 172.21.15.5:0/1423417220 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655003e040 con 0x7f655406bf50 2024-03-24T10:48:39.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.570+0000 7f654affd640 1 -- 172.21.15.5:0/1423417220 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6550039a40 con 0x7f655406bf50 2024-03-24T10:48:39.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.570+0000 7f655c655640 1 -- 172.21.15.5:0/1423417220 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f655415a6d0 con 0x7f655406bf50 2024-03-24T10:48:39.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.570+0000 7f654affd640 1 -- 172.21.15.5:0/1423417220 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6550039df0 con 0x7f655406bf50 2024-03-24T10:48:39.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.570+0000 7f655c655640 1 -- 172.21.15.5:0/1423417220 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f655415b3e0 con 0x7f655406bf50 2024-03-24T10:48:39.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.571+0000 7f655c655640 1 -- 172.21.15.5:0/1423417220 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f655415bff0 con 0x7f655406bf50 2024-03-24T10:48:39.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.604+0000 7f654affd640 1 -- 172.21.15.5:0/1423417220 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f655004d3b0 con 0x7f655406bf50 2024-03-24T10:48:39.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.604+0000 7f654affd640 1 --2- 172.21.15.5:0/1423417220 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f653806b860 0x7f653806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:39.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.604+0000 7f6559bf0640 1 --2- 172.21.15.5:0/1423417220 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f653806b860 0x7f653806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:39.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.605+0000 7f6559bf0640 1 --2- 172.21.15.5:0/1423417220 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f653806b860 0x7f653806dd20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f655414e360 tx=0x7f6544007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:39.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.874+0000 7f654affd640 1 -- 172.21.15.5:0/1423417220 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f655007b2d0 con 0x7f655406bf50 2024-03-24T10:48:39.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:39.874+0000 7f654affd640 1 -- 172.21.15.5:0/1423417220 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6550077570 con 0x7f655406bf50 2024-03-24T10:48:40.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:40.105+0000 7f655c655640 1 -- 172.21.15.5:0/1423417220 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65540681d0 con 0x7f655406bf50 2024-03-24T10:48:40.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:40.118+0000 7f654affd640 1 -- 172.21.15.5:0/1423417220 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f655007ac30 con 0x7f655406bf50 2024-03-24T10:48:40.119 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:40.119 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:40.119 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:40.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:40.121+0000 7f655c655640 1 -- 172.21.15.5:0/1423417220 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f653806b860 msgr2=0x7f653806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:40.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:40.121+0000 7f655c655640 1 --2- 172.21.15.5:0/1423417220 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f653806b860 0x7f653806dd20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f655414e360 tx=0x7f6544007e80 comp rx=0 tx=0).stop 2024-03-24T10:48:40.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:40.121+0000 7f655c655640 1 -- 172.21.15.5:0/1423417220 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f655406bf50 msgr2=0x7f6554159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:40.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:40.121+0000 7f655c655640 1 --2- 172.21.15.5:0/1423417220 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f655406bf50 0x7f6554159220 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f655002bf70 tx=0x7f6550004730 comp rx=0 tx=0).stop 2024-03-24T10:48:40.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:40.121+0000 7f655c655640 1 -- 172.21.15.5:0/1423417220 shutdown_connections 2024-03-24T10:48:40.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:40.121+0000 7f655c655640 1 --2- 172.21.15.5:0/1423417220 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f653806b860 0x7f653806dd20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:40.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:40.121+0000 7f655c655640 1 --2- 172.21.15.5:0/1423417220 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f65541543a0 0x7f65541effd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:40.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:40.121+0000 7f655c655640 1 --2- 172.21.15.5:0/1423417220 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f655406d4f0 0x7f65541efa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:40.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:40.121+0000 7f655c655640 1 --2- 172.21.15.5:0/1423417220 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f655406bf50 0x7f6554159220 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:40.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:40.121+0000 7f655c655640 1 -- 172.21.15.5:0/1423417220 >> 172.21.15.5:0/1423417220 conn(0x7f6554072ef0 msgr2=0x7f6554151ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:40.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:40.121+0000 7f655c655640 1 -- 172.21.15.5:0/1423417220 shutdown_connections 2024-03-24T10:48:40.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:40.121+0000 7f655c655640 1 -- 172.21.15.5:0/1423417220 wait complete. 2024-03-24T10:48:40.736 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:40.737 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:41.738 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:41.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.839+0000 7f82195a0640 1 Processor -- start 2024-03-24T10:48:41.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.840+0000 7f82195a0640 1 -- start start 2024-03-24T10:48:41.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.840+0000 7f82195a0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f821415ac00 0x7f82140cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:41.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.840+0000 7f82195a0640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82140cb2f0 con 0x7f82141543a0 2024-03-24T10:48:41.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.840+0000 7f82195a0640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f821415b070 con 0x7f82140ccaa0 2024-03-24T10:48:41.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.840+0000 7f82195a0640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f82140c6750 con 0x7f821415ac00 2024-03-24T10:48:41.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.841+0000 7f8212ffd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f82141543a0 0x7f821406d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:58436/0 (socket says 172.21.15.5:58436) 2024-03-24T10:48:41.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.841+0000 7f8212ffd640 1 -- 172.21.15.5:0/3430848021 learned_addr learned my addr 172.21.15.5:0/3430848021 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:48:41.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.842+0000 7f82127fc640 1 --2- 172.21.15.5:0/3430848021 >> v2:172.21.15.57:3300/0 conn(0x7f821415ac00 0x7f82140cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:41.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.844+0000 7f8211ffb640 1 -- 172.21.15.5:0/3430848021 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2497416712 0 0) 0x7f82140cb2f0 con 0x7f82141543a0 2024-03-24T10:48:41.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.844+0000 7f8211ffb640 1 -- 172.21.15.5:0/3430848021 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81fc003520 con 0x7f82141543a0 2024-03-24T10:48:41.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.848+0000 7f8211ffb640 1 -- 172.21.15.5:0/3430848021 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3272290959 0 0) 0x7f81fc003520 con 0x7f82141543a0 2024-03-24T10:48:41.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.848+0000 7f8211ffb640 1 -- 172.21.15.5:0/3430848021 >> v2:172.21.15.57:3300/0 conn(0x7f821415ac00 msgr2=0x7f82140cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:41.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.848+0000 7f8211ffb640 1 --2- 172.21.15.5:0/3430848021 >> v2:172.21.15.57:3300/0 conn(0x7f821415ac00 0x7f82140cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:41.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.848+0000 7f8211ffb640 1 -- 172.21.15.5:0/3430848021 >> v1:172.21.15.57:6789/0 conn(0x7f82140ccaa0 legacy=0x7f821406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:41.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.848+0000 7f8211ffb640 1 -- 172.21.15.5:0/3430848021 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82140ca7e0 con 0x7f82141543a0 2024-03-24T10:48:41.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.848+0000 7f8211ffb640 1 -- 172.21.15.5:0/3430848021 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 100681612 0 0) 0x7f8204001670 con 0x7f82141543a0 2024-03-24T10:48:41.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.848+0000 7f8211ffb640 1 -- 172.21.15.5:0/3430848021 >> v1:172.21.15.5:6789/0 conn(0x7f82141543a0 legacy=0x7f821406d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:41.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.848+0000 7f8211ffb640 1 --2- 172.21.15.5:0/3430848021 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f81fc0039c0 0x7f81fc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:41.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.848+0000 7f8211ffb640 1 --2- 172.21.15.5:0/3430848021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81fc0047c0 0x7f81fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:41.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.848+0000 7f8211ffb640 1 --2- 172.21.15.5:0/3430848021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81fc005180 0x7f81fc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:41.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.848+0000 7f8211ffb640 1 -- 172.21.15.5:0/3430848021 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f82140c6750 con 0x7f81fc0047c0 2024-03-24T10:48:41.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.848+0000 7f8211ffb640 1 -- 172.21.15.5:0/3430848021 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f81fc00c080 con 0x7f81fc005180 2024-03-24T10:48:41.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.848+0000 7f8211ffb640 1 -- 172.21.15.5:0/3430848021 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f81fc00c240 con 0x7f81fc0039c0 2024-03-24T10:48:41.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.848+0000 7f8212ffd640 1 --2- 172.21.15.5:0/3430848021 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f81fc0039c0 0x7f81fc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:41.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.851+0000 7f82127fc640 1 --2- 172.21.15.5:0/3430848021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81fc0047c0 0x7f81fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:41.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.852+0000 7f8212ffd640 1 -- 172.21.15.5:0/3430848021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81fc005180 msgr2=0x7f81fc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:41.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.852+0000 7f8212ffd640 1 --2- 172.21.15.5:0/3430848021 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81fc005180 0x7f81fc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:41.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.852+0000 7f8212ffd640 1 -- 172.21.15.5:0/3430848021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81fc0047c0 msgr2=0x7f81fc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:41.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.852+0000 7f8212ffd640 1 --2- 172.21.15.5:0/3430848021 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81fc0047c0 0x7f81fc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:41.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.852+0000 7f8212ffd640 1 -- 172.21.15.5:0/3430848021 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81fc003520 con 0x7f81fc0039c0 2024-03-24T10:48:41.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.852+0000 7f82137fe640 1 -- 172.21.15.5:0/3430848021 reap_dead start 2024-03-24T10:48:41.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.854+0000 7f8212ffd640 1 --2- 172.21.15.5:0/3430848021 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f81fc0039c0 0x7f81fc003de0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f820400c0e0 tx=0x7f81fc0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9377da8c757d74ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:41.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.859+0000 7f8211ffb640 1 -- 172.21.15.5:0/3430848021 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8204005ad0 con 0x7f81fc0039c0 2024-03-24T10:48:41.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.859+0000 7f8211ffb640 1 -- 172.21.15.5:0/3430848021 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8204005c50 con 0x7f81fc0039c0 2024-03-24T10:48:41.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.859+0000 7f8211ffb640 1 -- 172.21.15.5:0/3430848021 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820402c7d0 con 0x7f81fc0039c0 2024-03-24T10:48:41.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.860+0000 7f82195a0640 1 -- 172.21.15.5:0/3430848021 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f81fc0039c0 msgr2=0x7f81fc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:41.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.860+0000 7f82195a0640 1 --2- 172.21.15.5:0/3430848021 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f81fc0039c0 0x7f81fc003de0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f820400c0e0 tx=0x7f81fc0057f0 comp rx=0 tx=0).stop 2024-03-24T10:48:41.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.860+0000 7f82195a0640 1 -- 172.21.15.5:0/3430848021 shutdown_connections 2024-03-24T10:48:41.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.860+0000 7f82195a0640 1 --2- 172.21.15.5:0/3430848021 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f81fc0039c0 0x7f81fc003de0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:41.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.860+0000 7f82195a0640 1 -- 172.21.15.5:0/3430848021 >> 172.21.15.5:0/3430848021 conn(0x7f8214072ef0 msgr2=0x7f8214061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:41.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.860+0000 7f82195a0640 1 -- 172.21.15.5:0/3430848021 shutdown_connections 2024-03-24T10:48:41.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.860+0000 7f82195a0640 1 -- 172.21.15.5:0/3430848021 wait complete. 2024-03-24T10:48:41.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.861+0000 7f82195a0640 1 Processor -- start 2024-03-24T10:48:41.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.861+0000 7f82195a0640 1 -- start start 2024-03-24T10:48:41.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.862+0000 7f82195a0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f81fc0039c0 0x7f82141449a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:41.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.862+0000 7f82195a0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82140ce920 0x7f82140b4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:41.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.862+0000 7f82195a0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82140b4d70 0x7f82141ef840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:41.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.862+0000 7f82195a0640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f82140b3040 con 0x7f82140ce920 2024-03-24T10:48:41.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.862+0000 7f82195a0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f82140b3d70 con 0x7f82140b4d70 2024-03-24T10:48:41.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.862+0000 7f82195a0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f82141582c0 con 0x7f81fc0039c0 2024-03-24T10:48:41.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.863+0000 7f8212ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f81fc0039c0 0x7f82141449a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:41.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.863+0000 7f82127fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82140ce920 0x7f82140b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:41.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.863+0000 7f82137fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82140b4d70 0x7f82141ef840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:41.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.863+0000 7f82127fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82140ce920 0x7f82140b4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54022/0 (socket says 172.21.15.5:54022) 2024-03-24T10:48:41.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.863+0000 7f82127fc640 1 -- 172.21.15.5:0/1715166164 learned_addr learned my addr 172.21.15.5:0/1715166164 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:41.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.866+0000 7f82127fc640 1 -- 172.21.15.5:0/1715166164 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f81fc0039c0 msgr2=0x7f82141449a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:41.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.866+0000 7f82127fc640 1 --2- 172.21.15.5:0/1715166164 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f81fc0039c0 0x7f82141449a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:41.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.866+0000 7f82127fc640 1 -- 172.21.15.5:0/1715166164 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82140b4d70 msgr2=0x7f82141ef840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:41.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.866+0000 7f82127fc640 1 --2- 172.21.15.5:0/1715166164 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82140b4d70 0x7f82141ef840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:41.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.866+0000 7f82127fc640 1 -- 172.21.15.5:0/1715166164 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82140cee80 con 0x7f82140ce920 2024-03-24T10:48:41.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.869+0000 7f82127fc640 1 --2- 172.21.15.5:0/1715166164 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82140ce920 0x7f82140b4830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8208011a70 tx=0x7f8208011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:41.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.873+0000 7f81fb7fe640 1 -- 172.21.15.5:0/1715166164 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8208009680 con 0x7f82140ce920 2024-03-24T10:48:41.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.873+0000 7f81fb7fe640 1 -- 172.21.15.5:0/1715166164 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f820800fb50 con 0x7f82140ce920 2024-03-24T10:48:41.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.873+0000 7f82195a0640 1 -- 172.21.15.5:0/1715166164 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82140cf160 con 0x7f82140ce920 2024-03-24T10:48:41.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.873+0000 7f81fb7fe640 1 -- 172.21.15.5:0/1715166164 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820800fee0 con 0x7f82140ce920 2024-03-24T10:48:41.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.873+0000 7f82195a0640 1 -- 172.21.15.5:0/1715166164 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82140b3570 con 0x7f82140ce920 2024-03-24T10:48:41.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:41.874+0000 7f82195a0640 1 -- 172.21.15.5:0/1715166164 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82140681d0 con 0x7f82140ce920 2024-03-24T10:48:42.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:42.147+0000 7f81fb7fe640 1 -- 172.21.15.5:0/1715166164 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f820801d400 con 0x7f82140ce920 2024-03-24T10:48:42.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:42.147+0000 7f81fb7fe640 1 --2- 172.21.15.5:0/1715166164 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f820006b860 0x7f820006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:42.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:42.148+0000 7f8212ffd640 1 --2- 172.21.15.5:0/1715166164 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f820006b860 0x7f820006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:42.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:42.148+0000 7f8212ffd640 1 --2- 172.21.15.5:0/1715166164 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f820006b860 0x7f820006dd20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8204034000 tx=0x7f8204009030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:42.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:42.164+0000 7f81fb7fe640 1 -- 172.21.15.5:0/1715166164 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f82080596e0 con 0x7f82140ce920 2024-03-24T10:48:42.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:42.164+0000 7f81fb7fe640 1 -- 172.21.15.5:0/1715166164 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8208055980 con 0x7f82140ce920 2024-03-24T10:48:42.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:42.390+0000 7f82195a0640 1 -- 172.21.15.5:0/1715166164 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82140b2620 con 0x7f82140ce920 2024-03-24T10:48:42.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:42.411+0000 7f81fb7fe640 1 -- 172.21.15.5:0/1715166164 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f8208059040 con 0x7f82140ce920 2024-03-24T10:48:42.412 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:42.412 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:42.412 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:42.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:42.413+0000 7f82195a0640 1 -- 172.21.15.5:0/1715166164 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f820006b860 msgr2=0x7f820006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:42.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:42.413+0000 7f82195a0640 1 --2- 172.21.15.5:0/1715166164 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f820006b860 0x7f820006dd20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8204034000 tx=0x7f8204009030 comp rx=0 tx=0).stop 2024-03-24T10:48:42.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:42.414+0000 7f82195a0640 1 -- 172.21.15.5:0/1715166164 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82140ce920 msgr2=0x7f82140b4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:42.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:42.414+0000 7f82195a0640 1 --2- 172.21.15.5:0/1715166164 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82140ce920 0x7f82140b4830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8208011a70 tx=0x7f8208011f30 comp rx=0 tx=0).stop 2024-03-24T10:48:42.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:42.414+0000 7f82195a0640 1 -- 172.21.15.5:0/1715166164 shutdown_connections 2024-03-24T10:48:42.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:42.414+0000 7f82195a0640 1 --2- 172.21.15.5:0/1715166164 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f820006b860 0x7f820006dd20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:42.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:42.414+0000 7f82195a0640 1 --2- 172.21.15.5:0/1715166164 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82140ce920 0x7f82140b4830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:42.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:42.414+0000 7f82195a0640 1 --2- 172.21.15.5:0/1715166164 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82140b4d70 0x7f82141ef840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:42.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:42.414+0000 7f82195a0640 1 --2- 172.21.15.5:0/1715166164 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f81fc0039c0 0x7f82141449a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:42.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:42.414+0000 7f82195a0640 1 -- 172.21.15.5:0/1715166164 >> 172.21.15.5:0/1715166164 conn(0x7f8214072ef0 msgr2=0x7f8214150fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:42.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:42.414+0000 7f82195a0640 1 -- 172.21.15.5:0/1715166164 shutdown_connections 2024-03-24T10:48:42.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:42.414+0000 7f82195a0640 1 -- 172.21.15.5:0/1715166164 wait complete. 2024-03-24T10:48:43.029 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:43.029 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:44.030 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:44.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.133+0000 7fdb9f581640 1 Processor -- start 2024-03-24T10:48:44.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.133+0000 7fdb9f581640 1 -- start start 2024-03-24T10:48:44.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.133+0000 7fdb9f581640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb980cf530 0x7fdb980cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:44.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.133+0000 7fdb9f581640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb980cf9e0 con 0x7fdb980cae90 2024-03-24T10:48:44.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.133+0000 7fdb9f581640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb980ccba0 con 0x7fdb980cb270 2024-03-24T10:48:44.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.133+0000 7fdb9f581640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fdb980cc9b0 con 0x7fdb980cf530 2024-03-24T10:48:44.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.134+0000 7fdb9d31d640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fdb980cae90 0x7fdb980cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:51062/0 (socket says 172.21.15.5:51062) 2024-03-24T10:48:44.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.134+0000 7fdb9d31d640 1 -- 172.21.15.5:0/3326514471 learned_addr learned my addr 172.21.15.5:0/3326514471 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:48:44.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.134+0000 7fdb9db1e640 1 --2- 172.21.15.5:0/3326514471 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb980cf530 0x7fdb980cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:44.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.137+0000 7fdb8ffff640 1 -- 172.21.15.5:0/3326514471 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3376833569 0 0) 0x7fdb94001110 con 0x7fdb980cae90 2024-03-24T10:48:44.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.137+0000 7fdb8ffff640 1 -- 172.21.15.5:0/3326514471 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb84003520 con 0x7fdb980cae90 2024-03-24T10:48:44.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.138+0000 7fdb8ffff640 1 -- 172.21.15.5:0/3326514471 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4108700986 0 0) 0x7fdb980ccba0 con 0x7fdb980cb270 2024-03-24T10:48:44.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.138+0000 7fdb8ffff640 1 -- 172.21.15.5:0/3326514471 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb94001110 con 0x7fdb980cb270 2024-03-24T10:48:44.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.138+0000 7fdb9db1e640 1 -- 172.21.15.5:0/3326514471 >> v1:172.21.15.57:6789/0 conn(0x7fdb980cb270 legacy=0x7fdb980cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:44.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.138+0000 7fdb9db1e640 1 -- 172.21.15.5:0/3326514471 >> v1:172.21.15.5:6789/0 conn(0x7fdb980cae90 legacy=0x7fdb980cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:44.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.138+0000 7fdb9db1e640 1 -- 172.21.15.5:0/3326514471 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb980cce40 con 0x7fdb980cf530 2024-03-24T10:48:44.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.141+0000 7fdb9db1e640 1 --2- 172.21.15.5:0/3326514471 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb980cf530 0x7fdb980cb6f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdb90009dc0 tx=0x7fdb9002bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a5f120405b40054 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:44.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.144+0000 7fdb8ffff640 1 -- 172.21.15.5:0/3326514471 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9002cd50 con 0x7fdb980cf530 2024-03-24T10:48:44.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.146+0000 7fdb8ffff640 1 -- 172.21.15.5:0/3326514471 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb90034440 con 0x7fdb980cf530 2024-03-24T10:48:44.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.146+0000 7fdb8ffff640 1 -- 172.21.15.5:0/3326514471 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb900347b0 con 0x7fdb980cf530 2024-03-24T10:48:44.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.147+0000 7fdb9f581640 1 -- 172.21.15.5:0/3326514471 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb980cf530 msgr2=0x7fdb980cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:44.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.147+0000 7fdb9f581640 1 --2- 172.21.15.5:0/3326514471 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb980cf530 0x7fdb980cb6f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdb90009dc0 tx=0x7fdb9002bfa0 comp rx=0 tx=0).stop 2024-03-24T10:48:44.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.147+0000 7fdb9f581640 1 -- 172.21.15.5:0/3326514471 shutdown_connections 2024-03-24T10:48:44.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.147+0000 7fdb9f581640 1 --2- 172.21.15.5:0/3326514471 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb980cf530 0x7fdb980cb6f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:44.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.147+0000 7fdb9f581640 1 -- 172.21.15.5:0/3326514471 >> 172.21.15.5:0/3326514471 conn(0x7fdb980cfb80 msgr2=0x7fdb980cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:44.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.147+0000 7fdb9f581640 1 -- 172.21.15.5:0/3326514471 shutdown_connections 2024-03-24T10:48:44.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.147+0000 7fdb9f581640 1 -- 172.21.15.5:0/3326514471 wait complete. 2024-03-24T10:48:44.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.148+0000 7fdb9f581640 1 Processor -- start 2024-03-24T10:48:44.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.148+0000 7fdb9f581640 1 -- start start 2024-03-24T10:48:44.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.149+0000 7fdb9f581640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb980cae90 0x7fdb9815be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:44.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.149+0000 7fdb9f581640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb980cb270 0x7fdb9815d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:44.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.149+0000 7fdb9f581640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb980cf530 0x7fdb9815d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:44.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.149+0000 7fdb9f581640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fdb9815e6d0 con 0x7fdb980cb270 2024-03-24T10:48:44.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.149+0000 7fdb9f581640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fdb981f0b30 con 0x7fdb980cf530 2024-03-24T10:48:44.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.149+0000 7fdb9f581640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fdb981efc30 con 0x7fdb980cae90 2024-03-24T10:48:44.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.149+0000 7fdb9d31d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb980cae90 0x7fdb9815be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:44.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.149+0000 7fdb9cb1c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb980cb270 0x7fdb9815d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:44.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.149+0000 7fdb9db1e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb980cf530 0x7fdb9815d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:44.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.149+0000 7fdb9cb1c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb980cb270 0x7fdb9815d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39284/0 (socket says 172.21.15.5:39284) 2024-03-24T10:48:44.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.149+0000 7fdb9cb1c640 1 -- 172.21.15.5:0/3343010054 learned_addr learned my addr 172.21.15.5:0/3343010054 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:44.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.153+0000 7fdb9cb1c640 1 -- 172.21.15.5:0/3343010054 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb980cae90 msgr2=0x7fdb9815be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:44.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.153+0000 7fdb9cb1c640 1 --2- 172.21.15.5:0/3343010054 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb980cae90 0x7fdb9815be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:44.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.153+0000 7fdb9cb1c640 1 -- 172.21.15.5:0/3343010054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb980cf530 msgr2=0x7fdb9815d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:44.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.153+0000 7fdb9cb1c640 1 --2- 172.21.15.5:0/3343010054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb980cf530 0x7fdb9815d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:44.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.153+0000 7fdb9cb1c640 1 -- 172.21.15.5:0/3343010054 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb90009a70 con 0x7fdb980cb270 2024-03-24T10:48:44.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.156+0000 7fdb9cb1c640 1 --2- 172.21.15.5:0/3343010054 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb980cb270 0x7fdb9815d3d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800c530 tx=0x7fdb8800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:44.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.158+0000 7fdb8dffb640 1 -- 172.21.15.5:0/3343010054 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb880052d0 con 0x7fdb980cb270 2024-03-24T10:48:44.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.158+0000 7fdb9f581640 1 -- 172.21.15.5:0/3343010054 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb981efeb0 con 0x7fdb980cb270 2024-03-24T10:48:44.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.159+0000 7fdb9f581640 1 -- 172.21.15.5:0/3343010054 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb981f0470 con 0x7fdb980cb270 2024-03-24T10:48:44.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.160+0000 7fdb9f581640 1 -- 172.21.15.5:0/3343010054 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb9806d5a0 con 0x7fdb980cb270 2024-03-24T10:48:44.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.370+0000 7fdb8dffb640 1 -- 172.21.15.5:0/3343010054 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb88005430 con 0x7fdb980cb270 2024-03-24T10:48:44.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.443+0000 7fdb8dffb640 1 -- 172.21.15.5:0/3343010054 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb88012850 con 0x7fdb980cb270 2024-03-24T10:48:44.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.443+0000 7fdb8dffb640 1 -- 172.21.15.5:0/3343010054 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdb88012a30 con 0x7fdb980cb270 2024-03-24T10:48:44.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.444+0000 7fdb8dffb640 1 --2- 172.21.15.5:0/3343010054 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdb7c06b810 0x7fdb7c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:44.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.444+0000 7fdb9d31d640 1 --2- 172.21.15.5:0/3343010054 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdb7c06b810 0x7fdb7c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:44.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.445+0000 7fdb9d31d640 1 --2- 172.21.15.5:0/3343010054 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdb7c06b810 0x7fdb7c06dcd0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdb94000f60 tx=0x7fdb94009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:44.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.459+0000 7fdb8dffb640 1 -- 172.21.15.5:0/3343010054 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fdb88058f60 con 0x7fdb980cb270 2024-03-24T10:48:44.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.459+0000 7fdb8dffb640 1 -- 172.21.15.5:0/3343010054 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdb88055200 con 0x7fdb980cb270 2024-03-24T10:48:44.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.697+0000 7fdb9f581640 1 -- 172.21.15.5:0/3343010054 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb9815d220 con 0x7fdb980cb270 2024-03-24T10:48:44.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.714+0000 7fdb8dffb640 1 -- 172.21.15.5:0/3343010054 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fdb880588c0 con 0x7fdb980cb270 2024-03-24T10:48:44.714 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:44.715 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:44.715 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:44.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.717+0000 7fdb9f581640 1 -- 172.21.15.5:0/3343010054 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdb7c06b810 msgr2=0x7fdb7c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:44.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.717+0000 7fdb9f581640 1 --2- 172.21.15.5:0/3343010054 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdb7c06b810 0x7fdb7c06dcd0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdb94000f60 tx=0x7fdb94009040 comp rx=0 tx=0).stop 2024-03-24T10:48:44.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.717+0000 7fdb9f581640 1 -- 172.21.15.5:0/3343010054 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb980cb270 msgr2=0x7fdb9815d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:44.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.717+0000 7fdb9f581640 1 --2- 172.21.15.5:0/3343010054 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb980cb270 0x7fdb9815d3d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdb8800c530 tx=0x7fdb8800c9f0 comp rx=0 tx=0).stop 2024-03-24T10:48:44.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.717+0000 7fdb9f581640 1 -- 172.21.15.5:0/3343010054 shutdown_connections 2024-03-24T10:48:44.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.717+0000 7fdb9f581640 1 --2- 172.21.15.5:0/3343010054 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdb7c06b810 0x7fdb7c06dcd0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:44.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.717+0000 7fdb9f581640 1 --2- 172.21.15.5:0/3343010054 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb980cb270 0x7fdb9815d3d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:44.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.717+0000 7fdb9f581640 1 --2- 172.21.15.5:0/3343010054 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb980cf530 0x7fdb9815d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:44.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.717+0000 7fdb9f581640 1 --2- 172.21.15.5:0/3343010054 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb980cae90 0x7fdb9815be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:44.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.717+0000 7fdb9f581640 1 -- 172.21.15.5:0/3343010054 >> 172.21.15.5:0/3343010054 conn(0x7fdb980cfb80 msgr2=0x7fdb980c86b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:44.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.717+0000 7fdb9f581640 1 -- 172.21.15.5:0/3343010054 shutdown_connections 2024-03-24T10:48:44.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:44.717+0000 7fdb9f581640 1 -- 172.21.15.5:0/3343010054 wait complete. 2024-03-24T10:48:45.333 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:45.333 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:46.334 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:46.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.438+0000 7f9031708640 1 Processor -- start 2024-03-24T10:48:46.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.438+0000 7f9031708640 1 -- start start 2024-03-24T10:48:46.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.439+0000 7f9031708640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f902c0c92f0 0x7f902c0cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:46.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.439+0000 7f9031708640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f902c06c750 0x7f902c06cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:46.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.439+0000 7f9031708640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f902c0ce350 con 0x7f902c0c96d0 2024-03-24T10:48:46.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.439+0000 7f9031708640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f902c15e320 con 0x7f902c0c92f0 2024-03-24T10:48:46.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.439+0000 7f9031708640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f902c15e7b0 con 0x7f902c06c750 2024-03-24T10:48:46.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.440+0000 7f902affd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f902c0c92f0 0x7f902c0cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:46.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.440+0000 7f902b7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f902c06c750 0x7f902c06cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:46.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.440+0000 7f902affd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f902c0c92f0 0x7f902c0cd9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39296/0 (socket says 172.21.15.5:39296) 2024-03-24T10:48:46.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.440+0000 7f902affd640 1 -- 172.21.15.5:0/1517644707 learned_addr learned my addr 172.21.15.5:0/1517644707 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:46.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.444+0000 7f902affd640 1 -- 172.21.15.5:0/1517644707 >> v2:172.21.15.57:3300/0 conn(0x7f902c06c750 msgr2=0x7f902c06cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:46.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.444+0000 7f902affd640 1 --2- 172.21.15.5:0/1517644707 >> v2:172.21.15.57:3300/0 conn(0x7f902c06c750 0x7f902c06cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:46.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.444+0000 7f902affd640 1 -- 172.21.15.5:0/1517644707 >> v1:172.21.15.57:6789/0 conn(0x7f902c0c96d0 legacy=0x7f902c15ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:46.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.444+0000 7f902affd640 1 -- 172.21.15.5:0/1517644707 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f902c0ce070 con 0x7f902c0c92f0 2024-03-24T10:48:46.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.446+0000 7f902affd640 1 --2- 172.21.15.5:0/1517644707 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f902c0c92f0 0x7f902c0cd9c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f901c009870 tx=0x7f901c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ddff4425cbb5514 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:46.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.449+0000 7f902a7fc640 1 -- 172.21.15.5:0/1517644707 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c030860 con 0x7f902c0c92f0 2024-03-24T10:48:46.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.451+0000 7f902a7fc640 1 -- 172.21.15.5:0/1517644707 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f901c0309c0 con 0x7f902c0c92f0 2024-03-24T10:48:46.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.451+0000 7f902a7fc640 1 -- 172.21.15.5:0/1517644707 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c030d50 con 0x7f902c0c92f0 2024-03-24T10:48:46.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.451+0000 7f9031708640 1 -- 172.21.15.5:0/1517644707 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f902c0c92f0 msgr2=0x7f902c0cd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:46.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.451+0000 7f9031708640 1 --2- 172.21.15.5:0/1517644707 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f902c0c92f0 0x7f902c0cd9c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f901c009870 tx=0x7f901c02b9a0 comp rx=0 tx=0).stop 2024-03-24T10:48:46.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.451+0000 7f9031708640 1 -- 172.21.15.5:0/1517644707 shutdown_connections 2024-03-24T10:48:46.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.451+0000 7f9031708640 1 --2- 172.21.15.5:0/1517644707 >> v2:172.21.15.57:3300/0 conn(0x7f902c06c750 0x7f902c06cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:46.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.451+0000 7f9031708640 1 --2- 172.21.15.5:0/1517644707 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f902c0c92f0 0x7f902c0cd9c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:46.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.451+0000 7f9031708640 1 -- 172.21.15.5:0/1517644707 >> 172.21.15.5:0/1517644707 conn(0x7f902c06f820 msgr2=0x7f902c061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:46.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.452+0000 7f9031708640 1 -- 172.21.15.5:0/1517644707 shutdown_connections 2024-03-24T10:48:46.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.452+0000 7f9031708640 1 -- 172.21.15.5:0/1517644707 wait complete. 2024-03-24T10:48:46.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.453+0000 7f9031708640 1 Processor -- start 2024-03-24T10:48:46.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.453+0000 7f9031708640 1 -- start start 2024-03-24T10:48:46.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.453+0000 7f9031708640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f902c06c750 0x7f902c14dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:46.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.453+0000 7f9031708640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f902c0c92f0 0x7f902c14e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:46.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.453+0000 7f9031708640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f902c0c96d0 0x7f902c14edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:46.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.453+0000 7f9031708640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f902c1f0f50 con 0x7f902c06c750 2024-03-24T10:48:46.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.453+0000 7f9031708640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f902c0cd340 con 0x7f902c0c92f0 2024-03-24T10:48:46.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.453+0000 7f9031708640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f902c1f0340 con 0x7f902c0c96d0 2024-03-24T10:48:46.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.454+0000 7f902affd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f902c06c750 0x7f902c14dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:46.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.454+0000 7f902affd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f902c06c750 0x7f902c14dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39298/0 (socket says 172.21.15.5:39298) 2024-03-24T10:48:46.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.454+0000 7f902affd640 1 -- 172.21.15.5:0/1119020550 learned_addr learned my addr 172.21.15.5:0/1119020550 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:46.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.454+0000 7f9023fff640 1 --2- 172.21.15.5:0/1119020550 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f902c0c92f0 0x7f902c14e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:46.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.454+0000 7f902b7fe640 1 --2- 172.21.15.5:0/1119020550 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f902c0c96d0 0x7f902c14edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:46.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.458+0000 7f902affd640 1 -- 172.21.15.5:0/1119020550 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f902c0c96d0 msgr2=0x7f902c14edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:46.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.458+0000 7f902affd640 1 --2- 172.21.15.5:0/1119020550 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f902c0c96d0 0x7f902c14edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:46.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.458+0000 7f902affd640 1 -- 172.21.15.5:0/1119020550 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f902c0c92f0 msgr2=0x7f902c14e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:46.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.458+0000 7f902affd640 1 --2- 172.21.15.5:0/1119020550 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f902c0c92f0 0x7f902c14e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:46.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.458+0000 7f902affd640 1 -- 172.21.15.5:0/1119020550 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f901c0094d0 con 0x7f902c06c750 2024-03-24T10:48:46.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.461+0000 7f902affd640 1 --2- 172.21.15.5:0/1119020550 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f902c06c750 0x7f902c14dcd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f901c000c00 tx=0x7f901c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:46.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.465+0000 7f90237fe640 1 -- 172.21.15.5:0/1119020550 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c039c90 con 0x7f902c06c750 2024-03-24T10:48:46.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.465+0000 7f90237fe640 1 -- 172.21.15.5:0/1119020550 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f901c039df0 con 0x7f902c06c750 2024-03-24T10:48:46.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.465+0000 7f9031708640 1 -- 172.21.15.5:0/1119020550 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f902c1f0620 con 0x7f902c06c750 2024-03-24T10:48:46.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.465+0000 7f90237fe640 1 -- 172.21.15.5:0/1119020550 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c043640 con 0x7f902c06c750 2024-03-24T10:48:46.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.465+0000 7f9031708640 1 -- 172.21.15.5:0/1119020550 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f902c1f0be0 con 0x7f902c06c750 2024-03-24T10:48:46.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.466+0000 7f9031708640 1 -- 172.21.15.5:0/1119020550 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f902c0681d0 con 0x7f902c06c750 2024-03-24T10:48:46.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.486+0000 7f90237fe640 1 -- 172.21.15.5:0/1119020550 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f901c0437a0 con 0x7f902c06c750 2024-03-24T10:48:46.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.487+0000 7f90237fe640 1 --2- 172.21.15.5:0/1119020550 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f900c06b890 0x7f900c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:46.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.487+0000 7f9023fff640 1 --2- 172.21.15.5:0/1119020550 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f900c06b890 0x7f900c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:46.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.487+0000 7f9023fff640 1 --2- 172.21.15.5:0/1119020550 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f900c06b890 0x7f900c06dd50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f902c06e310 tx=0x7f9014009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:46.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.491+0000 7f90237fe640 1 -- 172.21.15.5:0/1119020550 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f901c03e070 con 0x7f902c06c750 2024-03-24T10:48:46.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:46.751+0000 7f90237fe640 1 -- 172.21.15.5:0/1119020550 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f901c0d9e10 con 0x7f902c06c750 2024-03-24T10:48:47.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:47.003+0000 7f9031708640 1 -- 172.21.15.5:0/1119020550 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f902c0c8a10 con 0x7f902c06c750 2024-03-24T10:48:47.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:47.024+0000 7f90237fe640 1 -- 172.21.15.5:0/1119020550 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f901c034020 con 0x7f902c06c750 2024-03-24T10:48:47.025 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:47.025 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:47.025 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:47.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:47.027+0000 7f9031708640 1 -- 172.21.15.5:0/1119020550 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f900c06b890 msgr2=0x7f900c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:47.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:47.027+0000 7f9031708640 1 --2- 172.21.15.5:0/1119020550 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f900c06b890 0x7f900c06dd50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f902c06e310 tx=0x7f9014009040 comp rx=0 tx=0).stop 2024-03-24T10:48:47.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:47.027+0000 7f9031708640 1 -- 172.21.15.5:0/1119020550 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f902c06c750 msgr2=0x7f902c14dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:47.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:47.027+0000 7f9031708640 1 --2- 172.21.15.5:0/1119020550 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f902c06c750 0x7f902c14dcd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f901c000c00 tx=0x7f901c004730 comp rx=0 tx=0).stop 2024-03-24T10:48:47.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:47.027+0000 7f9031708640 1 -- 172.21.15.5:0/1119020550 shutdown_connections 2024-03-24T10:48:47.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:47.027+0000 7f9031708640 1 --2- 172.21.15.5:0/1119020550 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f900c06b890 0x7f900c06dd50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:47.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:47.027+0000 7f9031708640 1 --2- 172.21.15.5:0/1119020550 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f902c0c96d0 0x7f902c14edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:47.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:47.027+0000 7f9031708640 1 --2- 172.21.15.5:0/1119020550 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f902c0c92f0 0x7f902c14e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:47.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:47.027+0000 7f9031708640 1 --2- 172.21.15.5:0/1119020550 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f902c06c750 0x7f902c14dcd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:47.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:47.027+0000 7f9031708640 1 -- 172.21.15.5:0/1119020550 >> 172.21.15.5:0/1119020550 conn(0x7f902c06f820 msgr2=0x7f902c151f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:47.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:47.028+0000 7f9031708640 1 -- 172.21.15.5:0/1119020550 shutdown_connections 2024-03-24T10:48:47.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:47.028+0000 7f9031708640 1 -- 172.21.15.5:0/1119020550 wait complete. 2024-03-24T10:48:47.647 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:47.647 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:48.648 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:48.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.750+0000 7f3e705b6640 1 Processor -- start 2024-03-24T10:48:48.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.750+0000 7f3e705b6640 1 -- start start 2024-03-24T10:48:48.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.751+0000 7f3e705b6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f3e681543a0 0x7f3e6815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:48.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.751+0000 7f3e705b6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e6806d4f0 0x7f3e680cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:48.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.751+0000 7f3e705b6640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e680cb2f0 con 0x7f3e680ccaa0 2024-03-24T10:48:48.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.751+0000 7f3e705b6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f3e680c6750 con 0x7f3e681543a0 2024-03-24T10:48:48.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.751+0000 7f3e705b6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f3e680cc380 con 0x7f3e6806d4f0 2024-03-24T10:48:48.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.752+0000 7f3e6eb53640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f3e680ccaa0 0x7f3e6806bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:51068/0 (socket says 172.21.15.5:51068) 2024-03-24T10:48:48.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.752+0000 7f3e6eb53640 1 -- 172.21.15.5:0/1476543771 learned_addr learned my addr 172.21.15.5:0/1476543771 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:48:48.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.752+0000 7f3e6e352640 1 --2- 172.21.15.5:0/1476543771 >> v2:172.21.15.57:3300/0 conn(0x7f3e681543a0 0x7f3e6815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:48.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.752+0000 7f3e6db51640 1 --2- 172.21.15.5:0/1476543771 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e6806d4f0 0x7f3e680cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:48.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.755+0000 7f3e6d350640 1 -- 172.21.15.5:0/1476543771 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2242776226 0 0) 0x7f3e680cb2f0 con 0x7f3e680ccaa0 2024-03-24T10:48:48.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.755+0000 7f3e6d350640 1 -- 172.21.15.5:0/1476543771 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e54003520 con 0x7f3e680ccaa0 2024-03-24T10:48:48.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.756+0000 7f3e6db51640 1 -- 172.21.15.5:0/1476543771 >> v2:172.21.15.57:3300/0 conn(0x7f3e681543a0 msgr2=0x7f3e6815ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:48.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.756+0000 7f3e6db51640 1 --2- 172.21.15.5:0/1476543771 >> v2:172.21.15.57:3300/0 conn(0x7f3e681543a0 0x7f3e6815ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:48.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.756+0000 7f3e6db51640 1 -- 172.21.15.5:0/1476543771 >> v1:172.21.15.5:6789/0 conn(0x7f3e680ccaa0 legacy=0x7f3e6806bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:48.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.756+0000 7f3e6db51640 1 -- 172.21.15.5:0/1476543771 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e680ca7e0 con 0x7f3e6806d4f0 2024-03-24T10:48:48.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.756+0000 7f3e6e352640 1 --2- 172.21.15.5:0/1476543771 >> v2:172.21.15.57:3300/0 conn(0x7f3e681543a0 0x7f3e6815ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:48:48.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.759+0000 7f3e6db51640 1 --2- 172.21.15.5:0/1476543771 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e6806d4f0 0x7f3e680cc560 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3e580097a0 tx=0x7f3e5802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb88c7f353dec827 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:48.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.764+0000 7f3e6d350640 1 -- 172.21.15.5:0/1476543771 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e58030610 con 0x7f3e6806d4f0 2024-03-24T10:48:48.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.764+0000 7f3e6d350640 1 -- 172.21.15.5:0/1476543771 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e58030770 con 0x7f3e6806d4f0 2024-03-24T10:48:48.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.764+0000 7f3e6d350640 1 -- 172.21.15.5:0/1476543771 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e58030b00 con 0x7f3e6806d4f0 2024-03-24T10:48:48.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.765+0000 7f3e705b6640 1 -- 172.21.15.5:0/1476543771 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e6806d4f0 msgr2=0x7f3e680cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:48.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.765+0000 7f3e705b6640 1 --2- 172.21.15.5:0/1476543771 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e6806d4f0 0x7f3e680cc560 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3e580097a0 tx=0x7f3e5802b960 comp rx=0 tx=0).stop 2024-03-24T10:48:48.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.765+0000 7f3e705b6640 1 -- 172.21.15.5:0/1476543771 shutdown_connections 2024-03-24T10:48:48.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.765+0000 7f3e705b6640 1 --2- 172.21.15.5:0/1476543771 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e6806d4f0 0x7f3e680cc560 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:48.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.765+0000 7f3e705b6640 1 --2- 172.21.15.5:0/1476543771 >> v2:172.21.15.57:3300/0 conn(0x7f3e681543a0 0x7f3e6815ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:48.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.765+0000 7f3e705b6640 1 -- 172.21.15.5:0/1476543771 >> 172.21.15.5:0/1476543771 conn(0x7f3e68072ef0 msgr2=0x7f3e68061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:48.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.765+0000 7f3e705b6640 1 -- 172.21.15.5:0/1476543771 shutdown_connections 2024-03-24T10:48:48.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.765+0000 7f3e705b6640 1 -- 172.21.15.5:0/1476543771 wait complete. 2024-03-24T10:48:48.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.767+0000 7f3e705b6640 1 Processor -- start 2024-03-24T10:48:48.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.767+0000 7f3e705b6640 1 -- start start 2024-03-24T10:48:48.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.768+0000 7f3e705b6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e6806d4f0 0x7f3e680b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:48.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.768+0000 7f3e705b6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e680ccaa0 0x7f3e68157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:48.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.768+0000 7f3e705b6640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e681543a0 0x7f3e6814b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:48.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.768+0000 7f3e705b6640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f3e681f0b20 con 0x7f3e681543a0 2024-03-24T10:48:48.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.768+0000 7f3e705b6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3e681eff10 con 0x7f3e6806d4f0 2024-03-24T10:48:48.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.768+0000 7f3e705b6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f3e681f01f0 con 0x7f3e680ccaa0 2024-03-24T10:48:48.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.769+0000 7f3e6eb53640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e681543a0 0x7f3e6814b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:48.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.769+0000 7f3e6db51640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e680ccaa0 0x7f3e68157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:48.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.769+0000 7f3e6e352640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e6806d4f0 0x7f3e680b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:48.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.769+0000 7f3e6eb53640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e681543a0 0x7f3e6814b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39300/0 (socket says 172.21.15.5:39300) 2024-03-24T10:48:48.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.769+0000 7f3e6eb53640 1 -- 172.21.15.5:0/3451400381 learned_addr learned my addr 172.21.15.5:0/3451400381 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:48.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.773+0000 7f3e6eb53640 1 -- 172.21.15.5:0/3451400381 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e680ccaa0 msgr2=0x7f3e68157420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:48.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.773+0000 7f3e6eb53640 1 --2- 172.21.15.5:0/3451400381 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e680ccaa0 0x7f3e68157420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:48.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.773+0000 7f3e6eb53640 1 -- 172.21.15.5:0/3451400381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e6806d4f0 msgr2=0x7f3e680b23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:48.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.773+0000 7f3e6eb53640 1 --2- 172.21.15.5:0/3451400381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e6806d4f0 0x7f3e680b23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:48.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.773+0000 7f3e6eb53640 1 -- 172.21.15.5:0/3451400381 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e64009400 con 0x7f3e681543a0 2024-03-24T10:48:48.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.776+0000 7f3e6eb53640 1 --2- 172.21.15.5:0/3451400381 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e681543a0 0x7f3e6814b330 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3e6000cb40 tx=0x7f3e6000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:48.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.781+0000 7f3e5effd640 1 -- 172.21.15.5:0/3451400381 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6000aca0 con 0x7f3e681543a0 2024-03-24T10:48:48.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.781+0000 7f3e5effd640 1 -- 172.21.15.5:0/3451400381 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e6000ae00 con 0x7f3e681543a0 2024-03-24T10:48:48.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.781+0000 7f3e5effd640 1 -- 172.21.15.5:0/3451400381 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e60012620 con 0x7f3e681543a0 2024-03-24T10:48:48.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.781+0000 7f3e705b6640 1 -- 172.21.15.5:0/3451400381 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e58009400 con 0x7f3e681543a0 2024-03-24T10:48:48.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.781+0000 7f3e705b6640 1 -- 172.21.15.5:0/3451400381 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e681f07b0 con 0x7f3e681543a0 2024-03-24T10:48:48.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.782+0000 7f3e705b6640 1 -- 172.21.15.5:0/3451400381 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e68157f60 con 0x7f3e681543a0 2024-03-24T10:48:48.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.806+0000 7f3e5effd640 1 -- 172.21.15.5:0/3451400381 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3e60007070 con 0x7f3e681543a0 2024-03-24T10:48:48.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.807+0000 7f3e5effd640 1 --2- 172.21.15.5:0/3451400381 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3e4c06b890 0x7f3e4c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:48.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.807+0000 7f3e6e352640 1 --2- 172.21.15.5:0/3451400381 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3e4c06b890 0x7f3e4c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:48.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:48.808+0000 7f3e6e352640 1 --2- 172.21.15.5:0/3451400381 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3e4c06b890 0x7f3e4c06dd50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3e64009c30 tx=0x7f3e64009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:49.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:49.087+0000 7f3e5effd640 1 -- 172.21.15.5:0/3451400381 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f3e60058c50 con 0x7f3e681543a0 2024-03-24T10:48:49.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:49.087+0000 7f3e5effd640 1 -- 172.21.15.5:0/3451400381 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3e60054ef0 con 0x7f3e681543a0 2024-03-24T10:48:49.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:49.312+0000 7f3e705b6640 1 -- 172.21.15.5:0/3451400381 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e68157f60 con 0x7f3e681543a0 2024-03-24T10:48:49.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:49.325+0000 7f3e5effd640 1 -- 172.21.15.5:0/3451400381 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f3e68157f60 con 0x7f3e681543a0 2024-03-24T10:48:49.325 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:49.325 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:49.325 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:49.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:49.327+0000 7f3e705b6640 1 -- 172.21.15.5:0/3451400381 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3e4c06b890 msgr2=0x7f3e4c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:49.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:49.327+0000 7f3e705b6640 1 --2- 172.21.15.5:0/3451400381 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3e4c06b890 0x7f3e4c06dd50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3e64009c30 tx=0x7f3e64009190 comp rx=0 tx=0).stop 2024-03-24T10:48:49.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:49.327+0000 7f3e705b6640 1 -- 172.21.15.5:0/3451400381 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e681543a0 msgr2=0x7f3e6814b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:49.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:49.327+0000 7f3e705b6640 1 --2- 172.21.15.5:0/3451400381 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e681543a0 0x7f3e6814b330 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3e6000cb40 tx=0x7f3e6000a060 comp rx=0 tx=0).stop 2024-03-24T10:48:49.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:49.327+0000 7f3e705b6640 1 -- 172.21.15.5:0/3451400381 shutdown_connections 2024-03-24T10:48:49.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:49.327+0000 7f3e705b6640 1 --2- 172.21.15.5:0/3451400381 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3e4c06b890 0x7f3e4c06dd50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:49.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:49.327+0000 7f3e705b6640 1 --2- 172.21.15.5:0/3451400381 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e681543a0 0x7f3e6814b330 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:49.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:49.327+0000 7f3e705b6640 1 --2- 172.21.15.5:0/3451400381 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e680ccaa0 0x7f3e68157420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:49.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:49.327+0000 7f3e705b6640 1 --2- 172.21.15.5:0/3451400381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e6806d4f0 0x7f3e680b23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:49.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:49.327+0000 7f3e705b6640 1 -- 172.21.15.5:0/3451400381 >> 172.21.15.5:0/3451400381 conn(0x7f3e68072ef0 msgr2=0x7f3e680cef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:49.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:49.327+0000 7f3e705b6640 1 -- 172.21.15.5:0/3451400381 shutdown_connections 2024-03-24T10:48:49.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:49.327+0000 7f3e705b6640 1 -- 172.21.15.5:0/3451400381 wait complete. 2024-03-24T10:48:49.945 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:49.946 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:50.947 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:51.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.050+0000 7f9490ceb640 1 Processor -- start 2024-03-24T10:48:51.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.050+0000 7f9490ceb640 1 -- start start 2024-03-24T10:48:51.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.051+0000 7f9490ceb640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f948c0c2ea0 0x7f948c0caab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.051+0000 7f9490ceb640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f948c0c9610 0x7f948c0c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.051+0000 7f9490ceb640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f948c0ca000 con 0x7f948c0c58c0 2024-03-24T10:48:51.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.051+0000 7f9490ceb640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f948c0cb210 con 0x7f948c0c2ea0 2024-03-24T10:48:51.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.051+0000 7f9490ceb640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f948c156db0 con 0x7f948c0c9610 2024-03-24T10:48:51.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.051+0000 7f948a59c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f948c0c2ea0 0x7f948c0caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.052+0000 7f948a59c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f948c0c2ea0 0x7f948c0caab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39306/0 (socket says 172.21.15.5:39306) 2024-03-24T10:48:51.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.052+0000 7f948a59c640 1 -- 172.21.15.5:0/1281956032 learned_addr learned my addr 172.21.15.5:0/1281956032 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:51.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.052+0000 7f948ad9d640 1 --2- 172.21.15.5:0/1281956032 >> v2:172.21.15.57:3300/0 conn(0x7f948c0c9610 0x7f948c0c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.054+0000 7f948a59c640 1 -- 172.21.15.5:0/1281956032 >> v2:172.21.15.57:3300/0 conn(0x7f948c0c9610 msgr2=0x7f948c0c9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:51.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.054+0000 7f948a59c640 1 --2- 172.21.15.5:0/1281956032 >> v2:172.21.15.57:3300/0 conn(0x7f948c0c9610 0x7f948c0c9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.054+0000 7f948a59c640 1 -- 172.21.15.5:0/1281956032 >> v1:172.21.15.57:6789/0 conn(0x7f948c0c58c0 legacy=0x7f948c0c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.054+0000 7f948a59c640 1 -- 172.21.15.5:0/1281956032 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f948c156fc0 con 0x7f948c0c2ea0 2024-03-24T10:48:51.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.056+0000 7f948a59c640 1 --2- 172.21.15.5:0/1281956032 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f948c0c2ea0 0x7f948c0caab0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f948c0c69a0 tx=0x7f947c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b53341749b52503b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.061+0000 7f9489d9b640 1 -- 172.21.15.5:0/1281956032 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c02f850 con 0x7f948c0c2ea0 2024-03-24T10:48:51.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.061+0000 7f9489d9b640 1 -- 172.21.15.5:0/1281956032 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f947c02f9b0 con 0x7f948c0c2ea0 2024-03-24T10:48:51.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.061+0000 7f9489d9b640 1 -- 172.21.15.5:0/1281956032 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c02fd40 con 0x7f948c0c2ea0 2024-03-24T10:48:51.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.061+0000 7f9490ceb640 1 -- 172.21.15.5:0/1281956032 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f948c0c2ea0 msgr2=0x7f948c0caab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.061+0000 7f9490ceb640 1 --2- 172.21.15.5:0/1281956032 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f948c0c2ea0 0x7f948c0caab0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f948c0c69a0 tx=0x7f947c02ba30 comp rx=0 tx=0).stop 2024-03-24T10:48:51.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.061+0000 7f9490ceb640 1 -- 172.21.15.5:0/1281956032 shutdown_connections 2024-03-24T10:48:51.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.061+0000 7f9490ceb640 1 --2- 172.21.15.5:0/1281956032 >> v2:172.21.15.57:3300/0 conn(0x7f948c0c9610 0x7f948c0c9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.061+0000 7f9490ceb640 1 --2- 172.21.15.5:0/1281956032 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f948c0c2ea0 0x7f948c0caab0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.061+0000 7f9490ceb640 1 -- 172.21.15.5:0/1281956032 >> 172.21.15.5:0/1281956032 conn(0x7f948c06b130 msgr2=0x7f948c06b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:51.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.061+0000 7f9490ceb640 1 -- 172.21.15.5:0/1281956032 shutdown_connections 2024-03-24T10:48:51.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.061+0000 7f9490ceb640 1 -- 172.21.15.5:0/1281956032 wait complete. 2024-03-24T10:48:51.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.063+0000 7f9490ceb640 1 Processor -- start 2024-03-24T10:48:51.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.063+0000 7f9490ceb640 1 -- start start 2024-03-24T10:48:51.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.063+0000 7f9490ceb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f948c0c2ea0 0x7f948c152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.063+0000 7f9490ceb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f948c0c58c0 0x7f948c15a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.063+0000 7f9490ceb640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f948c0c9610 0x7f948c145130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.063+0000 7f9490ceb640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f948c14b010 con 0x7f948c0c2ea0 2024-03-24T10:48:51.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.063+0000 7f9490ceb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f948c153c30 con 0x7f948c0c58c0 2024-03-24T10:48:51.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.063+0000 7f9490ceb640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f948c1516a0 con 0x7f948c0c9610 2024-03-24T10:48:51.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.064+0000 7f948a59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f948c0c2ea0 0x7f948c152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.064+0000 7f9483fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f948c0c58c0 0x7f948c15a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.064+0000 7f948ad9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f948c0c9610 0x7f948c145130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.064+0000 7f948a59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f948c0c2ea0 0x7f948c152160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39318/0 (socket says 172.21.15.5:39318) 2024-03-24T10:48:51.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.064+0000 7f948a59c640 1 -- 172.21.15.5:0/979044205 learned_addr learned my addr 172.21.15.5:0/979044205 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:51.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.069+0000 7f9483fff640 1 -- 172.21.15.5:0/979044205 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f948c0c9610 msgr2=0x7f948c145130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.069+0000 7f9483fff640 1 --2- 172.21.15.5:0/979044205 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f948c0c9610 0x7f948c145130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.069+0000 7f9483fff640 1 -- 172.21.15.5:0/979044205 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f948c0c2ea0 msgr2=0x7f948c152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.069+0000 7f9483fff640 1 --2- 172.21.15.5:0/979044205 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f948c0c2ea0 0x7f948c152160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.069+0000 7f9483fff640 1 -- 172.21.15.5:0/979044205 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f947c0094d0 con 0x7f948c0c58c0 2024-03-24T10:48:51.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.072+0000 7f9483fff640 1 --2- 172.21.15.5:0/979044205 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f948c0c58c0 0x7f948c15a5c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f947400c3c0 tx=0x7f947400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.078+0000 7f9482ffd640 1 -- 172.21.15.5:0/979044205 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94740052d0 con 0x7f948c0c58c0 2024-03-24T10:48:51.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.078+0000 7f9482ffd640 1 -- 172.21.15.5:0/979044205 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9474005450 con 0x7f948c0c58c0 2024-03-24T10:48:51.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.078+0000 7f9482ffd640 1 -- 172.21.15.5:0/979044205 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9474007260 con 0x7f948c0c58c0 2024-03-24T10:48:51.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.078+0000 7f9490ceb640 1 -- 172.21.15.5:0/979044205 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f948c153ef0 con 0x7f948c0c58c0 2024-03-24T10:48:51.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.078+0000 7f9490ceb640 1 -- 172.21.15.5:0/979044205 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f948c151960 con 0x7f948c0c58c0 2024-03-24T10:48:51.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.079+0000 7f9490ceb640 1 -- 172.21.15.5:0/979044205 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f948c062a00 con 0x7f948c0c58c0 2024-03-24T10:48:51.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.109+0000 7f9482ffd640 1 -- 172.21.15.5:0/979044205 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f94740073c0 con 0x7f948c0c58c0 2024-03-24T10:48:51.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.109+0000 7f9482ffd640 1 --2- 172.21.15.5:0/979044205 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f946c06b890 0x7f946c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:51.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.109+0000 7f948a59c640 1 --2- 172.21.15.5:0/979044205 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f946c06b890 0x7f946c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:51.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.110+0000 7f948a59c640 1 --2- 172.21.15.5:0/979044205 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f946c06b890 0x7f946c06dd50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f947c0090b0 tx=0x7f947c008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:51.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.113+0000 7f9482ffd640 1 -- 172.21.15.5:0/979044205 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f9474058730 con 0x7f948c0c58c0 2024-03-24T10:48:51.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.394+0000 7f9482ffd640 1 -- 172.21.15.5:0/979044205 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f94740549d0 con 0x7f948c0c58c0 2024-03-24T10:48:51.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.636+0000 7f9490ceb640 1 -- 172.21.15.5:0/979044205 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f948c0681d0 con 0x7f948c0c58c0 2024-03-24T10:48:51.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.694+0000 7f9482ffd640 1 -- 172.21.15.5:0/979044205 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f9474020020 con 0x7f948c0c58c0 2024-03-24T10:48:51.695 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:51.695 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:51.695 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:51.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.697+0000 7f9490ceb640 1 -- 172.21.15.5:0/979044205 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f946c06b890 msgr2=0x7f946c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.697+0000 7f9490ceb640 1 --2- 172.21.15.5:0/979044205 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f946c06b890 0x7f946c06dd50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f947c0090b0 tx=0x7f947c008030 comp rx=0 tx=0).stop 2024-03-24T10:48:51.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.697+0000 7f9490ceb640 1 -- 172.21.15.5:0/979044205 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f948c0c58c0 msgr2=0x7f948c15a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:51.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.697+0000 7f9490ceb640 1 --2- 172.21.15.5:0/979044205 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f948c0c58c0 0x7f948c15a5c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f947400c3c0 tx=0x7f947400c880 comp rx=0 tx=0).stop 2024-03-24T10:48:51.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.697+0000 7f9490ceb640 1 -- 172.21.15.5:0/979044205 shutdown_connections 2024-03-24T10:48:51.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.697+0000 7f9490ceb640 1 --2- 172.21.15.5:0/979044205 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f946c06b890 0x7f946c06dd50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.697+0000 7f9490ceb640 1 --2- 172.21.15.5:0/979044205 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f948c0c9610 0x7f948c145130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.697+0000 7f9490ceb640 1 --2- 172.21.15.5:0/979044205 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f948c0c58c0 0x7f948c15a5c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.697+0000 7f9490ceb640 1 --2- 172.21.15.5:0/979044205 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f948c0c2ea0 0x7f948c152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:51.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.697+0000 7f9490ceb640 1 -- 172.21.15.5:0/979044205 >> 172.21.15.5:0/979044205 conn(0x7f948c06b130 msgr2=0x7f948c06b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:51.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.697+0000 7f9490ceb640 1 -- 172.21.15.5:0/979044205 shutdown_connections 2024-03-24T10:48:51.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:51.697+0000 7f9490ceb640 1 -- 172.21.15.5:0/979044205 wait complete. 2024-03-24T10:48:52.312 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:52.313 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:53.314 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:53.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.415+0000 7febf61a2640 1 Processor -- start 2024-03-24T10:48:53.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.415+0000 7febf61a2640 1 -- start start 2024-03-24T10:48:53.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.416+0000 7febf61a2640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7febf00cbee0 0x7febf00cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.416+0000 7febf61a2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7febf00ca970 0x7febf00cfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.416+0000 7febf61a2640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febf015d7c0 con 0x7febf0154420 2024-03-24T10:48:53.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.416+0000 7febf61a2640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7febf015e400 con 0x7febf00cbee0 2024-03-24T10:48:53.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.416+0000 7febf61a2640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7febf00b35e0 con 0x7febf00ca970 2024-03-24T10:48:53.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.417+0000 7febf59a1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7febf00ca970 0x7febf00cfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.417+0000 7febf51a0640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7febf0154420 0x7febf015abb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:38182/0 (socket says 172.21.15.5:38182) 2024-03-24T10:48:53.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.417+0000 7febf51a0640 1 -- 172.21.15.5:0/2263801414 learned_addr learned my addr 172.21.15.5:0/2263801414 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:48:53.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.417+0000 7febf499f640 1 --2- 172.21.15.5:0/2263801414 >> v2:172.21.15.5:3300/0 conn(0x7febf00cbee0 0x7febf00cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.421+0000 7febf499f640 1 -- 172.21.15.5:0/2263801414 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7febf00ca970 msgr2=0x7febf00cfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:53.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.421+0000 7febf499f640 1 --2- 172.21.15.5:0/2263801414 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7febf00ca970 0x7febf00cfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.421+0000 7febf499f640 1 -- 172.21.15.5:0/2263801414 >> v1:172.21.15.5:6789/0 conn(0x7febf0154420 legacy=0x7febf015abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.421+0000 7febf499f640 1 -- 172.21.15.5:0/2263801414 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febf015da60 con 0x7febf00cbee0 2024-03-24T10:48:53.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.425+0000 7febf499f640 1 --2- 172.21.15.5:0/2263801414 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7febf00cbee0 0x7febf00cf980 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7febe00097a0 tx=0x7febe002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a938a27a658661ae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:53.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.458+0000 7febe7fff640 1 -- 172.21.15.5:0/2263801414 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe0030610 con 0x7febf00cbee0 2024-03-24T10:48:53.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.458+0000 7febe7fff640 1 -- 172.21.15.5:0/2263801414 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febe0030770 con 0x7febf00cbee0 2024-03-24T10:48:53.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.458+0000 7febe7fff640 1 -- 172.21.15.5:0/2263801414 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe0030b00 con 0x7febf00cbee0 2024-03-24T10:48:53.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.459+0000 7febf61a2640 1 -- 172.21.15.5:0/2263801414 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7febf00cbee0 msgr2=0x7febf00cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.459+0000 7febf61a2640 1 --2- 172.21.15.5:0/2263801414 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7febf00cbee0 0x7febf00cf980 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7febe00097a0 tx=0x7febe002b960 comp rx=0 tx=0).stop 2024-03-24T10:48:53.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.459+0000 7febf61a2640 1 -- 172.21.15.5:0/2263801414 shutdown_connections 2024-03-24T10:48:53.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.459+0000 7febf61a2640 1 --2- 172.21.15.5:0/2263801414 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7febf00ca970 0x7febf00cfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.459+0000 7febf61a2640 1 --2- 172.21.15.5:0/2263801414 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7febf00cbee0 0x7febf00cf980 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.459+0000 7febf61a2640 1 -- 172.21.15.5:0/2263801414 >> 172.21.15.5:0/2263801414 conn(0x7febf006c400 msgr2=0x7febf0072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:53.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.459+0000 7febf61a2640 1 -- 172.21.15.5:0/2263801414 shutdown_connections 2024-03-24T10:48:53.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.459+0000 7febf61a2640 1 -- 172.21.15.5:0/2263801414 wait complete. 2024-03-24T10:48:53.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.461+0000 7febf61a2640 1 Processor -- start 2024-03-24T10:48:53.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.461+0000 7febf61a2640 1 -- start start 2024-03-24T10:48:53.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.461+0000 7febf61a2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7febf00ca970 0x7febf00c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.461+0000 7febf61a2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7febf00cbee0 0x7febf014d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.461+0000 7febf61a2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf0154420 0x7febf0150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.461+0000 7febf61a2640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7febf01f0bd0 con 0x7febf00ca970 2024-03-24T10:48:53.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.461+0000 7febf61a2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7febf00c8200 con 0x7febf0154420 2024-03-24T10:48:53.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.461+0000 7febf61a2640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7febf00c8450 con 0x7febf00cbee0 2024-03-24T10:48:53.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.462+0000 7febf59a1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf0154420 0x7febf0150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.463+0000 7febf59a1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf0154420 0x7febf0150f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:56002/0 (socket says 172.21.15.5:56002) 2024-03-24T10:48:53.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.463+0000 7febf59a1640 1 -- 172.21.15.5:0/3219901045 learned_addr learned my addr 172.21.15.5:0/3219901045 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:53.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.463+0000 7febf51a0640 1 --2- 172.21.15.5:0/3219901045 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7febf00ca970 0x7febf00c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.464+0000 7febf499f640 1 --2- 172.21.15.5:0/3219901045 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7febf00cbee0 0x7febf014d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.467+0000 7febf499f640 1 -- 172.21.15.5:0/3219901045 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf0154420 msgr2=0x7febf0150f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.467+0000 7febf499f640 1 --2- 172.21.15.5:0/3219901045 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf0154420 0x7febf0150f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.467+0000 7febf499f640 1 -- 172.21.15.5:0/3219901045 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7febf00ca970 msgr2=0x7febf00c7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:53.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.467+0000 7febf499f640 1 --2- 172.21.15.5:0/3219901045 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7febf00ca970 0x7febf00c7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:53.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.467+0000 7febf499f640 1 -- 172.21.15.5:0/3219901045 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe0009400 con 0x7febf00cbee0 2024-03-24T10:48:53.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.470+0000 7febf499f640 1 --2- 172.21.15.5:0/3219901045 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7febf00cbee0 0x7febf014d980 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7febf0073510 tx=0x7febe0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:53.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.474+0000 7febe5ffb640 1 -- 172.21.15.5:0/3219901045 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe003e040 con 0x7febf00cbee0 2024-03-24T10:48:53.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.474+0000 7febe5ffb640 1 -- 172.21.15.5:0/3219901045 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febe0039a40 con 0x7febf00cbee0 2024-03-24T10:48:53.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.474+0000 7febf61a2640 1 -- 172.21.15.5:0/3219901045 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febf01efe10 con 0x7febf00cbee0 2024-03-24T10:48:53.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.474+0000 7febe5ffb640 1 -- 172.21.15.5:0/3219901045 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe0039df0 con 0x7febf00cbee0 2024-03-24T10:48:53.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.474+0000 7febf61a2640 1 -- 172.21.15.5:0/3219901045 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7febf01f03d0 con 0x7febf00cbee0 2024-03-24T10:48:53.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.475+0000 7febf61a2640 1 -- 172.21.15.5:0/3219901045 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febf004f470 con 0x7febf00cbee0 2024-03-24T10:48:53.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.494+0000 7febe5ffb640 1 -- 172.21.15.5:0/3219901045 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7febe004c3b0 con 0x7febf00cbee0 2024-03-24T10:48:53.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.494+0000 7febe5ffb640 1 --2- 172.21.15.5:0/3219901045 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7febd406b810 0x7febd406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:53.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.494+0000 7febf51a0640 1 --2- 172.21.15.5:0/3219901045 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7febd406b810 0x7febd406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:53.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.495+0000 7febf51a0640 1 --2- 172.21.15.5:0/3219901045 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7febd406b810 0x7febd406dcd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7febec007fd0 tx=0x7febec007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:53.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.749+0000 7febe5ffb640 1 -- 172.21.15.5:0/3219901045 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7febe007aea0 con 0x7febf00cbee0 2024-03-24T10:48:53.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:53.762+0000 7febe5ffb640 1 -- 172.21.15.5:0/3219901045 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7febe0077140 con 0x7febf00cbee0 2024-03-24T10:48:54.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:54.007+0000 7febf61a2640 1 -- 172.21.15.5:0/3219901045 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7febf00cb4b0 con 0x7febf00cbee0 2024-03-24T10:48:54.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:54.022+0000 7febe5ffb640 1 -- 172.21.15.5:0/3219901045 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7febe007fe60 con 0x7febf00cbee0 2024-03-24T10:48:54.022 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:54.022 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:54.022 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:54.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:54.024+0000 7febf61a2640 1 -- 172.21.15.5:0/3219901045 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7febd406b810 msgr2=0x7febd406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:54.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:54.024+0000 7febf61a2640 1 --2- 172.21.15.5:0/3219901045 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7febd406b810 0x7febd406dcd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7febec007fd0 tx=0x7febec007e80 comp rx=0 tx=0).stop 2024-03-24T10:48:54.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:54.024+0000 7febf61a2640 1 -- 172.21.15.5:0/3219901045 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7febf00cbee0 msgr2=0x7febf014d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:54.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:54.024+0000 7febf61a2640 1 --2- 172.21.15.5:0/3219901045 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7febf00cbee0 0x7febf014d980 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7febf0073510 tx=0x7febe0004730 comp rx=0 tx=0).stop 2024-03-24T10:48:54.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:54.024+0000 7febf61a2640 1 -- 172.21.15.5:0/3219901045 shutdown_connections 2024-03-24T10:48:54.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:54.024+0000 7febf61a2640 1 --2- 172.21.15.5:0/3219901045 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7febd406b810 0x7febd406dcd0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:54.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:54.024+0000 7febf61a2640 1 --2- 172.21.15.5:0/3219901045 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7febf0154420 0x7febf0150f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:54.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:54.024+0000 7febf61a2640 1 --2- 172.21.15.5:0/3219901045 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7febf00cbee0 0x7febf014d980 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:54.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:54.024+0000 7febf61a2640 1 --2- 172.21.15.5:0/3219901045 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7febf00ca970 0x7febf00c7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:54.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:54.024+0000 7febf61a2640 1 -- 172.21.15.5:0/3219901045 >> 172.21.15.5:0/3219901045 conn(0x7febf006c400 msgr2=0x7febf0072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:54.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:54.024+0000 7febf61a2640 1 -- 172.21.15.5:0/3219901045 shutdown_connections 2024-03-24T10:48:54.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:54.024+0000 7febf61a2640 1 -- 172.21.15.5:0/3219901045 wait complete. 2024-03-24T10:48:54.639 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:54.639 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:55.640 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:55.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.743+0000 7f760015c640 1 Processor -- start 2024-03-24T10:48:55.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.743+0000 7f760015c640 1 -- start start 2024-03-24T10:48:55.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.744+0000 7f760015c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f75f81543a0 0x7f75f815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:55.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.744+0000 7f760015c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f75f806d4f0 0x7f75f80cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:55.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.744+0000 7f760015c640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75f80cb2f0 con 0x7f75f80ccaa0 2024-03-24T10:48:55.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.744+0000 7f760015c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f75f80c6750 con 0x7f75f81543a0 2024-03-24T10:48:55.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.744+0000 7f760015c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f75f80cc380 con 0x7f75f806d4f0 2024-03-24T10:48:55.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.744+0000 7f75fe6f9640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f75f80ccaa0 0x7f75f806bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:38186/0 (socket says 172.21.15.5:38186) 2024-03-24T10:48:55.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.744+0000 7f75fe6f9640 1 -- 172.21.15.5:0/1580740678 learned_addr learned my addr 172.21.15.5:0/1580740678 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:48:55.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.744+0000 7f75fdef8640 1 --2- 172.21.15.5:0/1580740678 >> v2:172.21.15.57:3300/0 conn(0x7f75f81543a0 0x7f75f815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:55.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.745+0000 7f75fd6f7640 1 --2- 172.21.15.5:0/1580740678 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f75f806d4f0 0x7f75f80cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.747+0000 7f75fcef6640 1 -- 172.21.15.5:0/1580740678 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2246606446 0 0) 0x7f75f80cb2f0 con 0x7f75f80ccaa0 2024-03-24T10:48:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.747+0000 7f75fcef6640 1 -- 172.21.15.5:0/1580740678 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75e4003520 con 0x7f75f80ccaa0 2024-03-24T10:48:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.747+0000 7f75fdef8640 1 -- 172.21.15.5:0/1580740678 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f75f806d4f0 msgr2=0x7f75f80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.747+0000 7f75fdef8640 1 --2- 172.21.15.5:0/1580740678 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f75f806d4f0 0x7f75f80cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.747+0000 7f75fdef8640 1 -- 172.21.15.5:0/1580740678 >> v1:172.21.15.5:6789/0 conn(0x7f75f80ccaa0 legacy=0x7f75f806bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:55.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.747+0000 7f75fdef8640 1 -- 172.21.15.5:0/1580740678 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f80ca7e0 con 0x7f75f81543a0 2024-03-24T10:48:55.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.750+0000 7f75fdef8640 1 --2- 172.21.15.5:0/1580740678 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75f81543a0 0x7f75f815ac00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f75f40097a0 tx=0x7f75f402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97038ac0f709453c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:55.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.755+0000 7f75fcef6640 1 -- 172.21.15.5:0/1580740678 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f4030610 con 0x7f75f81543a0 2024-03-24T10:48:55.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.755+0000 7f75fcef6640 1 -- 172.21.15.5:0/1580740678 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75f4030770 con 0x7f75f81543a0 2024-03-24T10:48:55.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.755+0000 7f75fcef6640 1 -- 172.21.15.5:0/1580740678 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f4030b00 con 0x7f75f81543a0 2024-03-24T10:48:55.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.756+0000 7f760015c640 1 -- 172.21.15.5:0/1580740678 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75f81543a0 msgr2=0x7f75f815ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:55.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.756+0000 7f760015c640 1 --2- 172.21.15.5:0/1580740678 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75f81543a0 0x7f75f815ac00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f75f40097a0 tx=0x7f75f402b960 comp rx=0 tx=0).stop 2024-03-24T10:48:55.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.756+0000 7f760015c640 1 -- 172.21.15.5:0/1580740678 shutdown_connections 2024-03-24T10:48:55.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.756+0000 7f760015c640 1 --2- 172.21.15.5:0/1580740678 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f75f806d4f0 0x7f75f80cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:55.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.756+0000 7f760015c640 1 --2- 172.21.15.5:0/1580740678 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75f81543a0 0x7f75f815ac00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:55.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.756+0000 7f760015c640 1 -- 172.21.15.5:0/1580740678 >> 172.21.15.5:0/1580740678 conn(0x7f75f8072ef0 msgr2=0x7f75f8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:55.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.756+0000 7f760015c640 1 -- 172.21.15.5:0/1580740678 shutdown_connections 2024-03-24T10:48:55.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.756+0000 7f760015c640 1 -- 172.21.15.5:0/1580740678 wait complete. 2024-03-24T10:48:55.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.757+0000 7f760015c640 1 Processor -- start 2024-03-24T10:48:55.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.757+0000 7f760015c640 1 -- start start 2024-03-24T10:48:55.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.758+0000 7f760015c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f75f806d4f0 0x7f75f8158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:55.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.758+0000 7f760015c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75f80ccaa0 0x7f75f81401c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:55.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.758+0000 7f760015c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75f81543a0 0x7f75f8140700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:55.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.758+0000 7f760015c640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f75f80b4950 con 0x7f75f81543a0 2024-03-24T10:48:55.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.758+0000 7f760015c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f75f8140c40 con 0x7f75f80ccaa0 2024-03-24T10:48:55.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.758+0000 7f760015c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f75f80b3700 con 0x7f75f806d4f0 2024-03-24T10:48:55.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.759+0000 7f75fe6f9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75f81543a0 0x7f75f8140700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:55.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.759+0000 7f75fe6f9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75f81543a0 0x7f75f8140700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38576/0 (socket says 172.21.15.5:38576) 2024-03-24T10:48:55.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.759+0000 7f75fe6f9640 1 -- 172.21.15.5:0/38113584 learned_addr learned my addr 172.21.15.5:0/38113584 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:55.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.760+0000 7f75fd6f7640 1 --2- 172.21.15.5:0/38113584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75f80ccaa0 0x7f75f81401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:55.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.760+0000 7f75fdef8640 1 --2- 172.21.15.5:0/38113584 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f75f806d4f0 0x7f75f8158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:55.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.762+0000 7f75fe6f9640 1 -- 172.21.15.5:0/38113584 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f75f806d4f0 msgr2=0x7f75f8158e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:55.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.762+0000 7f75fe6f9640 1 --2- 172.21.15.5:0/38113584 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f75f806d4f0 0x7f75f8158e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:55.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.762+0000 7f75fe6f9640 1 -- 172.21.15.5:0/38113584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75f80ccaa0 msgr2=0x7f75f81401c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:55.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.763+0000 7f75fe6f9640 1 --2- 172.21.15.5:0/38113584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75f80ccaa0 0x7f75f81401c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:55.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.763+0000 7f75fe6f9640 1 -- 172.21.15.5:0/38113584 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f4009400 con 0x7f75f81543a0 2024-03-24T10:48:55.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.765+0000 7f75fe6f9640 1 --2- 172.21.15.5:0/38113584 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75f81543a0 0x7f75f8140700 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f75f000cb40 tx=0x7f75f000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:55.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.769+0000 7f75ee7fc640 1 -- 172.21.15.5:0/38113584 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f000aca0 con 0x7f75f81543a0 2024-03-24T10:48:55.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.769+0000 7f75ee7fc640 1 -- 172.21.15.5:0/38113584 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75f000ae00 con 0x7f75f81543a0 2024-03-24T10:48:55.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.769+0000 7f760015c640 1 -- 172.21.15.5:0/38113584 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f80b39e0 con 0x7f75f81543a0 2024-03-24T10:48:55.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.769+0000 7f75ee7fc640 1 -- 172.21.15.5:0/38113584 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f0012620 con 0x7f75f81543a0 2024-03-24T10:48:55.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.769+0000 7f760015c640 1 -- 172.21.15.5:0/38113584 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f80b4e80 con 0x7f75f81543a0 2024-03-24T10:48:55.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:55.770+0000 7f760015c640 1 -- 172.21.15.5:0/38113584 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75f80681d0 con 0x7f75f81543a0 2024-03-24T10:48:56.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:56.040+0000 7f75ee7fc640 1 -- 172.21.15.5:0/38113584 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f75f0007070 con 0x7f75f81543a0 2024-03-24T10:48:56.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:56.040+0000 7f75ee7fc640 1 --2- 172.21.15.5:0/38113584 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f75dc06b860 0x7f75dc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:56.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:56.041+0000 7f75fdef8640 1 --2- 172.21.15.5:0/38113584 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f75dc06b860 0x7f75dc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:56.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:56.041+0000 7f75fdef8640 1 --2- 172.21.15.5:0/38113584 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f75dc06b860 0x7f75dc06dd20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f75f4036000 tx=0x7f75f403b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:56.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:56.064+0000 7f75ee7fc640 1 -- 172.21.15.5:0/38113584 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f75f0058e40 con 0x7f75f81543a0 2024-03-24T10:48:56.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:56.064+0000 7f75ee7fc640 1 -- 172.21.15.5:0/38113584 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f75f00550e0 con 0x7f75f81543a0 2024-03-24T10:48:56.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:56.326+0000 7f760015c640 1 -- 172.21.15.5:0/38113584 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75f8150240 con 0x7f75f81543a0 2024-03-24T10:48:56.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:56.338+0000 7f75ee7fc640 1 -- 172.21.15.5:0/38113584 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f75f0017090 con 0x7f75f81543a0 2024-03-24T10:48:56.339 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:56.339 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:56.339 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:56.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:56.341+0000 7f760015c640 1 -- 172.21.15.5:0/38113584 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f75dc06b860 msgr2=0x7f75dc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:56.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:56.341+0000 7f760015c640 1 --2- 172.21.15.5:0/38113584 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f75dc06b860 0x7f75dc06dd20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f75f4036000 tx=0x7f75f403b040 comp rx=0 tx=0).stop 2024-03-24T10:48:56.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:56.341+0000 7f760015c640 1 -- 172.21.15.5:0/38113584 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75f81543a0 msgr2=0x7f75f8140700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:56.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:56.341+0000 7f760015c640 1 --2- 172.21.15.5:0/38113584 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75f81543a0 0x7f75f8140700 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f75f000cb40 tx=0x7f75f000a060 comp rx=0 tx=0).stop 2024-03-24T10:48:56.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:56.341+0000 7f760015c640 1 -- 172.21.15.5:0/38113584 shutdown_connections 2024-03-24T10:48:56.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:56.341+0000 7f760015c640 1 --2- 172.21.15.5:0/38113584 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f75dc06b860 0x7f75dc06dd20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:56.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:56.341+0000 7f760015c640 1 --2- 172.21.15.5:0/38113584 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f75f81543a0 0x7f75f8140700 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:56.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:56.341+0000 7f760015c640 1 --2- 172.21.15.5:0/38113584 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f75f80ccaa0 0x7f75f81401c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:56.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:56.341+0000 7f760015c640 1 --2- 172.21.15.5:0/38113584 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f75f806d4f0 0x7f75f8158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:56.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:56.341+0000 7f760015c640 1 -- 172.21.15.5:0/38113584 >> 172.21.15.5:0/38113584 conn(0x7f75f8072ef0 msgr2=0x7f75f806c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:56.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:56.341+0000 7f760015c640 1 -- 172.21.15.5:0/38113584 shutdown_connections 2024-03-24T10:48:56.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:56.341+0000 7f760015c640 1 -- 172.21.15.5:0/38113584 wait complete. 2024-03-24T10:48:56.955 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:56.955 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:48:57.956 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:48:58.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.058+0000 7f50be6f1640 1 Processor -- start 2024-03-24T10:48:58.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.058+0000 7f50be6f1640 1 -- start start 2024-03-24T10:48:58.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.060+0000 7f50be6f1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f50b81543a0 0x7f50b815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:58.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.060+0000 7f50be6f1640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f50b806d4f0 0x7f50b80cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:58.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.060+0000 7f50be6f1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f50b80ccaa0 0x7f50b806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:58.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.060+0000 7f50be6f1640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f50b80c6750 con 0x7f50b806d4f0 2024-03-24T10:48:58.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.060+0000 7f50be6f1640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f50b80cceb0 con 0x7f50b81543a0 2024-03-24T10:48:58.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.060+0000 7f50be6f1640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f50b80cc380 con 0x7f50b80ccaa0 2024-03-24T10:48:58.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.060+0000 7f50b77fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f50b806d4f0 0x7f50b80cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:58.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.061+0000 7f50bcc8e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f50b80ccaa0 0x7f50b806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:58.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.061+0000 7f50b7fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f50b81543a0 0x7f50b815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:58.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.061+0000 7f50b77fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f50b806d4f0 0x7f50b80cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38580/0 (socket says 172.21.15.5:38580) 2024-03-24T10:48:58.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.061+0000 7f50b77fe640 1 -- 172.21.15.5:0/2588529173 learned_addr learned my addr 172.21.15.5:0/2588529173 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:58.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.063+0000 7f50b77fe640 1 -- 172.21.15.5:0/2588529173 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f50b80ccaa0 msgr2=0x7f50b806bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:58.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.063+0000 7f50b77fe640 1 --2- 172.21.15.5:0/2588529173 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f50b80ccaa0 0x7f50b806bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:58.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.063+0000 7f50b77fe640 1 -- 172.21.15.5:0/2588529173 >> v2:172.21.15.57:3300/0 conn(0x7f50b81543a0 msgr2=0x7f50b815ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:48:58.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.063+0000 7f50b77fe640 1 --2- 172.21.15.5:0/2588529173 >> v2:172.21.15.57:3300/0 conn(0x7f50b81543a0 0x7f50b815ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:58.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.063+0000 7f50b77fe640 1 -- 172.21.15.5:0/2588529173 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50b80cb2f0 con 0x7f50b806d4f0 2024-03-24T10:48:58.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.065+0000 7f50b77fe640 1 --2- 172.21.15.5:0/2588529173 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f50b806d4f0 0x7f50b80cc560 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f50b80cd600 tx=0x7f50ac02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c4117dc1a3f11b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:58.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.069+0000 7f50b6ffd640 1 -- 172.21.15.5:0/2588529173 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ac030860 con 0x7f50b806d4f0 2024-03-24T10:48:58.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.069+0000 7f50b6ffd640 1 -- 172.21.15.5:0/2588529173 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50ac0309c0 con 0x7f50b806d4f0 2024-03-24T10:48:58.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.069+0000 7f50b6ffd640 1 -- 172.21.15.5:0/2588529173 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50ac030d50 con 0x7f50b806d4f0 2024-03-24T10:48:58.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.069+0000 7f50be6f1640 1 -- 172.21.15.5:0/2588529173 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f50b806d4f0 msgr2=0x7f50b80cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:58.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.069+0000 7f50be6f1640 1 --2- 172.21.15.5:0/2588529173 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f50b806d4f0 0x7f50b80cc560 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f50b80cd600 tx=0x7f50ac02b9a0 comp rx=0 tx=0).stop 2024-03-24T10:48:58.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.069+0000 7f50be6f1640 1 -- 172.21.15.5:0/2588529173 shutdown_connections 2024-03-24T10:48:58.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.069+0000 7f50be6f1640 1 --2- 172.21.15.5:0/2588529173 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f50b80ccaa0 0x7f50b806bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:58.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.069+0000 7f50be6f1640 1 --2- 172.21.15.5:0/2588529173 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f50b806d4f0 0x7f50b80cc560 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:58.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.069+0000 7f50be6f1640 1 --2- 172.21.15.5:0/2588529173 >> v2:172.21.15.57:3300/0 conn(0x7f50b81543a0 0x7f50b815ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:58.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.069+0000 7f50be6f1640 1 -- 172.21.15.5:0/2588529173 >> 172.21.15.5:0/2588529173 conn(0x7f50b8072ef0 msgr2=0x7f50b8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:58.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.070+0000 7f50be6f1640 1 -- 172.21.15.5:0/2588529173 shutdown_connections 2024-03-24T10:48:58.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.070+0000 7f50be6f1640 1 -- 172.21.15.5:0/2588529173 wait complete. 2024-03-24T10:48:58.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.071+0000 7f50be6f1640 1 Processor -- start 2024-03-24T10:48:58.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.071+0000 7f50be6f1640 1 -- start start 2024-03-24T10:48:58.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.072+0000 7f50be6f1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f50b806d4f0 0x7f50b80ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:58.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.072+0000 7f50be6f1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f50b80ccaa0 0x7f50b81f3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:58.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.072+0000 7f50be6f1640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f50b81543a0 0x7f50b81f37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:58.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.072+0000 7f50be6f1640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f50b8155d30 con 0x7f50b81543a0 2024-03-24T10:48:58.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.072+0000 7f50be6f1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f50b80cea10 con 0x7f50b806d4f0 2024-03-24T10:48:58.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.072+0000 7f50be6f1640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f50b8157760 con 0x7f50b80ccaa0 2024-03-24T10:48:58.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.072+0000 7f50bcc8e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f50b81543a0 0x7f50b81f37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:58.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.072+0000 7f50b77fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f50b80ccaa0 0x7f50b81f3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:58.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.072+0000 7f50bcc8e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f50b81543a0 0x7f50b81f37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38594/0 (socket says 172.21.15.5:38594) 2024-03-24T10:48:58.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.072+0000 7f50bcc8e640 1 -- 172.21.15.5:0/948818564 learned_addr learned my addr 172.21.15.5:0/948818564 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:48:58.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.073+0000 7f50b7fff640 1 --2- 172.21.15.5:0/948818564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f50b806d4f0 0x7f50b80ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:58.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.076+0000 7f50bcc8e640 1 -- 172.21.15.5:0/948818564 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f50b80ccaa0 msgr2=0x7f50b81f3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:58.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.076+0000 7f50bcc8e640 1 --2- 172.21.15.5:0/948818564 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f50b80ccaa0 0x7f50b81f3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:58.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.076+0000 7f50bcc8e640 1 -- 172.21.15.5:0/948818564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f50b806d4f0 msgr2=0x7f50b80ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:58.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.076+0000 7f50bcc8e640 1 --2- 172.21.15.5:0/948818564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f50b806d4f0 0x7f50b80ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:58.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.076+0000 7f50bcc8e640 1 -- 172.21.15.5:0/948818564 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50ac0094d0 con 0x7f50b81543a0 2024-03-24T10:48:58.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.079+0000 7f50bcc8e640 1 --2- 172.21.15.5:0/948818564 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f50b81543a0 0x7f50b81f37b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f50b000efc0 tx=0x7f50b000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:58.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.083+0000 7f50b4ff9640 1 -- 172.21.15.5:0/948818564 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b0014480 con 0x7f50b81543a0 2024-03-24T10:48:58.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.083+0000 7f50b4ff9640 1 -- 172.21.15.5:0/948818564 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50b000f040 con 0x7f50b81543a0 2024-03-24T10:48:58.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.083+0000 7f50be6f1640 1 -- 172.21.15.5:0/948818564 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50b81590f0 con 0x7f50b81543a0 2024-03-24T10:48:58.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.083+0000 7f50b4ff9640 1 -- 172.21.15.5:0/948818564 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b0014820 con 0x7f50b81543a0 2024-03-24T10:48:58.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.083+0000 7f50be6f1640 1 -- 172.21.15.5:0/948818564 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50b8157480 con 0x7f50b81543a0 2024-03-24T10:48:58.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.084+0000 7f50be6f1640 1 -- 172.21.15.5:0/948818564 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50b80cebc0 con 0x7f50b81543a0 2024-03-24T10:48:58.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.366+0000 7f50b4ff9640 1 -- 172.21.15.5:0/948818564 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f50b0014980 con 0x7f50b81543a0 2024-03-24T10:48:58.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.366+0000 7f50b4ff9640 1 --2- 172.21.15.5:0/948818564 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f50a406b890 0x7f50a406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:48:58.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.366+0000 7f50b7fff640 1 --2- 172.21.15.5:0/948818564 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f50a406b890 0x7f50a406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:48:58.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.367+0000 7f50b7fff640 1 --2- 172.21.15.5:0/948818564 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f50a406b890 0x7f50a406dd50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f50a8009a20 tx=0x7f50a8009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:48:58.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.387+0000 7f50b4ff9640 1 -- 172.21.15.5:0/948818564 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f50b0059b50 con 0x7f50b81543a0 2024-03-24T10:48:58.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.387+0000 7f50b4ff9640 1 -- 172.21.15.5:0/948818564 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f50b0055df0 con 0x7f50b81543a0 2024-03-24T10:48:58.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.608+0000 7f50be6f1640 1 -- 172.21.15.5:0/948818564 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50b80cebc0 con 0x7f50b81543a0 2024-03-24T10:48:58.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.673+0000 7f50b4ff9640 1 -- 172.21.15.5:0/948818564 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f50b80cebc0 con 0x7f50b81543a0 2024-03-24T10:48:58.673 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:48:58.673 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:48:58.674 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:48:58.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.675+0000 7f50be6f1640 1 -- 172.21.15.5:0/948818564 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f50a406b890 msgr2=0x7f50a406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:58.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.675+0000 7f50be6f1640 1 --2- 172.21.15.5:0/948818564 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f50a406b890 0x7f50a406dd50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f50a8009a20 tx=0x7f50a8009190 comp rx=0 tx=0).stop 2024-03-24T10:48:58.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.675+0000 7f50be6f1640 1 -- 172.21.15.5:0/948818564 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f50b81543a0 msgr2=0x7f50b81f37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:48:58.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.675+0000 7f50be6f1640 1 --2- 172.21.15.5:0/948818564 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f50b81543a0 0x7f50b81f37b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f50b000efc0 tx=0x7f50b000c490 comp rx=0 tx=0).stop 2024-03-24T10:48:58.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.675+0000 7f50be6f1640 1 -- 172.21.15.5:0/948818564 shutdown_connections 2024-03-24T10:48:58.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.675+0000 7f50be6f1640 1 --2- 172.21.15.5:0/948818564 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f50a406b890 0x7f50a406dd50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:58.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.675+0000 7f50be6f1640 1 --2- 172.21.15.5:0/948818564 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f50b81543a0 0x7f50b81f37b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:58.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.675+0000 7f50be6f1640 1 --2- 172.21.15.5:0/948818564 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f50b80ccaa0 0x7f50b81f3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:58.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.675+0000 7f50be6f1640 1 --2- 172.21.15.5:0/948818564 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f50b806d4f0 0x7f50b80ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:48:58.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.675+0000 7f50be6f1640 1 -- 172.21.15.5:0/948818564 >> 172.21.15.5:0/948818564 conn(0x7f50b8072ef0 msgr2=0x7f50b81525d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:48:58.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.676+0000 7f50be6f1640 1 -- 172.21.15.5:0/948818564 shutdown_connections 2024-03-24T10:48:58.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:48:58.676+0000 7f50be6f1640 1 -- 172.21.15.5:0/948818564 wait complete. 2024-03-24T10:48:59.290 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:48:59.290 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:00.292 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:00.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.394+0000 7f7d78023640 1 Processor -- start 2024-03-24T10:49:00.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.394+0000 7f7d78023640 1 -- start start 2024-03-24T10:49:00.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.395+0000 7f7d78023640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7d701543a0 0x7f7d7015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:00.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.395+0000 7f7d78023640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7d7006d4f0 0x7f7d700cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:00.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.395+0000 7f7d78023640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d700cb2f0 con 0x7f7d700ccaa0 2024-03-24T10:49:00.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.395+0000 7f7d78023640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f7d700c6750 con 0x7f7d7006d4f0 2024-03-24T10:49:00.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.395+0000 7f7d78023640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f7d700cc380 con 0x7f7d701543a0 2024-03-24T10:49:00.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.395+0000 7f7d755be640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7d7006d4f0 0x7f7d700cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:00.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.396+0000 7f7d755be640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7d7006d4f0 0x7f7d700cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38606/0 (socket says 172.21.15.5:38606) 2024-03-24T10:49:00.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.396+0000 7f7d755be640 1 -- 172.21.15.5:0/2015220389 learned_addr learned my addr 172.21.15.5:0/2015220389 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:00.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.396+0000 7f7d75dbf640 1 --2- 172.21.15.5:0/2015220389 >> v2:172.21.15.57:3300/0 conn(0x7f7d701543a0 0x7f7d7015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:00.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.398+0000 7f7d755be640 1 -- 172.21.15.5:0/2015220389 >> v2:172.21.15.57:3300/0 conn(0x7f7d701543a0 msgr2=0x7f7d7015ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:00.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.398+0000 7f7d755be640 1 --2- 172.21.15.5:0/2015220389 >> v2:172.21.15.57:3300/0 conn(0x7f7d701543a0 0x7f7d7015ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:00.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.398+0000 7f7d755be640 1 -- 172.21.15.5:0/2015220389 >> v1:172.21.15.57:6789/0 conn(0x7f7d700ccaa0 legacy=0x7f7d7006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:00.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.398+0000 7f7d755be640 1 -- 172.21.15.5:0/2015220389 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d700ca7e0 con 0x7f7d7006d4f0 2024-03-24T10:49:00.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.400+0000 7f7d755be640 1 --2- 172.21.15.5:0/2015220389 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d7006d4f0 0x7f7d700cc560 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7d700cd600 tx=0x7f7d6002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=522694aa7d3b93a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:00.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.404+0000 7f7d74dbd640 1 -- 172.21.15.5:0/2015220389 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6002f850 con 0x7f7d7006d4f0 2024-03-24T10:49:00.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.405+0000 7f7d74dbd640 1 -- 172.21.15.5:0/2015220389 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d6002f9b0 con 0x7f7d7006d4f0 2024-03-24T10:49:00.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.405+0000 7f7d74dbd640 1 -- 172.21.15.5:0/2015220389 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6002fd40 con 0x7f7d7006d4f0 2024-03-24T10:49:00.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.405+0000 7f7d78023640 1 -- 172.21.15.5:0/2015220389 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d7006d4f0 msgr2=0x7f7d700cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:00.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.405+0000 7f7d78023640 1 --2- 172.21.15.5:0/2015220389 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d7006d4f0 0x7f7d700cc560 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7d700cd600 tx=0x7f7d6002ba30 comp rx=0 tx=0).stop 2024-03-24T10:49:00.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.405+0000 7f7d78023640 1 -- 172.21.15.5:0/2015220389 shutdown_connections 2024-03-24T10:49:00.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.405+0000 7f7d78023640 1 --2- 172.21.15.5:0/2015220389 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d7006d4f0 0x7f7d700cc560 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:00.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.405+0000 7f7d78023640 1 --2- 172.21.15.5:0/2015220389 >> v2:172.21.15.57:3300/0 conn(0x7f7d701543a0 0x7f7d7015ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:00.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.405+0000 7f7d78023640 1 -- 172.21.15.5:0/2015220389 >> 172.21.15.5:0/2015220389 conn(0x7f7d70072ef0 msgr2=0x7f7d70061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:00.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.406+0000 7f7d78023640 1 -- 172.21.15.5:0/2015220389 shutdown_connections 2024-03-24T10:49:00.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.406+0000 7f7d78023640 1 -- 172.21.15.5:0/2015220389 wait complete. 2024-03-24T10:49:00.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.407+0000 7f7d78023640 1 Processor -- start 2024-03-24T10:49:00.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.407+0000 7f7d78023640 1 -- start start 2024-03-24T10:49:00.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.408+0000 7f7d78023640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7d7006d4f0 0x7f7d700c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:00.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.408+0000 7f7d78023640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d700ccaa0 0x7f7d701f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:00.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.408+0000 7f7d78023640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d701543a0 0x7f7d70159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:00.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.408+0000 7f7d78023640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f7d701f40f0 con 0x7f7d700ccaa0 2024-03-24T10:49:00.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.408+0000 7f7d78023640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7d70151690 con 0x7f7d701543a0 2024-03-24T10:49:00.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.408+0000 7f7d78023640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7d70150a90 con 0x7f7d7006d4f0 2024-03-24T10:49:00.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.408+0000 7f7d765c0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d701543a0 0x7f7d70159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:00.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.408+0000 7f7d755be640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d700ccaa0 0x7f7d701f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:00.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.408+0000 7f7d75dbf640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7d7006d4f0 0x7f7d700c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:00.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.409+0000 7f7d755be640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d700ccaa0 0x7f7d701f3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38618/0 (socket says 172.21.15.5:38618) 2024-03-24T10:49:00.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.409+0000 7f7d755be640 1 -- 172.21.15.5:0/3056718359 learned_addr learned my addr 172.21.15.5:0/3056718359 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:00.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.412+0000 7f7d755be640 1 -- 172.21.15.5:0/3056718359 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7d7006d4f0 msgr2=0x7f7d700c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:00.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.412+0000 7f7d755be640 1 --2- 172.21.15.5:0/3056718359 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7d7006d4f0 0x7f7d700c9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:00.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.412+0000 7f7d755be640 1 -- 172.21.15.5:0/3056718359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d701543a0 msgr2=0x7f7d70159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:00.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.412+0000 7f7d755be640 1 --2- 172.21.15.5:0/3056718359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d701543a0 0x7f7d70159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:00.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.412+0000 7f7d755be640 1 -- 172.21.15.5:0/3056718359 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d600094d0 con 0x7f7d700ccaa0 2024-03-24T10:49:00.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.415+0000 7f7d755be640 1 --2- 172.21.15.5:0/3056718359 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d700ccaa0 0x7f7d701f3bb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7d60000c00 tx=0x7f7d600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:00.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.419+0000 7f7d667fc640 1 -- 172.21.15.5:0/3056718359 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6003b040 con 0x7f7d700ccaa0 2024-03-24T10:49:00.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.419+0000 7f7d667fc640 1 -- 172.21.15.5:0/3056718359 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d60037c60 con 0x7f7d700ccaa0 2024-03-24T10:49:00.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.419+0000 7f7d78023640 1 -- 172.21.15.5:0/3056718359 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d70150d30 con 0x7f7d700ccaa0 2024-03-24T10:49:00.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.419+0000 7f7d667fc640 1 -- 172.21.15.5:0/3056718359 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d60042480 con 0x7f7d700ccaa0 2024-03-24T10:49:00.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.419+0000 7f7d78023640 1 -- 172.21.15.5:0/3056718359 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d701efdd0 con 0x7f7d700ccaa0 2024-03-24T10:49:00.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.420+0000 7f7d78023640 1 -- 172.21.15.5:0/3056718359 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d700c9ff0 con 0x7f7d700ccaa0 2024-03-24T10:49:00.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.692+0000 7f7d667fc640 1 -- 172.21.15.5:0/3056718359 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7d600425e0 con 0x7f7d700ccaa0 2024-03-24T10:49:00.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.692+0000 7f7d667fc640 1 --2- 172.21.15.5:0/3056718359 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7d5406b860 0x7f7d5406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:00.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.693+0000 7f7d75dbf640 1 --2- 172.21.15.5:0/3056718359 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7d5406b860 0x7f7d5406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:00.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.693+0000 7f7d75dbf640 1 --2- 172.21.15.5:0/3056718359 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7d5406b860 0x7f7d5406dd20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c009aa0 tx=0x7f7d6c009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:00.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.707+0000 7f7d667fc640 1 -- 172.21.15.5:0/3056718359 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f7d6007c4a0 con 0x7f7d700ccaa0 2024-03-24T10:49:00.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.707+0000 7f7d667fc640 1 -- 172.21.15.5:0/3056718359 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7d60078740 con 0x7f7d700ccaa0 2024-03-24T10:49:00.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.932+0000 7f7d78023640 1 -- 172.21.15.5:0/3056718359 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d701efc30 con 0x7f7d700ccaa0 2024-03-24T10:49:00.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.944+0000 7f7d667fc640 1 -- 172.21.15.5:0/3056718359 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f7d701efc30 con 0x7f7d700ccaa0 2024-03-24T10:49:00.945 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:00.945 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:00.945 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:00.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.947+0000 7f7d78023640 1 -- 172.21.15.5:0/3056718359 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7d5406b860 msgr2=0x7f7d5406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:00.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.947+0000 7f7d78023640 1 --2- 172.21.15.5:0/3056718359 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7d5406b860 0x7f7d5406dd20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c009aa0 tx=0x7f7d6c009190 comp rx=0 tx=0).stop 2024-03-24T10:49:00.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.947+0000 7f7d78023640 1 -- 172.21.15.5:0/3056718359 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d700ccaa0 msgr2=0x7f7d701f3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:00.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.947+0000 7f7d78023640 1 --2- 172.21.15.5:0/3056718359 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d700ccaa0 0x7f7d701f3bb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7d60000c00 tx=0x7f7d600046c0 comp rx=0 tx=0).stop 2024-03-24T10:49:00.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.947+0000 7f7d78023640 1 -- 172.21.15.5:0/3056718359 shutdown_connections 2024-03-24T10:49:00.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.947+0000 7f7d78023640 1 --2- 172.21.15.5:0/3056718359 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7d5406b860 0x7f7d5406dd20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:00.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.947+0000 7f7d78023640 1 --2- 172.21.15.5:0/3056718359 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7d700ccaa0 0x7f7d701f3bb0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:00.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.947+0000 7f7d78023640 1 --2- 172.21.15.5:0/3056718359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d701543a0 0x7f7d70159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:00.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.947+0000 7f7d78023640 1 --2- 172.21.15.5:0/3056718359 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7d7006d4f0 0x7f7d700c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:00.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.947+0000 7f7d78023640 1 -- 172.21.15.5:0/3056718359 >> 172.21.15.5:0/3056718359 conn(0x7f7d70072ef0 msgr2=0x7f7d7006be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:00.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.947+0000 7f7d78023640 1 -- 172.21.15.5:0/3056718359 shutdown_connections 2024-03-24T10:49:00.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:00.947+0000 7f7d78023640 1 -- 172.21.15.5:0/3056718359 wait complete. 2024-03-24T10:49:01.563 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:01.563 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:01.649 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:49:01.652 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:49:02.564 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:02.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.666+0000 7f1457c64640 1 Processor -- start 2024-03-24T10:49:02.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.666+0000 7f1457c64640 1 -- start start 2024-03-24T10:49:02.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.667+0000 7f1457c64640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f14500c4300 0x7f14500cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:02.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.667+0000 7f1457c64640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f14500cdfb0 0x7f145006c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:02.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.667+0000 7f1457c64640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14500c4720 con 0x7f1450154220 2024-03-24T10:49:02.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.667+0000 7f1457c64640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f14500c8bf0 con 0x7f14500cdfb0 2024-03-24T10:49:02.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.667+0000 7f1457c64640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f14500c9050 con 0x7f14500c4300 2024-03-24T10:49:02.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.668+0000 7f14551ff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f14500c4300 0x7f14500cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:02.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.668+0000 7f1455a00640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1450154220 0x7f14500c84e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:38764/0 (socket says 172.21.15.5:38764) 2024-03-24T10:49:02.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.668+0000 7f1455a00640 1 -- 172.21.15.5:0/281082331 learned_addr learned my addr 172.21.15.5:0/281082331 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:49:02.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.668+0000 7f1456201640 1 --2- 172.21.15.5:0/281082331 >> v2:172.21.15.57:3300/0 conn(0x7f14500cdfb0 0x7f145006c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:02.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.671+0000 7f14551ff640 1 -- 172.21.15.5:0/281082331 >> v2:172.21.15.57:3300/0 conn(0x7f14500cdfb0 msgr2=0x7f145006c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:02.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.671+0000 7f14551ff640 1 --2- 172.21.15.5:0/281082331 >> v2:172.21.15.57:3300/0 conn(0x7f14500cdfb0 0x7f145006c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:02.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.671+0000 7f14551ff640 1 -- 172.21.15.5:0/281082331 >> v1:172.21.15.57:6789/0 conn(0x7f1450154220 legacy=0x7f14500c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:02.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.671+0000 7f14551ff640 1 -- 172.21.15.5:0/281082331 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145015a5f0 con 0x7f14500c4300 2024-03-24T10:49:02.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.673+0000 7f14551ff640 1 --2- 172.21.15.5:0/281082331 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f14500c4300 0x7f14500cda70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f14400097a0 tx=0x7f144002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24b50d86424223af server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:02.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.676+0000 7f14549fe640 1 -- 172.21.15.5:0/281082331 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1440030610 con 0x7f14500c4300 2024-03-24T10:49:02.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.680+0000 7f14549fe640 1 -- 172.21.15.5:0/281082331 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1440030770 con 0x7f14500c4300 2024-03-24T10:49:02.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.680+0000 7f14549fe640 1 -- 172.21.15.5:0/281082331 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1440030b00 con 0x7f14500c4300 2024-03-24T10:49:02.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.681+0000 7f1457c64640 1 -- 172.21.15.5:0/281082331 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f14500c4300 msgr2=0x7f14500cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:02.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.681+0000 7f1457c64640 1 --2- 172.21.15.5:0/281082331 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f14500c4300 0x7f14500cda70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f14400097a0 tx=0x7f144002b960 comp rx=0 tx=0).stop 2024-03-24T10:49:02.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.681+0000 7f1457c64640 1 -- 172.21.15.5:0/281082331 shutdown_connections 2024-03-24T10:49:02.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.681+0000 7f1457c64640 1 --2- 172.21.15.5:0/281082331 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f14500c4300 0x7f14500cda70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:02.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.681+0000 7f1457c64640 1 --2- 172.21.15.5:0/281082331 >> v2:172.21.15.57:3300/0 conn(0x7f14500cdfb0 0x7f145006c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:02.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.681+0000 7f1457c64640 1 -- 172.21.15.5:0/281082331 >> 172.21.15.5:0/281082331 conn(0x7f145006f830 msgr2=0x7f145006fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:02.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.681+0000 7f1457c64640 1 -- 172.21.15.5:0/281082331 shutdown_connections 2024-03-24T10:49:02.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.681+0000 7f1457c64640 1 -- 172.21.15.5:0/281082331 wait complete. 2024-03-24T10:49:02.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.682+0000 7f1457c64640 1 Processor -- start 2024-03-24T10:49:02.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.682+0000 7f1457c64640 1 -- start start 2024-03-24T10:49:02.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.683+0000 7f1457c64640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f14500c4300 0x7f14500ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:02.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.683+0000 7f1457c64640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14500cdfb0 0x7f145015b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:02.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.683+0000 7f1457c64640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1450154220 0x7f145014d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:02.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.683+0000 7f1457c64640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f14500c7f80 con 0x7f1450154220 2024-03-24T10:49:02.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.683+0000 7f1457c64640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f14501f0600 con 0x7f14500cdfb0 2024-03-24T10:49:02.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.683+0000 7f1457c64640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f14500ada50 con 0x7f14500c4300 2024-03-24T10:49:02.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.683+0000 7f14551ff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14500cdfb0 0x7f145015b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:02.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.683+0000 7f1455a00640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f14500c4300 0x7f14500ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:02.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.684+0000 7f14551ff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14500cdfb0 0x7f145015b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:55010/0 (socket says 172.21.15.5:55010) 2024-03-24T10:49:02.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.684+0000 7f14551ff640 1 -- 172.21.15.5:0/1503477930 learned_addr learned my addr 172.21.15.5:0/1503477930 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:02.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.687+0000 7f1455a00640 1 -- 172.21.15.5:0/1503477930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14500cdfb0 msgr2=0x7f145015b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:02.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.687+0000 7f1455a00640 1 --2- 172.21.15.5:0/1503477930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14500cdfb0 0x7f145015b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:02.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.687+0000 7f1455a00640 1 -- 172.21.15.5:0/1503477930 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1450154220 msgr2=0x7f145014d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:02.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.687+0000 7f1455a00640 1 --2- 172.21.15.5:0/1503477930 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1450154220 0x7f145014d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:02.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.687+0000 7f1455a00640 1 -- 172.21.15.5:0/1503477930 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1440009400 con 0x7f14500c4300 2024-03-24T10:49:02.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.690+0000 7f1455a00640 1 --2- 172.21.15.5:0/1503477930 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f14500c4300 0x7f14500ca2b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f14500c48b0 tx=0x7f144c00c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:02.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.696+0000 7f14467fc640 1 -- 172.21.15.5:0/1503477930 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144c005ce0 con 0x7f14500c4300 2024-03-24T10:49:02.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.696+0000 7f14467fc640 1 -- 172.21.15.5:0/1503477930 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f144c005e60 con 0x7f14500c4300 2024-03-24T10:49:02.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.696+0000 7f1457c64640 1 -- 172.21.15.5:0/1503477930 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14501f0870 con 0x7f14500c4300 2024-03-24T10:49:02.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.696+0000 7f14467fc640 1 -- 172.21.15.5:0/1503477930 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144c007310 con 0x7f14500c4300 2024-03-24T10:49:02.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.696+0000 7f1457c64640 1 -- 172.21.15.5:0/1503477930 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f14501f0e30 con 0x7f14500c4300 2024-03-24T10:49:02.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.698+0000 7f1457c64640 1 -- 172.21.15.5:0/1503477930 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f145006df50 con 0x7f14500c4300 2024-03-24T10:49:02.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.995+0000 7f14467fc640 1 -- 172.21.15.5:0/1503477930 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f144c007470 con 0x7f14500c4300 2024-03-24T10:49:02.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.996+0000 7f14467fc640 1 --2- 172.21.15.5:0/1503477930 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f143406b810 0x7f143406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:02.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.996+0000 7f14551ff640 1 --2- 172.21.15.5:0/1503477930 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f143406b810 0x7f143406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:02.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:02.996+0000 7f14551ff640 1 --2- 172.21.15.5:0/1503477930 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f143406b810 0x7f143406dcd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f145015cf80 tx=0x7f1440008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:03.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:03.000+0000 7f14467fc640 1 -- 172.21.15.5:0/1503477930 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f144c007720 con 0x7f14500c4300 2024-03-24T10:49:03.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:03.011+0000 7f14467fc640 1 -- 172.21.15.5:0/1503477930 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f144c054490 con 0x7f14500c4300 2024-03-24T10:49:03.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:03.231+0000 7f1457c64640 1 -- 172.21.15.5:0/1503477930 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f145015a2c0 con 0x7f14500c4300 2024-03-24T10:49:03.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:03.252+0000 7f14467fc640 1 -- 172.21.15.5:0/1503477930 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f144c017030 con 0x7f14500c4300 2024-03-24T10:49:03.254 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:03.254 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:03.254 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:03.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:03.255+0000 7f1457c64640 1 -- 172.21.15.5:0/1503477930 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f143406b810 msgr2=0x7f143406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:03.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:03.255+0000 7f1457c64640 1 --2- 172.21.15.5:0/1503477930 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f143406b810 0x7f143406dcd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f145015cf80 tx=0x7f1440008030 comp rx=0 tx=0).stop 2024-03-24T10:49:03.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:03.255+0000 7f1457c64640 1 -- 172.21.15.5:0/1503477930 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f14500c4300 msgr2=0x7f14500ca2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:03.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:03.255+0000 7f1457c64640 1 --2- 172.21.15.5:0/1503477930 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f14500c4300 0x7f14500ca2b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f14500c48b0 tx=0x7f144c00c760 comp rx=0 tx=0).stop 2024-03-24T10:49:03.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:03.255+0000 7f1457c64640 1 -- 172.21.15.5:0/1503477930 shutdown_connections 2024-03-24T10:49:03.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:03.255+0000 7f1457c64640 1 --2- 172.21.15.5:0/1503477930 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f143406b810 0x7f143406dcd0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:03.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:03.255+0000 7f1457c64640 1 --2- 172.21.15.5:0/1503477930 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1450154220 0x7f145014d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:03.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:03.255+0000 7f1457c64640 1 --2- 172.21.15.5:0/1503477930 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14500cdfb0 0x7f145015b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:03.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:03.255+0000 7f1457c64640 1 --2- 172.21.15.5:0/1503477930 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f14500c4300 0x7f14500ca2b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:03.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:03.255+0000 7f1457c64640 1 -- 172.21.15.5:0/1503477930 >> 172.21.15.5:0/1503477930 conn(0x7f145006f830 msgr2=0x7f1450152e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:03.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:03.255+0000 7f1457c64640 1 -- 172.21.15.5:0/1503477930 shutdown_connections 2024-03-24T10:49:03.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:03.255+0000 7f1457c64640 1 -- 172.21.15.5:0/1503477930 wait complete. 2024-03-24T10:49:03.871 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:03.872 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:04.873 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:04.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.974+0000 7f2577dc3640 1 Processor -- start 2024-03-24T10:49:04.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.974+0000 7f2577dc3640 1 -- start start 2024-03-24T10:49:04.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.975+0000 7f2577dc3640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f25700cb270 0x7f25700cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:04.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.975+0000 7f2577dc3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f25700cd960 0x7f25700cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:04.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.975+0000 7f2577dc3640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25700ccba0 con 0x7f25700cae90 2024-03-24T10:49:04.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.975+0000 7f2577dc3640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f25700cc9b0 con 0x7f25700cd960 2024-03-24T10:49:04.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.975+0000 7f2577dc3640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f25700cce40 con 0x7f25700cb270 2024-03-24T10:49:04.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.976+0000 7f2575b5f640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f25700cae90 0x7f25700cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:52950/0 (socket says 172.21.15.5:52950) 2024-03-24T10:49:04.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.976+0000 7f2575b5f640 1 -- 172.21.15.5:0/1897563227 learned_addr learned my addr 172.21.15.5:0/1897563227 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:49:04.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.976+0000 7f2576360640 1 --2- 172.21.15.5:0/1897563227 >> v2:172.21.15.57:3300/0 conn(0x7f25700cd960 0x7f25700cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:04.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.977+0000 7f257535e640 1 --2- 172.21.15.5:0/1897563227 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f25700cb270 0x7f25700cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:04.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.978+0000 7f2574b5d640 1 -- 172.21.15.5:0/1897563227 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2735771733 0 0) 0x7f25700ccba0 con 0x7f25700cae90 2024-03-24T10:49:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.979+0000 7f2574b5d640 1 -- 172.21.15.5:0/1897563227 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f255c003520 con 0x7f25700cae90 2024-03-24T10:49:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.979+0000 7f2576360640 1 -- 172.21.15.5:0/1897563227 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f25700cb270 msgr2=0x7f25700cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.979+0000 7f2576360640 1 --2- 172.21.15.5:0/1897563227 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f25700cb270 0x7f25700cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.979+0000 7f2576360640 1 -- 172.21.15.5:0/1897563227 >> v1:172.21.15.5:6789/0 conn(0x7f25700cae90 legacy=0x7f25700cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:04.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.979+0000 7f2576360640 1 -- 172.21.15.5:0/1897563227 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f257015a590 con 0x7f25700cd960 2024-03-24T10:49:04.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.981+0000 7f2576360640 1 --2- 172.21.15.5:0/1897563227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25700cd960 0x7f25700cb6f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2568009d90 tx=0x7f256802bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6630c50b7eef88e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:04.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.987+0000 7f2574b5d640 1 -- 172.21.15.5:0/1897563227 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256802cce0 con 0x7f25700cd960 2024-03-24T10:49:04.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.987+0000 7f2574b5d640 1 -- 172.21.15.5:0/1897563227 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f256802ce40 con 0x7f25700cd960 2024-03-24T10:49:04.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.987+0000 7f2574b5d640 1 -- 172.21.15.5:0/1897563227 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2568034660 con 0x7f25700cd960 2024-03-24T10:49:04.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.987+0000 7f2577dc3640 1 -- 172.21.15.5:0/1897563227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25700cd960 msgr2=0x7f25700cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:04.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.987+0000 7f2577dc3640 1 --2- 172.21.15.5:0/1897563227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25700cd960 0x7f25700cb6f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2568009d90 tx=0x7f256802bf30 comp rx=0 tx=0).stop 2024-03-24T10:49:04.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.987+0000 7f2577dc3640 1 -- 172.21.15.5:0/1897563227 shutdown_connections 2024-03-24T10:49:04.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.987+0000 7f2577dc3640 1 --2- 172.21.15.5:0/1897563227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25700cd960 0x7f25700cb6f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:04.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.987+0000 7f2577dc3640 1 --2- 172.21.15.5:0/1897563227 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f25700cb270 0x7f25700cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:04.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.987+0000 7f2577dc3640 1 -- 172.21.15.5:0/1897563227 >> 172.21.15.5:0/1897563227 conn(0x7f25700cfb80 msgr2=0x7f25700cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:04.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.987+0000 7f2577dc3640 1 -- 172.21.15.5:0/1897563227 shutdown_connections 2024-03-24T10:49:04.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.987+0000 7f2577dc3640 1 -- 172.21.15.5:0/1897563227 wait complete. 2024-03-24T10:49:04.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.989+0000 7f2577dc3640 1 Processor -- start 2024-03-24T10:49:04.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.989+0000 7f2577dc3640 1 -- start start 2024-03-24T10:49:04.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.989+0000 7f2577dc3640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25700cae90 0x7f2570159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:04.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.989+0000 7f2577dc3640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f25700cb270 0x7f25701efa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:04.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.989+0000 7f2577dc3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25700cd960 0x7f25701eff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:04.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.989+0000 7f2577dc3640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f257015e7a0 con 0x7f25700cae90 2024-03-24T10:49:04.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.989+0000 7f2577dc3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f25700b5770 con 0x7f25700cd960 2024-03-24T10:49:04.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.989+0000 7f2577dc3640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f257015e260 con 0x7f25700cb270 2024-03-24T10:49:04.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.990+0000 7f2575b5f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25700cae90 0x7f2570159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:04.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.990+0000 7f257535e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f25700cb270 0x7f25701efa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:04.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.990+0000 7f2575b5f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25700cae90 0x7f2570159c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:58850/0 (socket says 172.21.15.5:58850) 2024-03-24T10:49:04.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.990+0000 7f2575b5f640 1 -- 172.21.15.5:0/2649017118 learned_addr learned my addr 172.21.15.5:0/2649017118 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:04.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.991+0000 7f2576360640 1 --2- 172.21.15.5:0/2649017118 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25700cd960 0x7f25701eff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:04.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.993+0000 7f2575b5f640 1 -- 172.21.15.5:0/2649017118 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f25700cb270 msgr2=0x7f25701efa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:04.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.993+0000 7f2575b5f640 1 --2- 172.21.15.5:0/2649017118 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f25700cb270 0x7f25701efa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:04.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.993+0000 7f2575b5f640 1 -- 172.21.15.5:0/2649017118 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25700cd960 msgr2=0x7f25701eff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:04.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.993+0000 7f2575b5f640 1 --2- 172.21.15.5:0/2649017118 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25700cd960 0x7f25701eff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:04.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.993+0000 7f2575b5f640 1 -- 172.21.15.5:0/2649017118 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2568009a70 con 0x7f25700cae90 2024-03-24T10:49:04.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.996+0000 7f2575b5f640 1 --2- 172.21.15.5:0/2649017118 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25700cae90 0x7f2570159c10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f256c00c430 tx=0x7f256c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:05.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.999+0000 7f25667fc640 1 -- 172.21.15.5:0/2649017118 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c005ce0 con 0x7f25700cae90 2024-03-24T10:49:05.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.999+0000 7f25667fc640 1 -- 172.21.15.5:0/2649017118 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f256c005e40 con 0x7f25700cae90 2024-03-24T10:49:05.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.999+0000 7f2577dc3640 1 -- 172.21.15.5:0/2649017118 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f257015e500 con 0x7f25700cae90 2024-03-24T10:49:05.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:04.999+0000 7f25667fc640 1 -- 172.21.15.5:0/2649017118 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256c0072f0 con 0x7f25700cae90 2024-03-24T10:49:05.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.000+0000 7f2577dc3640 1 -- 172.21.15.5:0/2649017118 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25700b5140 con 0x7f25700cae90 2024-03-24T10:49:05.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.001+0000 7f2577dc3640 1 -- 172.21.15.5:0/2649017118 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25700681d0 con 0x7f25700cae90 2024-03-24T10:49:05.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.031+0000 7f25667fc640 1 -- 172.21.15.5:0/2649017118 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f256c007450 con 0x7f25700cae90 2024-03-24T10:49:05.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.031+0000 7f25667fc640 1 --2- 172.21.15.5:0/2649017118 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f255406b860 0x7f255406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:05.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.031+0000 7f257535e640 1 --2- 172.21.15.5:0/2649017118 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f255406b860 0x7f255406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:05.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.031+0000 7f257535e640 1 --2- 172.21.15.5:0/2649017118 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f255406b860 0x7f255406dd20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2570153ae0 tx=0x7f2560009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:05.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.038+0000 7f25667fc640 1 -- 172.21.15.5:0/2649017118 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f256c058460 con 0x7f25700cae90 2024-03-24T10:49:05.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.306+0000 7f25667fc640 1 -- 172.21.15.5:0/2649017118 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f256c054700 con 0x7f25700cae90 2024-03-24T10:49:05.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.561+0000 7f2577dc3640 1 -- 172.21.15.5:0/2649017118 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25700d0240 con 0x7f25700cae90 2024-03-24T10:49:05.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.579+0000 7f25667fc640 1 -- 172.21.15.5:0/2649017118 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f256c017070 con 0x7f25700cae90 2024-03-24T10:49:05.582 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:05.582 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:05.584 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:05.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.581+0000 7f2577dc3640 1 -- 172.21.15.5:0/2649017118 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f255406b860 msgr2=0x7f255406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:05.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.581+0000 7f2577dc3640 1 --2- 172.21.15.5:0/2649017118 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f255406b860 0x7f255406dd20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2570153ae0 tx=0x7f2560009190 comp rx=0 tx=0).stop 2024-03-24T10:49:05.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.581+0000 7f2577dc3640 1 -- 172.21.15.5:0/2649017118 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25700cae90 msgr2=0x7f2570159c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:05.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.581+0000 7f2577dc3640 1 --2- 172.21.15.5:0/2649017118 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25700cae90 0x7f2570159c10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f256c00c430 tx=0x7f256c00c8f0 comp rx=0 tx=0).stop 2024-03-24T10:49:05.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.581+0000 7f2577dc3640 1 -- 172.21.15.5:0/2649017118 shutdown_connections 2024-03-24T10:49:05.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.581+0000 7f2577dc3640 1 --2- 172.21.15.5:0/2649017118 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f255406b860 0x7f255406dd20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:05.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.581+0000 7f2577dc3640 1 --2- 172.21.15.5:0/2649017118 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f25700cd960 0x7f25701eff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:05.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.581+0000 7f2577dc3640 1 --2- 172.21.15.5:0/2649017118 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f25700cb270 0x7f25701efa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:05.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.581+0000 7f2577dc3640 1 --2- 172.21.15.5:0/2649017118 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25700cae90 0x7f2570159c10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:05.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.581+0000 7f2577dc3640 1 -- 172.21.15.5:0/2649017118 >> 172.21.15.5:0/2649017118 conn(0x7f25700cfb80 msgr2=0x7f257006bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:05.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.581+0000 7f2577dc3640 1 -- 172.21.15.5:0/2649017118 shutdown_connections 2024-03-24T10:49:05.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:05.581+0000 7f2577dc3640 1 -- 172.21.15.5:0/2649017118 wait complete. 2024-03-24T10:49:06.197 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:06.198 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:07.198 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:07.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.299+0000 7f1da6925640 1 Processor -- start 2024-03-24T10:49:07.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.299+0000 7f1da6925640 1 -- start start 2024-03-24T10:49:07.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.300+0000 7f1da6925640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1da00cb270 0x7f1da00cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:07.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.300+0000 7f1da6925640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1da00cd960 0x7f1da00cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:07.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.300+0000 7f1da6925640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1da00ccba0 con 0x7f1da00cae90 2024-03-24T10:49:07.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.300+0000 7f1da6925640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f1da00cc9b0 con 0x7f1da00cd960 2024-03-24T10:49:07.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.300+0000 7f1da6925640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1da00cce40 con 0x7f1da00cb270 2024-03-24T10:49:07.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.301+0000 7f1d9f7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1da00cb270 0x7f1da00cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:07.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.301+0000 7f1da4ec2640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1da00cd960 0x7f1da00cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:07.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.301+0000 7f1d9ffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1da00cae90 0x7f1da00cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:38778/0 (socket says 172.21.15.5:38778) 2024-03-24T10:49:07.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.301+0000 7f1d9ffff640 1 -- 172.21.15.5:0/2780367547 learned_addr learned my addr 172.21.15.5:0/2780367547 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:49:07.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.304+0000 7f1da4ec2640 1 -- 172.21.15.5:0/2780367547 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1da00cb270 msgr2=0x7f1da00cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:07.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.304+0000 7f1da4ec2640 1 --2- 172.21.15.5:0/2780367547 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1da00cb270 0x7f1da00cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:07.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.304+0000 7f1da4ec2640 1 -- 172.21.15.5:0/2780367547 >> v1:172.21.15.57:6789/0 conn(0x7f1da00cae90 legacy=0x7f1da00cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:07.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.304+0000 7f1da4ec2640 1 -- 172.21.15.5:0/2780367547 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da015a590 con 0x7f1da00cd960 2024-03-24T10:49:07.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.310+0000 7f1da4ec2640 1 --2- 172.21.15.5:0/2780367547 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1da00cd960 0x7f1da00cb6f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1d98009d90 tx=0x7f1d9802bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87e19b6b39c104da server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:07.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.316+0000 7f1d9effd640 1 -- 172.21.15.5:0/2780367547 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9802cce0 con 0x7f1da00cd960 2024-03-24T10:49:07.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.316+0000 7f1d9effd640 1 -- 172.21.15.5:0/2780367547 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d9802ce40 con 0x7f1da00cd960 2024-03-24T10:49:07.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.316+0000 7f1d9effd640 1 -- 172.21.15.5:0/2780367547 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d98034660 con 0x7f1da00cd960 2024-03-24T10:49:07.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.317+0000 7f1da6925640 1 -- 172.21.15.5:0/2780367547 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1da00cd960 msgr2=0x7f1da00cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:07.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.317+0000 7f1da6925640 1 --2- 172.21.15.5:0/2780367547 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1da00cd960 0x7f1da00cb6f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f1d98009d90 tx=0x7f1d9802bf30 comp rx=0 tx=0).stop 2024-03-24T10:49:07.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.317+0000 7f1da6925640 1 -- 172.21.15.5:0/2780367547 shutdown_connections 2024-03-24T10:49:07.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.317+0000 7f1da6925640 1 --2- 172.21.15.5:0/2780367547 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1da00cb270 0x7f1da00cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:07.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.317+0000 7f1da6925640 1 --2- 172.21.15.5:0/2780367547 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1da00cd960 0x7f1da00cb6f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:07.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.317+0000 7f1da6925640 1 -- 172.21.15.5:0/2780367547 >> 172.21.15.5:0/2780367547 conn(0x7f1da00cfb80 msgr2=0x7f1da00cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:07.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.317+0000 7f1da6925640 1 -- 172.21.15.5:0/2780367547 shutdown_connections 2024-03-24T10:49:07.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.317+0000 7f1da6925640 1 -- 172.21.15.5:0/2780367547 wait complete. 2024-03-24T10:49:07.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.318+0000 7f1da6925640 1 Processor -- start 2024-03-24T10:49:07.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.318+0000 7f1da6925640 1 -- start start 2024-03-24T10:49:07.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.319+0000 7f1da6925640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1da00cae90 0x7f1da015e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:07.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.319+0000 7f1da6925640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1da00cb270 0x7f1da006bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:07.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.319+0000 7f1da6925640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1da00cd960 0x7f1da01f1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:07.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.319+0000 7f1da6925640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f1da01f2ff0 con 0x7f1da00cb270 2024-03-24T10:49:07.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.319+0000 7f1da6925640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1da01f32d0 con 0x7f1da00cae90 2024-03-24T10:49:07.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.319+0000 7f1da6925640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1da01f35b0 con 0x7f1da00cd960 2024-03-24T10:49:07.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.320+0000 7f1d9f7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1da00cb270 0x7f1da006bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:07.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.320+0000 7f1d9ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1da00cae90 0x7f1da015e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:07.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.320+0000 7f1da4ec2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1da00cd960 0x7f1da01f1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:07.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.320+0000 7f1d9f7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1da00cb270 0x7f1da006bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:58862/0 (socket says 172.21.15.5:58862) 2024-03-24T10:49:07.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.320+0000 7f1d9f7fe640 1 -- 172.21.15.5:0/4042234104 learned_addr learned my addr 172.21.15.5:0/4042234104 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:07.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.324+0000 7f1d9f7fe640 1 -- 172.21.15.5:0/4042234104 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1da00cd960 msgr2=0x7f1da01f1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:07.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.324+0000 7f1d9f7fe640 1 --2- 172.21.15.5:0/4042234104 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1da00cd960 0x7f1da01f1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:07.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.324+0000 7f1d9f7fe640 1 -- 172.21.15.5:0/4042234104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1da00cae90 msgr2=0x7f1da015e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:07.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.324+0000 7f1d9f7fe640 1 --2- 172.21.15.5:0/4042234104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1da00cae90 0x7f1da015e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:07.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.324+0000 7f1d9f7fe640 1 -- 172.21.15.5:0/4042234104 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d98009a70 con 0x7f1da00cb270 2024-03-24T10:49:07.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.327+0000 7f1d9f7fe640 1 --2- 172.21.15.5:0/4042234104 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1da00cb270 0x7f1da006bdc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1d9400e9b0 tx=0x7f1d9400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:07.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.331+0000 7f1d9cff9640 1 -- 172.21.15.5:0/4042234104 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d94009670 con 0x7f1da00cb270 2024-03-24T10:49:07.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.331+0000 7f1d9cff9640 1 -- 172.21.15.5:0/4042234104 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d9400c9a0 con 0x7f1da00cb270 2024-03-24T10:49:07.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.331+0000 7f1da6925640 1 -- 172.21.15.5:0/4042234104 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da01f3890 con 0x7f1da00cb270 2024-03-24T10:49:07.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.331+0000 7f1d9cff9640 1 -- 172.21.15.5:0/4042234104 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9400cd50 con 0x7f1da00cb270 2024-03-24T10:49:07.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.331+0000 7f1da6925640 1 -- 172.21.15.5:0/4042234104 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da00c4660 con 0x7f1da00cb270 2024-03-24T10:49:07.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.332+0000 7f1da6925640 1 -- 172.21.15.5:0/4042234104 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1da00681d0 con 0x7f1da00cb270 2024-03-24T10:49:07.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.608+0000 7f1d9cff9640 1 -- 172.21.15.5:0/4042234104 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1d94013470 con 0x7f1da00cb270 2024-03-24T10:49:07.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.609+0000 7f1d9cff9640 1 --2- 172.21.15.5:0/4042234104 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1d8c06b860 0x7f1d8c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:07.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.609+0000 7f1d9ffff640 1 --2- 172.21.15.5:0/4042234104 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1d8c06b860 0x7f1d8c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:07.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.609+0000 7f1d9ffff640 1 --2- 172.21.15.5:0/4042234104 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1d8c06b860 0x7f1d8c06dd20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1d90007fd0 tx=0x7f1d90007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:07.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.631+0000 7f1d9cff9640 1 -- 172.21.15.5:0/4042234104 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f1d94059430 con 0x7f1da00cb270 2024-03-24T10:49:07.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.631+0000 7f1d9cff9640 1 -- 172.21.15.5:0/4042234104 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1d940556d0 con 0x7f1da00cb270 2024-03-24T10:49:07.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.877+0000 7f1da6925640 1 -- 172.21.15.5:0/4042234104 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1da015a1b0 con 0x7f1da00cb270 2024-03-24T10:49:07.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.941+0000 7f1d9cff9640 1 -- 172.21.15.5:0/4042234104 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f1d94017090 con 0x7f1da00cb270 2024-03-24T10:49:07.942 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:07.942 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:07.942 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:07.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.943+0000 7f1da6925640 1 -- 172.21.15.5:0/4042234104 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1d8c06b860 msgr2=0x7f1d8c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:07.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.943+0000 7f1da6925640 1 --2- 172.21.15.5:0/4042234104 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1d8c06b860 0x7f1d8c06dd20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1d90007fd0 tx=0x7f1d90007e80 comp rx=0 tx=0).stop 2024-03-24T10:49:07.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.943+0000 7f1da6925640 1 -- 172.21.15.5:0/4042234104 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1da00cb270 msgr2=0x7f1da006bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:07.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.943+0000 7f1da6925640 1 --2- 172.21.15.5:0/4042234104 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1da00cb270 0x7f1da006bdc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1d9400e9b0 tx=0x7f1d9400ee70 comp rx=0 tx=0).stop 2024-03-24T10:49:07.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.943+0000 7f1da6925640 1 -- 172.21.15.5:0/4042234104 shutdown_connections 2024-03-24T10:49:07.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.943+0000 7f1da6925640 1 --2- 172.21.15.5:0/4042234104 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1d8c06b860 0x7f1d8c06dd20 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:07.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.943+0000 7f1da6925640 1 --2- 172.21.15.5:0/4042234104 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1da00cb270 0x7f1da006bdc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:07.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.943+0000 7f1da6925640 1 --2- 172.21.15.5:0/4042234104 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1da00cd960 0x7f1da01f1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:07.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.943+0000 7f1da6925640 1 --2- 172.21.15.5:0/4042234104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1da00cae90 0x7f1da015e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:07.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.943+0000 7f1da6925640 1 -- 172.21.15.5:0/4042234104 >> 172.21.15.5:0/4042234104 conn(0x7f1da00cfb80 msgr2=0x7f1da00b5350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:07.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.943+0000 7f1da6925640 1 -- 172.21.15.5:0/4042234104 shutdown_connections 2024-03-24T10:49:07.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:07.943+0000 7f1da6925640 1 -- 172.21.15.5:0/4042234104 wait complete. 2024-03-24T10:49:08.563 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:08.563 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:09.564 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:09.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.666+0000 7fe3cc274640 1 Processor -- start 2024-03-24T10:49:09.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.666+0000 7fe3cc274640 1 -- start start 2024-03-24T10:49:09.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.667+0000 7fe3cc274640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3c415ac00 0x7fe3c40cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:09.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.667+0000 7fe3cc274640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe3c40ccaa0 0x7fe3c406bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:09.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.667+0000 7fe3cc274640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3c40cb2f0 con 0x7fe3c41543a0 2024-03-24T10:49:09.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.667+0000 7fe3cc274640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fe3c40c6750 con 0x7fe3c40ccaa0 2024-03-24T10:49:09.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.667+0000 7fe3cc274640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe3c40cc380 con 0x7fe3c415ac00 2024-03-24T10:49:09.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.668+0000 7fe3c980f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3c415ac00 0x7fe3c40cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:09.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.668+0000 7fe3c980f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3c415ac00 0x7fe3c40cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:55590/0 (socket says 172.21.15.5:55590) 2024-03-24T10:49:09.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.668+0000 7fe3c980f640 1 -- 172.21.15.5:0/3151174284 learned_addr learned my addr 172.21.15.5:0/3151174284 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:09.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.672+0000 7fe3c980f640 1 -- 172.21.15.5:0/3151174284 >> v2:172.21.15.5:3300/0 conn(0x7fe3c40ccaa0 msgr2=0x7fe3c406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:09.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.672+0000 7fe3c980f640 1 --2- 172.21.15.5:0/3151174284 >> v2:172.21.15.5:3300/0 conn(0x7fe3c40ccaa0 0x7fe3c406bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:09.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.672+0000 7fe3c980f640 1 -- 172.21.15.5:0/3151174284 >> v1:172.21.15.5:6789/0 conn(0x7fe3c41543a0 legacy=0x7fe3c406d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:09.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.672+0000 7fe3c980f640 1 -- 172.21.15.5:0/3151174284 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c415b070 con 0x7fe3c415ac00 2024-03-24T10:49:09.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.675+0000 7fe3c980f640 1 --2- 172.21.15.5:0/3151174284 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3c415ac00 0x7fe3c40cc560 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009870 tx=0x7fe3b402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc9d9f26ce0458a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:09.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.677+0000 7fe3c900e640 1 -- 172.21.15.5:0/3151174284 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4031640 con 0x7fe3c415ac00 2024-03-24T10:49:09.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.680+0000 7fe3c900e640 1 -- 172.21.15.5:0/3151174284 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3b40317a0 con 0x7fe3c415ac00 2024-03-24T10:49:09.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.680+0000 7fe3c900e640 1 -- 172.21.15.5:0/3151174284 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4031b30 con 0x7fe3c415ac00 2024-03-24T10:49:09.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.681+0000 7fe3cc274640 1 -- 172.21.15.5:0/3151174284 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3c415ac00 msgr2=0x7fe3c40cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:09.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.681+0000 7fe3cc274640 1 --2- 172.21.15.5:0/3151174284 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3c415ac00 0x7fe3c40cc560 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009870 tx=0x7fe3b402b8f0 comp rx=0 tx=0).stop 2024-03-24T10:49:09.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.681+0000 7fe3cc274640 1 -- 172.21.15.5:0/3151174284 shutdown_connections 2024-03-24T10:49:09.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.681+0000 7fe3cc274640 1 --2- 172.21.15.5:0/3151174284 >> v2:172.21.15.5:3300/0 conn(0x7fe3c40ccaa0 0x7fe3c406bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:09.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.681+0000 7fe3cc274640 1 --2- 172.21.15.5:0/3151174284 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3c415ac00 0x7fe3c40cc560 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:09.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.681+0000 7fe3cc274640 1 -- 172.21.15.5:0/3151174284 >> 172.21.15.5:0/3151174284 conn(0x7fe3c4072ef0 msgr2=0x7fe3c4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:09.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.681+0000 7fe3cc274640 1 -- 172.21.15.5:0/3151174284 shutdown_connections 2024-03-24T10:49:09.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.681+0000 7fe3cc274640 1 -- 172.21.15.5:0/3151174284 wait complete. 2024-03-24T10:49:09.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.682+0000 7fe3cc274640 1 Processor -- start 2024-03-24T10:49:09.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.682+0000 7fe3cc274640 1 -- start start 2024-03-24T10:49:09.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.683+0000 7fe3cc274640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3c40ccaa0 0x7fe3c40b3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:09.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.683+0000 7fe3cc274640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3c41543a0 0x7fe3c41e6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:09.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.683+0000 7fe3cc274640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3c415ac00 0x7fe3c41494f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:09.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.683+0000 7fe3cc274640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c40b4f20 con 0x7fe3c415ac00 2024-03-24T10:49:09.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.683+0000 7fe3cc274640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c40b2790 con 0x7fe3c41543a0 2024-03-24T10:49:09.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.683+0000 7fe3cc274640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe3c40b4960 con 0x7fe3c40ccaa0 2024-03-24T10:49:09.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.683+0000 7fe3ca811640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3c415ac00 0x7fe3c41494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:09.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.684+0000 7fe3ca010640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3c40ccaa0 0x7fe3c40b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:09.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.684+0000 7fe3c980f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3c41543a0 0x7fe3c41e6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:09.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.684+0000 7fe3ca811640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3c415ac00 0x7fe3c41494f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:58882/0 (socket says 172.21.15.5:58882) 2024-03-24T10:49:09.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.684+0000 7fe3ca811640 1 -- 172.21.15.5:0/4053820296 learned_addr learned my addr 172.21.15.5:0/4053820296 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:09.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.687+0000 7fe3ca811640 1 -- 172.21.15.5:0/4053820296 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3c40ccaa0 msgr2=0x7fe3c40b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:09.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.687+0000 7fe3ca811640 1 --2- 172.21.15.5:0/4053820296 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3c40ccaa0 0x7fe3c40b3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:09.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.687+0000 7fe3ca811640 1 -- 172.21.15.5:0/4053820296 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3c41543a0 msgr2=0x7fe3c41e6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:09.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.687+0000 7fe3ca811640 1 --2- 172.21.15.5:0/4053820296 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3c41543a0 0x7fe3c41e6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:09.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.687+0000 7fe3ca811640 1 -- 172.21.15.5:0/4053820296 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3b40094d0 con 0x7fe3c415ac00 2024-03-24T10:49:09.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.690+0000 7fe3ca811640 1 --2- 172.21.15.5:0/4053820296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3c415ac00 0x7fe3c41494f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc00b970 tx=0x7fe3bc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:09.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.692+0000 7fe3baffd640 1 -- 172.21.15.5:0/4053820296 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc00cb40 con 0x7fe3c415ac00 2024-03-24T10:49:09.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.692+0000 7fe3cc274640 1 -- 172.21.15.5:0/4053820296 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c40b4c40 con 0x7fe3c415ac00 2024-03-24T10:49:09.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.692+0000 7fe3cc274640 1 -- 172.21.15.5:0/4053820296 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c40b3450 con 0x7fe3c415ac00 2024-03-24T10:49:09.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.693+0000 7fe3cc274640 1 -- 172.21.15.5:0/4053820296 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3c40b2fd0 con 0x7fe3c415ac00 2024-03-24T10:49:09.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.694+0000 7fe3baffd640 1 -- 172.21.15.5:0/4053820296 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3bc00cca0 con 0x7fe3c415ac00 2024-03-24T10:49:09.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.694+0000 7fe3baffd640 1 -- 172.21.15.5:0/4053820296 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc01e660 con 0x7fe3c415ac00 2024-03-24T10:49:09.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.978+0000 7fe3baffd640 1 -- 172.21.15.5:0/4053820296 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe3bc014440 con 0x7fe3c415ac00 2024-03-24T10:49:09.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.979+0000 7fe3baffd640 1 --2- 172.21.15.5:0/4053820296 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe3a806b810 0x7fe3a806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:09.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.979+0000 7fe3ca010640 1 --2- 172.21.15.5:0/4053820296 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe3a806b810 0x7fe3a806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:09.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.979+0000 7fe3ca010640 1 --2- 172.21.15.5:0/4053820296 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe3a806b810 0x7fe3a806dcd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe3c00017f0 tx=0x7fe3c0009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:09.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.998+0000 7fe3baffd640 1 -- 172.21.15.5:0/4053820296 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fe3bc058f30 con 0x7fe3c415ac00 2024-03-24T10:49:09.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:09.999+0000 7fe3baffd640 1 -- 172.21.15.5:0/4053820296 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe3bc0551d0 con 0x7fe3c415ac00 2024-03-24T10:49:10.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:10.246+0000 7fe3cc274640 1 -- 172.21.15.5:0/4053820296 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3c40b2fd0 con 0x7fe3c415ac00 2024-03-24T10:49:10.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:10.268+0000 7fe3baffd640 1 -- 172.21.15.5:0/4053820296 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fe3c40b2fd0 con 0x7fe3c415ac00 2024-03-24T10:49:10.269 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:10.269 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:10.269 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:10.270+0000 7fe3cc274640 1 -- 172.21.15.5:0/4053820296 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe3a806b810 msgr2=0x7fe3a806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:10.270+0000 7fe3cc274640 1 --2- 172.21.15.5:0/4053820296 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe3a806b810 0x7fe3a806dcd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe3c00017f0 tx=0x7fe3c0009040 comp rx=0 tx=0).stop 2024-03-24T10:49:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:10.270+0000 7fe3cc274640 1 -- 172.21.15.5:0/4053820296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3c415ac00 msgr2=0x7fe3c41494f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:10.270+0000 7fe3cc274640 1 --2- 172.21.15.5:0/4053820296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3c415ac00 0x7fe3c41494f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc00b970 tx=0x7fe3bc00be30 comp rx=0 tx=0).stop 2024-03-24T10:49:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:10.270+0000 7fe3cc274640 1 -- 172.21.15.5:0/4053820296 shutdown_connections 2024-03-24T10:49:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:10.270+0000 7fe3cc274640 1 --2- 172.21.15.5:0/4053820296 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe3a806b810 0x7fe3a806dcd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:10.270+0000 7fe3cc274640 1 --2- 172.21.15.5:0/4053820296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3c415ac00 0x7fe3c41494f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:10.270+0000 7fe3cc274640 1 --2- 172.21.15.5:0/4053820296 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3c41543a0 0x7fe3c41e6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:10.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:10.270+0000 7fe3cc274640 1 --2- 172.21.15.5:0/4053820296 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3c40ccaa0 0x7fe3c40b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:10.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:10.270+0000 7fe3cc274640 1 -- 172.21.15.5:0/4053820296 >> 172.21.15.5:0/4053820296 conn(0x7fe3c4072ef0 msgr2=0x7fe3c4157f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:10.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:10.271+0000 7fe3cc274640 1 -- 172.21.15.5:0/4053820296 shutdown_connections 2024-03-24T10:49:10.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:10.271+0000 7fe3cc274640 1 -- 172.21.15.5:0/4053820296 wait complete. 2024-03-24T10:49:10.886 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:10.886 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:11.888 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:11.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:11.990+0000 7f96299e2640 1 Processor -- start 2024-03-24T10:49:11.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:11.990+0000 7f96299e2640 1 -- start start 2024-03-24T10:49:11.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:11.991+0000 7f96299e2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f962415ac00 0x7f96240cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:11.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:11.991+0000 7f96299e2640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f96240ccaa0 0x7f962406bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:11.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:11.991+0000 7f96299e2640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96240cb2f0 con 0x7f96241543a0 2024-03-24T10:49:11.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:11.991+0000 7f96299e2640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f96240c6750 con 0x7f96240ccaa0 2024-03-24T10:49:11.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:11.991+0000 7f96299e2640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f96240cc380 con 0x7f962415ac00 2024-03-24T10:49:11.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:11.991+0000 7f9622ffd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f96241543a0 0x7f962406d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:52970/0 (socket says 172.21.15.5:52970) 2024-03-24T10:49:11.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:11.991+0000 7f9622ffd640 1 -- 172.21.15.5:0/4227082219 learned_addr learned my addr 172.21.15.5:0/4227082219 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:49:11.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:11.991+0000 7f96237fe640 1 --2- 172.21.15.5:0/4227082219 >> v2:172.21.15.57:3300/0 conn(0x7f96240ccaa0 0x7f962406bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:11.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:11.993+0000 7f96227fc640 1 --2- 172.21.15.5:0/4227082219 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f962415ac00 0x7f96240cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:11.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:11.994+0000 7f9621ffb640 1 -- 172.21.15.5:0/4227082219 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1796996121 0 0) 0x7f96240cb2f0 con 0x7f96241543a0 2024-03-24T10:49:11.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:11.994+0000 7f9621ffb640 1 -- 172.21.15.5:0/4227082219 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f960c003520 con 0x7f96241543a0 2024-03-24T10:49:11.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:11.994+0000 7f96237fe640 1 -- 172.21.15.5:0/4227082219 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f962415ac00 msgr2=0x7f96240cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:11.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:11.994+0000 7f96237fe640 1 --2- 172.21.15.5:0/4227082219 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f962415ac00 0x7f96240cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:11.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:11.994+0000 7f96237fe640 1 -- 172.21.15.5:0/4227082219 >> v1:172.21.15.5:6789/0 conn(0x7f96241543a0 legacy=0x7f962406d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:11.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:11.994+0000 7f96237fe640 1 -- 172.21.15.5:0/4227082219 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f962415b070 con 0x7f96240ccaa0 2024-03-24T10:49:11.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:11.999+0000 7f96237fe640 1 --2- 172.21.15.5:0/4227082219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96240ccaa0 0x7f962406bf50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f961c009d90 tx=0x7f961c02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=408e455b76a0701 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:12.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.004+0000 7f9621ffb640 1 -- 172.21.15.5:0/4227082219 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c02cc50 con 0x7f96240ccaa0 2024-03-24T10:49:12.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.004+0000 7f9621ffb640 1 -- 172.21.15.5:0/4227082219 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f961c02cdb0 con 0x7f96240ccaa0 2024-03-24T10:49:12.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.004+0000 7f9621ffb640 1 -- 172.21.15.5:0/4227082219 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f961c034660 con 0x7f96240ccaa0 2024-03-24T10:49:12.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.005+0000 7f96299e2640 1 -- 172.21.15.5:0/4227082219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96240ccaa0 msgr2=0x7f962406bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:12.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.005+0000 7f96299e2640 1 --2- 172.21.15.5:0/4227082219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96240ccaa0 0x7f962406bf50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f961c009d90 tx=0x7f961c02bea0 comp rx=0 tx=0).stop 2024-03-24T10:49:12.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.005+0000 7f96299e2640 1 -- 172.21.15.5:0/4227082219 shutdown_connections 2024-03-24T10:49:12.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.005+0000 7f96299e2640 1 --2- 172.21.15.5:0/4227082219 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96240ccaa0 0x7f962406bf50 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:12.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.005+0000 7f96299e2640 1 --2- 172.21.15.5:0/4227082219 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f962415ac00 0x7f96240cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:12.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.005+0000 7f96299e2640 1 -- 172.21.15.5:0/4227082219 >> 172.21.15.5:0/4227082219 conn(0x7f9624072ef0 msgr2=0x7f9624061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:12.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.005+0000 7f96299e2640 1 -- 172.21.15.5:0/4227082219 shutdown_connections 2024-03-24T10:49:12.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.005+0000 7f96299e2640 1 -- 172.21.15.5:0/4227082219 wait complete. 2024-03-24T10:49:12.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.006+0000 7f96299e2640 1 Processor -- start 2024-03-24T10:49:12.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.006+0000 7f96299e2640 1 -- start start 2024-03-24T10:49:12.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.007+0000 7f96299e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96240ccaa0 0x7f96241f3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:12.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.007+0000 7f96299e2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f96241543a0 0x7f96240cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:12.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.007+0000 7f96299e2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f962415ac00 0x7f96240b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:12.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.007+0000 7f96299e2640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f9624154d10 con 0x7f962415ac00 2024-03-24T10:49:12.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.007+0000 7f96299e2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f96241f3070 con 0x7f96240ccaa0 2024-03-24T10:49:12.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.007+0000 7f96299e2640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f96241f3a40 con 0x7f96241543a0 2024-03-24T10:49:12.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.008+0000 7f96227fc640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f96241543a0 0x7f96240cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:12.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.008+0000 7f96227fc640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f96241543a0 0x7f96240cecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:55616/0 (socket says 172.21.15.5:55616) 2024-03-24T10:49:12.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.008+0000 7f96227fc640 1 -- 172.21.15.5:0/2168893451 learned_addr learned my addr 172.21.15.5:0/2168893451 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:12.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.008+0000 7f96237fe640 1 --2- 172.21.15.5:0/2168893451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f962415ac00 0x7f96240b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:12.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.009+0000 7f9622ffd640 1 --2- 172.21.15.5:0/2168893451 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96240ccaa0 0x7f96241f3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:12.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.013+0000 7f9622ffd640 1 -- 172.21.15.5:0/2168893451 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f96241543a0 msgr2=0x7f96240cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:12.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.013+0000 7f9622ffd640 1 --2- 172.21.15.5:0/2168893451 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f96241543a0 0x7f96240cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:12.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.013+0000 7f9622ffd640 1 -- 172.21.15.5:0/2168893451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f962415ac00 msgr2=0x7f96240b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:12.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.013+0000 7f9622ffd640 1 --2- 172.21.15.5:0/2168893451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f962415ac00 0x7f96240b3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:12.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.013+0000 7f9622ffd640 1 -- 172.21.15.5:0/2168893451 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f961c009a70 con 0x7f96240ccaa0 2024-03-24T10:49:12.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.017+0000 7f9622ffd640 1 --2- 172.21.15.5:0/2168893451 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96240ccaa0 0x7f96241f3500 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f961400c430 tx=0x7f961400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:12.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.021+0000 7f960bfff640 1 -- 172.21.15.5:0/2168893451 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9614005ce0 con 0x7f96240ccaa0 2024-03-24T10:49:12.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.021+0000 7f960bfff640 1 -- 172.21.15.5:0/2168893451 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9614005e40 con 0x7f96240ccaa0 2024-03-24T10:49:12.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.021+0000 7f960bfff640 1 -- 172.21.15.5:0/2168893451 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96140072f0 con 0x7f96240ccaa0 2024-03-24T10:49:12.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.021+0000 7f96299e2640 1 -- 172.21.15.5:0/2168893451 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9624154f00 con 0x7f96240ccaa0 2024-03-24T10:49:12.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.021+0000 7f96299e2640 1 -- 172.21.15.5:0/2168893451 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96241f2d00 con 0x7f96240ccaa0 2024-03-24T10:49:12.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.023+0000 7f96299e2640 1 -- 172.21.15.5:0/2168893451 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f962414d550 con 0x7f96240ccaa0 2024-03-24T10:49:12.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.047+0000 7f960bfff640 1 -- 172.21.15.5:0/2168893451 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9614007450 con 0x7f96240ccaa0 2024-03-24T10:49:12.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.047+0000 7f960bfff640 1 --2- 172.21.15.5:0/2168893451 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f961006b860 0x7f961006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:12.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.047+0000 7f96227fc640 1 --2- 172.21.15.5:0/2168893451 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f961006b860 0x7f961006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:12.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.048+0000 7f96227fc640 1 --2- 172.21.15.5:0/2168893451 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f961006b860 0x7f961006dd20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f962414d940 tx=0x7f9618009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:12.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.322+0000 7f960bfff640 1 -- 172.21.15.5:0/2168893451 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f9614007e40 con 0x7f96240ccaa0 2024-03-24T10:49:12.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.334+0000 7f960bfff640 1 -- 172.21.15.5:0/2168893451 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9614054440 con 0x7f96240ccaa0 2024-03-24T10:49:12.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.558+0000 7f96299e2640 1 -- 172.21.15.5:0/2168893451 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f962414d550 con 0x7f96240ccaa0 2024-03-24T10:49:12.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.588+0000 7f960bfff640 1 -- 172.21.15.5:0/2168893451 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f962414d550 con 0x7f96240ccaa0 2024-03-24T10:49:12.588 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:12.589 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:12.589 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:12.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.590+0000 7f96299e2640 1 -- 172.21.15.5:0/2168893451 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f961006b860 msgr2=0x7f961006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:12.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.590+0000 7f96299e2640 1 --2- 172.21.15.5:0/2168893451 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f961006b860 0x7f961006dd20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f962414d940 tx=0x7f9618009190 comp rx=0 tx=0).stop 2024-03-24T10:49:12.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.590+0000 7f96299e2640 1 -- 172.21.15.5:0/2168893451 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96240ccaa0 msgr2=0x7f96241f3500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:12.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.590+0000 7f96299e2640 1 --2- 172.21.15.5:0/2168893451 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96240ccaa0 0x7f96241f3500 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f961400c430 tx=0x7f961400c8f0 comp rx=0 tx=0).stop 2024-03-24T10:49:12.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.590+0000 7f96299e2640 1 -- 172.21.15.5:0/2168893451 shutdown_connections 2024-03-24T10:49:12.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.590+0000 7f96299e2640 1 --2- 172.21.15.5:0/2168893451 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f961006b860 0x7f961006dd20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:12.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.590+0000 7f96299e2640 1 --2- 172.21.15.5:0/2168893451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f962415ac00 0x7f96240b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:12.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.590+0000 7f96299e2640 1 --2- 172.21.15.5:0/2168893451 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f96241543a0 0x7f96240cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:12.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.590+0000 7f96299e2640 1 --2- 172.21.15.5:0/2168893451 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f96240ccaa0 0x7f96241f3500 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:12.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.590+0000 7f96299e2640 1 -- 172.21.15.5:0/2168893451 >> 172.21.15.5:0/2168893451 conn(0x7f9624072ef0 msgr2=0x7f962406be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:12.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.591+0000 7f96299e2640 1 -- 172.21.15.5:0/2168893451 shutdown_connections 2024-03-24T10:49:12.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:12.591+0000 7f96299e2640 1 -- 172.21.15.5:0/2168893451 wait complete. 2024-03-24T10:49:13.205 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:13.205 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:14.206 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:14.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.308+0000 7f848f7af640 1 Processor -- start 2024-03-24T10:49:14.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.309+0000 7f848f7af640 1 -- start start 2024-03-24T10:49:14.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.310+0000 7f848f7af640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f84881543a0 0x7f848815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:14.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.310+0000 7f848f7af640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84880cb2f0 con 0x7f848806d4f0 2024-03-24T10:49:14.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.310+0000 7f848f7af640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84880ca7e0 con 0x7f848806bf50 2024-03-24T10:49:14.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.310+0000 7f848f7af640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f84880c6750 con 0x7f84881543a0 2024-03-24T10:49:14.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.310+0000 7f848cd4a640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f848806d4f0 0x7f84880cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60390/0 (socket says 172.21.15.5:60390) 2024-03-24T10:49:14.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.310+0000 7f848cd4a640 1 -- 172.21.15.5:0/3523945953 learned_addr learned my addr 172.21.15.5:0/3523945953 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:49:14.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.311+0000 7f848d54b640 1 --2- 172.21.15.5:0/3523945953 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f84881543a0 0x7f848815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:14.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.314+0000 7f847ffff640 1 -- 172.21.15.5:0/3523945953 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2885748775 0 0) 0x7f84880cb2f0 con 0x7f848806d4f0 2024-03-24T10:49:14.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.314+0000 7f847ffff640 1 -- 172.21.15.5:0/3523945953 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8474003520 con 0x7f848806d4f0 2024-03-24T10:49:14.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.314+0000 7f847ffff640 1 -- 172.21.15.5:0/3523945953 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 598219591 0 0) 0x7f84880ca7e0 con 0x7f848806bf50 2024-03-24T10:49:14.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.314+0000 7f847ffff640 1 -- 172.21.15.5:0/3523945953 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84880cb2f0 con 0x7f848806bf50 2024-03-24T10:49:14.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.315+0000 7f848d54b640 1 -- 172.21.15.5:0/3523945953 >> v1:172.21.15.57:6789/0 conn(0x7f848806bf50 legacy=0x7f848806c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:14.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.315+0000 7f848d54b640 1 -- 172.21.15.5:0/3523945953 >> v1:172.21.15.5:6789/0 conn(0x7f848806d4f0 legacy=0x7f84880cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:14.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.315+0000 7f848d54b640 1 -- 172.21.15.5:0/3523945953 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84880cb520 con 0x7f84881543a0 2024-03-24T10:49:14.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.317+0000 7f848d54b640 1 --2- 172.21.15.5:0/3523945953 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f84881543a0 0x7f848815ac00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f84840097d0 tx=0x7f848402b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b3eb894f896fe43 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:14.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.322+0000 7f847ffff640 1 -- 172.21.15.5:0/3523945953 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8484030880 con 0x7f84881543a0 2024-03-24T10:49:14.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.322+0000 7f847ffff640 1 -- 172.21.15.5:0/3523945953 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84840309e0 con 0x7f84881543a0 2024-03-24T10:49:14.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.322+0000 7f847ffff640 1 -- 172.21.15.5:0/3523945953 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8484030d70 con 0x7f84881543a0 2024-03-24T10:49:14.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.323+0000 7f848f7af640 1 -- 172.21.15.5:0/3523945953 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f84881543a0 msgr2=0x7f848815ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:14.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.323+0000 7f848f7af640 1 --2- 172.21.15.5:0/3523945953 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f84881543a0 0x7f848815ac00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f84840097d0 tx=0x7f848402b9c0 comp rx=0 tx=0).stop 2024-03-24T10:49:14.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.323+0000 7f848f7af640 1 -- 172.21.15.5:0/3523945953 shutdown_connections 2024-03-24T10:49:14.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.323+0000 7f848f7af640 1 --2- 172.21.15.5:0/3523945953 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f84881543a0 0x7f848815ac00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:14.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.323+0000 7f848f7af640 1 -- 172.21.15.5:0/3523945953 >> 172.21.15.5:0/3523945953 conn(0x7f8488072ef0 msgr2=0x7f8488061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:14.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.323+0000 7f848f7af640 1 -- 172.21.15.5:0/3523945953 shutdown_connections 2024-03-24T10:49:14.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.323+0000 7f848f7af640 1 -- 172.21.15.5:0/3523945953 wait complete. 2024-03-24T10:49:14.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.325+0000 7f848f7af640 1 Processor -- start 2024-03-24T10:49:14.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.325+0000 7f848f7af640 1 -- start start 2024-03-24T10:49:14.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.326+0000 7f848f7af640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f848806bf50 0x7f8488157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:14.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.326+0000 7f848f7af640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f848806d4f0 0x7f84880b4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:14.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.326+0000 7f848f7af640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f84881543a0 0x7f84880b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:14.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.326+0000 7f848f7af640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f84880b3be0 con 0x7f848806bf50 2024-03-24T10:49:14.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.326+0000 7f848f7af640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f848814e580 con 0x7f848806d4f0 2024-03-24T10:49:14.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.326+0000 7f848f7af640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f84880b35b0 con 0x7f84881543a0 2024-03-24T10:49:14.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.327+0000 7f848d54b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f848806bf50 0x7f8488157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:14.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.327+0000 7f848cd4a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f848806d4f0 0x7f84880b4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:14.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.327+0000 7f848dd4c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f84881543a0 0x7f84880b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:14.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.327+0000 7f848d54b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f848806bf50 0x7f8488157f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36354/0 (socket says 172.21.15.5:36354) 2024-03-24T10:49:14.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.327+0000 7f848d54b640 1 -- 172.21.15.5:0/364225777 learned_addr learned my addr 172.21.15.5:0/364225777 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:14.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.332+0000 7f848d54b640 1 -- 172.21.15.5:0/364225777 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f84881543a0 msgr2=0x7f84880b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:14.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.332+0000 7f848d54b640 1 --2- 172.21.15.5:0/364225777 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f84881543a0 0x7f84880b45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:14.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.332+0000 7f848d54b640 1 -- 172.21.15.5:0/364225777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f848806d4f0 msgr2=0x7f84880b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:14.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.332+0000 7f848d54b640 1 --2- 172.21.15.5:0/364225777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f848806d4f0 0x7f84880b4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:14.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.332+0000 7f848d54b640 1 -- 172.21.15.5:0/364225777 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8484009400 con 0x7f848806bf50 2024-03-24T10:49:14.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.335+0000 7f848d54b640 1 --2- 172.21.15.5:0/364225777 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f848806bf50 0x7f8488157f60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f84840098f0 tx=0x7f8484004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:14.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.339+0000 7f847dffb640 1 -- 172.21.15.5:0/364225777 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f848403e040 con 0x7f848806bf50 2024-03-24T10:49:14.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.339+0000 7f847dffb640 1 -- 172.21.15.5:0/364225777 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8484039cd0 con 0x7f848806bf50 2024-03-24T10:49:14.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.339+0000 7f848f7af640 1 -- 172.21.15.5:0/364225777 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84881594b0 con 0x7f848806bf50 2024-03-24T10:49:14.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.339+0000 7f847dffb640 1 -- 172.21.15.5:0/364225777 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8484044630 con 0x7f848806bf50 2024-03-24T10:49:14.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.339+0000 7f848f7af640 1 -- 172.21.15.5:0/364225777 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8488159a70 con 0x7f848806bf50 2024-03-24T10:49:14.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.341+0000 7f848f7af640 1 -- 172.21.15.5:0/364225777 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8488152f00 con 0x7f848806bf50 2024-03-24T10:49:14.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.609+0000 7f847dffb640 1 -- 172.21.15.5:0/364225777 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8484044790 con 0x7f848806bf50 2024-03-24T10:49:14.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.610+0000 7f847dffb640 1 --2- 172.21.15.5:0/364225777 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f846c06b840 0x7f846c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:14.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.610+0000 7f848cd4a640 1 --2- 172.21.15.5:0/364225777 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f846c06b840 0x7f846c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:14.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.610+0000 7f848cd4a640 1 --2- 172.21.15.5:0/364225777 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f846c06b840 0x7f846c06dd00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f84880ca3a0 tx=0x7f8478007d70 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.624+0000 7f847dffb640 1 -- 172.21.15.5:0/364225777 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f8484039e40 con 0x7f848806bf50 2024-03-24T10:49:14.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.624+0000 7f847dffb640 1 -- 172.21.15.5:0/364225777 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8484077c90 con 0x7f848806bf50 2024-03-24T10:49:14.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.859+0000 7f848f7af640 1 -- 172.21.15.5:0/364225777 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8488159c00 con 0x7f848806bf50 2024-03-24T10:49:14.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.873+0000 7f847dffb640 1 -- 172.21.15.5:0/364225777 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f8484049090 con 0x7f848806bf50 2024-03-24T10:49:14.874 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:14.874 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:14.874 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:14.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.875+0000 7f848f7af640 1 -- 172.21.15.5:0/364225777 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f846c06b840 msgr2=0x7f846c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.875+0000 7f848f7af640 1 --2- 172.21.15.5:0/364225777 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f846c06b840 0x7f846c06dd00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f84880ca3a0 tx=0x7f8478007d70 comp rx=0 tx=0).stop 2024-03-24T10:49:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.875+0000 7f848f7af640 1 -- 172.21.15.5:0/364225777 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f848806bf50 msgr2=0x7f8488157f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.875+0000 7f848f7af640 1 --2- 172.21.15.5:0/364225777 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f848806bf50 0x7f8488157f60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f84840098f0 tx=0x7f8484004730 comp rx=0 tx=0).stop 2024-03-24T10:49:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.875+0000 7f848f7af640 1 -- 172.21.15.5:0/364225777 shutdown_connections 2024-03-24T10:49:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.875+0000 7f848f7af640 1 --2- 172.21.15.5:0/364225777 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f846c06b840 0x7f846c06dd00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.875+0000 7f848f7af640 1 --2- 172.21.15.5:0/364225777 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f84881543a0 0x7f84880b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.875+0000 7f848f7af640 1 --2- 172.21.15.5:0/364225777 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f848806d4f0 0x7f84880b4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.875+0000 7f848f7af640 1 --2- 172.21.15.5:0/364225777 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f848806bf50 0x7f8488157f60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.875+0000 7f848f7af640 1 -- 172.21.15.5:0/364225777 >> 172.21.15.5:0/364225777 conn(0x7f8488072ef0 msgr2=0x7f8488150370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.876+0000 7f848f7af640 1 -- 172.21.15.5:0/364225777 shutdown_connections 2024-03-24T10:49:14.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:14.876+0000 7f848f7af640 1 -- 172.21.15.5:0/364225777 wait complete. 2024-03-24T10:49:15.491 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:15.491 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:16.492 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:16.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.596+0000 7feead82e640 1 Processor -- start 2024-03-24T10:49:16.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.596+0000 7feead82e640 1 -- start start 2024-03-24T10:49:16.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.597+0000 7feead82e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feea806bf50 0x7feea806c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:16.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.597+0000 7feead82e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feea80cb2f0 con 0x7feea815ac00 2024-03-24T10:49:16.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.597+0000 7feead82e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feea815b070 con 0x7feea81543a0 2024-03-24T10:49:16.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.597+0000 7feead82e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7feea80c6750 con 0x7feea806bf50 2024-03-24T10:49:16.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.598+0000 7feea67fc640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7feea815ac00 0x7feea80cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60396/0 (socket says 172.21.15.5:60396) 2024-03-24T10:49:16.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.598+0000 7feea67fc640 1 -- 172.21.15.5:0/3429494071 learned_addr learned my addr 172.21.15.5:0/3429494071 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:49:16.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.598+0000 7feea77fe640 1 --2- 172.21.15.5:0/3429494071 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feea806bf50 0x7feea806c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:16.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.601+0000 7feea5ffb640 1 -- 172.21.15.5:0/3429494071 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2497029693 0 0) 0x7feea80cb2f0 con 0x7feea815ac00 2024-03-24T10:49:16.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.601+0000 7feea5ffb640 1 -- 172.21.15.5:0/3429494071 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee90003520 con 0x7feea815ac00 2024-03-24T10:49:16.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.601+0000 7feea77fe640 1 -- 172.21.15.5:0/3429494071 >> v1:172.21.15.57:6789/0 conn(0x7feea81543a0 legacy=0x7feea806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:16.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.601+0000 7feea77fe640 1 -- 172.21.15.5:0/3429494071 >> v1:172.21.15.5:6789/0 conn(0x7feea815ac00 legacy=0x7feea80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:16.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.601+0000 7feea77fe640 1 -- 172.21.15.5:0/3429494071 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feea80ca7e0 con 0x7feea806bf50 2024-03-24T10:49:16.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.604+0000 7feea77fe640 1 --2- 172.21.15.5:0/3429494071 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feea806bf50 0x7feea806c360 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7feea0009dc0 tx=0x7feea002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66a1abfbe5bf5c5d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:16.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.608+0000 7feea5ffb640 1 -- 172.21.15.5:0/3429494071 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea002ccd0 con 0x7feea806bf50 2024-03-24T10:49:16.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.608+0000 7feea5ffb640 1 -- 172.21.15.5:0/3429494071 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feea002ce30 con 0x7feea806bf50 2024-03-24T10:49:16.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.608+0000 7feea5ffb640 1 -- 172.21.15.5:0/3429494071 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea0034660 con 0x7feea806bf50 2024-03-24T10:49:16.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.609+0000 7feead82e640 1 -- 172.21.15.5:0/3429494071 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feea806bf50 msgr2=0x7feea806c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:16.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.609+0000 7feead82e640 1 --2- 172.21.15.5:0/3429494071 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feea806bf50 0x7feea806c360 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7feea0009dc0 tx=0x7feea002bf20 comp rx=0 tx=0).stop 2024-03-24T10:49:16.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.609+0000 7feead82e640 1 -- 172.21.15.5:0/3429494071 shutdown_connections 2024-03-24T10:49:16.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.609+0000 7feead82e640 1 --2- 172.21.15.5:0/3429494071 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feea806bf50 0x7feea806c360 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:16.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.609+0000 7feead82e640 1 -- 172.21.15.5:0/3429494071 >> 172.21.15.5:0/3429494071 conn(0x7feea8072ef0 msgr2=0x7feea8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:16.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.609+0000 7feead82e640 1 -- 172.21.15.5:0/3429494071 shutdown_connections 2024-03-24T10:49:16.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.609+0000 7feead82e640 1 -- 172.21.15.5:0/3429494071 wait complete. 2024-03-24T10:49:16.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.610+0000 7feead82e640 1 Processor -- start 2024-03-24T10:49:16.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.611+0000 7feead82e640 1 -- start start 2024-03-24T10:49:16.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.611+0000 7feead82e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feea806bf50 0x7feea8147b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:16.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.611+0000 7feead82e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feea81543a0 0x7feea814d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:16.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.611+0000 7feead82e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feea815ac00 0x7feea814db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:16.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.611+0000 7feead82e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7feea814ef50 con 0x7feea806bf50 2024-03-24T10:49:16.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.611+0000 7feead82e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7feea81480a0 con 0x7feea815ac00 2024-03-24T10:49:16.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.611+0000 7feead82e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7feea81500a0 con 0x7feea81543a0 2024-03-24T10:49:16.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.612+0000 7feea6ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feea806bf50 0x7feea8147b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:16.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.612+0000 7feea67fc640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feea81543a0 0x7feea814d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:16.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.612+0000 7feea77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feea815ac00 0x7feea814db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:16.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.612+0000 7feea6ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feea806bf50 0x7feea8147b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36366/0 (socket says 172.21.15.5:36366) 2024-03-24T10:49:16.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.612+0000 7feea6ffd640 1 -- 172.21.15.5:0/3595706665 learned_addr learned my addr 172.21.15.5:0/3595706665 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:16.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.615+0000 7feea6ffd640 1 -- 172.21.15.5:0/3595706665 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feea81543a0 msgr2=0x7feea814d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:16.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.615+0000 7feea6ffd640 1 --2- 172.21.15.5:0/3595706665 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feea81543a0 0x7feea814d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:16.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.615+0000 7feea6ffd640 1 -- 172.21.15.5:0/3595706665 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feea815ac00 msgr2=0x7feea814db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:16.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.615+0000 7feea6ffd640 1 --2- 172.21.15.5:0/3595706665 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feea815ac00 0x7feea814db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:16.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.615+0000 7feea6ffd640 1 -- 172.21.15.5:0/3595706665 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feea0009a70 con 0x7feea806bf50 2024-03-24T10:49:16.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.618+0000 7feea6ffd640 1 --2- 172.21.15.5:0/3595706665 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feea806bf50 0x7feea8147b60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fee9800c4b0 tx=0x7fee9800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:16.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.622+0000 7fee8ffff640 1 -- 172.21.15.5:0/3595706665 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee980052d0 con 0x7feea806bf50 2024-03-24T10:49:16.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.622+0000 7fee8ffff640 1 -- 172.21.15.5:0/3595706665 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee98005430 con 0x7feea806bf50 2024-03-24T10:49:16.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.622+0000 7fee8ffff640 1 -- 172.21.15.5:0/3595706665 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee98012870 con 0x7feea806bf50 2024-03-24T10:49:16.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.622+0000 7feead82e640 1 -- 172.21.15.5:0/3595706665 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feea814c0c0 con 0x7feea806bf50 2024-03-24T10:49:16.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.622+0000 7feead82e640 1 -- 172.21.15.5:0/3595706665 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feea814c680 con 0x7feea806bf50 2024-03-24T10:49:16.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.623+0000 7feead82e640 1 -- 172.21.15.5:0/3595706665 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feea80681d0 con 0x7feea806bf50 2024-03-24T10:49:16.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.648+0000 7fee8ffff640 1 -- 172.21.15.5:0/3595706665 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fee980129d0 con 0x7feea806bf50 2024-03-24T10:49:16.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.649+0000 7fee8ffff640 1 --2- 172.21.15.5:0/3595706665 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fee9406b860 0x7fee9406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:16.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.649+0000 7feea67fc640 1 --2- 172.21.15.5:0/3595706665 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fee9406b860 0x7fee9406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:16.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.649+0000 7feea67fc640 1 --2- 172.21.15.5:0/3595706665 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fee9406b860 0x7fee9406dd20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7feea80cf2f0 tx=0x7fee9c007d30 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:16.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.922+0000 7fee8ffff640 1 -- 172.21.15.5:0/3595706665 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fee98058ea0 con 0x7feea806bf50 2024-03-24T10:49:16.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:16.922+0000 7fee8ffff640 1 -- 172.21.15.5:0/3595706665 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fee98055140 con 0x7feea806bf50 2024-03-24T10:49:17.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:17.165+0000 7feead82e640 1 -- 172.21.15.5:0/3595706665 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feea815e580 con 0x7feea806bf50 2024-03-24T10:49:17.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:17.178+0000 7fee8ffff640 1 -- 172.21.15.5:0/3595706665 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fee9805de60 con 0x7feea806bf50 2024-03-24T10:49:17.178 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:17.178 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:17.179 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:17.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:17.180+0000 7feead82e640 1 -- 172.21.15.5:0/3595706665 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fee9406b860 msgr2=0x7fee9406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:17.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:17.180+0000 7feead82e640 1 --2- 172.21.15.5:0/3595706665 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fee9406b860 0x7fee9406dd20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7feea80cf2f0 tx=0x7fee9c007d30 comp rx=0 tx=0).stop 2024-03-24T10:49:17.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:17.180+0000 7feead82e640 1 -- 172.21.15.5:0/3595706665 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feea806bf50 msgr2=0x7feea8147b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:17.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:17.180+0000 7feead82e640 1 --2- 172.21.15.5:0/3595706665 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feea806bf50 0x7feea8147b60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fee9800c4b0 tx=0x7fee9800c970 comp rx=0 tx=0).stop 2024-03-24T10:49:17.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:17.180+0000 7feead82e640 1 -- 172.21.15.5:0/3595706665 shutdown_connections 2024-03-24T10:49:17.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:17.180+0000 7feead82e640 1 --2- 172.21.15.5:0/3595706665 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fee9406b860 0x7fee9406dd20 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:17.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:17.180+0000 7feead82e640 1 --2- 172.21.15.5:0/3595706665 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feea815ac00 0x7feea814db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:17.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:17.180+0000 7feead82e640 1 --2- 172.21.15.5:0/3595706665 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feea81543a0 0x7feea814d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:17.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:17.180+0000 7feead82e640 1 --2- 172.21.15.5:0/3595706665 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feea806bf50 0x7feea8147b60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:17.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:17.180+0000 7feead82e640 1 -- 172.21.15.5:0/3595706665 >> 172.21.15.5:0/3595706665 conn(0x7feea8072ef0 msgr2=0x7feea8152ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:17.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:17.181+0000 7feead82e640 1 -- 172.21.15.5:0/3595706665 shutdown_connections 2024-03-24T10:49:17.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:17.181+0000 7feead82e640 1 -- 172.21.15.5:0/3595706665 wait complete. 2024-03-24T10:49:17.795 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:17.795 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:18.796 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:18.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.898+0000 7f5088a3b640 1 Processor -- start 2024-03-24T10:49:18.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.898+0000 7f5088a3b640 1 -- start start 2024-03-24T10:49:18.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.899+0000 7f5088a3b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5084154220 0x7f50840c4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:18.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.899+0000 7f5088a3b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f50840c84e0 0x7f50840cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:18.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.899+0000 7f5088a3b640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50840c9050 con 0x7f50840c8900 2024-03-24T10:49:18.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.899+0000 7f5088a3b640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f508415a5f0 con 0x7f50840c84e0 2024-03-24T10:49:18.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.899+0000 7f5088a3b640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f508415a8d0 con 0x7f5084154220 2024-03-24T10:49:18.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.900+0000 7f5081d9b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f50840c84e0 0x7f50840cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:18.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.900+0000 7f5082d9d640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f50840c8900 0x7f508406c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60408/0 (socket says 172.21.15.5:60408) 2024-03-24T10:49:18.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.900+0000 7f5082d9d640 1 -- 172.21.15.5:0/660944939 learned_addr learned my addr 172.21.15.5:0/660944939 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:49:18.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.900+0000 7f508259c640 1 --2- 172.21.15.5:0/660944939 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5084154220 0x7f50840c4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:18.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.902+0000 7f508159a640 1 -- 172.21.15.5:0/660944939 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 207837969 0 0) 0x7f50840c9050 con 0x7f50840c8900 2024-03-24T10:49:18.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.902+0000 7f508159a640 1 -- 172.21.15.5:0/660944939 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f506c003520 con 0x7f50840c8900 2024-03-24T10:49:18.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.903+0000 7f508259c640 1 -- 172.21.15.5:0/660944939 >> v2:172.21.15.57:3300/0 conn(0x7f50840c84e0 msgr2=0x7f50840cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:18.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.903+0000 7f508259c640 1 --2- 172.21.15.5:0/660944939 >> v2:172.21.15.57:3300/0 conn(0x7f50840c84e0 0x7f50840cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:18.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.903+0000 7f508259c640 1 -- 172.21.15.5:0/660944939 >> v1:172.21.15.5:6789/0 conn(0x7f50840c8900 legacy=0x7f508406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:18.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.903+0000 7f508259c640 1 -- 172.21.15.5:0/660944939 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f508415abb0 con 0x7f5084154220 2024-03-24T10:49:18.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.905+0000 7f508259c640 1 --2- 172.21.15.5:0/660944939 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5084154220 0x7f50840c4300 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f50740097a0 tx=0x7f507402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=508e38fa1045bf7e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:18.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.910+0000 7f508159a640 1 -- 172.21.15.5:0/660944939 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5074030610 con 0x7f5084154220 2024-03-24T10:49:18.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.910+0000 7f508159a640 1 -- 172.21.15.5:0/660944939 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5074030770 con 0x7f5084154220 2024-03-24T10:49:18.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.910+0000 7f508159a640 1 -- 172.21.15.5:0/660944939 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5074030b00 con 0x7f5084154220 2024-03-24T10:49:18.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.910+0000 7f5088a3b640 1 -- 172.21.15.5:0/660944939 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5084154220 msgr2=0x7f50840c4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:18.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.910+0000 7f5088a3b640 1 --2- 172.21.15.5:0/660944939 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5084154220 0x7f50840c4300 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f50740097a0 tx=0x7f507402b960 comp rx=0 tx=0).stop 2024-03-24T10:49:18.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.911+0000 7f5088a3b640 1 -- 172.21.15.5:0/660944939 shutdown_connections 2024-03-24T10:49:18.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.911+0000 7f5088a3b640 1 --2- 172.21.15.5:0/660944939 >> v2:172.21.15.57:3300/0 conn(0x7f50840c84e0 0x7f50840cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:18.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.911+0000 7f5088a3b640 1 --2- 172.21.15.5:0/660944939 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5084154220 0x7f50840c4300 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:18.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.911+0000 7f5088a3b640 1 -- 172.21.15.5:0/660944939 >> 172.21.15.5:0/660944939 conn(0x7f508406f830 msgr2=0x7f508406fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:18.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.911+0000 7f5088a3b640 1 -- 172.21.15.5:0/660944939 shutdown_connections 2024-03-24T10:49:18.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.911+0000 7f5088a3b640 1 -- 172.21.15.5:0/660944939 wait complete. 2024-03-24T10:49:18.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.912+0000 7f5088a3b640 1 Processor -- start 2024-03-24T10:49:18.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.912+0000 7f5088a3b640 1 -- start start 2024-03-24T10:49:18.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.913+0000 7f5088a3b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f50840c84e0 0x7f50841f06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:18.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.913+0000 7f5088a3b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f50840c8900 0x7f50840ae510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:18.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.913+0000 7f5088a3b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5084154220 0x7f50840cbb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:18.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.913+0000 7f5088a3b640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f50840afa60 con 0x7f5084154220 2024-03-24T10:49:18.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.913+0000 7f5088a3b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f50840af780 con 0x7f50840c8900 2024-03-24T10:49:18.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.913+0000 7f5088a3b640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f50840ae080 con 0x7f50840c84e0 2024-03-24T10:49:18.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.913+0000 7f5082d9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5084154220 0x7f50840cbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:18.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.913+0000 7f508259c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f50840c84e0 0x7f50841f06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:18.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.914+0000 7f5081d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f50840c8900 0x7f50840ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:18.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.914+0000 7f5082d9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5084154220 0x7f50840cbb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36378/0 (socket says 172.21.15.5:36378) 2024-03-24T10:49:18.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.914+0000 7f5082d9d640 1 -- 172.21.15.5:0/2224752866 learned_addr learned my addr 172.21.15.5:0/2224752866 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:18.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.919+0000 7f5082d9d640 1 -- 172.21.15.5:0/2224752866 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f50840c84e0 msgr2=0x7f50841f06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:18.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.919+0000 7f5082d9d640 1 --2- 172.21.15.5:0/2224752866 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f50840c84e0 0x7f50841f06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:18.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.919+0000 7f5082d9d640 1 -- 172.21.15.5:0/2224752866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f50840c8900 msgr2=0x7f50840ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:18.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.919+0000 7f5082d9d640 1 --2- 172.21.15.5:0/2224752866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f50840c8900 0x7f50840ae510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:18.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.919+0000 7f5082d9d640 1 -- 172.21.15.5:0/2224752866 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5074009400 con 0x7f5084154220 2024-03-24T10:49:18.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.922+0000 7f5082d9d640 1 --2- 172.21.15.5:0/2224752866 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5084154220 0x7f50840cbb50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f507c00cae0 tx=0x7f507c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:18.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.927+0000 7f5072ffd640 1 -- 172.21.15.5:0/2224752866 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c00ac40 con 0x7f5084154220 2024-03-24T10:49:18.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.927+0000 7f5072ffd640 1 -- 172.21.15.5:0/2224752866 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f507c00ada0 con 0x7f5084154220 2024-03-24T10:49:18.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.927+0000 7f5088a3b640 1 -- 172.21.15.5:0/2224752866 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50840b00b0 con 0x7f5084154220 2024-03-24T10:49:18.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.927+0000 7f5072ffd640 1 -- 172.21.15.5:0/2224752866 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c012640 con 0x7f5084154220 2024-03-24T10:49:18.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.927+0000 7f5088a3b640 1 -- 172.21.15.5:0/2224752866 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50840aea50 con 0x7f5084154220 2024-03-24T10:49:18.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:18.928+0000 7f5088a3b640 1 -- 172.21.15.5:0/2224752866 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50840ae1f0 con 0x7f5084154220 2024-03-24T10:49:19.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:19.211+0000 7f5072ffd640 1 -- 172.21.15.5:0/2224752866 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f507c007070 con 0x7f5084154220 2024-03-24T10:49:19.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:19.211+0000 7f5072ffd640 1 --2- 172.21.15.5:0/2224752866 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f506406b810 0x7f506406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:19.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:19.212+0000 7f508259c640 1 --2- 172.21.15.5:0/2224752866 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f506406b810 0x7f506406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:19.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:19.212+0000 7f508259c640 1 --2- 172.21.15.5:0/2224752866 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f506406b810 0x7f506406dcd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5074036000 tx=0x7f5074008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:19.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:19.231+0000 7f5072ffd640 1 -- 172.21.15.5:0/2224752866 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f507c058db0 con 0x7f5084154220 2024-03-24T10:49:19.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:19.231+0000 7f5072ffd640 1 -- 172.21.15.5:0/2224752866 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f507c055050 con 0x7f5084154220 2024-03-24T10:49:19.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:19.468+0000 7f5088a3b640 1 -- 172.21.15.5:0/2224752866 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50840681d0 con 0x7f5084154220 2024-03-24T10:49:19.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:19.481+0000 7f5072ffd640 1 -- 172.21.15.5:0/2224752866 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f507c017090 con 0x7f5084154220 2024-03-24T10:49:19.481 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:19.481 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:19.482 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:19.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:19.483+0000 7f5088a3b640 1 -- 172.21.15.5:0/2224752866 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f506406b810 msgr2=0x7f506406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:19.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:19.483+0000 7f5088a3b640 1 --2- 172.21.15.5:0/2224752866 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f506406b810 0x7f506406dcd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5074036000 tx=0x7f5074008030 comp rx=0 tx=0).stop 2024-03-24T10:49:19.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:19.483+0000 7f5088a3b640 1 -- 172.21.15.5:0/2224752866 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5084154220 msgr2=0x7f50840cbb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:19.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:19.483+0000 7f5088a3b640 1 --2- 172.21.15.5:0/2224752866 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5084154220 0x7f50840cbb50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f507c00cae0 tx=0x7f507c00a060 comp rx=0 tx=0).stop 2024-03-24T10:49:19.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:19.483+0000 7f5088a3b640 1 -- 172.21.15.5:0/2224752866 shutdown_connections 2024-03-24T10:49:19.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:19.483+0000 7f5088a3b640 1 --2- 172.21.15.5:0/2224752866 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f506406b810 0x7f506406dcd0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:19.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:19.483+0000 7f5088a3b640 1 --2- 172.21.15.5:0/2224752866 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5084154220 0x7f50840cbb50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:19.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:19.483+0000 7f5088a3b640 1 --2- 172.21.15.5:0/2224752866 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f50840c8900 0x7f50840ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:19.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:19.483+0000 7f5088a3b640 1 --2- 172.21.15.5:0/2224752866 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f50840c84e0 0x7f50841f06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:19.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:19.483+0000 7f5088a3b640 1 -- 172.21.15.5:0/2224752866 >> 172.21.15.5:0/2224752866 conn(0x7f508406f830 msgr2=0x7f5084159350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:19.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:19.483+0000 7f5088a3b640 1 -- 172.21.15.5:0/2224752866 shutdown_connections 2024-03-24T10:49:19.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:19.483+0000 7f5088a3b640 1 -- 172.21.15.5:0/2224752866 wait complete. 2024-03-24T10:49:20.098 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:20.098 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:21.099 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:21.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.201+0000 7f7ddc04e640 1 Processor -- start 2024-03-24T10:49:21.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.202+0000 7f7ddc04e640 1 -- start start 2024-03-24T10:49:21.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.203+0000 7f7ddc04e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7dd40cc8f0 0x7f7dd415b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:21.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.203+0000 7f7ddc04e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7dd406c580 0x7f7dd41538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:21.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.203+0000 7f7ddc04e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7dd40cf110 0x7f7dd4153df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:21.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.203+0000 7f7ddc04e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f7dd40cd890 con 0x7f7dd40cc8f0 2024-03-24T10:49:21.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.203+0000 7f7ddc04e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f7dd406cae0 con 0x7f7dd40cf110 2024-03-24T10:49:21.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.203+0000 7f7ddc04e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7dd40cdb50 con 0x7f7dd406c580 2024-03-24T10:49:21.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.204+0000 7f7dd9dea640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7dd40cc8f0 0x7f7dd415b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:21.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.204+0000 7f7dda5eb640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7dd40cf110 0x7f7dd4153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:21.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.204+0000 7f7dd95e9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7dd406c580 0x7f7dd41538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:21.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.204+0000 7f7dd9dea640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7dd40cc8f0 0x7f7dd415b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36382/0 (socket says 172.21.15.5:36382) 2024-03-24T10:49:21.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.204+0000 7f7dd9dea640 1 -- 172.21.15.5:0/1896580394 learned_addr learned my addr 172.21.15.5:0/1896580394 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:21.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.206+0000 7f7dd9dea640 1 -- 172.21.15.5:0/1896580394 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7dd406c580 msgr2=0x7f7dd41538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:21.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.206+0000 7f7dd9dea640 1 --2- 172.21.15.5:0/1896580394 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7dd406c580 0x7f7dd41538b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:21.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.206+0000 7f7dd9dea640 1 -- 172.21.15.5:0/1896580394 >> v2:172.21.15.57:3300/0 conn(0x7f7dd40cf110 msgr2=0x7f7dd4153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:21.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.206+0000 7f7dd9dea640 1 --2- 172.21.15.5:0/1896580394 >> v2:172.21.15.57:3300/0 conn(0x7f7dd40cf110 0x7f7dd4153df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:21.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.206+0000 7f7dd9dea640 1 -- 172.21.15.5:0/1896580394 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd40cde30 con 0x7f7dd40cc8f0 2024-03-24T10:49:21.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.208+0000 7f7dd9dea640 1 --2- 172.21.15.5:0/1896580394 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7dd40cc8f0 0x7f7dd415b5d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7dd40cfa60 tx=0x7f7dd002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de737f81e7a5d63f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:21.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.212+0000 7f7dd8de8640 1 -- 172.21.15.5:0/1896580394 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd0031870 con 0x7f7dd40cc8f0 2024-03-24T10:49:21.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.213+0000 7f7dd8de8640 1 -- 172.21.15.5:0/1896580394 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dd00319d0 con 0x7f7dd40cc8f0 2024-03-24T10:49:21.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.213+0000 7f7dd8de8640 1 -- 172.21.15.5:0/1896580394 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd0031d60 con 0x7f7dd40cc8f0 2024-03-24T10:49:21.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.213+0000 7f7ddc04e640 1 -- 172.21.15.5:0/1896580394 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7dd40cc8f0 msgr2=0x7f7dd415b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:21.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.213+0000 7f7ddc04e640 1 --2- 172.21.15.5:0/1896580394 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7dd40cc8f0 0x7f7dd415b5d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7dd40cfa60 tx=0x7f7dd002b910 comp rx=0 tx=0).stop 2024-03-24T10:49:21.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.213+0000 7f7ddc04e640 1 -- 172.21.15.5:0/1896580394 shutdown_connections 2024-03-24T10:49:21.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.213+0000 7f7ddc04e640 1 --2- 172.21.15.5:0/1896580394 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7dd406c580 0x7f7dd41538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:21.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.213+0000 7f7ddc04e640 1 --2- 172.21.15.5:0/1896580394 >> v2:172.21.15.57:3300/0 conn(0x7f7dd40cf110 0x7f7dd4153df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:21.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.213+0000 7f7ddc04e640 1 --2- 172.21.15.5:0/1896580394 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7dd40cc8f0 0x7f7dd415b5d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:21.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.213+0000 7f7ddc04e640 1 -- 172.21.15.5:0/1896580394 >> 172.21.15.5:0/1896580394 conn(0x7f7dd40b1820 msgr2=0x7f7dd40b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:21.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.213+0000 7f7ddc04e640 1 -- 172.21.15.5:0/1896580394 shutdown_connections 2024-03-24T10:49:21.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.214+0000 7f7ddc04e640 1 -- 172.21.15.5:0/1896580394 wait complete. 2024-03-24T10:49:21.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.215+0000 7f7ddc04e640 1 Processor -- start 2024-03-24T10:49:21.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.215+0000 7f7ddc04e640 1 -- start start 2024-03-24T10:49:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.217+0000 7f7ddc04e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7dd406c580 0x7f7dd4159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.217+0000 7f7ddc04e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dd40cc8f0 0x7f7dd4157170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.217+0000 7f7ddc04e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7dd40cf110 0x7f7dd41576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:21.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.217+0000 7f7ddc04e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd40c44b0 con 0x7f7dd406c580 2024-03-24T10:49:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.217+0000 7f7ddc04e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7dd40c35e0 con 0x7f7dd40cc8f0 2024-03-24T10:49:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.217+0000 7f7ddc04e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7dd40c3830 con 0x7f7dd40cf110 2024-03-24T10:49:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.217+0000 7f7dd9dea640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7dd406c580 0x7f7dd4159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.217+0000 7f7dd95e9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dd40cc8f0 0x7f7dd4157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.217+0000 7f7dda5eb640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7dd40cf110 0x7f7dd41576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.217+0000 7f7dd9dea640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7dd406c580 0x7f7dd4159d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36384/0 (socket says 172.21.15.5:36384) 2024-03-24T10:49:21.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.217+0000 7f7dd9dea640 1 -- 172.21.15.5:0/2455692929 learned_addr learned my addr 172.21.15.5:0/2455692929 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:21.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.221+0000 7f7dd9dea640 1 -- 172.21.15.5:0/2455692929 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7dd40cf110 msgr2=0x7f7dd41576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:21.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.221+0000 7f7dd9dea640 1 --2- 172.21.15.5:0/2455692929 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7dd40cf110 0x7f7dd41576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:21.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.221+0000 7f7dd9dea640 1 -- 172.21.15.5:0/2455692929 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dd40cc8f0 msgr2=0x7f7dd4157170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:21.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.221+0000 7f7dd9dea640 1 --2- 172.21.15.5:0/2455692929 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dd40cc8f0 0x7f7dd4157170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:21.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.221+0000 7f7dd9dea640 1 -- 172.21.15.5:0/2455692929 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dd00094d0 con 0x7f7dd406c580 2024-03-24T10:49:21.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.223+0000 7f7dd9dea640 1 --2- 172.21.15.5:0/2455692929 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7dd406c580 0x7f7dd4159d40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7dd002b950 tx=0x7f7dd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:21.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.227+0000 7f7dca7fc640 1 -- 172.21.15.5:0/2455692929 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd003ac60 con 0x7f7dd406c580 2024-03-24T10:49:21.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.227+0000 7f7dca7fc640 1 -- 172.21.15.5:0/2455692929 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dd003adc0 con 0x7f7dd406c580 2024-03-24T10:49:21.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.227+0000 7f7ddc04e640 1 -- 172.21.15.5:0/2455692929 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dd40c3b10 con 0x7f7dd406c580 2024-03-24T10:49:21.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.227+0000 7f7dca7fc640 1 -- 172.21.15.5:0/2455692929 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd0044630 con 0x7f7dd406c580 2024-03-24T10:49:21.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.227+0000 7f7ddc04e640 1 -- 172.21.15.5:0/2455692929 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dd40c40d0 con 0x7f7dd406c580 2024-03-24T10:49:21.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.228+0000 7f7ddc04e640 1 -- 172.21.15.5:0/2455692929 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dd40ce910 con 0x7f7dd406c580 2024-03-24T10:49:21.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.247+0000 7f7dca7fc640 1 -- 172.21.15.5:0/2455692929 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7dd004d450 con 0x7f7dd406c580 2024-03-24T10:49:21.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.247+0000 7f7dca7fc640 1 --2- 172.21.15.5:0/2455692929 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7db806b860 0x7f7db806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:21.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.248+0000 7f7dd95e9640 1 --2- 172.21.15.5:0/2455692929 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7db806b860 0x7f7db806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:21.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.248+0000 7f7dd95e9640 1 --2- 172.21.15.5:0/2455692929 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7db806b860 0x7f7db806dd20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4154390 tx=0x7f7dc4009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:21.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.253+0000 7f7dca7fc640 1 -- 172.21.15.5:0/2455692929 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f7dd002d040 con 0x7f7dd406c580 2024-03-24T10:49:21.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.510+0000 7f7dca7fc640 1 -- 172.21.15.5:0/2455692929 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7dd00775f0 con 0x7f7dd406c580 2024-03-24T10:49:21.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.747+0000 7f7ddc04e640 1 -- 172.21.15.5:0/2455692929 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7dd40c4260 con 0x7f7dd406c580 2024-03-24T10:49:21.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.760+0000 7f7dca7fc640 1 -- 172.21.15.5:0/2455692929 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f7dd0035070 con 0x7f7dd406c580 2024-03-24T10:49:21.761 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:21.761 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:21.761 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:21.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.762+0000 7f7ddc04e640 1 -- 172.21.15.5:0/2455692929 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7db806b860 msgr2=0x7f7db806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:21.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.762+0000 7f7ddc04e640 1 --2- 172.21.15.5:0/2455692929 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7db806b860 0x7f7db806dd20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4154390 tx=0x7f7dc4009190 comp rx=0 tx=0).stop 2024-03-24T10:49:21.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.762+0000 7f7ddc04e640 1 -- 172.21.15.5:0/2455692929 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7dd406c580 msgr2=0x7f7dd4159d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:21.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.762+0000 7f7ddc04e640 1 --2- 172.21.15.5:0/2455692929 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7dd406c580 0x7f7dd4159d40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7dd002b950 tx=0x7f7dd00046c0 comp rx=0 tx=0).stop 2024-03-24T10:49:21.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.762+0000 7f7ddc04e640 1 -- 172.21.15.5:0/2455692929 shutdown_connections 2024-03-24T10:49:21.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.762+0000 7f7ddc04e640 1 --2- 172.21.15.5:0/2455692929 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7db806b860 0x7f7db806dd20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:21.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.763+0000 7f7ddc04e640 1 --2- 172.21.15.5:0/2455692929 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7dd40cf110 0x7f7dd41576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:21.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.763+0000 7f7ddc04e640 1 --2- 172.21.15.5:0/2455692929 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7dd40cc8f0 0x7f7dd4157170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:21.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.763+0000 7f7ddc04e640 1 --2- 172.21.15.5:0/2455692929 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7dd406c580 0x7f7dd4159d40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:21.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.763+0000 7f7ddc04e640 1 -- 172.21.15.5:0/2455692929 >> 172.21.15.5:0/2455692929 conn(0x7f7dd40b1820 msgr2=0x7f7dd40b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:21.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.763+0000 7f7ddc04e640 1 -- 172.21.15.5:0/2455692929 shutdown_connections 2024-03-24T10:49:21.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:21.763+0000 7f7ddc04e640 1 -- 172.21.15.5:0/2455692929 wait complete. 2024-03-24T10:49:22.378 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:22.378 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:23.379 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:23.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.482+0000 7f24d0e23640 1 Processor -- start 2024-03-24T10:49:23.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.482+0000 7f24d0e23640 1 -- start start 2024-03-24T10:49:23.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.483+0000 7f24d0e23640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f24cc15ac00 0x7f24cc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.483+0000 7f24d0e23640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24cc0cb2f0 con 0x7f24cc1543a0 2024-03-24T10:49:23.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.483+0000 7f24d0e23640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24cc15b070 con 0x7f24cc0ccaa0 2024-03-24T10:49:23.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.483+0000 7f24d0e23640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f24cc0c6750 con 0x7f24cc15ac00 2024-03-24T10:49:23.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.483+0000 7f24cad9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f24cc0ccaa0 0x7f24cc06bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:60496/0 (socket says 172.21.15.5:60496) 2024-03-24T10:49:23.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.483+0000 7f24cad9d640 1 -- 172.21.15.5:0/3485563561 learned_addr learned my addr 172.21.15.5:0/3485563561 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:49:23.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.483+0000 7f24c3fff640 1 --2- 172.21.15.5:0/3485563561 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f24cc15ac00 0x7f24cc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.486+0000 7f24c9d9b640 1 -- 172.21.15.5:0/3485563561 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3764375387 0 0) 0x7f24cc0cb2f0 con 0x7f24cc1543a0 2024-03-24T10:49:23.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.486+0000 7f24c9d9b640 1 -- 172.21.15.5:0/3485563561 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24b8003520 con 0x7f24cc1543a0 2024-03-24T10:49:23.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.486+0000 7f24c3fff640 1 -- 172.21.15.5:0/3485563561 >> v1:172.21.15.57:6789/0 conn(0x7f24cc0ccaa0 legacy=0x7f24cc06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:23.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.486+0000 7f24c3fff640 1 -- 172.21.15.5:0/3485563561 >> v1:172.21.15.5:6789/0 conn(0x7f24cc1543a0 legacy=0x7f24cc06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:23.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.486+0000 7f24c3fff640 1 -- 172.21.15.5:0/3485563561 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24cc0ca7e0 con 0x7f24cc15ac00 2024-03-24T10:49:23.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.489+0000 7f24c3fff640 1 --2- 172.21.15.5:0/3485563561 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f24cc15ac00 0x7f24cc0cc560 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f24b40097d0 tx=0x7f24b402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67a78a24b2204e9e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:23.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.494+0000 7f24c9d9b640 1 -- 172.21.15.5:0/3485563561 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b4031890 con 0x7f24cc15ac00 2024-03-24T10:49:23.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.494+0000 7f24c9d9b640 1 -- 172.21.15.5:0/3485563561 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24b40319f0 con 0x7f24cc15ac00 2024-03-24T10:49:23.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.494+0000 7f24c9d9b640 1 -- 172.21.15.5:0/3485563561 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b4031d80 con 0x7f24cc15ac00 2024-03-24T10:49:23.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.495+0000 7f24d0e23640 1 -- 172.21.15.5:0/3485563561 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f24cc15ac00 msgr2=0x7f24cc0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:23.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.495+0000 7f24d0e23640 1 --2- 172.21.15.5:0/3485563561 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f24cc15ac00 0x7f24cc0cc560 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f24b40097d0 tx=0x7f24b402b930 comp rx=0 tx=0).stop 2024-03-24T10:49:23.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.495+0000 7f24d0e23640 1 -- 172.21.15.5:0/3485563561 shutdown_connections 2024-03-24T10:49:23.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.495+0000 7f24d0e23640 1 --2- 172.21.15.5:0/3485563561 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f24cc15ac00 0x7f24cc0cc560 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.495+0000 7f24d0e23640 1 -- 172.21.15.5:0/3485563561 >> 172.21.15.5:0/3485563561 conn(0x7f24cc072ef0 msgr2=0x7f24cc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:23.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.495+0000 7f24d0e23640 1 -- 172.21.15.5:0/3485563561 shutdown_connections 2024-03-24T10:49:23.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.495+0000 7f24d0e23640 1 -- 172.21.15.5:0/3485563561 wait complete. 2024-03-24T10:49:23.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.496+0000 7f24d0e23640 1 Processor -- start 2024-03-24T10:49:23.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.496+0000 7f24d0e23640 1 -- start start 2024-03-24T10:49:23.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.497+0000 7f24d0e23640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f24cc0ccaa0 0x7f24cc0b3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.497+0000 7f24d0e23640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f24cc1543a0 0x7f24cc158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.497+0000 7f24d0e23640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24cc15ac00 0x7f24cc1592c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.497+0000 7f24d0e23640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc158960 con 0x7f24cc0ccaa0 2024-03-24T10:49:23.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.497+0000 7f24d0e23640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc0cf210 con 0x7f24cc15ac00 2024-03-24T10:49:23.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.497+0000 7f24d0e23640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f24cc1ef940 con 0x7f24cc1543a0 2024-03-24T10:49:23.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.497+0000 7f24cad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24cc15ac00 0x7f24cc1592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.497+0000 7f24ca59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f24cc0ccaa0 0x7f24cc0b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.497+0000 7f24c3fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f24cc1543a0 0x7f24cc158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.498+0000 7f24cad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24cc15ac00 0x7f24cc1592c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:35952/0 (socket says 172.21.15.5:35952) 2024-03-24T10:49:23.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.498+0000 7f24cad9d640 1 -- 172.21.15.5:0/4031489853 learned_addr learned my addr 172.21.15.5:0/4031489853 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:23.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.501+0000 7f24cad9d640 1 -- 172.21.15.5:0/4031489853 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f24cc1543a0 msgr2=0x7f24cc158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:23.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.501+0000 7f24cad9d640 1 --2- 172.21.15.5:0/4031489853 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f24cc1543a0 0x7f24cc158d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.501+0000 7f24cad9d640 1 -- 172.21.15.5:0/4031489853 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f24cc0ccaa0 msgr2=0x7f24cc0b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:23.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.501+0000 7f24cad9d640 1 --2- 172.21.15.5:0/4031489853 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f24cc0ccaa0 0x7f24cc0b3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:23.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.501+0000 7f24cad9d640 1 -- 172.21.15.5:0/4031489853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b4009400 con 0x7f24cc15ac00 2024-03-24T10:49:23.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.504+0000 7f24cad9d640 1 --2- 172.21.15.5:0/4031489853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24cc15ac00 0x7f24cc1592c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f24c400cbc0 tx=0x7f24c400a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:23.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.509+0000 7f24c2ffd640 1 -- 172.21.15.5:0/4031489853 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c400ad40 con 0x7f24cc15ac00 2024-03-24T10:49:23.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.509+0000 7f24c2ffd640 1 -- 172.21.15.5:0/4031489853 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24c4012400 con 0x7f24cc15ac00 2024-03-24T10:49:23.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.509+0000 7f24c2ffd640 1 -- 172.21.15.5:0/4031489853 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c400aea0 con 0x7f24cc15ac00 2024-03-24T10:49:23.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.509+0000 7f24d0e23640 1 -- 172.21.15.5:0/4031489853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24cc1efb50 con 0x7f24cc15ac00 2024-03-24T10:49:23.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.509+0000 7f24d0e23640 1 -- 172.21.15.5:0/4031489853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24cc1f0110 con 0x7f24cc15ac00 2024-03-24T10:49:23.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.510+0000 7f24d0e23640 1 -- 172.21.15.5:0/4031489853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24cc158aa0 con 0x7f24cc15ac00 2024-03-24T10:49:23.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.800+0000 7f24c2ffd640 1 -- 172.21.15.5:0/4031489853 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f24c4007070 con 0x7f24cc15ac00 2024-03-24T10:49:23.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.800+0000 7f24c2ffd640 1 --2- 172.21.15.5:0/4031489853 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f24ac06b890 0x7f24ac06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:23.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.800+0000 7f24ca59c640 1 --2- 172.21.15.5:0/4031489853 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f24ac06b890 0x7f24ac06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:23.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.801+0000 7f24ca59c640 1 --2- 172.21.15.5:0/4031489853 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f24ac06b890 0x7f24ac06dd50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f24bc007fd0 tx=0x7f24bc009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:23.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.804+0000 7f24c2ffd640 1 -- 172.21.15.5:0/4031489853 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f24c4058c40 con 0x7f24cc15ac00 2024-03-24T10:49:23.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:23.816+0000 7f24c2ffd640 1 -- 172.21.15.5:0/4031489853 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f24c4054ee0 con 0x7f24cc15ac00 2024-03-24T10:49:24.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:24.042+0000 7f24d0e23640 1 -- 172.21.15.5:0/4031489853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24cc158aa0 con 0x7f24cc15ac00 2024-03-24T10:49:24.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:24.076+0000 7f24c2ffd640 1 -- 172.21.15.5:0/4031489853 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f24cc158aa0 con 0x7f24cc15ac00 2024-03-24T10:49:24.076 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:24.076 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:24.076 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:24.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:24.079+0000 7f24d0e23640 1 -- 172.21.15.5:0/4031489853 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f24ac06b890 msgr2=0x7f24ac06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:24.079+0000 7f24d0e23640 1 --2- 172.21.15.5:0/4031489853 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f24ac06b890 0x7f24ac06dd50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f24bc007fd0 tx=0x7f24bc009040 comp rx=0 tx=0).stop 2024-03-24T10:49:24.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:24.079+0000 7f24d0e23640 1 -- 172.21.15.5:0/4031489853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24cc15ac00 msgr2=0x7f24cc1592c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:24.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:24.079+0000 7f24d0e23640 1 --2- 172.21.15.5:0/4031489853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24cc15ac00 0x7f24cc1592c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f24c400cbc0 tx=0x7f24c400a430 comp rx=0 tx=0).stop 2024-03-24T10:49:24.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:24.079+0000 7f24d0e23640 1 -- 172.21.15.5:0/4031489853 shutdown_connections 2024-03-24T10:49:24.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:24.079+0000 7f24d0e23640 1 --2- 172.21.15.5:0/4031489853 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f24ac06b890 0x7f24ac06dd50 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:24.079+0000 7f24d0e23640 1 --2- 172.21.15.5:0/4031489853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f24cc15ac00 0x7f24cc1592c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:24.079+0000 7f24d0e23640 1 --2- 172.21.15.5:0/4031489853 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f24cc1543a0 0x7f24cc158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:24.079+0000 7f24d0e23640 1 --2- 172.21.15.5:0/4031489853 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f24cc0ccaa0 0x7f24cc0b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:24.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:24.079+0000 7f24d0e23640 1 -- 172.21.15.5:0/4031489853 >> 172.21.15.5:0/4031489853 conn(0x7f24cc072ef0 msgr2=0x7f24cc1494c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:24.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:24.079+0000 7f24d0e23640 1 -- 172.21.15.5:0/4031489853 shutdown_connections 2024-03-24T10:49:24.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:24.079+0000 7f24d0e23640 1 -- 172.21.15.5:0/4031489853 wait complete. 2024-03-24T10:49:24.694 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:24.694 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:25.695 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:25.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.796+0000 7ffa7afc7640 1 Processor -- start 2024-03-24T10:49:25.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.796+0000 7ffa7afc7640 1 -- start start 2024-03-24T10:49:25.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.797+0000 7ffa7afc7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ffa741543a0 0x7ffa7415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.797+0000 7ffa7afc7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ffa7406d4f0 0x7ffa740cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.797+0000 7ffa7afc7640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa740cb2f0 con 0x7ffa740ccaa0 2024-03-24T10:49:25.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.797+0000 7ffa7afc7640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7ffa740c6750 con 0x7ffa741543a0 2024-03-24T10:49:25.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.797+0000 7ffa7afc7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ffa740cc380 con 0x7ffa7406d4f0 2024-03-24T10:49:25.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.798+0000 7ffa78d63640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ffa741543a0 0x7ffa7415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.798+0000 7ffa79564640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ffa740ccaa0 0x7ffa7406bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:60498/0 (socket says 172.21.15.5:60498) 2024-03-24T10:49:25.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.798+0000 7ffa79564640 1 -- 172.21.15.5:0/2312451544 learned_addr learned my addr 172.21.15.5:0/2312451544 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:49:25.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.798+0000 7ffa73fff640 1 --2- 172.21.15.5:0/2312451544 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ffa7406d4f0 0x7ffa740cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.800+0000 7ffa78d63640 1 -- 172.21.15.5:0/2312451544 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ffa7406d4f0 msgr2=0x7ffa740cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:25.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.800+0000 7ffa78d63640 1 --2- 172.21.15.5:0/2312451544 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ffa7406d4f0 0x7ffa740cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.801+0000 7ffa78d63640 1 -- 172.21.15.5:0/2312451544 >> v1:172.21.15.57:6789/0 conn(0x7ffa740ccaa0 legacy=0x7ffa7406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.801+0000 7ffa78d63640 1 -- 172.21.15.5:0/2312451544 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa740ca7e0 con 0x7ffa741543a0 2024-03-24T10:49:25.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.803+0000 7ffa78d63640 1 --2- 172.21.15.5:0/2312451544 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa741543a0 0x7ffa7415ac00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c009d90 tx=0x7ffa6c02bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57186383a7fab018 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:25.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.807+0000 7ffa737fe640 1 -- 172.21.15.5:0/2312451544 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c02cce0 con 0x7ffa741543a0 2024-03-24T10:49:25.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.807+0000 7ffa737fe640 1 -- 172.21.15.5:0/2312451544 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa6c02ce40 con 0x7ffa741543a0 2024-03-24T10:49:25.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.807+0000 7ffa737fe640 1 -- 172.21.15.5:0/2312451544 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c034660 con 0x7ffa741543a0 2024-03-24T10:49:25.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.807+0000 7ffa7afc7640 1 -- 172.21.15.5:0/2312451544 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa741543a0 msgr2=0x7ffa7415ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.807+0000 7ffa7afc7640 1 --2- 172.21.15.5:0/2312451544 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa741543a0 0x7ffa7415ac00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c009d90 tx=0x7ffa6c02bf30 comp rx=0 tx=0).stop 2024-03-24T10:49:25.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.807+0000 7ffa7afc7640 1 -- 172.21.15.5:0/2312451544 shutdown_connections 2024-03-24T10:49:25.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.807+0000 7ffa7afc7640 1 --2- 172.21.15.5:0/2312451544 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ffa7406d4f0 0x7ffa740cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.807+0000 7ffa7afc7640 1 --2- 172.21.15.5:0/2312451544 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa741543a0 0x7ffa7415ac00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.807+0000 7ffa7afc7640 1 -- 172.21.15.5:0/2312451544 >> 172.21.15.5:0/2312451544 conn(0x7ffa74072ef0 msgr2=0x7ffa74061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:25.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.807+0000 7ffa7afc7640 1 -- 172.21.15.5:0/2312451544 shutdown_connections 2024-03-24T10:49:25.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.807+0000 7ffa7afc7640 1 -- 172.21.15.5:0/2312451544 wait complete. 2024-03-24T10:49:25.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.809+0000 7ffa7afc7640 1 Processor -- start 2024-03-24T10:49:25.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.809+0000 7ffa7afc7640 1 -- start start 2024-03-24T10:49:25.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.809+0000 7ffa7afc7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa7406d4f0 0x7ffa740c5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.809+0000 7ffa7afc7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa740ccaa0 0x7ffa740c8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.809+0000 7ffa7afc7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ffa741543a0 0x7ffa740cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.809+0000 7ffa7afc7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ffa741f0360 con 0x7ffa740ccaa0 2024-03-24T10:49:25.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.809+0000 7ffa7afc7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ffa741efa50 con 0x7ffa7406d4f0 2024-03-24T10:49:25.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.809+0000 7ffa7afc7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ffa740cf800 con 0x7ffa741543a0 2024-03-24T10:49:25.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.810+0000 7ffa79564640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ffa741543a0 0x7ffa740cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.810+0000 7ffa73fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa740ccaa0 0x7ffa740c8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.811+0000 7ffa79564640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ffa741543a0 0x7ffa740cecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:41064/0 (socket says 172.21.15.5:41064) 2024-03-24T10:49:25.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.811+0000 7ffa79564640 1 -- 172.21.15.5:0/1498312422 learned_addr learned my addr 172.21.15.5:0/1498312422 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:25.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.811+0000 7ffa78d63640 1 --2- 172.21.15.5:0/1498312422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa7406d4f0 0x7ffa740c5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.815+0000 7ffa73fff640 1 -- 172.21.15.5:0/1498312422 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ffa741543a0 msgr2=0x7ffa740cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.815+0000 7ffa73fff640 1 --2- 172.21.15.5:0/1498312422 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ffa741543a0 0x7ffa740cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.815+0000 7ffa73fff640 1 -- 172.21.15.5:0/1498312422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa7406d4f0 msgr2=0x7ffa740c5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:25.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.815+0000 7ffa73fff640 1 --2- 172.21.15.5:0/1498312422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa7406d4f0 0x7ffa740c5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:25.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.815+0000 7ffa73fff640 1 -- 172.21.15.5:0/1498312422 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6c009a70 con 0x7ffa740ccaa0 2024-03-24T10:49:25.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.817+0000 7ffa73fff640 1 --2- 172.21.15.5:0/1498312422 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa740ccaa0 0x7ffa740c8420 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c00e850 tx=0x7ffa5c00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:25.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.821+0000 7ffa717fa640 1 -- 172.21.15.5:0/1498312422 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c0095b0 con 0x7ffa740ccaa0 2024-03-24T10:49:25.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.822+0000 7ffa717fa640 1 -- 172.21.15.5:0/1498312422 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa5c00c940 con 0x7ffa740ccaa0 2024-03-24T10:49:25.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.822+0000 7ffa717fa640 1 -- 172.21.15.5:0/1498312422 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c00ccd0 con 0x7ffa740ccaa0 2024-03-24T10:49:25.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.822+0000 7ffa7afc7640 1 -- 172.21.15.5:0/1498312422 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa741f3a70 con 0x7ffa740ccaa0 2024-03-24T10:49:25.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.822+0000 7ffa7afc7640 1 -- 172.21.15.5:0/1498312422 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa741efd10 con 0x7ffa740ccaa0 2024-03-24T10:49:25.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.823+0000 7ffa7afc7640 1 -- 172.21.15.5:0/1498312422 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa741f0510 con 0x7ffa740ccaa0 2024-03-24T10:49:25.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.847+0000 7ffa717fa640 1 -- 172.21.15.5:0/1498312422 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ffa5c013470 con 0x7ffa740ccaa0 2024-03-24T10:49:25.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.848+0000 7ffa717fa640 1 --2- 172.21.15.5:0/1498312422 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ffa5406b890 0x7ffa5406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:25.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.848+0000 7ffa78d63640 1 --2- 172.21.15.5:0/1498312422 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ffa5406b890 0x7ffa5406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:25.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:25.848+0000 7ffa78d63640 1 --2- 172.21.15.5:0/1498312422 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ffa5406b890 0x7ffa5406dd50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c036000 tx=0x7ffa6c030000 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:26.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:26.123+0000 7ffa717fa640 1 -- 172.21.15.5:0/1498312422 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7ffa5c00ce30 con 0x7ffa740ccaa0 2024-03-24T10:49:26.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:26.123+0000 7ffa717fa640 1 -- 172.21.15.5:0/1498312422 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffa5c055370 con 0x7ffa740ccaa0 2024-03-24T10:49:26.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:26.363+0000 7ffa7afc7640 1 -- 172.21.15.5:0/1498312422 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa740681d0 con 0x7ffa740ccaa0 2024-03-24T10:49:26.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:26.384+0000 7ffa717fa640 1 -- 172.21.15.5:0/1498312422 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7ffa5c021090 con 0x7ffa740ccaa0 2024-03-24T10:49:26.384 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:26.384 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:26.385 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:26.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:26.387+0000 7ffa7afc7640 1 -- 172.21.15.5:0/1498312422 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ffa5406b890 msgr2=0x7ffa5406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:26.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:26.387+0000 7ffa7afc7640 1 --2- 172.21.15.5:0/1498312422 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ffa5406b890 0x7ffa5406dd50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c036000 tx=0x7ffa6c030000 comp rx=0 tx=0).stop 2024-03-24T10:49:26.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:26.387+0000 7ffa7afc7640 1 -- 172.21.15.5:0/1498312422 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa740ccaa0 msgr2=0x7ffa740c8420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:26.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:26.387+0000 7ffa7afc7640 1 --2- 172.21.15.5:0/1498312422 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa740ccaa0 0x7ffa740c8420 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c00e850 tx=0x7ffa5c00ed10 comp rx=0 tx=0).stop 2024-03-24T10:49:26.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:26.387+0000 7ffa7afc7640 1 -- 172.21.15.5:0/1498312422 shutdown_connections 2024-03-24T10:49:26.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:26.387+0000 7ffa7afc7640 1 --2- 172.21.15.5:0/1498312422 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ffa5406b890 0x7ffa5406dd50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:26.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:26.387+0000 7ffa7afc7640 1 --2- 172.21.15.5:0/1498312422 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ffa740ccaa0 0x7ffa740c8420 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:26.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:26.387+0000 7ffa7afc7640 1 --2- 172.21.15.5:0/1498312422 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ffa741543a0 0x7ffa740cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:26.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:26.387+0000 7ffa7afc7640 1 --2- 172.21.15.5:0/1498312422 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ffa7406d4f0 0x7ffa740c5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:26.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:26.387+0000 7ffa7afc7640 1 -- 172.21.15.5:0/1498312422 >> 172.21.15.5:0/1498312422 conn(0x7ffa74072ef0 msgr2=0x7ffa740c4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:26.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:26.387+0000 7ffa7afc7640 1 -- 172.21.15.5:0/1498312422 shutdown_connections 2024-03-24T10:49:26.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:26.387+0000 7ffa7afc7640 1 -- 172.21.15.5:0/1498312422 wait complete. 2024-03-24T10:49:27.002 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:27.002 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:28.003 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:28.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.104+0000 7f1ee0f9a640 1 Processor -- start 2024-03-24T10:49:28.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.104+0000 7f1ee0f9a640 1 -- start start 2024-03-24T10:49:28.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.105+0000 7f1ee0f9a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1edc15ac00 0x7f1edc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:28.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.105+0000 7f1ee0f9a640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1edc0ccaa0 0x7f1edc06bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:28.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.105+0000 7f1ee0f9a640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1edc0cb2f0 con 0x7f1edc1543a0 2024-03-24T10:49:28.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.105+0000 7f1ee0f9a640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f1edc0c6750 con 0x7f1edc0ccaa0 2024-03-24T10:49:28.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.105+0000 7f1ee0f9a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1edc0cc380 con 0x7f1edc15ac00 2024-03-24T10:49:28.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.106+0000 7f1edad9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1edc0ccaa0 0x7f1edc06bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:28.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.106+0000 7f1ed9d9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1edc15ac00 0x7f1edc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:28.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.106+0000 7f1eda59c640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f1edc1543a0 0x7f1edc06d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:50764/0 (socket says 172.21.15.5:50764) 2024-03-24T10:49:28.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.106+0000 7f1eda59c640 1 -- 172.21.15.5:0/4087291723 learned_addr learned my addr 172.21.15.5:0/4087291723 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:49:28.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.109+0000 7f1ed9d9b640 1 -- 172.21.15.5:0/4087291723 >> v2:172.21.15.57:3300/0 conn(0x7f1edc0ccaa0 msgr2=0x7f1edc06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:28.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.109+0000 7f1ed9d9b640 1 --2- 172.21.15.5:0/4087291723 >> v2:172.21.15.57:3300/0 conn(0x7f1edc0ccaa0 0x7f1edc06bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:28.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.109+0000 7f1ed9d9b640 1 -- 172.21.15.5:0/4087291723 >> v1:172.21.15.5:6789/0 conn(0x7f1edc1543a0 legacy=0x7f1edc06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:28.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.109+0000 7f1ed9d9b640 1 -- 172.21.15.5:0/4087291723 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1edc15b070 con 0x7f1edc15ac00 2024-03-24T10:49:28.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.111+0000 7f1ed9d9b640 1 --2- 172.21.15.5:0/4087291723 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1edc15ac00 0x7f1edc0cc560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1ed00097a0 tx=0x7f1ed002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93172506dca5ab72 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:28.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.116+0000 7f1ed959a640 1 -- 172.21.15.5:0/4087291723 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0031830 con 0x7f1edc15ac00 2024-03-24T10:49:28.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.116+0000 7f1ed959a640 1 -- 172.21.15.5:0/4087291723 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ed0031990 con 0x7f1edc15ac00 2024-03-24T10:49:28.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.116+0000 7f1ed959a640 1 -- 172.21.15.5:0/4087291723 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0031d20 con 0x7f1edc15ac00 2024-03-24T10:49:28.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.117+0000 7f1ee0f9a640 1 -- 172.21.15.5:0/4087291723 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1edc15ac00 msgr2=0x7f1edc0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:28.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.117+0000 7f1ee0f9a640 1 --2- 172.21.15.5:0/4087291723 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1edc15ac00 0x7f1edc0cc560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1ed00097a0 tx=0x7f1ed002b8d0 comp rx=0 tx=0).stop 2024-03-24T10:49:28.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.117+0000 7f1ee0f9a640 1 -- 172.21.15.5:0/4087291723 shutdown_connections 2024-03-24T10:49:28.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.117+0000 7f1ee0f9a640 1 --2- 172.21.15.5:0/4087291723 >> v2:172.21.15.57:3300/0 conn(0x7f1edc0ccaa0 0x7f1edc06bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:28.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.117+0000 7f1ee0f9a640 1 --2- 172.21.15.5:0/4087291723 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1edc15ac00 0x7f1edc0cc560 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:28.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.117+0000 7f1ee0f9a640 1 -- 172.21.15.5:0/4087291723 >> 172.21.15.5:0/4087291723 conn(0x7f1edc072ef0 msgr2=0x7f1edc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:28.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.117+0000 7f1ee0f9a640 1 -- 172.21.15.5:0/4087291723 shutdown_connections 2024-03-24T10:49:28.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.117+0000 7f1ee0f9a640 1 -- 172.21.15.5:0/4087291723 wait complete. 2024-03-24T10:49:28.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.119+0000 7f1ee0f9a640 1 Processor -- start 2024-03-24T10:49:28.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.119+0000 7f1ee0f9a640 1 -- start start 2024-03-24T10:49:28.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.120+0000 7f1ee0f9a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1edc0ccaa0 0x7f1edc0cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:28.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.120+0000 7f1ee0f9a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1edc1543a0 0x7f1edc14c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:28.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.120+0000 7f1ee0f9a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1edc15ac00 0x7f1edc1f3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:28.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.120+0000 7f1ee0f9a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc14b080 con 0x7f1edc1543a0 2024-03-24T10:49:28.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.120+0000 7f1ee0f9a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc14b320 con 0x7f1edc15ac00 2024-03-24T10:49:28.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.120+0000 7f1ee0f9a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1edc14b600 con 0x7f1edc0ccaa0 2024-03-24T10:49:28.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.121+0000 7f1ed9d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1edc1543a0 0x7f1edc14c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:28.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.121+0000 7f1eda59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1edc0ccaa0 0x7f1edc0cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:28.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.121+0000 7f1edad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1edc15ac00 0x7f1edc1f3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:28.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.121+0000 7f1eda59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1edc0ccaa0 0x7f1edc0cecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:41086/0 (socket says 172.21.15.5:41086) 2024-03-24T10:49:28.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.121+0000 7f1eda59c640 1 -- 172.21.15.5:0/4189416249 learned_addr learned my addr 172.21.15.5:0/4189416249 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:28.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.125+0000 7f1eda59c640 1 -- 172.21.15.5:0/4189416249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1edc15ac00 msgr2=0x7f1edc1f3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:28.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.125+0000 7f1eda59c640 1 --2- 172.21.15.5:0/4189416249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1edc15ac00 0x7f1edc1f3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:28.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.125+0000 7f1eda59c640 1 -- 172.21.15.5:0/4189416249 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1edc1543a0 msgr2=0x7f1edc14c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:28.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.125+0000 7f1eda59c640 1 --2- 172.21.15.5:0/4189416249 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1edc1543a0 0x7f1edc14c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:28.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.125+0000 7f1eda59c640 1 -- 172.21.15.5:0/4189416249 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed0009400 con 0x7f1edc0ccaa0 2024-03-24T10:49:28.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.128+0000 7f1eda59c640 1 --2- 172.21.15.5:0/4189416249 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1edc0ccaa0 0x7f1edc0cecc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc00c400 tx=0x7f1ecc00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:28.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.132+0000 7f1ecaffd640 1 -- 172.21.15.5:0/4189416249 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc005ce0 con 0x7f1edc0ccaa0 2024-03-24T10:49:28.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.132+0000 7f1ecaffd640 1 -- 172.21.15.5:0/4189416249 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ecc005e40 con 0x7f1edc0ccaa0 2024-03-24T10:49:28.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.132+0000 7f1ee0f9a640 1 -- 172.21.15.5:0/4189416249 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1edc14b8e0 con 0x7f1edc0ccaa0 2024-03-24T10:49:28.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.132+0000 7f1ecaffd640 1 -- 172.21.15.5:0/4189416249 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc0072d0 con 0x7f1edc0ccaa0 2024-03-24T10:49:28.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.132+0000 7f1ee0f9a640 1 -- 172.21.15.5:0/4189416249 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1edc0cf200 con 0x7f1edc0ccaa0 2024-03-24T10:49:28.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.133+0000 7f1ee0f9a640 1 -- 172.21.15.5:0/4189416249 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1edc0681d0 con 0x7f1edc0ccaa0 2024-03-24T10:49:28.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.152+0000 7f1ecaffd640 1 -- 172.21.15.5:0/4189416249 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1ecc007430 con 0x7f1edc0ccaa0 2024-03-24T10:49:28.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.153+0000 7f1ecaffd640 1 --2- 172.21.15.5:0/4189416249 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1ebc06b890 0x7f1ebc06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:28.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.153+0000 7f1ed9d9b640 1 --2- 172.21.15.5:0/4189416249 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1ebc06b890 0x7f1ebc06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:28.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.153+0000 7f1ed9d9b640 1 --2- 172.21.15.5:0/4189416249 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1ebc06b890 0x7f1ebc06dd50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0036000 tx=0x7f1ed003c040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:28.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.407+0000 7f1ecaffd640 1 -- 172.21.15.5:0/4189416249 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f1ecc058430 con 0x7f1edc0ccaa0 2024-03-24T10:49:28.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.418+0000 7f1ecaffd640 1 -- 172.21.15.5:0/4189416249 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1ecc0546d0 con 0x7f1edc0ccaa0 2024-03-24T10:49:28.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.659+0000 7f1ee0f9a640 1 -- 172.21.15.5:0/4189416249 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1edc14b740 con 0x7f1edc0ccaa0 2024-03-24T10:49:28.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.674+0000 7f1ecaffd640 1 -- 172.21.15.5:0/4189416249 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f1edc14b740 con 0x7f1edc0ccaa0 2024-03-24T10:49:28.675 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:28.675 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:28.675 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:28.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.676+0000 7f1ee0f9a640 1 -- 172.21.15.5:0/4189416249 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1ebc06b890 msgr2=0x7f1ebc06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:28.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.676+0000 7f1ee0f9a640 1 --2- 172.21.15.5:0/4189416249 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1ebc06b890 0x7f1ebc06dd50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0036000 tx=0x7f1ed003c040 comp rx=0 tx=0).stop 2024-03-24T10:49:28.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.676+0000 7f1ee0f9a640 1 -- 172.21.15.5:0/4189416249 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1edc0ccaa0 msgr2=0x7f1edc0cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:28.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.676+0000 7f1ee0f9a640 1 --2- 172.21.15.5:0/4189416249 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1edc0ccaa0 0x7f1edc0cecc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc00c400 tx=0x7f1ecc00c8c0 comp rx=0 tx=0).stop 2024-03-24T10:49:28.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.676+0000 7f1ee0f9a640 1 -- 172.21.15.5:0/4189416249 shutdown_connections 2024-03-24T10:49:28.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.677+0000 7f1ee0f9a640 1 --2- 172.21.15.5:0/4189416249 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1ebc06b890 0x7f1ebc06dd50 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:28.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.677+0000 7f1ee0f9a640 1 --2- 172.21.15.5:0/4189416249 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1edc1543a0 0x7f1edc14c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:28.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.677+0000 7f1ee0f9a640 1 --2- 172.21.15.5:0/4189416249 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1edc15ac00 0x7f1edc1f3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:28.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.677+0000 7f1ee0f9a640 1 --2- 172.21.15.5:0/4189416249 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1edc0ccaa0 0x7f1edc0cecc0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:28.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.677+0000 7f1ee0f9a640 1 -- 172.21.15.5:0/4189416249 >> 172.21.15.5:0/4189416249 conn(0x7f1edc072ef0 msgr2=0x7f1edc158d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:28.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.677+0000 7f1ee0f9a640 1 -- 172.21.15.5:0/4189416249 shutdown_connections 2024-03-24T10:49:28.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:28.677+0000 7f1ee0f9a640 1 -- 172.21.15.5:0/4189416249 wait complete. 2024-03-24T10:49:29.295 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:29.295 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:30.296 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:30.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.401+0000 7f0241713640 1 Processor -- start 2024-03-24T10:49:30.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.401+0000 7f0241713640 1 -- start start 2024-03-24T10:49:30.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.402+0000 7f0241713640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f023c1543a0 0x7f023c15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:30.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.402+0000 7f0241713640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f023c06d4f0 0x7f023c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:30.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.402+0000 7f0241713640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f023c0cb2f0 con 0x7f023c0ccaa0 2024-03-24T10:49:30.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.402+0000 7f0241713640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f023c0c6750 con 0x7f023c1543a0 2024-03-24T10:49:30.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.402+0000 7f0241713640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f023c0cc380 con 0x7f023c06d4f0 2024-03-24T10:49:30.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.403+0000 7f023a7fc640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f023c06d4f0 0x7f023c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:30.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.403+0000 7f023affd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f023c1543a0 0x7f023c15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:30.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.403+0000 7f023b7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f023c0ccaa0 0x7f023c06bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:60508/0 (socket says 172.21.15.5:60508) 2024-03-24T10:49:30.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.403+0000 7f023b7fe640 1 -- 172.21.15.5:0/1153026125 learned_addr learned my addr 172.21.15.5:0/1153026125 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:49:30.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.406+0000 7f0239ffb640 1 -- 172.21.15.5:0/1153026125 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3691743450 0 0) 0x7f023c0cb2f0 con 0x7f023c0ccaa0 2024-03-24T10:49:30.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.406+0000 7f0239ffb640 1 -- 172.21.15.5:0/1153026125 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0224003520 con 0x7f023c0ccaa0 2024-03-24T10:49:30.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.406+0000 7f023affd640 1 -- 172.21.15.5:0/1153026125 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f023c06d4f0 msgr2=0x7f023c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:30.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.406+0000 7f023affd640 1 --2- 172.21.15.5:0/1153026125 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f023c06d4f0 0x7f023c0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:30.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.406+0000 7f023affd640 1 -- 172.21.15.5:0/1153026125 >> v1:172.21.15.57:6789/0 conn(0x7f023c0ccaa0 legacy=0x7f023c06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:30.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.407+0000 7f023affd640 1 -- 172.21.15.5:0/1153026125 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f023c0ca7e0 con 0x7f023c1543a0 2024-03-24T10:49:30.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.409+0000 7f023affd640 1 --2- 172.21.15.5:0/1153026125 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f023c1543a0 0x7f023c15ac00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f022c0097a0 tx=0x7f022c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b6bb47c6a96cd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:30.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.414+0000 7f0239ffb640 1 -- 172.21.15.5:0/1153026125 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c030610 con 0x7f023c1543a0 2024-03-24T10:49:30.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.414+0000 7f0239ffb640 1 -- 172.21.15.5:0/1153026125 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f022c030770 con 0x7f023c1543a0 2024-03-24T10:49:30.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.414+0000 7f0239ffb640 1 -- 172.21.15.5:0/1153026125 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022c030b00 con 0x7f023c1543a0 2024-03-24T10:49:30.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.414+0000 7f0241713640 1 -- 172.21.15.5:0/1153026125 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f023c1543a0 msgr2=0x7f023c15ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:30.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.414+0000 7f0241713640 1 --2- 172.21.15.5:0/1153026125 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f023c1543a0 0x7f023c15ac00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f022c0097a0 tx=0x7f022c02b960 comp rx=0 tx=0).stop 2024-03-24T10:49:30.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.414+0000 7f0241713640 1 -- 172.21.15.5:0/1153026125 shutdown_connections 2024-03-24T10:49:30.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.414+0000 7f0241713640 1 --2- 172.21.15.5:0/1153026125 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f023c06d4f0 0x7f023c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:30.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.414+0000 7f0241713640 1 --2- 172.21.15.5:0/1153026125 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f023c1543a0 0x7f023c15ac00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:30.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.414+0000 7f0241713640 1 -- 172.21.15.5:0/1153026125 >> 172.21.15.5:0/1153026125 conn(0x7f023c072ef0 msgr2=0x7f023c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:30.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.414+0000 7f0241713640 1 -- 172.21.15.5:0/1153026125 shutdown_connections 2024-03-24T10:49:30.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.414+0000 7f0241713640 1 -- 172.21.15.5:0/1153026125 wait complete. 2024-03-24T10:49:30.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.416+0000 7f0241713640 1 Processor -- start 2024-03-24T10:49:30.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.416+0000 7f0241713640 1 -- start start 2024-03-24T10:49:30.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.417+0000 7f0241713640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f023c06d4f0 0x7f023c1f0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:30.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.417+0000 7f0241713640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f023c0ccaa0 0x7f023c0cec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:30.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.417+0000 7f0241713640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f023c1543a0 0x7f023c154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:30.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.417+0000 7f0241713640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f023c157f60 con 0x7f023c0ccaa0 2024-03-24T10:49:30.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.417+0000 7f0241713640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f023c1f0eb0 con 0x7f023c06d4f0 2024-03-24T10:49:30.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.417+0000 7f0241713640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f023c158190 con 0x7f023c1543a0 2024-03-24T10:49:30.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.417+0000 7f023a7fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f023c0ccaa0 0x7f023c0cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:30.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.417+0000 7f023b7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f023c1543a0 0x7f023c154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:30.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.417+0000 7f023affd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f023c06d4f0 0x7f023c1f0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:30.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.418+0000 7f023a7fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f023c0ccaa0 0x7f023c0cec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50008/0 (socket says 172.21.15.5:50008) 2024-03-24T10:49:30.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.418+0000 7f023a7fc640 1 -- 172.21.15.5:0/1936791979 learned_addr learned my addr 172.21.15.5:0/1936791979 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:30.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.421+0000 7f023a7fc640 1 -- 172.21.15.5:0/1936791979 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f023c1543a0 msgr2=0x7f023c154d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:30.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.421+0000 7f023a7fc640 1 --2- 172.21.15.5:0/1936791979 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f023c1543a0 0x7f023c154d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:30.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.421+0000 7f023a7fc640 1 -- 172.21.15.5:0/1936791979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f023c06d4f0 msgr2=0x7f023c1f0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:30.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.421+0000 7f023a7fc640 1 --2- 172.21.15.5:0/1936791979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f023c06d4f0 0x7f023c1f0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:30.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.421+0000 7f023a7fc640 1 -- 172.21.15.5:0/1936791979 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f022c009400 con 0x7f023c0ccaa0 2024-03-24T10:49:30.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.424+0000 7f023a7fc640 1 --2- 172.21.15.5:0/1936791979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f023c0ccaa0 0x7f023c0cec00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f023000e9b0 tx=0x7f023000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:30.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.426+0000 7f02237fe640 1 -- 172.21.15.5:0/1936791979 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0230009670 con 0x7f023c0ccaa0 2024-03-24T10:49:30.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.426+0000 7f0241713640 1 -- 172.21.15.5:0/1936791979 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f023c158470 con 0x7f023c0ccaa0 2024-03-24T10:49:30.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.426+0000 7f0241713640 1 -- 172.21.15.5:0/1936791979 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f023c158a30 con 0x7f023c0ccaa0 2024-03-24T10:49:30.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.427+0000 7f0241713640 1 -- 172.21.15.5:0/1936791979 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f023c0681d0 con 0x7f023c0ccaa0 2024-03-24T10:49:30.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.428+0000 7f02237fe640 1 -- 172.21.15.5:0/1936791979 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f023000c9a0 con 0x7f023c0ccaa0 2024-03-24T10:49:30.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.428+0000 7f02237fe640 1 -- 172.21.15.5:0/1936791979 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f023000cd10 con 0x7f023c0ccaa0 2024-03-24T10:49:30.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.717+0000 7f02237fe640 1 -- 172.21.15.5:0/1936791979 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0230013470 con 0x7f023c0ccaa0 2024-03-24T10:49:30.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.718+0000 7f02237fe640 1 --2- 172.21.15.5:0/1936791979 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f022806b840 0x7f022806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:30.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.718+0000 7f023affd640 1 --2- 172.21.15.5:0/1936791979 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f022806b840 0x7f022806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:30.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.719+0000 7f023affd640 1 --2- 172.21.15.5:0/1936791979 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f022806b840 0x7f022806dd00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f022c036000 tx=0x7f022c03b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:30.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.743+0000 7f02237fe640 1 -- 172.21.15.5:0/1936791979 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f0230059320 con 0x7f023c0ccaa0 2024-03-24T10:49:30.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.743+0000 7f02237fe640 1 -- 172.21.15.5:0/1936791979 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f02300555c0 con 0x7f023c0ccaa0 2024-03-24T10:49:30.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.971+0000 7f0241713640 1 -- 172.21.15.5:0/1936791979 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f023c0cc880 con 0x7f023c0ccaa0 2024-03-24T10:49:30.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.987+0000 7f02237fe640 1 -- 172.21.15.5:0/1936791979 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f0230021090 con 0x7f023c0ccaa0 2024-03-24T10:49:30.987 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:30.987 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:30.987 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:30.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.989+0000 7f0241713640 1 -- 172.21.15.5:0/1936791979 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f022806b840 msgr2=0x7f022806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:30.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.989+0000 7f0241713640 1 --2- 172.21.15.5:0/1936791979 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f022806b840 0x7f022806dd00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f022c036000 tx=0x7f022c03b040 comp rx=0 tx=0).stop 2024-03-24T10:49:30.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.989+0000 7f0241713640 1 -- 172.21.15.5:0/1936791979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f023c0ccaa0 msgr2=0x7f023c0cec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:30.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.989+0000 7f0241713640 1 --2- 172.21.15.5:0/1936791979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f023c0ccaa0 0x7f023c0cec00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f023000e9b0 tx=0x7f023000ee70 comp rx=0 tx=0).stop 2024-03-24T10:49:30.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.989+0000 7f0241713640 1 -- 172.21.15.5:0/1936791979 shutdown_connections 2024-03-24T10:49:30.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.989+0000 7f0241713640 1 --2- 172.21.15.5:0/1936791979 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f022806b840 0x7f022806dd00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:30.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.989+0000 7f0241713640 1 --2- 172.21.15.5:0/1936791979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f023c0ccaa0 0x7f023c0cec00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:30.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.989+0000 7f0241713640 1 --2- 172.21.15.5:0/1936791979 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f023c1543a0 0x7f023c154d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:30.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.989+0000 7f0241713640 1 --2- 172.21.15.5:0/1936791979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f023c06d4f0 0x7f023c1f0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:30.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.989+0000 7f0241713640 1 -- 172.21.15.5:0/1936791979 >> 172.21.15.5:0/1936791979 conn(0x7f023c072ef0 msgr2=0x7f023c153050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:30.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.989+0000 7f0241713640 1 -- 172.21.15.5:0/1936791979 shutdown_connections 2024-03-24T10:49:30.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:30.989+0000 7f0241713640 1 -- 172.21.15.5:0/1936791979 wait complete. 2024-03-24T10:49:31.604 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:31.604 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:31.707 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:49:31.710 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:49:32.604 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:32.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.706+0000 7fc4460f4640 1 Processor -- start 2024-03-24T10:49:32.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.706+0000 7fc4460f4640 1 -- start start 2024-03-24T10:49:32.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.707+0000 7fc4460f4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc44015ac00 0x7fc4400cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:32.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.707+0000 7fc4460f4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4400ccaa0 0x7fc44006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:32.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.707+0000 7fc4460f4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4400cb2f0 con 0x7fc4401543a0 2024-03-24T10:49:32.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.707+0000 7fc4460f4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fc4400c6750 con 0x7fc44015ac00 2024-03-24T10:49:32.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.707+0000 7fc4460f4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc4400cceb0 con 0x7fc4400ccaa0 2024-03-24T10:49:32.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.708+0000 7fc43effd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc44015ac00 0x7fc4400cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:32.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.708+0000 7fc43ffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4400ccaa0 0x7fc44006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:32.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.708+0000 7fc43f7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fc4401543a0 0x7fc44006d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:57090/0 (socket says 172.21.15.5:57090) 2024-03-24T10:49:32.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.708+0000 7fc43f7fe640 1 -- 172.21.15.5:0/3859662926 learned_addr learned my addr 172.21.15.5:0/3859662926 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:49:32.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.712+0000 7fc43ffff640 1 -- 172.21.15.5:0/3859662926 >> v2:172.21.15.57:3300/0 conn(0x7fc44015ac00 msgr2=0x7fc4400cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:32.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.712+0000 7fc43ffff640 1 --2- 172.21.15.5:0/3859662926 >> v2:172.21.15.57:3300/0 conn(0x7fc44015ac00 0x7fc4400cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:32.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.712+0000 7fc43ffff640 1 -- 172.21.15.5:0/3859662926 >> v1:172.21.15.57:6789/0 conn(0x7fc4401543a0 legacy=0x7fc44006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:32.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.712+0000 7fc43ffff640 1 -- 172.21.15.5:0/3859662926 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc44015b070 con 0x7fc4400ccaa0 2024-03-24T10:49:32.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.715+0000 7fc43ffff640 1 --2- 172.21.15.5:0/3859662926 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4400ccaa0 0x7fc44006bf50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc4380097a0 tx=0x7fc43802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edb3908f98e4fdf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:32.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.720+0000 7fc43e7fc640 1 -- 172.21.15.5:0/3859662926 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc438031830 con 0x7fc4400ccaa0 2024-03-24T10:49:32.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.721+0000 7fc43e7fc640 1 -- 172.21.15.5:0/3859662926 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc438031990 con 0x7fc4400ccaa0 2024-03-24T10:49:32.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.721+0000 7fc43e7fc640 1 -- 172.21.15.5:0/3859662926 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc438031d20 con 0x7fc4400ccaa0 2024-03-24T10:49:32.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.721+0000 7fc4460f4640 1 -- 172.21.15.5:0/3859662926 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4400ccaa0 msgr2=0x7fc44006bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:32.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.721+0000 7fc4460f4640 1 --2- 172.21.15.5:0/3859662926 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4400ccaa0 0x7fc44006bf50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc4380097a0 tx=0x7fc43802b8d0 comp rx=0 tx=0).stop 2024-03-24T10:49:32.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.722+0000 7fc4460f4640 1 -- 172.21.15.5:0/3859662926 shutdown_connections 2024-03-24T10:49:32.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.722+0000 7fc4460f4640 1 --2- 172.21.15.5:0/3859662926 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4400ccaa0 0x7fc44006bf50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:32.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.722+0000 7fc4460f4640 1 --2- 172.21.15.5:0/3859662926 >> v2:172.21.15.57:3300/0 conn(0x7fc44015ac00 0x7fc4400cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:32.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.722+0000 7fc4460f4640 1 -- 172.21.15.5:0/3859662926 >> 172.21.15.5:0/3859662926 conn(0x7fc440072ef0 msgr2=0x7fc440061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:32.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.722+0000 7fc4460f4640 1 -- 172.21.15.5:0/3859662926 shutdown_connections 2024-03-24T10:49:32.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.722+0000 7fc4460f4640 1 -- 172.21.15.5:0/3859662926 wait complete. 2024-03-24T10:49:32.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.723+0000 7fc4460f4640 1 Processor -- start 2024-03-24T10:49:32.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.724+0000 7fc4460f4640 1 -- start start 2024-03-24T10:49:32.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.725+0000 7fc4460f4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4400ccaa0 0x7fc4401f2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:32.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.725+0000 7fc4460f4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4401543a0 0x7fc4400ce730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:32.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.725+0000 7fc4460f4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc44015ac00 0x7fc4400cec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:32.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.725+0000 7fc4460f4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fc440159b40 con 0x7fc44015ac00 2024-03-24T10:49:32.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.725+0000 7fc4460f4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc4400b2810 con 0x7fc4401543a0 2024-03-24T10:49:32.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.725+0000 7fc4460f4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc440150b80 con 0x7fc4400ccaa0 2024-03-24T10:49:32.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.725+0000 7fc43effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4401543a0 0x7fc4400ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:32.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.725+0000 7fc43f7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4400ccaa0 0x7fc4401f2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:32.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.726+0000 7fc43ffff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc44015ac00 0x7fc4400cec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:32.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.726+0000 7fc43effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4401543a0 0x7fc4400ce730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:43666/0 (socket says 172.21.15.5:43666) 2024-03-24T10:49:32.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.726+0000 7fc43effd640 1 -- 172.21.15.5:0/1068671018 learned_addr learned my addr 172.21.15.5:0/1068671018 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:32.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.729+0000 7fc43effd640 1 -- 172.21.15.5:0/1068671018 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4400ccaa0 msgr2=0x7fc4401f2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:32.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.729+0000 7fc43effd640 1 --2- 172.21.15.5:0/1068671018 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4400ccaa0 0x7fc4401f2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:32.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.729+0000 7fc43effd640 1 -- 172.21.15.5:0/1068671018 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc44015ac00 msgr2=0x7fc4400cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:32.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.729+0000 7fc43effd640 1 --2- 172.21.15.5:0/1068671018 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc44015ac00 0x7fc4400cec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:32.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.729+0000 7fc43effd640 1 -- 172.21.15.5:0/1068671018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc438009400 con 0x7fc4401543a0 2024-03-24T10:49:32.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.732+0000 7fc43effd640 1 --2- 172.21.15.5:0/1068671018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4401543a0 0x7fc4400ce730 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc43400e870 tx=0x7fc43400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:32.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.736+0000 7fc427fff640 1 -- 172.21.15.5:0/1068671018 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4340094f0 con 0x7fc4401543a0 2024-03-24T10:49:32.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.736+0000 7fc427fff640 1 -- 172.21.15.5:0/1068671018 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc43400c940 con 0x7fc4401543a0 2024-03-24T10:49:32.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.736+0000 7fc427fff640 1 -- 172.21.15.5:0/1068671018 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43400ccf0 con 0x7fc4401543a0 2024-03-24T10:49:32.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.736+0000 7fc4460f4640 1 -- 172.21.15.5:0/1068671018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4401f3970 con 0x7fc4401543a0 2024-03-24T10:49:32.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.736+0000 7fc4460f4640 1 -- 172.21.15.5:0/1068671018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4400cd970 con 0x7fc4401543a0 2024-03-24T10:49:32.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.737+0000 7fc4460f4640 1 -- 172.21.15.5:0/1068671018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4400681d0 con 0x7fc4401543a0 2024-03-24T10:49:32.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.761+0000 7fc427fff640 1 -- 172.21.15.5:0/1068671018 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc434013470 con 0x7fc4401543a0 2024-03-24T10:49:32.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.762+0000 7fc427fff640 1 --2- 172.21.15.5:0/1068671018 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc42c06b890 0x7fc42c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:32.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.762+0000 7fc43f7fe640 1 --2- 172.21.15.5:0/1068671018 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc42c06b890 0x7fc42c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:32.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:32.762+0000 7fc43f7fe640 1 --2- 172.21.15.5:0/1068671018 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc42c06b890 0x7fc42c06dd50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc430005c00 tx=0x7fc430009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:33.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:33.071+0000 7fc427fff640 1 -- 172.21.15.5:0/1068671018 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fc434059160 con 0x7fc4401543a0 2024-03-24T10:49:33.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:33.083+0000 7fc427fff640 1 -- 172.21.15.5:0/1068671018 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc434055400 con 0x7fc4401543a0 2024-03-24T10:49:33.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:33.309+0000 7fc4460f4640 1 -- 172.21.15.5:0/1068671018 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc440157b30 con 0x7fc4401543a0 2024-03-24T10:49:33.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:33.323+0000 7fc427fff640 1 -- 172.21.15.5:0/1068671018 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fc434021020 con 0x7fc4401543a0 2024-03-24T10:49:33.324 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:33.324 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:33.324 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:33.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:33.325+0000 7fc4460f4640 1 -- 172.21.15.5:0/1068671018 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc42c06b890 msgr2=0x7fc42c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:33.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:33.325+0000 7fc4460f4640 1 --2- 172.21.15.5:0/1068671018 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc42c06b890 0x7fc42c06dd50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc430005c00 tx=0x7fc430009040 comp rx=0 tx=0).stop 2024-03-24T10:49:33.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:33.325+0000 7fc4460f4640 1 -- 172.21.15.5:0/1068671018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4401543a0 msgr2=0x7fc4400ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:33.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:33.325+0000 7fc4460f4640 1 --2- 172.21.15.5:0/1068671018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4401543a0 0x7fc4400ce730 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc43400e870 tx=0x7fc43400ed30 comp rx=0 tx=0).stop 2024-03-24T10:49:33.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:33.325+0000 7fc4460f4640 1 -- 172.21.15.5:0/1068671018 shutdown_connections 2024-03-24T10:49:33.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:33.325+0000 7fc4460f4640 1 --2- 172.21.15.5:0/1068671018 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc42c06b890 0x7fc42c06dd50 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:33.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:33.325+0000 7fc4460f4640 1 --2- 172.21.15.5:0/1068671018 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc44015ac00 0x7fc4400cec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:33.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:33.325+0000 7fc4460f4640 1 --2- 172.21.15.5:0/1068671018 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4401543a0 0x7fc4400ce730 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:33.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:33.325+0000 7fc4460f4640 1 --2- 172.21.15.5:0/1068671018 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4400ccaa0 0x7fc4401f2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:33.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:33.325+0000 7fc4460f4640 1 -- 172.21.15.5:0/1068671018 >> 172.21.15.5:0/1068671018 conn(0x7fc440072ef0 msgr2=0x7fc4401533a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:33.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:33.325+0000 7fc4460f4640 1 -- 172.21.15.5:0/1068671018 shutdown_connections 2024-03-24T10:49:33.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:33.325+0000 7fc4460f4640 1 -- 172.21.15.5:0/1068671018 wait complete. 2024-03-24T10:49:33.941 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:33.941 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:34.942 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:35.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.045+0000 7f70f815c640 1 Processor -- start 2024-03-24T10:49:35.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.045+0000 7f70f815c640 1 -- start start 2024-03-24T10:49:35.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.046+0000 7f70f815c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f70f015ac00 0x7f70f00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:35.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.046+0000 7f70f815c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70f00ccaa0 0x7f70f006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:35.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.046+0000 7f70f815c640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70f00cb2f0 con 0x7f70f01543a0 2024-03-24T10:49:35.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.046+0000 7f70f815c640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f70f00c6750 con 0x7f70f015ac00 2024-03-24T10:49:35.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.046+0000 7f70f815c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f70f00cceb0 con 0x7f70f00ccaa0 2024-03-24T10:49:35.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.046+0000 7f70f66f9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70f00ccaa0 0x7f70f006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:35.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.047+0000 7f70f56f7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f70f015ac00 0x7f70f00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:35.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.047+0000 7f70f56f7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f70f015ac00 0x7f70f00cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:58742/0 (socket says 172.21.15.5:58742) 2024-03-24T10:49:35.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.047+0000 7f70f66f9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70f00ccaa0 0x7f70f006bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:38984/0 (socket says 172.21.15.5:38984) 2024-03-24T10:49:35.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.047+0000 7f70f56f7640 1 -- 172.21.15.5:0/1156569340 learned_addr learned my addr 172.21.15.5:0/1156569340 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:35.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.050+0000 7f70f56f7640 1 -- 172.21.15.5:0/1156569340 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70f00ccaa0 msgr2=0x7f70f006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:35.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.050+0000 7f70f56f7640 1 --2- 172.21.15.5:0/1156569340 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70f00ccaa0 0x7f70f006bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:35.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.050+0000 7f70f56f7640 1 -- 172.21.15.5:0/1156569340 >> v1:172.21.15.5:6789/0 conn(0x7f70f01543a0 legacy=0x7f70f006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:35.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.050+0000 7f70f56f7640 1 -- 172.21.15.5:0/1156569340 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70f015b070 con 0x7f70f015ac00 2024-03-24T10:49:35.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.052+0000 7f70f56f7640 1 --2- 172.21.15.5:0/1156569340 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70f015ac00 0x7f70f00cc560 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f70e0009870 tx=0x7f70e002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cadf19bacebbeb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:35.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.057+0000 7f70f4ef6640 1 -- 172.21.15.5:0/1156569340 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e0031640 con 0x7f70f015ac00 2024-03-24T10:49:35.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.057+0000 7f70f4ef6640 1 -- 172.21.15.5:0/1156569340 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70e00317a0 con 0x7f70f015ac00 2024-03-24T10:49:35.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.057+0000 7f70f4ef6640 1 -- 172.21.15.5:0/1156569340 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e0031b30 con 0x7f70f015ac00 2024-03-24T10:49:35.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.058+0000 7f70f815c640 1 -- 172.21.15.5:0/1156569340 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70f015ac00 msgr2=0x7f70f00cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:35.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.058+0000 7f70f815c640 1 --2- 172.21.15.5:0/1156569340 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70f015ac00 0x7f70f00cc560 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f70e0009870 tx=0x7f70e002b8f0 comp rx=0 tx=0).stop 2024-03-24T10:49:35.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.058+0000 7f70f815c640 1 -- 172.21.15.5:0/1156569340 shutdown_connections 2024-03-24T10:49:35.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.058+0000 7f70f815c640 1 --2- 172.21.15.5:0/1156569340 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70f00ccaa0 0x7f70f006bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:35.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.058+0000 7f70f815c640 1 --2- 172.21.15.5:0/1156569340 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70f015ac00 0x7f70f00cc560 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:35.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.058+0000 7f70f815c640 1 -- 172.21.15.5:0/1156569340 >> 172.21.15.5:0/1156569340 conn(0x7f70f0072ef0 msgr2=0x7f70f0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:35.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.058+0000 7f70f815c640 1 -- 172.21.15.5:0/1156569340 shutdown_connections 2024-03-24T10:49:35.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.058+0000 7f70f815c640 1 -- 172.21.15.5:0/1156569340 wait complete. 2024-03-24T10:49:35.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.059+0000 7f70f815c640 1 Processor -- start 2024-03-24T10:49:35.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.059+0000 7f70f815c640 1 -- start start 2024-03-24T10:49:35.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.060+0000 7f70f815c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70f00ccaa0 0x7f70f015dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:35.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.060+0000 7f70f815c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70f01543a0 0x7f70f014be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:35.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.060+0000 7f70f815c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70f015ac00 0x7f70f014cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:35.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.060+0000 7f70f815c640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f70f015bb10 con 0x7f70f01543a0 2024-03-24T10:49:35.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.060+0000 7f70f815c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f70f014b400 con 0x7f70f00ccaa0 2024-03-24T10:49:35.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.060+0000 7f70f815c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f70f014b120 con 0x7f70f015ac00 2024-03-24T10:49:35.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.061+0000 7f70f56f7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70f01543a0 0x7f70f014be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:35.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.061+0000 7f70f66f9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70f015ac00 0x7f70f014cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:35.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.061+0000 7f70f5ef8640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70f00ccaa0 0x7f70f015dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:35.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.061+0000 7f70f56f7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70f01543a0 0x7f70f014be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:58750/0 (socket says 172.21.15.5:58750) 2024-03-24T10:49:35.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.061+0000 7f70f56f7640 1 -- 172.21.15.5:0/4189337827 learned_addr learned my addr 172.21.15.5:0/4189337827 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:35.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.065+0000 7f70f56f7640 1 -- 172.21.15.5:0/4189337827 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70f015ac00 msgr2=0x7f70f014cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:35.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.065+0000 7f70f56f7640 1 --2- 172.21.15.5:0/4189337827 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70f015ac00 0x7f70f014cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:35.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.065+0000 7f70f56f7640 1 -- 172.21.15.5:0/4189337827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70f00ccaa0 msgr2=0x7f70f015dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:35.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.065+0000 7f70f56f7640 1 --2- 172.21.15.5:0/4189337827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70f00ccaa0 0x7f70f015dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:35.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.065+0000 7f70f56f7640 1 -- 172.21.15.5:0/4189337827 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70e00094d0 con 0x7f70f01543a0 2024-03-24T10:49:35.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.067+0000 7f70f56f7640 1 --2- 172.21.15.5:0/4189337827 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70f01543a0 0x7f70f014be00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f70e0009870 tx=0x7f70e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:35.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.072+0000 7f70e67fc640 1 -- 172.21.15.5:0/4189337827 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e003a930 con 0x7f70f01543a0 2024-03-24T10:49:35.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.072+0000 7f70e67fc640 1 -- 172.21.15.5:0/4189337827 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70e003aa90 con 0x7f70f01543a0 2024-03-24T10:49:35.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.072+0000 7f70f815c640 1 -- 172.21.15.5:0/4189337827 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70f015b350 con 0x7f70f01543a0 2024-03-24T10:49:35.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.072+0000 7f70e67fc640 1 -- 172.21.15.5:0/4189337827 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e003ae40 con 0x7f70f01543a0 2024-03-24T10:49:35.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.072+0000 7f70f815c640 1 -- 172.21.15.5:0/4189337827 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70f014fcd0 con 0x7f70f01543a0 2024-03-24T10:49:35.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.073+0000 7f70f815c640 1 -- 172.21.15.5:0/4189337827 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70f014cae0 con 0x7f70f01543a0 2024-03-24T10:49:35.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.099+0000 7f70e67fc640 1 -- 172.21.15.5:0/4189337827 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f70e002f050 con 0x7f70f01543a0 2024-03-24T10:49:35.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.099+0000 7f70e67fc640 1 --2- 172.21.15.5:0/4189337827 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f70d406b890 0x7f70d406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:35.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.100+0000 7f70f5ef8640 1 --2- 172.21.15.5:0/4189337827 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f70d406b890 0x7f70d406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:35.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.100+0000 7f70f5ef8640 1 --2- 172.21.15.5:0/4189337827 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f70d406b890 0x7f70d406dd50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f70ec004f60 tx=0x7f70ec007e30 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:35.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.381+0000 7f70e67fc640 1 -- 172.21.15.5:0/4189337827 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f70e007aa00 con 0x7f70f01543a0 2024-03-24T10:49:35.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.381+0000 7f70e67fc640 1 -- 172.21.15.5:0/4189337827 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f70e0076ca0 con 0x7f70f01543a0 2024-03-24T10:49:35.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.606+0000 7f70f815c640 1 -- 172.21.15.5:0/4189337827 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70f014cae0 con 0x7f70f01543a0 2024-03-24T10:49:35.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.632+0000 7f70e67fc640 1 -- 172.21.15.5:0/4189337827 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f70f014cae0 con 0x7f70f01543a0 2024-03-24T10:49:35.633 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:35.633 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:35.633 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:35.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.635+0000 7f70f815c640 1 -- 172.21.15.5:0/4189337827 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f70d406b890 msgr2=0x7f70d406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:35.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.635+0000 7f70f815c640 1 --2- 172.21.15.5:0/4189337827 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f70d406b890 0x7f70d406dd50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f70ec004f60 tx=0x7f70ec007e30 comp rx=0 tx=0).stop 2024-03-24T10:49:35.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.635+0000 7f70f815c640 1 -- 172.21.15.5:0/4189337827 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70f01543a0 msgr2=0x7f70f014be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:35.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.635+0000 7f70f815c640 1 --2- 172.21.15.5:0/4189337827 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70f01543a0 0x7f70f014be00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f70e0009870 tx=0x7f70e00046c0 comp rx=0 tx=0).stop 2024-03-24T10:49:35.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.635+0000 7f70f815c640 1 -- 172.21.15.5:0/4189337827 shutdown_connections 2024-03-24T10:49:35.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.635+0000 7f70f815c640 1 --2- 172.21.15.5:0/4189337827 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f70d406b890 0x7f70d406dd50 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:35.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.635+0000 7f70f815c640 1 --2- 172.21.15.5:0/4189337827 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f70f01543a0 0x7f70f014be00 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:35.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.635+0000 7f70f815c640 1 --2- 172.21.15.5:0/4189337827 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f70f015ac00 0x7f70f014cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:35.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.635+0000 7f70f815c640 1 --2- 172.21.15.5:0/4189337827 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70f00ccaa0 0x7f70f015dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:35.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.635+0000 7f70f815c640 1 -- 172.21.15.5:0/4189337827 >> 172.21.15.5:0/4189337827 conn(0x7f70f0072ef0 msgr2=0x7f70f01507f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:35.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.635+0000 7f70f815c640 1 -- 172.21.15.5:0/4189337827 shutdown_connections 2024-03-24T10:49:35.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:35.635+0000 7f70f815c640 1 -- 172.21.15.5:0/4189337827 wait complete. 2024-03-24T10:49:36.250 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:36.250 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:37.251 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:37.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.357+0000 7f5c533a0640 1 Processor -- start 2024-03-24T10:49:37.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.357+0000 7f5c533a0640 1 -- start start 2024-03-24T10:49:37.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.358+0000 7f5c533a0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5c4c0ca970 0x7f5c4c0c4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:37.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.358+0000 7f5c533a0640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c4c15d7c0 con 0x7f5c4c154420 2024-03-24T10:49:37.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.358+0000 7f5c533a0640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c4c15da60 con 0x7f5c4c0cbee0 2024-03-24T10:49:37.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.358+0000 7f5c533a0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f5c4c15e400 con 0x7f5c4c0ca970 2024-03-24T10:49:37.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.359+0000 7f5c51b9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f5c4c0cbee0 0x7f5c4c0cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:57096/0 (socket says 172.21.15.5:57096) 2024-03-24T10:49:37.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.359+0000 7f5c51b9d640 1 -- 172.21.15.5:0/3732827279 learned_addr learned my addr 172.21.15.5:0/3732827279 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:49:37.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.359+0000 7f5c52b9f640 1 --2- 172.21.15.5:0/3732827279 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5c4c0ca970 0x7f5c4c0c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:37.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.362+0000 7f5c5139c640 1 -- 172.21.15.5:0/3732827279 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2403929483 0 0) 0x7f5c4c15d7c0 con 0x7f5c4c154420 2024-03-24T10:49:37.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.362+0000 7f5c5139c640 1 -- 172.21.15.5:0/3732827279 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c38003520 con 0x7f5c4c154420 2024-03-24T10:49:37.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.362+0000 7f5c5139c640 1 -- 172.21.15.5:0/3732827279 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1788064613 0 0) 0x7f5c4c15da60 con 0x7f5c4c0cbee0 2024-03-24T10:49:37.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.362+0000 7f5c5139c640 1 -- 172.21.15.5:0/3732827279 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c4c15d7c0 con 0x7f5c4c0cbee0 2024-03-24T10:49:37.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.363+0000 7f5c52b9f640 1 -- 172.21.15.5:0/3732827279 >> v1:172.21.15.57:6789/0 conn(0x7f5c4c0cbee0 legacy=0x7f5c4c0cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:37.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.363+0000 7f5c52b9f640 1 -- 172.21.15.5:0/3732827279 >> v1:172.21.15.5:6789/0 conn(0x7f5c4c154420 legacy=0x7f5c4c15abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:37.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.363+0000 7f5c52b9f640 1 -- 172.21.15.5:0/3732827279 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c4c15dd40 con 0x7f5c4c0ca970 2024-03-24T10:49:37.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.366+0000 7f5c52b9f640 1 --2- 172.21.15.5:0/3732827279 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5c4c0ca970 0x7f5c4c0c4910 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009dc0 tx=0x7f5c4402bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab48598188b9e0d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:37.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.371+0000 7f5c5139c640 1 -- 172.21.15.5:0/3732827279 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4402cd50 con 0x7f5c4c0ca970 2024-03-24T10:49:37.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.371+0000 7f5c5139c640 1 -- 172.21.15.5:0/3732827279 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c44034440 con 0x7f5c4c0ca970 2024-03-24T10:49:37.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.371+0000 7f5c5139c640 1 -- 172.21.15.5:0/3732827279 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c440347d0 con 0x7f5c4c0ca970 2024-03-24T10:49:37.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.372+0000 7f5c533a0640 1 -- 172.21.15.5:0/3732827279 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5c4c0ca970 msgr2=0x7f5c4c0c4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:37.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.372+0000 7f5c533a0640 1 --2- 172.21.15.5:0/3732827279 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5c4c0ca970 0x7f5c4c0c4910 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009dc0 tx=0x7f5c4402bfa0 comp rx=0 tx=0).stop 2024-03-24T10:49:37.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.372+0000 7f5c533a0640 1 -- 172.21.15.5:0/3732827279 shutdown_connections 2024-03-24T10:49:37.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.372+0000 7f5c533a0640 1 --2- 172.21.15.5:0/3732827279 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5c4c0ca970 0x7f5c4c0c4910 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:37.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.372+0000 7f5c533a0640 1 -- 172.21.15.5:0/3732827279 >> 172.21.15.5:0/3732827279 conn(0x7f5c4c06c400 msgr2=0x7f5c4c072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:37.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.372+0000 7f5c533a0640 1 -- 172.21.15.5:0/3732827279 shutdown_connections 2024-03-24T10:49:37.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.372+0000 7f5c533a0640 1 -- 172.21.15.5:0/3732827279 wait complete. 2024-03-24T10:49:37.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.374+0000 7f5c533a0640 1 Processor -- start 2024-03-24T10:49:37.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.374+0000 7f5c533a0640 1 -- start start 2024-03-24T10:49:37.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.375+0000 7f5c533a0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5c4c0ca970 0x7f5c4c154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:37.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.375+0000 7f5c533a0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5c4c0cbee0 0x7f5c4c0b3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:37.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.375+0000 7f5c533a0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c4c154420 0x7f5c4c1f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:37.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.375+0000 7f5c533a0640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4c1f00a0 con 0x7f5c4c0cbee0 2024-03-24T10:49:37.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.375+0000 7f5c533a0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4c157b60 con 0x7f5c4c154420 2024-03-24T10:49:37.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.375+0000 7f5c533a0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f5c4c0c99c0 con 0x7f5c4c0ca970 2024-03-24T10:49:37.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.376+0000 7f5c51b9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5c4c0cbee0 0x7f5c4c0b3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:37.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.376+0000 7f5c52b9f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c4c154420 0x7f5c4c1f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:37.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.376+0000 7f5c5239e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5c4c0ca970 0x7f5c4c154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:37.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.376+0000 7f5c52b9f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c4c154420 0x7f5c4c1f2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:43670/0 (socket says 172.21.15.5:43670) 2024-03-24T10:49:37.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.376+0000 7f5c52b9f640 1 -- 172.21.15.5:0/1723058441 learned_addr learned my addr 172.21.15.5:0/1723058441 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:37.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.380+0000 7f5c51b9d640 1 -- 172.21.15.5:0/1723058441 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5c4c0ca970 msgr2=0x7f5c4c154d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:37.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.380+0000 7f5c51b9d640 1 --2- 172.21.15.5:0/1723058441 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5c4c0ca970 0x7f5c4c154d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:37.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.380+0000 7f5c51b9d640 1 -- 172.21.15.5:0/1723058441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c4c154420 msgr2=0x7f5c4c1f2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:37.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.380+0000 7f5c51b9d640 1 --2- 172.21.15.5:0/1723058441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c4c154420 0x7f5c4c1f2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:37.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.380+0000 7f5c51b9d640 1 -- 172.21.15.5:0/1723058441 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c44009a70 con 0x7f5c4c0cbee0 2024-03-24T10:49:37.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.383+0000 7f5c51b9d640 1 --2- 172.21.15.5:0/1723058441 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5c4c0cbee0 0x7f5c4c0b3ae0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c00c430 tx=0x7f5c3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:37.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.388+0000 7f5c42ffd640 1 -- 172.21.15.5:0/1723058441 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c005ce0 con 0x7f5c4c0cbee0 2024-03-24T10:49:37.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.388+0000 7f5c42ffd640 1 -- 172.21.15.5:0/1723058441 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c3c005e40 con 0x7f5c4c0cbee0 2024-03-24T10:49:37.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.388+0000 7f5c42ffd640 1 -- 172.21.15.5:0/1723058441 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c0072d0 con 0x7f5c4c0cbee0 2024-03-24T10:49:37.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.388+0000 7f5c533a0640 1 -- 172.21.15.5:0/1723058441 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c4c0c9ca0 con 0x7f5c4c0cbee0 2024-03-24T10:49:37.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.388+0000 7f5c533a0640 1 -- 172.21.15.5:0/1723058441 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c4c1ef9e0 con 0x7f5c4c0cbee0 2024-03-24T10:49:37.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.390+0000 7f5c533a0640 1 -- 172.21.15.5:0/1723058441 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c4c04f470 con 0x7f5c4c0cbee0 2024-03-24T10:49:37.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.603+0000 7f5c42ffd640 1 -- 172.21.15.5:0/1723058441 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5c3c007430 con 0x7f5c4c0cbee0 2024-03-24T10:49:37.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.603+0000 7f5c42ffd640 1 --2- 172.21.15.5:0/1723058441 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5c3006b840 0x7f5c3006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:37.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.604+0000 7f5c5239e640 1 --2- 172.21.15.5:0/1723058441 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5c3006b840 0x7f5c3006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:37.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.604+0000 7f5c5239e640 1 --2- 172.21.15.5:0/1723058441 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5c3006b840 0x7f5c3006dd00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5c480017b0 tx=0x7f5c48009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:37.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.690+0000 7f5c42ffd640 1 -- 172.21.15.5:0/1723058441 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f5c3c058380 con 0x7f5c4c0cbee0 2024-03-24T10:49:37.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.690+0000 7f5c42ffd640 1 -- 172.21.15.5:0/1723058441 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5c3c054620 con 0x7f5c4c0cbee0 2024-03-24T10:49:37.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.917+0000 7f5c533a0640 1 -- 172.21.15.5:0/1723058441 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c4c0cb4b0 con 0x7f5c4c0cbee0 2024-03-24T10:49:37.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:37.998+0000 7f5c42ffd640 1 -- 172.21.15.5:0/1723058441 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f5c3c017020 con 0x7f5c4c0cbee0 2024-03-24T10:49:37.999 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:37.999 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:37.999 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:38.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:38.000+0000 7f5c533a0640 1 -- 172.21.15.5:0/1723058441 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5c3006b840 msgr2=0x7f5c3006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:38.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:38.000+0000 7f5c533a0640 1 --2- 172.21.15.5:0/1723058441 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5c3006b840 0x7f5c3006dd00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5c480017b0 tx=0x7f5c48009040 comp rx=0 tx=0).stop 2024-03-24T10:49:38.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:38.000+0000 7f5c533a0640 1 -- 172.21.15.5:0/1723058441 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5c4c0cbee0 msgr2=0x7f5c4c0b3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:38.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:38.000+0000 7f5c533a0640 1 --2- 172.21.15.5:0/1723058441 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5c4c0cbee0 0x7f5c4c0b3ae0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c00c430 tx=0x7f5c3c00c8f0 comp rx=0 tx=0).stop 2024-03-24T10:49:38.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:38.000+0000 7f5c533a0640 1 -- 172.21.15.5:0/1723058441 shutdown_connections 2024-03-24T10:49:38.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:38.000+0000 7f5c533a0640 1 --2- 172.21.15.5:0/1723058441 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5c3006b840 0x7f5c3006dd00 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:38.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:38.000+0000 7f5c533a0640 1 --2- 172.21.15.5:0/1723058441 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5c4c0cbee0 0x7f5c4c0b3ae0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:38.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:38.000+0000 7f5c533a0640 1 --2- 172.21.15.5:0/1723058441 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5c4c154420 0x7f5c4c1f2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:38.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:38.000+0000 7f5c533a0640 1 --2- 172.21.15.5:0/1723058441 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5c4c0ca970 0x7f5c4c154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:38.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:38.000+0000 7f5c533a0640 1 -- 172.21.15.5:0/1723058441 >> 172.21.15.5:0/1723058441 conn(0x7f5c4c06c400 msgr2=0x7f5c4c153390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:38.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:38.001+0000 7f5c533a0640 1 -- 172.21.15.5:0/1723058441 shutdown_connections 2024-03-24T10:49:38.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:38.001+0000 7f5c533a0640 1 -- 172.21.15.5:0/1723058441 wait complete. 2024-03-24T10:49:38.615 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:38.616 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:39.616 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:39.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.720+0000 7f91fa963640 1 Processor -- start 2024-03-24T10:49:39.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.720+0000 7f91fa963640 1 -- start start 2024-03-24T10:49:39.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.721+0000 7f91fa963640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f91f41543a0 0x7f91f415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:39.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.721+0000 7f91fa963640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f91f406d4f0 0x7f91f40cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:39.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.721+0000 7f91fa963640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91f40cb2f0 con 0x7f91f40ccaa0 2024-03-24T10:49:39.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.721+0000 7f91fa963640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f91f40c6750 con 0x7f91f406d4f0 2024-03-24T10:49:39.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.721+0000 7f91fa963640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f91f40cc380 con 0x7f91f41543a0 2024-03-24T10:49:39.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.722+0000 7f91f37fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f91f406d4f0 0x7f91f40cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:39.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.722+0000 7f91f3fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f91f41543a0 0x7f91f415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:39.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.722+0000 7f91f37fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f91f406d4f0 0x7f91f40cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:58764/0 (socket says 172.21.15.5:58764) 2024-03-24T10:49:39.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.722+0000 7f91f37fe640 1 -- 172.21.15.5:0/450470287 learned_addr learned my addr 172.21.15.5:0/450470287 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:39.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.725+0000 7f91f37fe640 1 -- 172.21.15.5:0/450470287 >> v2:172.21.15.57:3300/0 conn(0x7f91f41543a0 msgr2=0x7f91f415ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:39.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.725+0000 7f91f37fe640 1 --2- 172.21.15.5:0/450470287 >> v2:172.21.15.57:3300/0 conn(0x7f91f41543a0 0x7f91f415ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:39.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.725+0000 7f91f37fe640 1 -- 172.21.15.5:0/450470287 >> v1:172.21.15.5:6789/0 conn(0x7f91f40ccaa0 legacy=0x7f91f406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:39.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.725+0000 7f91f37fe640 1 -- 172.21.15.5:0/450470287 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91f40ca7e0 con 0x7f91f406d4f0 2024-03-24T10:49:39.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.729+0000 7f91f37fe640 1 --2- 172.21.15.5:0/450470287 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f91f406d4f0 0x7f91f40cc560 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f91f40cd600 tx=0x7f91e802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de910ea890722ca4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:39.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.733+0000 7f91f2ffd640 1 -- 172.21.15.5:0/450470287 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e8030860 con 0x7f91f406d4f0 2024-03-24T10:49:39.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.733+0000 7f91f2ffd640 1 -- 172.21.15.5:0/450470287 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91e80309c0 con 0x7f91f406d4f0 2024-03-24T10:49:39.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.733+0000 7f91f2ffd640 1 -- 172.21.15.5:0/450470287 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e8030d50 con 0x7f91f406d4f0 2024-03-24T10:49:39.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.733+0000 7f91fa963640 1 -- 172.21.15.5:0/450470287 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f91f406d4f0 msgr2=0x7f91f40cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:39.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.733+0000 7f91fa963640 1 --2- 172.21.15.5:0/450470287 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f91f406d4f0 0x7f91f40cc560 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f91f40cd600 tx=0x7f91e802b9a0 comp rx=0 tx=0).stop 2024-03-24T10:49:39.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.733+0000 7f91fa963640 1 -- 172.21.15.5:0/450470287 shutdown_connections 2024-03-24T10:49:39.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.733+0000 7f91fa963640 1 --2- 172.21.15.5:0/450470287 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f91f406d4f0 0x7f91f40cc560 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:39.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.733+0000 7f91fa963640 1 --2- 172.21.15.5:0/450470287 >> v2:172.21.15.57:3300/0 conn(0x7f91f41543a0 0x7f91f415ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:39.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.733+0000 7f91fa963640 1 -- 172.21.15.5:0/450470287 >> 172.21.15.5:0/450470287 conn(0x7f91f4072ef0 msgr2=0x7f91f4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:39.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.733+0000 7f91fa963640 1 -- 172.21.15.5:0/450470287 shutdown_connections 2024-03-24T10:49:39.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.734+0000 7f91fa963640 1 -- 172.21.15.5:0/450470287 wait complete. 2024-03-24T10:49:39.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.735+0000 7f91fa963640 1 Processor -- start 2024-03-24T10:49:39.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.735+0000 7f91fa963640 1 -- start start 2024-03-24T10:49:39.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.736+0000 7f91fa963640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91f406d4f0 0x7f91f40b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:39.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.736+0000 7f91fa963640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f91f40ccaa0 0x7f91f40b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:39.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.736+0000 7f91fa963640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f91f41543a0 0x7f91f415b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:39.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.736+0000 7f91fa963640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f91f415d390 con 0x7f91f41543a0 2024-03-24T10:49:39.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.736+0000 7f91fa963640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f91f415c780 con 0x7f91f406d4f0 2024-03-24T10:49:39.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.736+0000 7f91fa963640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f91f415ca60 con 0x7f91f40ccaa0 2024-03-24T10:49:39.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.736+0000 7f91f3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91f406d4f0 0x7f91f40b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:39.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.736+0000 7f91f8f00640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f91f41543a0 0x7f91f415b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:39.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.736+0000 7f91f37fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f91f40ccaa0 0x7f91f40b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:39.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.737+0000 7f91f3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91f406d4f0 0x7f91f40b3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:43684/0 (socket says 172.21.15.5:43684) 2024-03-24T10:49:39.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.737+0000 7f91f3fff640 1 -- 172.21.15.5:0/3944312099 learned_addr learned my addr 172.21.15.5:0/3944312099 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:39.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.741+0000 7f91f37fe640 1 -- 172.21.15.5:0/3944312099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91f406d4f0 msgr2=0x7f91f40b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:39.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.741+0000 7f91f37fe640 1 --2- 172.21.15.5:0/3944312099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91f406d4f0 0x7f91f40b3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:39.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.741+0000 7f91f37fe640 1 -- 172.21.15.5:0/3944312099 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f91f41543a0 msgr2=0x7f91f415b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:39.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.741+0000 7f91f37fe640 1 --2- 172.21.15.5:0/3944312099 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f91f41543a0 0x7f91f415b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:39.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.741+0000 7f91f37fe640 1 -- 172.21.15.5:0/3944312099 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e80094d0 con 0x7f91f40ccaa0 2024-03-24T10:49:39.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.744+0000 7f91f37fe640 1 --2- 172.21.15.5:0/3944312099 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f91f40ccaa0 0x7f91f40b42d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f91e8033000 tx=0x7f91e8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:39.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.746+0000 7f91f0ff9640 1 -- 172.21.15.5:0/3944312099 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e8039c90 con 0x7f91f40ccaa0 2024-03-24T10:49:39.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.746+0000 7f91fa963640 1 -- 172.21.15.5:0/3944312099 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91f415cd40 con 0x7f91f40ccaa0 2024-03-24T10:49:39.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.747+0000 7f91fa963640 1 -- 172.21.15.5:0/3944312099 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f91f40b4e90 con 0x7f91f40ccaa0 2024-03-24T10:49:39.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:39.748+0000 7f91fa963640 1 -- 172.21.15.5:0/3944312099 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91f415d720 con 0x7f91f40ccaa0 2024-03-24T10:49:40.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.041+0000 7f91f0ff9640 1 -- 172.21.15.5:0/3944312099 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91e8039df0 con 0x7f91f40ccaa0 2024-03-24T10:49:40.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.041+0000 7f91f0ff9640 1 -- 172.21.15.5:0/3944312099 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91e804d660 con 0x7f91f40ccaa0 2024-03-24T10:49:40.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.042+0000 7f91f0ff9640 1 -- 172.21.15.5:0/3944312099 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f91e804d800 con 0x7f91f40ccaa0 2024-03-24T10:49:40.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.042+0000 7f91f0ff9640 1 --2- 172.21.15.5:0/3944312099 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f91e006b810 0x7f91e006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:40.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.043+0000 7f91f3fff640 1 --2- 172.21.15.5:0/3944312099 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f91e006b810 0x7f91e006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:40.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.043+0000 7f91f3fff640 1 --2- 172.21.15.5:0/3944312099 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f91e006b810 0x7f91e006dcd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f91e4009b60 tx=0x7f91e4009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:40.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.048+0000 7f91f0ff9640 1 -- 172.21.15.5:0/3944312099 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f91e803e070 con 0x7f91f40ccaa0 2024-03-24T10:49:40.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.060+0000 7f91f0ff9640 1 -- 172.21.15.5:0/3944312099 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f91e8078490 con 0x7f91f40ccaa0 2024-03-24T10:49:40.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.290+0000 7f91fa963640 1 -- 172.21.15.5:0/3944312099 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91f415d720 con 0x7f91f40ccaa0 2024-03-24T10:49:40.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.314+0000 7f91f0ff9640 1 -- 172.21.15.5:0/3944312099 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f91f415d720 con 0x7f91f40ccaa0 2024-03-24T10:49:40.314 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:40.315 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:40.315 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:40.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.316+0000 7f91fa963640 1 -- 172.21.15.5:0/3944312099 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f91e006b810 msgr2=0x7f91e006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:40.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.316+0000 7f91fa963640 1 --2- 172.21.15.5:0/3944312099 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f91e006b810 0x7f91e006dcd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f91e4009b60 tx=0x7f91e4009190 comp rx=0 tx=0).stop 2024-03-24T10:49:40.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.316+0000 7f91fa963640 1 -- 172.21.15.5:0/3944312099 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f91f40ccaa0 msgr2=0x7f91f40b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:40.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.316+0000 7f91fa963640 1 --2- 172.21.15.5:0/3944312099 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f91f40ccaa0 0x7f91f40b42d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f91e8033000 tx=0x7f91e8004730 comp rx=0 tx=0).stop 2024-03-24T10:49:40.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.317+0000 7f91fa963640 1 -- 172.21.15.5:0/3944312099 shutdown_connections 2024-03-24T10:49:40.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.317+0000 7f91fa963640 1 --2- 172.21.15.5:0/3944312099 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f91e006b810 0x7f91e006dcd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:40.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.317+0000 7f91fa963640 1 --2- 172.21.15.5:0/3944312099 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f91f41543a0 0x7f91f415b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:40.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.317+0000 7f91fa963640 1 --2- 172.21.15.5:0/3944312099 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f91f40ccaa0 0x7f91f40b42d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:40.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.317+0000 7f91fa963640 1 --2- 172.21.15.5:0/3944312099 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91f406d4f0 0x7f91f40b3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:40.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.317+0000 7f91fa963640 1 -- 172.21.15.5:0/3944312099 >> 172.21.15.5:0/3944312099 conn(0x7f91f4072ef0 msgr2=0x7f91f406be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:40.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.317+0000 7f91fa963640 1 -- 172.21.15.5:0/3944312099 shutdown_connections 2024-03-24T10:49:40.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:40.317+0000 7f91fa963640 1 -- 172.21.15.5:0/3944312099 wait complete. 2024-03-24T10:49:40.931 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:40.931 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:41.932 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:42.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.034+0000 7f57e4c6f640 1 Processor -- start 2024-03-24T10:49:42.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.034+0000 7f57e4c6f640 1 -- start start 2024-03-24T10:49:42.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.035+0000 7f57e4c6f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f57e01543a0 0x7f57e015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:42.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.035+0000 7f57e4c6f640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57e00cb2f0 con 0x7f57e006d4f0 2024-03-24T10:49:42.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.035+0000 7f57e4c6f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57e00ca7e0 con 0x7f57e006bf50 2024-03-24T10:49:42.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.035+0000 7f57e4c6f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f57e00c6750 con 0x7f57e01543a0 2024-03-24T10:49:42.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.035+0000 7f57de59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f57e01543a0 0x7f57e015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:42.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.035+0000 7f57ded9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f57e006bf50 0x7f57e006c380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:57110/0 (socket says 172.21.15.5:57110) 2024-03-24T10:49:42.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.035+0000 7f57ded9d640 1 -- 172.21.15.5:0/3250782876 learned_addr learned my addr 172.21.15.5:0/3250782876 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:49:42.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.038+0000 7f57ddd9b640 1 -- 172.21.15.5:0/3250782876 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4284893888 0 0) 0x7f57e00ca7e0 con 0x7f57e006bf50 2024-03-24T10:49:42.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.038+0000 7f57ddd9b640 1 -- 172.21.15.5:0/3250782876 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57cc003520 con 0x7f57e006bf50 2024-03-24T10:49:42.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.039+0000 7f57ddd9b640 1 -- 172.21.15.5:0/3250782876 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1193426988 0 0) 0x7f57e00cb2f0 con 0x7f57e006d4f0 2024-03-24T10:49:42.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.039+0000 7f57ddd9b640 1 -- 172.21.15.5:0/3250782876 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57e00ca7e0 con 0x7f57e006d4f0 2024-03-24T10:49:42.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.039+0000 7f57de59c640 1 -- 172.21.15.5:0/3250782876 >> v1:172.21.15.57:6789/0 conn(0x7f57e006bf50 legacy=0x7f57e006c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:42.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.039+0000 7f57de59c640 1 -- 172.21.15.5:0/3250782876 >> v1:172.21.15.5:6789/0 conn(0x7f57e006d4f0 legacy=0x7f57e00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:42.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.039+0000 7f57de59c640 1 -- 172.21.15.5:0/3250782876 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57e00cb520 con 0x7f57e01543a0 2024-03-24T10:49:42.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.042+0000 7f57de59c640 1 --2- 172.21.15.5:0/3250782876 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f57e01543a0 0x7f57e015ac00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f57d00097d0 tx=0x7f57d002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5d4531cb5eaa516 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:42.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.046+0000 7f57ddd9b640 1 -- 172.21.15.5:0/3250782876 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d0030880 con 0x7f57e01543a0 2024-03-24T10:49:42.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.048+0000 7f57ddd9b640 1 -- 172.21.15.5:0/3250782876 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57d00309e0 con 0x7f57e01543a0 2024-03-24T10:49:42.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.049+0000 7f57e4c6f640 1 -- 172.21.15.5:0/3250782876 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f57e01543a0 msgr2=0x7f57e015ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:42.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.049+0000 7f57e4c6f640 1 --2- 172.21.15.5:0/3250782876 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f57e01543a0 0x7f57e015ac00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f57d00097d0 tx=0x7f57d002b9c0 comp rx=0 tx=0).stop 2024-03-24T10:49:42.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.049+0000 7f57e4c6f640 1 -- 172.21.15.5:0/3250782876 shutdown_connections 2024-03-24T10:49:42.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.049+0000 7f57e4c6f640 1 --2- 172.21.15.5:0/3250782876 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f57e01543a0 0x7f57e015ac00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:42.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.049+0000 7f57e4c6f640 1 -- 172.21.15.5:0/3250782876 >> 172.21.15.5:0/3250782876 conn(0x7f57e0072ef0 msgr2=0x7f57e0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:42.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.049+0000 7f57e4c6f640 1 -- 172.21.15.5:0/3250782876 shutdown_connections 2024-03-24T10:49:42.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.049+0000 7f57e4c6f640 1 -- 172.21.15.5:0/3250782876 wait complete. 2024-03-24T10:49:42.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.051+0000 7f57e4c6f640 1 Processor -- start 2024-03-24T10:49:42.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.051+0000 7f57e4c6f640 1 -- start start 2024-03-24T10:49:42.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.052+0000 7f57e4c6f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f57e006bf50 0x7f57e0158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:42.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.052+0000 7f57e4c6f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57e006d4f0 0x7f57e01eff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:42.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.052+0000 7f57e4c6f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57e01543a0 0x7f57e00cffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:42.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.052+0000 7f57e4c6f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f57e01594c0 con 0x7f57e01543a0 2024-03-24T10:49:42.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.052+0000 7f57e4c6f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f57e015e170 con 0x7f57e006d4f0 2024-03-24T10:49:42.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.052+0000 7f57e4c6f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f57e00cf300 con 0x7f57e006bf50 2024-03-24T10:49:42.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.053+0000 7f57d7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57e006d4f0 0x7f57e01eff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:42.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.053+0000 7f57ded9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57e01543a0 0x7f57e00cffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:42.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.053+0000 7f57de59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f57e006bf50 0x7f57e0158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:42.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.053+0000 7f57d7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57e006d4f0 0x7f57e01eff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:43690/0 (socket says 172.21.15.5:43690) 2024-03-24T10:49:42.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.053+0000 7f57d7fff640 1 -- 172.21.15.5:0/299060262 learned_addr learned my addr 172.21.15.5:0/299060262 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:42.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.057+0000 7f57ded9d640 1 -- 172.21.15.5:0/299060262 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f57e006bf50 msgr2=0x7f57e0158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:42.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.057+0000 7f57ded9d640 1 --2- 172.21.15.5:0/299060262 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f57e006bf50 0x7f57e0158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:42.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.057+0000 7f57ded9d640 1 -- 172.21.15.5:0/299060262 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57e006d4f0 msgr2=0x7f57e01eff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:42.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.057+0000 7f57ded9d640 1 --2- 172.21.15.5:0/299060262 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57e006d4f0 0x7f57e01eff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:42.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.057+0000 7f57ded9d640 1 -- 172.21.15.5:0/299060262 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57d0009400 con 0x7f57e01543a0 2024-03-24T10:49:42.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.059+0000 7f57ded9d640 1 --2- 172.21.15.5:0/299060262 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57e01543a0 0x7f57e00cffc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f57d800cad0 tx=0x7f57d800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:42.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.062+0000 7f57d6ffd640 1 -- 172.21.15.5:0/299060262 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d800abd0 con 0x7f57e01543a0 2024-03-24T10:49:42.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.062+0000 7f57e4c6f640 1 -- 172.21.15.5:0/299060262 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57e00cdd70 con 0x7f57e01543a0 2024-03-24T10:49:42.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.062+0000 7f57e4c6f640 1 -- 172.21.15.5:0/299060262 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57e01efa20 con 0x7f57e01543a0 2024-03-24T10:49:42.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.063+0000 7f57e4c6f640 1 -- 172.21.15.5:0/299060262 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57e01efc00 con 0x7f57e01543a0 2024-03-24T10:49:42.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.064+0000 7f57d6ffd640 1 -- 172.21.15.5:0/299060262 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57d8001ac0 con 0x7f57e01543a0 2024-03-24T10:49:42.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.064+0000 7f57d6ffd640 1 -- 172.21.15.5:0/299060262 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d801c440 con 0x7f57e01543a0 2024-03-24T10:49:42.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.329+0000 7f57d6ffd640 1 -- 172.21.15.5:0/299060262 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f57d8012400 con 0x7f57e01543a0 2024-03-24T10:49:42.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.329+0000 7f57d6ffd640 1 --2- 172.21.15.5:0/299060262 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f57c006b810 0x7f57c006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:42.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.329+0000 7f57de59c640 1 --2- 172.21.15.5:0/299060262 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f57c006b810 0x7f57c006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:42.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.330+0000 7f57de59c640 1 --2- 172.21.15.5:0/299060262 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f57c006b810 0x7f57c006dcd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f57d0036000 tx=0x7f57d0008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:42.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.346+0000 7f57d6ffd640 1 -- 172.21.15.5:0/299060262 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f57d8059030 con 0x7f57e01543a0 2024-03-24T10:49:42.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.346+0000 7f57d6ffd640 1 -- 172.21.15.5:0/299060262 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f57d80552d0 con 0x7f57e01543a0 2024-03-24T10:49:42.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.592+0000 7f57e4c6f640 1 -- 172.21.15.5:0/299060262 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57e00681d0 con 0x7f57e01543a0 2024-03-24T10:49:42.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.613+0000 7f57d6ffd640 1 -- 172.21.15.5:0/299060262 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f57d8058990 con 0x7f57e01543a0 2024-03-24T10:49:42.613 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:42.614 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:42.614 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:42.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.615+0000 7f57e4c6f640 1 -- 172.21.15.5:0/299060262 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f57c006b810 msgr2=0x7f57c006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:42.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.615+0000 7f57e4c6f640 1 --2- 172.21.15.5:0/299060262 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f57c006b810 0x7f57c006dcd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f57d0036000 tx=0x7f57d0008030 comp rx=0 tx=0).stop 2024-03-24T10:49:42.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.615+0000 7f57e4c6f640 1 -- 172.21.15.5:0/299060262 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57e01543a0 msgr2=0x7f57e00cffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:42.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.615+0000 7f57e4c6f640 1 --2- 172.21.15.5:0/299060262 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57e01543a0 0x7f57e00cffc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f57d800cad0 tx=0x7f57d800cf90 comp rx=0 tx=0).stop 2024-03-24T10:49:42.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.615+0000 7f57e4c6f640 1 -- 172.21.15.5:0/299060262 shutdown_connections 2024-03-24T10:49:42.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.615+0000 7f57e4c6f640 1 --2- 172.21.15.5:0/299060262 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f57c006b810 0x7f57c006dcd0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:42.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.615+0000 7f57e4c6f640 1 --2- 172.21.15.5:0/299060262 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57e01543a0 0x7f57e00cffc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:42.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.615+0000 7f57e4c6f640 1 --2- 172.21.15.5:0/299060262 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57e006d4f0 0x7f57e01eff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:42.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.615+0000 7f57e4c6f640 1 --2- 172.21.15.5:0/299060262 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f57e006bf50 0x7f57e0158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:42.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.615+0000 7f57e4c6f640 1 -- 172.21.15.5:0/299060262 >> 172.21.15.5:0/299060262 conn(0x7f57e0072ef0 msgr2=0x7f57e014fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:42.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.616+0000 7f57e4c6f640 1 -- 172.21.15.5:0/299060262 shutdown_connections 2024-03-24T10:49:42.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:42.616+0000 7f57e4c6f640 1 -- 172.21.15.5:0/299060262 wait complete. 2024-03-24T10:49:43.231 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:43.231 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:44.233 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:44.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.335+0000 7f62bbb25640 1 Processor -- start 2024-03-24T10:49:44.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.335+0000 7f62bbb25640 1 -- start start 2024-03-24T10:49:44.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.336+0000 7f62bbb25640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f62b40cae90 0x7f62b40cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:44.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.336+0000 7f62bbb25640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62b40ccba0 con 0x7f62b40cb270 2024-03-24T10:49:44.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.336+0000 7f62bbb25640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62b40cce40 con 0x7f62b40cd960 2024-03-24T10:49:44.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.336+0000 7f62bbb25640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f62b40cc9b0 con 0x7f62b40cae90 2024-03-24T10:49:44.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.336+0000 7f62b98c1640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f62b40cae90 0x7f62b40cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:44.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.336+0000 7f62ba0c2640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f62b40cd960 0x7f62b40cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:33098/0 (socket says 172.21.15.5:33098) 2024-03-24T10:49:44.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.336+0000 7f62ba0c2640 1 -- 172.21.15.5:0/3057040140 learned_addr learned my addr 172.21.15.5:0/3057040140 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:49:44.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.339+0000 7f62b88bf640 1 -- 172.21.15.5:0/3057040140 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3747320420 0 0) 0x7f62b40cce40 con 0x7f62b40cd960 2024-03-24T10:49:44.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.339+0000 7f62b88bf640 1 -- 172.21.15.5:0/3057040140 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62a0003520 con 0x7f62b40cd960 2024-03-24T10:49:44.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.339+0000 7f62b98c1640 1 -- 172.21.15.5:0/3057040140 >> v1:172.21.15.57:6789/0 conn(0x7f62b40cd960 legacy=0x7f62b40cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:44.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.339+0000 7f62b98c1640 1 -- 172.21.15.5:0/3057040140 >> v1:172.21.15.5:6789/0 conn(0x7f62b40cb270 legacy=0x7f62b40cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:44.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.339+0000 7f62b98c1640 1 -- 172.21.15.5:0/3057040140 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b415a590 con 0x7f62b40cae90 2024-03-24T10:49:44.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.345+0000 7f62b98c1640 1 --2- 172.21.15.5:0/3057040140 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62b40cae90 0x7f62b40cf530 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f62b00097a0 tx=0x7f62b002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c041793d14143d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:44.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.350+0000 7f62b88bf640 1 -- 172.21.15.5:0/3057040140 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b002f810 con 0x7f62b40cae90 2024-03-24T10:49:44.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.350+0000 7f62b88bf640 1 -- 172.21.15.5:0/3057040140 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62b002f970 con 0x7f62b40cae90 2024-03-24T10:49:44.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.350+0000 7f62b88bf640 1 -- 172.21.15.5:0/3057040140 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62b002fd00 con 0x7f62b40cae90 2024-03-24T10:49:44.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.351+0000 7f62bbb25640 1 -- 172.21.15.5:0/3057040140 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62b40cae90 msgr2=0x7f62b40cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:44.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.351+0000 7f62bbb25640 1 --2- 172.21.15.5:0/3057040140 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62b40cae90 0x7f62b40cf530 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f62b00097a0 tx=0x7f62b002b9f0 comp rx=0 tx=0).stop 2024-03-24T10:49:44.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.351+0000 7f62bbb25640 1 -- 172.21.15.5:0/3057040140 shutdown_connections 2024-03-24T10:49:44.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.351+0000 7f62bbb25640 1 --2- 172.21.15.5:0/3057040140 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62b40cae90 0x7f62b40cf530 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:44.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.351+0000 7f62bbb25640 1 -- 172.21.15.5:0/3057040140 >> 172.21.15.5:0/3057040140 conn(0x7f62b40cfb80 msgr2=0x7f62b40cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:44.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.351+0000 7f62bbb25640 1 -- 172.21.15.5:0/3057040140 shutdown_connections 2024-03-24T10:49:44.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.351+0000 7f62bbb25640 1 -- 172.21.15.5:0/3057040140 wait complete. 2024-03-24T10:49:44.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.352+0000 7f62bbb25640 1 Processor -- start 2024-03-24T10:49:44.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.352+0000 7f62bbb25640 1 -- start start 2024-03-24T10:49:44.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.353+0000 7f62bbb25640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62b40cae90 0x7f62b40c7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:44.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.353+0000 7f62bbb25640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62b40cb270 0x7f62b415bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:44.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.353+0000 7f62bbb25640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f62b40cd960 0x7f62b415c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:44.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.353+0000 7f62bbb25640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f62b4157e30 con 0x7f62b40cae90 2024-03-24T10:49:44.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.353+0000 7f62bbb25640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f62b415b880 con 0x7f62b40cb270 2024-03-24T10:49:44.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.353+0000 7f62bbb25640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f62b41f0a20 con 0x7f62b40cd960 2024-03-24T10:49:44.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.354+0000 7f62b90c0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62b40cb270 0x7f62b415bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:44.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.354+0000 7f62b98c1640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62b40cae90 0x7f62b40c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:44.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.354+0000 7f62b90c0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62b40cb270 0x7f62b415bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:45506/0 (socket says 172.21.15.5:45506) 2024-03-24T10:49:44.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.354+0000 7f62b90c0640 1 -- 172.21.15.5:0/1462508353 learned_addr learned my addr 172.21.15.5:0/1462508353 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:44.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.354+0000 7f62ba0c2640 1 --2- 172.21.15.5:0/1462508353 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f62b40cd960 0x7f62b415c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:44.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.358+0000 7f62b90c0640 1 -- 172.21.15.5:0/1462508353 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f62b40cd960 msgr2=0x7f62b415c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:44.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.358+0000 7f62b90c0640 1 --2- 172.21.15.5:0/1462508353 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f62b40cd960 0x7f62b415c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:44.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.358+0000 7f62b90c0640 1 -- 172.21.15.5:0/1462508353 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62b40cae90 msgr2=0x7f62b40c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:44.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.358+0000 7f62b90c0640 1 --2- 172.21.15.5:0/1462508353 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62b40cae90 0x7f62b40c7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:44.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.358+0000 7f62b90c0640 1 -- 172.21.15.5:0/1462508353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b0009400 con 0x7f62b40cb270 2024-03-24T10:49:44.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.361+0000 7f62b90c0640 1 --2- 172.21.15.5:0/1462508353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62b40cb270 0x7f62b415bd10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f62a400c430 tx=0x7f62a400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:44.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.366+0000 7f62aa7fc640 1 -- 172.21.15.5:0/1462508353 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a40052d0 con 0x7f62b40cb270 2024-03-24T10:49:44.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.366+0000 7f62aa7fc640 1 -- 172.21.15.5:0/1462508353 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62a4005450 con 0x7f62b40cb270 2024-03-24T10:49:44.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.366+0000 7f62bbb25640 1 -- 172.21.15.5:0/1462508353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62b415e140 con 0x7f62b40cb270 2024-03-24T10:49:44.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.366+0000 7f62aa7fc640 1 -- 172.21.15.5:0/1462508353 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a4007260 con 0x7f62b40cb270 2024-03-24T10:49:44.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.366+0000 7f62bbb25640 1 -- 172.21.15.5:0/1462508353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62b41f0cc0 con 0x7f62b40cb270 2024-03-24T10:49:44.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.367+0000 7f62bbb25640 1 -- 172.21.15.5:0/1462508353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62b415e660 con 0x7f62b40cb270 2024-03-24T10:49:44.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.392+0000 7f62aa7fc640 1 -- 172.21.15.5:0/1462508353 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f62a40073c0 con 0x7f62b40cb270 2024-03-24T10:49:44.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.392+0000 7f62aa7fc640 1 --2- 172.21.15.5:0/1462508353 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f629806b840 0x7f629806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:44.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.392+0000 7f62b98c1640 1 --2- 172.21.15.5:0/1462508353 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f629806b840 0x7f629806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:44.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.392+0000 7f62b98c1640 1 --2- 172.21.15.5:0/1462508353 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f629806b840 0x7f629806dd00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f62b0036000 tx=0x7f62b0008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:44.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.674+0000 7f62aa7fc640 1 -- 172.21.15.5:0/1462508353 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f62a4058310 con 0x7f62b40cb270 2024-03-24T10:49:44.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.690+0000 7f62aa7fc640 1 -- 172.21.15.5:0/1462508353 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f62a40545b0 con 0x7f62b40cb270 2024-03-24T10:49:44.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.913+0000 7f62bbb25640 1 -- 172.21.15.5:0/1462508353 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62b40681d0 con 0x7f62b40cb270 2024-03-24T10:49:44.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.929+0000 7f62aa7fc640 1 -- 172.21.15.5:0/1462508353 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f62a4020020 con 0x7f62b40cb270 2024-03-24T10:49:44.929 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:44.929 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:44.930 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:44.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.931+0000 7f62bbb25640 1 -- 172.21.15.5:0/1462508353 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f629806b840 msgr2=0x7f629806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:44.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.931+0000 7f62bbb25640 1 --2- 172.21.15.5:0/1462508353 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f629806b840 0x7f629806dd00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f62b0036000 tx=0x7f62b0008030 comp rx=0 tx=0).stop 2024-03-24T10:49:44.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.931+0000 7f62bbb25640 1 -- 172.21.15.5:0/1462508353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62b40cb270 msgr2=0x7f62b415bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:44.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.931+0000 7f62bbb25640 1 --2- 172.21.15.5:0/1462508353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62b40cb270 0x7f62b415bd10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f62a400c430 tx=0x7f62a400c8f0 comp rx=0 tx=0).stop 2024-03-24T10:49:44.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.931+0000 7f62bbb25640 1 -- 172.21.15.5:0/1462508353 shutdown_connections 2024-03-24T10:49:44.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.931+0000 7f62bbb25640 1 --2- 172.21.15.5:0/1462508353 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f629806b840 0x7f629806dd00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:44.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.931+0000 7f62bbb25640 1 --2- 172.21.15.5:0/1462508353 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f62b40cd960 0x7f62b415c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:44.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.931+0000 7f62bbb25640 1 --2- 172.21.15.5:0/1462508353 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62b40cb270 0x7f62b415bd10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:44.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.931+0000 7f62bbb25640 1 --2- 172.21.15.5:0/1462508353 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62b40cae90 0x7f62b40c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:44.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.931+0000 7f62bbb25640 1 -- 172.21.15.5:0/1462508353 >> 172.21.15.5:0/1462508353 conn(0x7f62b40cfb80 msgr2=0x7f62b4152a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:44.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.931+0000 7f62bbb25640 1 -- 172.21.15.5:0/1462508353 shutdown_connections 2024-03-24T10:49:44.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:44.931+0000 7f62bbb25640 1 -- 172.21.15.5:0/1462508353 wait complete. 2024-03-24T10:49:45.548 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:45.548 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:46.549 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:46.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.653+0000 7f4d017b6640 1 Processor -- start 2024-03-24T10:49:46.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.653+0000 7f4d017b6640 1 -- start start 2024-03-24T10:49:46.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.653+0000 7f4d017b6640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4cfc1543a0 0x7f4cfc15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:46.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.653+0000 7f4d017b6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4cfc06d4f0 0x7f4cfc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:46.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.653+0000 7f4d017b6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4cfc0ccaa0 0x7f4cfc06bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:46.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.653+0000 7f4d017b6640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f4cfc0c6750 con 0x7f4cfc1543a0 2024-03-24T10:49:46.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.653+0000 7f4d017b6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f4cfc0cceb0 con 0x7f4cfc06d4f0 2024-03-24T10:49:46.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.653+0000 7f4d017b6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4cfc0cc380 con 0x7f4cfc0ccaa0 2024-03-24T10:49:46.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.654+0000 7f4cf3fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4cfc06d4f0 0x7f4cfc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:46.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.654+0000 7f4cfb7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4cfc0ccaa0 0x7f4cfc06bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:46.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.654+0000 7f4cfaffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4cfc1543a0 0x7f4cfc15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:46.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.654+0000 7f4cf3fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4cfc06d4f0 0x7f4cfc0cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:45518/0 (socket says 172.21.15.5:45518) 2024-03-24T10:49:46.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.654+0000 7f4cf3fff640 1 -- 172.21.15.5:0/1507170399 learned_addr learned my addr 172.21.15.5:0/1507170399 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:46.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.657+0000 7f4cf3fff640 1 -- 172.21.15.5:0/1507170399 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4cfc0ccaa0 msgr2=0x7f4cfc06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:46.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.657+0000 7f4cf3fff640 1 --2- 172.21.15.5:0/1507170399 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4cfc0ccaa0 0x7f4cfc06bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:46.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.657+0000 7f4cf3fff640 1 -- 172.21.15.5:0/1507170399 >> v2:172.21.15.5:3300/0 conn(0x7f4cfc1543a0 msgr2=0x7f4cfc15ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:46.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.657+0000 7f4cf3fff640 1 --2- 172.21.15.5:0/1507170399 >> v2:172.21.15.5:3300/0 conn(0x7f4cfc1543a0 0x7f4cfc15ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:46.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.657+0000 7f4cf3fff640 1 -- 172.21.15.5:0/1507170399 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cfc0cb2f0 con 0x7f4cfc06d4f0 2024-03-24T10:49:46.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.660+0000 7f4cf3fff640 1 --2- 172.21.15.5:0/1507170399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cfc06d4f0 0x7f4cfc0cc560 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc0c6990 tx=0x7f4ce402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf54a1b61d39f3b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:46.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.664+0000 7f4cfa7fc640 1 -- 172.21.15.5:0/1507170399 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4030860 con 0x7f4cfc06d4f0 2024-03-24T10:49:46.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.664+0000 7f4cfa7fc640 1 -- 172.21.15.5:0/1507170399 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ce40309c0 con 0x7f4cfc06d4f0 2024-03-24T10:49:46.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.664+0000 7f4cfa7fc640 1 -- 172.21.15.5:0/1507170399 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4030d50 con 0x7f4cfc06d4f0 2024-03-24T10:49:46.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.665+0000 7f4d017b6640 1 -- 172.21.15.5:0/1507170399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cfc06d4f0 msgr2=0x7f4cfc0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:46.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.665+0000 7f4d017b6640 1 --2- 172.21.15.5:0/1507170399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cfc06d4f0 0x7f4cfc0cc560 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4cfc0c6990 tx=0x7f4ce402b9a0 comp rx=0 tx=0).stop 2024-03-24T10:49:46.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.665+0000 7f4d017b6640 1 -- 172.21.15.5:0/1507170399 shutdown_connections 2024-03-24T10:49:46.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.665+0000 7f4d017b6640 1 --2- 172.21.15.5:0/1507170399 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4cfc0ccaa0 0x7f4cfc06bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:46.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.665+0000 7f4d017b6640 1 --2- 172.21.15.5:0/1507170399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cfc06d4f0 0x7f4cfc0cc560 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:46.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.665+0000 7f4d017b6640 1 --2- 172.21.15.5:0/1507170399 >> v2:172.21.15.5:3300/0 conn(0x7f4cfc1543a0 0x7f4cfc15ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:46.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.665+0000 7f4d017b6640 1 -- 172.21.15.5:0/1507170399 >> 172.21.15.5:0/1507170399 conn(0x7f4cfc072ef0 msgr2=0x7f4cfc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:46.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.665+0000 7f4d017b6640 1 -- 172.21.15.5:0/1507170399 shutdown_connections 2024-03-24T10:49:46.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.665+0000 7f4d017b6640 1 -- 172.21.15.5:0/1507170399 wait complete. 2024-03-24T10:49:46.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.666+0000 7f4d017b6640 1 Processor -- start 2024-03-24T10:49:46.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.666+0000 7f4d017b6640 1 -- start start 2024-03-24T10:49:46.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.667+0000 7f4d017b6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cfc06d4f0 0x7f4cfc14f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:46.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.667+0000 7f4d017b6640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4cfc0ccaa0 0x7f4cfc0b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:46.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.667+0000 7f4d017b6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4cfc1543a0 0x7f4cfc14e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:46.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.667+0000 7f4d017b6640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc0b5070 con 0x7f4cfc0ccaa0 2024-03-24T10:49:46.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.667+0000 7f4d017b6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc15b9d0 con 0x7f4cfc06d4f0 2024-03-24T10:49:46.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.667+0000 7f4d017b6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4cfc15c7e0 con 0x7f4cfc1543a0 2024-03-24T10:49:46.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.668+0000 7f4cf3fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4cfc0ccaa0 0x7f4cfc0b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:46.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.668+0000 7f4cfb7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4cfc1543a0 0x7f4cfc14e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:46.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.668+0000 7f4cfaffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cfc06d4f0 0x7f4cfc14f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:46.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.668+0000 7f4cf3fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4cfc0ccaa0 0x7f4cfc0b4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:33240/0 (socket says 172.21.15.5:33240) 2024-03-24T10:49:46.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.668+0000 7f4cfb7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4cfc1543a0 0x7f4cfc14e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:57830/0 (socket says 172.21.15.5:57830) 2024-03-24T10:49:46.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.668+0000 7f4cf3fff640 1 -- 172.21.15.5:0/1395809676 learned_addr learned my addr 172.21.15.5:0/1395809676 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:46.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.671+0000 7f4cf3fff640 1 -- 172.21.15.5:0/1395809676 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4cfc1543a0 msgr2=0x7f4cfc14e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:46.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.672+0000 7f4cf3fff640 1 --2- 172.21.15.5:0/1395809676 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4cfc1543a0 0x7f4cfc14e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:46.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.672+0000 7f4cf3fff640 1 -- 172.21.15.5:0/1395809676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cfc06d4f0 msgr2=0x7f4cfc14f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:46.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.672+0000 7f4cf3fff640 1 --2- 172.21.15.5:0/1395809676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cfc06d4f0 0x7f4cfc14f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:46.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.672+0000 7f4cf3fff640 1 -- 172.21.15.5:0/1395809676 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce40094d0 con 0x7f4cfc0ccaa0 2024-03-24T10:49:46.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.674+0000 7f4cf3fff640 1 --2- 172.21.15.5:0/1395809676 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4cfc0ccaa0 0x7f4cfc0b4220 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4ce402b9e0 tx=0x7f4ce4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:46.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.677+0000 7f4cf37fe640 1 -- 172.21.15.5:0/1395809676 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4039c90 con 0x7f4cfc0ccaa0 2024-03-24T10:49:46.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.677+0000 7f4cf37fe640 1 -- 172.21.15.5:0/1395809676 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ce4039e10 con 0x7f4cfc0ccaa0 2024-03-24T10:49:46.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.677+0000 7f4d017b6640 1 -- 172.21.15.5:0/1395809676 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cfc15b3c0 con 0x7f4cfc0ccaa0 2024-03-24T10:49:46.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.677+0000 7f4cf37fe640 1 -- 172.21.15.5:0/1395809676 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4043640 con 0x7f4cfc0ccaa0 2024-03-24T10:49:46.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.677+0000 7f4d017b6640 1 -- 172.21.15.5:0/1395809676 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cfc0b4d90 con 0x7f4cfc0ccaa0 2024-03-24T10:49:46.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.678+0000 7f4d017b6640 1 -- 172.21.15.5:0/1395809676 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cfc0681d0 con 0x7f4cfc0ccaa0 2024-03-24T10:49:46.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.959+0000 7f4cf37fe640 1 -- 172.21.15.5:0/1395809676 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4ce40437a0 con 0x7f4cfc0ccaa0 2024-03-24T10:49:46.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.959+0000 7f4cf37fe640 1 --2- 172.21.15.5:0/1395809676 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4cdc06b860 0x7f4cdc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:46.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.960+0000 7f4cfaffd640 1 --2- 172.21.15.5:0/1395809676 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4cdc06b860 0x7f4cdc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:46.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.960+0000 7f4cfaffd640 1 --2- 172.21.15.5:0/1395809676 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4cdc06b860 0x7f4cdc06dd20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009a20 tx=0x7f4cec009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:46.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.984+0000 7f4cf37fe640 1 -- 172.21.15.5:0/1395809676 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f4ce403e070 con 0x7f4cfc0ccaa0 2024-03-24T10:49:46.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:46.984+0000 7f4cf37fe640 1 -- 172.21.15.5:0/1395809676 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4ce4078620 con 0x7f4cfc0ccaa0 2024-03-24T10:49:47.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:47.209+0000 7f4d017b6640 1 -- 172.21.15.5:0/1395809676 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4cfc0cb0d0 con 0x7f4cfc0ccaa0 2024-03-24T10:49:47.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:47.224+0000 7f4cf37fe640 1 -- 172.21.15.5:0/1395809676 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f4ce4034070 con 0x7f4cfc0ccaa0 2024-03-24T10:49:47.224 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:47.225 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:47.225 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:47.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:47.226+0000 7f4d017b6640 1 -- 172.21.15.5:0/1395809676 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4cdc06b860 msgr2=0x7f4cdc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:47.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:47.226+0000 7f4d017b6640 1 --2- 172.21.15.5:0/1395809676 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4cdc06b860 0x7f4cdc06dd20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009a20 tx=0x7f4cec009190 comp rx=0 tx=0).stop 2024-03-24T10:49:47.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:47.226+0000 7f4d017b6640 1 -- 172.21.15.5:0/1395809676 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4cfc0ccaa0 msgr2=0x7f4cfc0b4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:47.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:47.226+0000 7f4d017b6640 1 --2- 172.21.15.5:0/1395809676 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4cfc0ccaa0 0x7f4cfc0b4220 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4ce402b9e0 tx=0x7f4ce4004730 comp rx=0 tx=0).stop 2024-03-24T10:49:47.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:47.226+0000 7f4d017b6640 1 -- 172.21.15.5:0/1395809676 shutdown_connections 2024-03-24T10:49:47.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:47.226+0000 7f4d017b6640 1 --2- 172.21.15.5:0/1395809676 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4cdc06b860 0x7f4cdc06dd20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:47.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:47.226+0000 7f4d017b6640 1 --2- 172.21.15.5:0/1395809676 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4cfc0ccaa0 0x7f4cfc0b4220 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:47.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:47.226+0000 7f4d017b6640 1 --2- 172.21.15.5:0/1395809676 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4cfc1543a0 0x7f4cfc14e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:47.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:47.226+0000 7f4d017b6640 1 --2- 172.21.15.5:0/1395809676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4cfc06d4f0 0x7f4cfc14f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:47.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:47.226+0000 7f4d017b6640 1 -- 172.21.15.5:0/1395809676 >> 172.21.15.5:0/1395809676 conn(0x7f4cfc072ef0 msgr2=0x7f4cfc0d0390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:47.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:47.226+0000 7f4d017b6640 1 -- 172.21.15.5:0/1395809676 shutdown_connections 2024-03-24T10:49:47.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:47.226+0000 7f4d017b6640 1 -- 172.21.15.5:0/1395809676 wait complete. 2024-03-24T10:49:47.841 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:47.841 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:48.842 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:48.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.946+0000 7f73af264640 1 Processor -- start 2024-03-24T10:49:48.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.946+0000 7f73af264640 1 -- start start 2024-03-24T10:49:48.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.947+0000 7f73af264640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f73a81543a0 0x7f73a815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:48.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.947+0000 7f73af264640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f73a806bf50 0x7f73a806c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:48.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.947+0000 7f73af264640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73a80cb2f0 con 0x7f73a806d4f0 2024-03-24T10:49:48.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.947+0000 7f73af264640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f73a80c6750 con 0x7f73a806bf50 2024-03-24T10:49:48.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.947+0000 7f73af264640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f73a80cc380 con 0x7f73a81543a0 2024-03-24T10:49:48.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.948+0000 7f73ad000640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f73a81543a0 0x7f73a815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:48.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.948+0000 7f73ad801640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f73a806bf50 0x7f73a806c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:48.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.948+0000 7f73a7fff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f73a806d4f0 0x7f73a80cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:37828/0 (socket says 172.21.15.5:37828) 2024-03-24T10:49:48.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.948+0000 7f73a7fff640 1 -- 172.21.15.5:0/2550368642 learned_addr learned my addr 172.21.15.5:0/2550368642 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:49:48.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.951+0000 7f73ad000640 1 -- 172.21.15.5:0/2550368642 >> v2:172.21.15.5:3300/0 conn(0x7f73a806bf50 msgr2=0x7f73a806c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:48.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.951+0000 7f73ad000640 1 --2- 172.21.15.5:0/2550368642 >> v2:172.21.15.5:3300/0 conn(0x7f73a806bf50 0x7f73a806c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:48.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.951+0000 7f73ad000640 1 -- 172.21.15.5:0/2550368642 >> v1:172.21.15.5:6789/0 conn(0x7f73a806d4f0 legacy=0x7f73a80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:48.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.951+0000 7f73ad000640 1 -- 172.21.15.5:0/2550368642 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a80ca7e0 con 0x7f73a81543a0 2024-03-24T10:49:48.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.953+0000 7f73ad000640 1 --2- 172.21.15.5:0/2550368642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f73a81543a0 0x7f73a815ac00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f739c0097a0 tx=0x7f739c02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=435e53223dcfb987 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:48.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.955+0000 7f73a77fe640 1 -- 172.21.15.5:0/2550368642 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c030610 con 0x7f73a81543a0 2024-03-24T10:49:48.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.958+0000 7f73a77fe640 1 -- 172.21.15.5:0/2550368642 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f739c030770 con 0x7f73a81543a0 2024-03-24T10:49:48.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.958+0000 7f73a77fe640 1 -- 172.21.15.5:0/2550368642 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c030b00 con 0x7f73a81543a0 2024-03-24T10:49:48.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.959+0000 7f73af264640 1 -- 172.21.15.5:0/2550368642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f73a81543a0 msgr2=0x7f73a815ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:48.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.959+0000 7f73af264640 1 --2- 172.21.15.5:0/2550368642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f73a81543a0 0x7f73a815ac00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f739c0097a0 tx=0x7f739c02b960 comp rx=0 tx=0).stop 2024-03-24T10:49:48.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.959+0000 7f73af264640 1 -- 172.21.15.5:0/2550368642 shutdown_connections 2024-03-24T10:49:48.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.959+0000 7f73af264640 1 --2- 172.21.15.5:0/2550368642 >> v2:172.21.15.5:3300/0 conn(0x7f73a806bf50 0x7f73a806c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:48.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.959+0000 7f73af264640 1 --2- 172.21.15.5:0/2550368642 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f73a81543a0 0x7f73a815ac00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:48.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.959+0000 7f73af264640 1 -- 172.21.15.5:0/2550368642 >> 172.21.15.5:0/2550368642 conn(0x7f73a8072ef0 msgr2=0x7f73a8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:48.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.959+0000 7f73af264640 1 -- 172.21.15.5:0/2550368642 shutdown_connections 2024-03-24T10:49:48.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.959+0000 7f73af264640 1 -- 172.21.15.5:0/2550368642 wait complete. 2024-03-24T10:49:48.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.961+0000 7f73af264640 1 Processor -- start 2024-03-24T10:49:48.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.961+0000 7f73af264640 1 -- start start 2024-03-24T10:49:48.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.962+0000 7f73af264640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73a806bf50 0x7f73a81f0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:48.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.962+0000 7f73af264640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f73a806d4f0 0x7f73a81e4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:48.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.962+0000 7f73af264640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f73a81543a0 0x7f73a81e6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:48.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.962+0000 7f73af264640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f73a81f0b80 con 0x7f73a806d4f0 2024-03-24T10:49:48.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.962+0000 7f73af264640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f73a81567d0 con 0x7f73a806bf50 2024-03-24T10:49:48.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.962+0000 7f73af264640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f73a81e4c60 con 0x7f73a81543a0 2024-03-24T10:49:48.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.963+0000 7f73ad801640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f73a81543a0 0x7f73a81e6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:48.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.963+0000 7f73ad000640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73a806bf50 0x7f73a81f0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:48.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.963+0000 7f73ad000640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73a806bf50 0x7f73a81f0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:45540/0 (socket says 172.21.15.5:45540) 2024-03-24T10:49:48.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.963+0000 7f73ad000640 1 -- 172.21.15.5:0/2454274899 learned_addr learned my addr 172.21.15.5:0/2454274899 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:48.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.968+0000 7f73ad000640 1 -- 172.21.15.5:0/2454274899 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f73a81543a0 msgr2=0x7f73a81e6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:48.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.968+0000 7f73ad000640 1 --2- 172.21.15.5:0/2454274899 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f73a81543a0 0x7f73a81e6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:48.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.968+0000 7f73ad000640 1 -- 172.21.15.5:0/2454274899 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f73a806d4f0 msgr2=0x7f73a81e4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:48.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.968+0000 7f73ad000640 1 --2- 172.21.15.5:0/2454274899 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f73a806d4f0 0x7f73a81e4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:48.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.969+0000 7f73ad000640 1 -- 172.21.15.5:0/2454274899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739c009400 con 0x7f73a806bf50 2024-03-24T10:49:48.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.972+0000 7f73ad000640 1 --2- 172.21.15.5:0/2454274899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73a806bf50 0x7f73a81f0640 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f739c0098c0 tx=0x7f739c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:48.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.977+0000 7f73a57fa640 1 -- 172.21.15.5:0/2454274899 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c03e070 con 0x7f73a806bf50 2024-03-24T10:49:48.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.977+0000 7f73a57fa640 1 -- 172.21.15.5:0/2454274899 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f739c030e10 con 0x7f73a806bf50 2024-03-24T10:49:48.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.977+0000 7f73a57fa640 1 -- 172.21.15.5:0/2454274899 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f739c039c30 con 0x7f73a806bf50 2024-03-24T10:49:48.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.977+0000 7f73af264640 1 -- 172.21.15.5:0/2454274899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73a8155ee0 con 0x7f73a806bf50 2024-03-24T10:49:48.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.977+0000 7f73af264640 1 -- 172.21.15.5:0/2454274899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73a8156180 con 0x7f73a806bf50 2024-03-24T10:49:48.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:48.978+0000 7f73af264640 1 -- 172.21.15.5:0/2454274899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73a80681d0 con 0x7f73a806bf50 2024-03-24T10:49:49.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:49.015+0000 7f73a57fa640 1 -- 172.21.15.5:0/2454274899 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f739c0423f0 con 0x7f73a806bf50 2024-03-24T10:49:49.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:49.015+0000 7f73a57fa640 1 --2- 172.21.15.5:0/2454274899 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f738c06b860 0x7f738c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:49.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:49.016+0000 7f73a7fff640 1 --2- 172.21.15.5:0/2454274899 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f738c06b860 0x7f738c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:49.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:49.016+0000 7f73a7fff640 1 --2- 172.21.15.5:0/2454274899 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f738c06b860 0x7f738c06dd20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7394000f60 tx=0x7f7394007a80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:49.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:49.284+0000 7f73a57fa640 1 -- 172.21.15.5:0/2454274899 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f739c07ab80 con 0x7f73a806bf50 2024-03-24T10:49:49.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:49.296+0000 7f73a57fa640 1 -- 172.21.15.5:0/2454274899 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f739c076e20 con 0x7f73a806bf50 2024-03-24T10:49:49.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:49.516+0000 7f73af264640 1 -- 172.21.15.5:0/2454274899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73a814f100 con 0x7f73a806bf50 2024-03-24T10:49:49.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:49.541+0000 7f73a57fa640 1 -- 172.21.15.5:0/2454274899 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f739c048020 con 0x7f73a806bf50 2024-03-24T10:49:49.541 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:49.541 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:49.541 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:49.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:49.543+0000 7f73af264640 1 -- 172.21.15.5:0/2454274899 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f738c06b860 msgr2=0x7f738c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:49.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:49.543+0000 7f73af264640 1 --2- 172.21.15.5:0/2454274899 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f738c06b860 0x7f738c06dd20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7394000f60 tx=0x7f7394007a80 comp rx=0 tx=0).stop 2024-03-24T10:49:49.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:49.543+0000 7f73af264640 1 -- 172.21.15.5:0/2454274899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73a806bf50 msgr2=0x7f73a81f0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:49.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:49.543+0000 7f73af264640 1 --2- 172.21.15.5:0/2454274899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73a806bf50 0x7f73a81f0640 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f739c0098c0 tx=0x7f739c0046c0 comp rx=0 tx=0).stop 2024-03-24T10:49:49.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:49.543+0000 7f73af264640 1 -- 172.21.15.5:0/2454274899 shutdown_connections 2024-03-24T10:49:49.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:49.543+0000 7f73af264640 1 --2- 172.21.15.5:0/2454274899 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f738c06b860 0x7f738c06dd20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:49.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:49.543+0000 7f73af264640 1 --2- 172.21.15.5:0/2454274899 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f73a806d4f0 0x7f73a81e4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:49.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:49.543+0000 7f73af264640 1 --2- 172.21.15.5:0/2454274899 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f73a81543a0 0x7f73a81e6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:49.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:49.543+0000 7f73af264640 1 --2- 172.21.15.5:0/2454274899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f73a806bf50 0x7f73a81f0640 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:49.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:49.543+0000 7f73af264640 1 -- 172.21.15.5:0/2454274899 >> 172.21.15.5:0/2454274899 conn(0x7f73a8072ef0 msgr2=0x7f73a8153650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:49.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:49.543+0000 7f73af264640 1 -- 172.21.15.5:0/2454274899 shutdown_connections 2024-03-24T10:49:49.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:49.543+0000 7f73af264640 1 -- 172.21.15.5:0/2454274899 wait complete. 2024-03-24T10:49:50.160 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:50.161 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:51.162 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:51.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.264+0000 7ff85813e640 1 Processor -- start 2024-03-24T10:49:51.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.264+0000 7ff85813e640 1 -- start start 2024-03-24T10:49:51.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.265+0000 7ff85813e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff8500c8c30 0x7ff8500c96d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:51.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.265+0000 7ff85813e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff85015be90 0x7ff850162a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:51.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.265+0000 7ff85813e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8500c8850 con 0x7ff8500c92f0 2024-03-24T10:49:51.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.265+0000 7ff85813e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7ff8500c9c10 con 0x7ff8500c8c30 2024-03-24T10:49:51.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.265+0000 7ff85813e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff850152560 con 0x7ff85015be90 2024-03-24T10:49:51.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.265+0000 7ff8556d9640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff8500c8c30 0x7ff8500c96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:51.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.265+0000 7ff855eda640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff8500c92f0 0x7ff85015b780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:33112/0 (socket says 172.21.15.5:33112) 2024-03-24T10:49:51.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.266+0000 7ff855eda640 1 -- 172.21.15.5:0/2389727323 learned_addr learned my addr 172.21.15.5:0/2389727323 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:49:51.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.266+0000 7ff8566db640 1 --2- 172.21.15.5:0/2389727323 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff85015be90 0x7ff850162a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:51.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.268+0000 7ff854ed8640 1 -- 172.21.15.5:0/2389727323 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1578840128 0 0) 0x7ff8500c8850 con 0x7ff8500c92f0 2024-03-24T10:49:51.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.268+0000 7ff854ed8640 1 -- 172.21.15.5:0/2389727323 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff83c003520 con 0x7ff8500c92f0 2024-03-24T10:49:51.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.269+0000 7ff8556d9640 1 -- 172.21.15.5:0/2389727323 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff85015be90 msgr2=0x7ff850162a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:51.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.269+0000 7ff8556d9640 1 --2- 172.21.15.5:0/2389727323 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff85015be90 0x7ff850162a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:51.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.269+0000 7ff8556d9640 1 -- 172.21.15.5:0/2389727323 >> v1:172.21.15.57:6789/0 conn(0x7ff8500c92f0 legacy=0x7ff85015b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:51.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.269+0000 7ff8556d9640 1 -- 172.21.15.5:0/2389727323 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8500b3b80 con 0x7ff8500c8c30 2024-03-24T10:49:51.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.272+0000 7ff8556d9640 1 --2- 172.21.15.5:0/2389727323 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff8500c8c30 0x7ff8500c96d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff8400097a0 tx=0x7ff84002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc993bceee032da server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:51.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.276+0000 7ff854ed8640 1 -- 172.21.15.5:0/2389727323 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff840030610 con 0x7ff8500c8c30 2024-03-24T10:49:51.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.276+0000 7ff854ed8640 1 -- 172.21.15.5:0/2389727323 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff840030770 con 0x7ff8500c8c30 2024-03-24T10:49:51.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.276+0000 7ff854ed8640 1 -- 172.21.15.5:0/2389727323 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff840030b00 con 0x7ff8500c8c30 2024-03-24T10:49:51.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.277+0000 7ff85813e640 1 -- 172.21.15.5:0/2389727323 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff8500c8c30 msgr2=0x7ff8500c96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:51.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.277+0000 7ff85813e640 1 --2- 172.21.15.5:0/2389727323 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff8500c8c30 0x7ff8500c96d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff8400097a0 tx=0x7ff84002b960 comp rx=0 tx=0).stop 2024-03-24T10:49:51.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.277+0000 7ff85813e640 1 -- 172.21.15.5:0/2389727323 shutdown_connections 2024-03-24T10:49:51.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.277+0000 7ff85813e640 1 --2- 172.21.15.5:0/2389727323 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff85015be90 0x7ff850162a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:51.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.277+0000 7ff85813e640 1 --2- 172.21.15.5:0/2389727323 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff8500c8c30 0x7ff8500c96d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:51.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.277+0000 7ff85813e640 1 -- 172.21.15.5:0/2389727323 >> 172.21.15.5:0/2389727323 conn(0x7ff85006f820 msgr2=0x7ff850061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:51.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.277+0000 7ff85813e640 1 -- 172.21.15.5:0/2389727323 shutdown_connections 2024-03-24T10:49:51.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.277+0000 7ff85813e640 1 -- 172.21.15.5:0/2389727323 wait complete. 2024-03-24T10:49:51.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.279+0000 7ff85813e640 1 Processor -- start 2024-03-24T10:49:51.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.279+0000 7ff85813e640 1 -- start start 2024-03-24T10:49:51.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.279+0000 7ff85813e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8500c8c30 0x7ff8501f03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:51.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.279+0000 7ff85813e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff8500c92f0 0x7ff850148070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:51.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.279+0000 7ff85813e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff85015be90 0x7ff85006d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:51.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.279+0000 7ff85813e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff8501e1350 con 0x7ff85015be90 2024-03-24T10:49:51.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.279+0000 7ff85813e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff8501f0900 con 0x7ff8500c8c30 2024-03-24T10:49:51.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.279+0000 7ff85813e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff850150290 con 0x7ff8500c92f0 2024-03-24T10:49:51.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.280+0000 7ff8566db640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff85015be90 0x7ff85006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:51.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.280+0000 7ff855eda640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8500c8c30 0x7ff8501f03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:51.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.280+0000 7ff8556d9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff8500c92f0 0x7ff850148070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:51.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.280+0000 7ff855eda640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8500c8c30 0x7ff8501f03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:45546/0 (socket says 172.21.15.5:45546) 2024-03-24T10:49:51.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.280+0000 7ff855eda640 1 -- 172.21.15.5:0/3407784402 learned_addr learned my addr 172.21.15.5:0/3407784402 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:51.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.284+0000 7ff855eda640 1 -- 172.21.15.5:0/3407784402 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff8500c92f0 msgr2=0x7ff850148070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:51.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.284+0000 7ff855eda640 1 --2- 172.21.15.5:0/3407784402 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff8500c92f0 0x7ff850148070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:51.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.284+0000 7ff855eda640 1 -- 172.21.15.5:0/3407784402 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff85015be90 msgr2=0x7ff85006d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:51.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.284+0000 7ff855eda640 1 --2- 172.21.15.5:0/3407784402 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff85015be90 0x7ff85006d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:51.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.284+0000 7ff855eda640 1 -- 172.21.15.5:0/3407784402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff840009400 con 0x7ff8500c8c30 2024-03-24T10:49:51.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.287+0000 7ff855eda640 1 --2- 172.21.15.5:0/3407784402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8500c8c30 0x7ff8501f03c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff84c00c430 tx=0x7ff84c00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:51.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.292+0000 7ff8467fc640 1 -- 172.21.15.5:0/3407784402 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c005ce0 con 0x7ff8500c8c30 2024-03-24T10:49:51.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.292+0000 7ff8467fc640 1 -- 172.21.15.5:0/3407784402 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff84c005e40 con 0x7ff8500c8c30 2024-03-24T10:49:51.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.292+0000 7ff85813e640 1 -- 172.21.15.5:0/3407784402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff850150530 con 0x7ff8500c8c30 2024-03-24T10:49:51.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.292+0000 7ff8467fc640 1 -- 172.21.15.5:0/3407784402 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c0072f0 con 0x7ff8500c8c30 2024-03-24T10:49:51.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.292+0000 7ff85813e640 1 -- 172.21.15.5:0/3407784402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff85006ca60 con 0x7ff8500c8c30 2024-03-24T10:49:51.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.293+0000 7ff85813e640 1 -- 172.21.15.5:0/3407784402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff85006c5e0 con 0x7ff8500c8c30 2024-03-24T10:49:51.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.312+0000 7ff8467fc640 1 -- 172.21.15.5:0/3407784402 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff84c007450 con 0x7ff8500c8c30 2024-03-24T10:49:51.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.313+0000 7ff8467fc640 1 --2- 172.21.15.5:0/3407784402 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff83406b840 0x7ff83406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:51.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.313+0000 7ff8556d9640 1 --2- 172.21.15.5:0/3407784402 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff83406b840 0x7ff83406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:51.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.313+0000 7ff8556d9640 1 --2- 172.21.15.5:0/3407784402 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff83406b840 0x7ff83406dd00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff840036000 tx=0x7ff840008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:51.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.318+0000 7ff8467fc640 1 -- 172.21.15.5:0/3407784402 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7ff84c0582c0 con 0x7ff8500c8c30 2024-03-24T10:49:51.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.590+0000 7ff8467fc640 1 -- 172.21.15.5:0/3407784402 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff84c054560 con 0x7ff8500c8c30 2024-03-24T10:49:51.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.834+0000 7ff85813e640 1 -- 172.21.15.5:0/3407784402 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff85006c5e0 con 0x7ff8500c8c30 2024-03-24T10:49:51.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.857+0000 7ff8467fc640 1 -- 172.21.15.5:0/3407784402 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7ff85006c5e0 con 0x7ff8500c8c30 2024-03-24T10:49:51.858 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:51.858 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:51.858 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:51.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.859+0000 7ff85813e640 1 -- 172.21.15.5:0/3407784402 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff83406b840 msgr2=0x7ff83406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:51.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.859+0000 7ff85813e640 1 --2- 172.21.15.5:0/3407784402 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff83406b840 0x7ff83406dd00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff840036000 tx=0x7ff840008030 comp rx=0 tx=0).stop 2024-03-24T10:49:51.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.859+0000 7ff85813e640 1 -- 172.21.15.5:0/3407784402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8500c8c30 msgr2=0x7ff8501f03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:51.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.859+0000 7ff85813e640 1 --2- 172.21.15.5:0/3407784402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8500c8c30 0x7ff8501f03c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff84c00c430 tx=0x7ff84c00c8f0 comp rx=0 tx=0).stop 2024-03-24T10:49:51.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.860+0000 7ff85813e640 1 -- 172.21.15.5:0/3407784402 shutdown_connections 2024-03-24T10:49:51.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.860+0000 7ff85813e640 1 --2- 172.21.15.5:0/3407784402 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff83406b840 0x7ff83406dd00 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:51.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.860+0000 7ff85813e640 1 --2- 172.21.15.5:0/3407784402 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff85015be90 0x7ff85006d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:51.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.860+0000 7ff85813e640 1 --2- 172.21.15.5:0/3407784402 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff8500c92f0 0x7ff850148070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:51.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.860+0000 7ff85813e640 1 --2- 172.21.15.5:0/3407784402 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff8500c8c30 0x7ff8501f03c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:51.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.860+0000 7ff85813e640 1 -- 172.21.15.5:0/3407784402 >> 172.21.15.5:0/3407784402 conn(0x7ff85006f820 msgr2=0x7ff850157900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:51.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.860+0000 7ff85813e640 1 -- 172.21.15.5:0/3407784402 shutdown_connections 2024-03-24T10:49:51.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:51.860+0000 7ff85813e640 1 -- 172.21.15.5:0/3407784402 wait complete. 2024-03-24T10:49:52.475 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:52.475 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:53.476 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:53.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.577+0000 7ff083c8f640 1 Processor -- start 2024-03-24T10:49:53.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.578+0000 7ff083c8f640 1 -- start start 2024-03-24T10:49:53.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.578+0000 7ff083c8f640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff07c1543a0 0x7ff07c15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:53.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.578+0000 7ff083c8f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff07c06bf50 0x7ff07c06c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:53.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.578+0000 7ff083c8f640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff07c0cb2f0 con 0x7ff07c06d4f0 2024-03-24T10:49:53.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.578+0000 7ff083c8f640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7ff07c0c6750 con 0x7ff07c1543a0 2024-03-24T10:49:53.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.578+0000 7ff083c8f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7ff07c0cc380 con 0x7ff07c06bf50 2024-03-24T10:49:53.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.579+0000 7ff081a2b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff07c1543a0 0x7ff07c15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:53.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.579+0000 7ff08222c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff07c06bf50 0x7ff07c06c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:53.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.580+0000 7ff08122a640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7ff07c06d4f0 0x7ff07c0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:45732/0 (socket says 172.21.15.5:45732) 2024-03-24T10:49:53.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.580+0000 7ff08122a640 1 -- 172.21.15.5:0/1640358740 learned_addr learned my addr 172.21.15.5:0/1640358740 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:49:53.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.583+0000 7ff08222c640 1 -- 172.21.15.5:0/1640358740 >> v2:172.21.15.5:3300/0 conn(0x7ff07c1543a0 msgr2=0x7ff07c15ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:53.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.583+0000 7ff08222c640 1 --2- 172.21.15.5:0/1640358740 >> v2:172.21.15.5:3300/0 conn(0x7ff07c1543a0 0x7ff07c15ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:53.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.583+0000 7ff08222c640 1 -- 172.21.15.5:0/1640358740 >> v1:172.21.15.5:6789/0 conn(0x7ff07c06d4f0 legacy=0x7ff07c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:53.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.583+0000 7ff08222c640 1 -- 172.21.15.5:0/1640358740 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff07c0ca7e0 con 0x7ff07c06bf50 2024-03-24T10:49:53.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.587+0000 7ff08222c640 1 --2- 172.21.15.5:0/1640358740 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff07c06bf50 0x7ff07c06c330 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff07c0b51c0 tx=0x7ff07402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57be5d2ac6b55008 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:53.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.592+0000 7ff080a29640 1 -- 172.21.15.5:0/1640358740 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07402cd00 con 0x7ff07c06bf50 2024-03-24T10:49:53.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.592+0000 7ff080a29640 1 -- 172.21.15.5:0/1640358740 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff07402ce60 con 0x7ff07c06bf50 2024-03-24T10:49:53.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.592+0000 7ff080a29640 1 -- 172.21.15.5:0/1640358740 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff074034660 con 0x7ff07c06bf50 2024-03-24T10:49:53.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.592+0000 7ff083c8f640 1 -- 172.21.15.5:0/1640358740 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff07c06bf50 msgr2=0x7ff07c06c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:53.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.592+0000 7ff083c8f640 1 --2- 172.21.15.5:0/1640358740 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff07c06bf50 0x7ff07c06c330 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff07c0b51c0 tx=0x7ff07402bf50 comp rx=0 tx=0).stop 2024-03-24T10:49:53.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.592+0000 7ff083c8f640 1 -- 172.21.15.5:0/1640358740 shutdown_connections 2024-03-24T10:49:53.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.592+0000 7ff083c8f640 1 --2- 172.21.15.5:0/1640358740 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff07c06bf50 0x7ff07c06c330 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:53.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.592+0000 7ff083c8f640 1 --2- 172.21.15.5:0/1640358740 >> v2:172.21.15.5:3300/0 conn(0x7ff07c1543a0 0x7ff07c15ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:53.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.592+0000 7ff083c8f640 1 -- 172.21.15.5:0/1640358740 >> 172.21.15.5:0/1640358740 conn(0x7ff07c072ef0 msgr2=0x7ff07c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:53.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.593+0000 7ff083c8f640 1 -- 172.21.15.5:0/1640358740 shutdown_connections 2024-03-24T10:49:53.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.593+0000 7ff083c8f640 1 -- 172.21.15.5:0/1640358740 wait complete. 2024-03-24T10:49:53.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.594+0000 7ff083c8f640 1 Processor -- start 2024-03-24T10:49:53.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.594+0000 7ff083c8f640 1 -- start start 2024-03-24T10:49:53.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.594+0000 7ff083c8f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff07c06bf50 0x7ff07c0cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:53.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.595+0000 7ff083c8f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff07c06d4f0 0x7ff07c152820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:53.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.595+0000 7ff083c8f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff07c1543a0 0x7ff07c152d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:53.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.595+0000 7ff083c8f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c1502b0 con 0x7ff07c1543a0 2024-03-24T10:49:53.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.595+0000 7ff083c8f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c15bad0 con 0x7ff07c06bf50 2024-03-24T10:49:53.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.595+0000 7ff083c8f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff07c14f210 con 0x7ff07c06d4f0 2024-03-24T10:49:53.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.595+0000 7ff08222c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff07c1543a0 0x7ff07c152d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:53.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.595+0000 7ff08122a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff07c06d4f0 0x7ff07c152820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:53.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.595+0000 7ff081a2b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff07c06bf50 0x7ff07c0cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:53.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.596+0000 7ff08222c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff07c1543a0 0x7ff07c152d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:47672/0 (socket says 172.21.15.5:47672) 2024-03-24T10:49:53.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.596+0000 7ff08222c640 1 -- 172.21.15.5:0/4121147621 learned_addr learned my addr 172.21.15.5:0/4121147621 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:53.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.601+0000 7ff08222c640 1 -- 172.21.15.5:0/4121147621 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff07c06d4f0 msgr2=0x7ff07c152820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:53.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.601+0000 7ff08222c640 1 --2- 172.21.15.5:0/4121147621 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff07c06d4f0 0x7ff07c152820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:53.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.601+0000 7ff08222c640 1 -- 172.21.15.5:0/4121147621 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff07c06bf50 msgr2=0x7ff07c0cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:53.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.601+0000 7ff08222c640 1 --2- 172.21.15.5:0/4121147621 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff07c06bf50 0x7ff07c0cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:53.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.601+0000 7ff08222c640 1 -- 172.21.15.5:0/4121147621 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff074009a70 con 0x7ff07c1543a0 2024-03-24T10:49:53.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.604+0000 7ff08222c640 1 --2- 172.21.15.5:0/4121147621 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff07c1543a0 0x7ff07c152d60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff074009e80 tx=0x7ff074004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:53.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.612+0000 7ff0727fc640 1 -- 172.21.15.5:0/4121147621 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07403d600 con 0x7ff07c1543a0 2024-03-24T10:49:53.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.612+0000 7ff0727fc640 1 -- 172.21.15.5:0/4121147621 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff07403d780 con 0x7ff07c1543a0 2024-03-24T10:49:53.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.612+0000 7ff0727fc640 1 -- 172.21.15.5:0/4121147621 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07403db30 con 0x7ff07c1543a0 2024-03-24T10:49:53.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.612+0000 7ff083c8f640 1 -- 172.21.15.5:0/4121147621 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff07c14f4f0 con 0x7ff07c1543a0 2024-03-24T10:49:53.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.612+0000 7ff083c8f640 1 -- 172.21.15.5:0/4121147621 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff07c14fab0 con 0x7ff07c1543a0 2024-03-24T10:49:53.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.613+0000 7ff083c8f640 1 -- 172.21.15.5:0/4121147621 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff07c0681d0 con 0x7ff07c1543a0 2024-03-24T10:49:53.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.914+0000 7ff0727fc640 1 -- 172.21.15.5:0/4121147621 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff07403b040 con 0x7ff07c1543a0 2024-03-24T10:49:53.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.914+0000 7ff0727fc640 1 --2- 172.21.15.5:0/4121147621 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff06006b890 0x7ff06006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:53.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.914+0000 7ff081a2b640 1 --2- 172.21.15.5:0/4121147621 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff06006b890 0x7ff06006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:53.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.915+0000 7ff081a2b640 1 --2- 172.21.15.5:0/4121147621 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff06006b890 0x7ff06006dd50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff078009aa0 tx=0x7ff078009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:53.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.932+0000 7ff0727fc640 1 -- 172.21.15.5:0/4121147621 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7ff07407b310 con 0x7ff07c1543a0 2024-03-24T10:49:53.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:53.933+0000 7ff0727fc640 1 -- 172.21.15.5:0/4121147621 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff0740775b0 con 0x7ff07c1543a0 2024-03-24T10:49:54.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:54.157+0000 7ff083c8f640 1 -- 172.21.15.5:0/4121147621 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff07c061f90 con 0x7ff07c1543a0 2024-03-24T10:49:54.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:54.171+0000 7ff0727fc640 1 -- 172.21.15.5:0/4121147621 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7ff07404e020 con 0x7ff07c1543a0 2024-03-24T10:49:54.174 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:54.174 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:54.175 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:54.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:54.173+0000 7ff083c8f640 1 -- 172.21.15.5:0/4121147621 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff06006b890 msgr2=0x7ff06006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:54.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:54.173+0000 7ff083c8f640 1 --2- 172.21.15.5:0/4121147621 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff06006b890 0x7ff06006dd50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff078009aa0 tx=0x7ff078009190 comp rx=0 tx=0).stop 2024-03-24T10:49:54.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:54.173+0000 7ff083c8f640 1 -- 172.21.15.5:0/4121147621 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff07c1543a0 msgr2=0x7ff07c152d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:54.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:54.173+0000 7ff083c8f640 1 --2- 172.21.15.5:0/4121147621 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff07c1543a0 0x7ff07c152d60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff074009e80 tx=0x7ff074004a40 comp rx=0 tx=0).stop 2024-03-24T10:49:54.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:54.173+0000 7ff083c8f640 1 -- 172.21.15.5:0/4121147621 shutdown_connections 2024-03-24T10:49:54.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:54.173+0000 7ff083c8f640 1 --2- 172.21.15.5:0/4121147621 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff06006b890 0x7ff06006dd50 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:54.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:54.173+0000 7ff083c8f640 1 --2- 172.21.15.5:0/4121147621 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff07c1543a0 0x7ff07c152d60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:54.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:54.173+0000 7ff083c8f640 1 --2- 172.21.15.5:0/4121147621 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff07c06d4f0 0x7ff07c152820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:54.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:54.173+0000 7ff083c8f640 1 --2- 172.21.15.5:0/4121147621 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff07c06bf50 0x7ff07c0cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:54.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:54.173+0000 7ff083c8f640 1 -- 172.21.15.5:0/4121147621 >> 172.21.15.5:0/4121147621 conn(0x7ff07c072ef0 msgr2=0x7ff07c15a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:54.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:54.173+0000 7ff083c8f640 1 -- 172.21.15.5:0/4121147621 shutdown_connections 2024-03-24T10:49:54.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:54.173+0000 7ff083c8f640 1 -- 172.21.15.5:0/4121147621 wait complete. 2024-03-24T10:49:54.790 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:54.790 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:55.791 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.893+0000 7f53286c2640 1 Processor -- start 2024-03-24T10:49:55.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.893+0000 7f53286c2640 1 -- start start 2024-03-24T10:49:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.894+0000 7f53286c2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f53200ccb50 0x7f532015ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.894+0000 7f53286c2640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f53200ccf30 0x7f5320072ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.894+0000 7f53286c2640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53200c5ac0 con 0x7f5320150d80 2024-03-24T10:49:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.894+0000 7f53286c2640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f53200c4270 con 0x7f53200ccf30 2024-03-24T10:49:55.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.894+0000 7f53286c2640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f53200c4550 con 0x7f53200ccb50 2024-03-24T10:49:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.894+0000 7f5325c5d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f53200ccf30 0x7f5320072ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.894+0000 7f5326c5f640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f5320150d80 0x7f53200c93c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:45742/0 (socket says 172.21.15.5:45742) 2024-03-24T10:49:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.895+0000 7f5326c5f640 1 -- 172.21.15.5:0/1259278338 learned_addr learned my addr 172.21.15.5:0/1259278338 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:49:55.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.895+0000 7f532645e640 1 --2- 172.21.15.5:0/1259278338 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f53200ccb50 0x7f532015ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.897+0000 7f532545c640 1 -- 172.21.15.5:0/1259278338 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3874253584 0 0) 0x7f53200c5ac0 con 0x7f5320150d80 2024-03-24T10:49:55.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.897+0000 7f532545c640 1 -- 172.21.15.5:0/1259278338 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f530c005730 con 0x7f5320150d80 2024-03-24T10:49:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.898+0000 7f5325c5d640 1 -- 172.21.15.5:0/1259278338 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f53200ccb50 msgr2=0x7f532015ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.898+0000 7f5325c5d640 1 --2- 172.21.15.5:0/1259278338 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f53200ccb50 0x7f532015ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.898+0000 7f5325c5d640 1 -- 172.21.15.5:0/1259278338 >> v1:172.21.15.5:6789/0 conn(0x7f5320150d80 legacy=0x7f53200c93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:55.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.898+0000 7f5325c5d640 1 -- 172.21.15.5:0/1259278338 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53200c4830 con 0x7f53200ccf30 2024-03-24T10:49:55.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.901+0000 7f5325c5d640 1 --2- 172.21.15.5:0/1259278338 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53200ccf30 0x7f5320072ce0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f53100097a0 tx=0x7f531002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74660a1b5cacbe17 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.906+0000 7f532545c640 1 -- 172.21.15.5:0/1259278338 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5310030610 con 0x7f53200ccf30 2024-03-24T10:49:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.906+0000 7f532545c640 1 -- 172.21.15.5:0/1259278338 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5310030770 con 0x7f53200ccf30 2024-03-24T10:49:55.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.906+0000 7f532545c640 1 -- 172.21.15.5:0/1259278338 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5310030b00 con 0x7f53200ccf30 2024-03-24T10:49:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.907+0000 7f53286c2640 1 -- 172.21.15.5:0/1259278338 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53200ccf30 msgr2=0x7f5320072ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.907+0000 7f53286c2640 1 --2- 172.21.15.5:0/1259278338 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53200ccf30 0x7f5320072ce0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f53100097a0 tx=0x7f531002b960 comp rx=0 tx=0).stop 2024-03-24T10:49:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.907+0000 7f53286c2640 1 -- 172.21.15.5:0/1259278338 shutdown_connections 2024-03-24T10:49:55.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.907+0000 7f53286c2640 1 --2- 172.21.15.5:0/1259278338 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53200ccf30 0x7f5320072ce0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.907+0000 7f53286c2640 1 --2- 172.21.15.5:0/1259278338 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f53200ccb50 0x7f532015ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.907+0000 7f53286c2640 1 -- 172.21.15.5:0/1259278338 >> 172.21.15.5:0/1259278338 conn(0x7f532006c240 msgr2=0x7f532006c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.907+0000 7f53286c2640 1 -- 172.21.15.5:0/1259278338 shutdown_connections 2024-03-24T10:49:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.907+0000 7f53286c2640 1 -- 172.21.15.5:0/1259278338 wait complete. 2024-03-24T10:49:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.908+0000 7f53286c2640 1 Processor -- start 2024-03-24T10:49:55.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.908+0000 7f53286c2640 1 -- start start 2024-03-24T10:49:55.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.909+0000 7f53286c2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53200ccb50 0x7f5320159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.909+0000 7f53286c2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53200ccf30 0x7f532014c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.909+0000 7f53286c2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5320150d80 0x7f532014ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:55.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.909+0000 7f53286c2640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f53201f0c80 con 0x7f53200ccb50 2024-03-24T10:49:55.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.909+0000 7f53286c2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5320144950 con 0x7f53200ccf30 2024-03-24T10:49:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.909+0000 7f53286c2640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f5320144bf0 con 0x7f5320150d80 2024-03-24T10:49:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.909+0000 7f532645e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53200ccb50 0x7f5320159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.910+0000 7f5326c5f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5320150d80 0x7f532014ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.910+0000 7f5325c5d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53200ccf30 0x7f532014c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.910+0000 7f532645e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53200ccb50 0x7f5320159790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:47688/0 (socket says 172.21.15.5:47688) 2024-03-24T10:49:55.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.910+0000 7f532645e640 1 -- 172.21.15.5:0/3734992574 learned_addr learned my addr 172.21.15.5:0/3734992574 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:55.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.914+0000 7f532645e640 1 -- 172.21.15.5:0/3734992574 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5320150d80 msgr2=0x7f532014ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:55.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.914+0000 7f532645e640 1 --2- 172.21.15.5:0/3734992574 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5320150d80 0x7f532014ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.914+0000 7f532645e640 1 -- 172.21.15.5:0/3734992574 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53200ccf30 msgr2=0x7f532014c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:55.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.914+0000 7f532645e640 1 --2- 172.21.15.5:0/3734992574 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53200ccf30 0x7f532014c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:55.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.914+0000 7f532645e640 1 -- 172.21.15.5:0/3734992574 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5310009400 con 0x7f53200ccb50 2024-03-24T10:49:55.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.916+0000 7f532645e640 1 --2- 172.21.15.5:0/3734992574 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53200ccb50 0x7f5320159790 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f531c00e870 tx=0x7f531c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:55.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.920+0000 7f5316ffd640 1 -- 172.21.15.5:0/3734992574 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531c0094f0 con 0x7f53200ccb50 2024-03-24T10:49:55.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.920+0000 7f5316ffd640 1 -- 172.21.15.5:0/3734992574 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f531c00c940 con 0x7f53200ccb50 2024-03-24T10:49:55.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.920+0000 7f53286c2640 1 -- 172.21.15.5:0/3734992574 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53201f0070 con 0x7f53200ccb50 2024-03-24T10:49:55.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.920+0000 7f5316ffd640 1 -- 172.21.15.5:0/3734992574 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531c00ccf0 con 0x7f53200ccb50 2024-03-24T10:49:55.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.920+0000 7f53286c2640 1 -- 172.21.15.5:0/3734992574 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53201f0630 con 0x7f53200ccb50 2024-03-24T10:49:55.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:55.921+0000 7f53286c2640 1 -- 172.21.15.5:0/3734992574 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53200681d0 con 0x7f53200ccb50 2024-03-24T10:49:56.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:56.202+0000 7f5316ffd640 1 -- 172.21.15.5:0/3734992574 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f531c013470 con 0x7f53200ccb50 2024-03-24T10:49:56.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:56.202+0000 7f5316ffd640 1 --2- 172.21.15.5:0/3734992574 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f530406b890 0x7f530406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:56.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:56.202+0000 7f5325c5d640 1 --2- 172.21.15.5:0/3734992574 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f530406b890 0x7f530406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:56.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:56.203+0000 7f5325c5d640 1 --2- 172.21.15.5:0/3734992574 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f530406b890 0x7f530406dd50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5310036000 tx=0x7f5310008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:56.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:56.218+0000 7f5316ffd640 1 -- 172.21.15.5:0/3734992574 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f531c059190 con 0x7f53200ccb50 2024-03-24T10:49:56.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:56.219+0000 7f5316ffd640 1 -- 172.21.15.5:0/3734992574 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f531c055430 con 0x7f53200ccb50 2024-03-24T10:49:56.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:56.476+0000 7f53286c2640 1 -- 172.21.15.5:0/3734992574 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53200c3e50 con 0x7f53200ccb50 2024-03-24T10:49:56.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:56.498+0000 7f5316ffd640 1 -- 172.21.15.5:0/3734992574 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f531c021090 con 0x7f53200ccb50 2024-03-24T10:49:56.500 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:56.500 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:56.500 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:56.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:56.500+0000 7f53286c2640 1 -- 172.21.15.5:0/3734992574 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f530406b890 msgr2=0x7f530406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:56.500+0000 7f53286c2640 1 --2- 172.21.15.5:0/3734992574 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f530406b890 0x7f530406dd50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5310036000 tx=0x7f5310008030 comp rx=0 tx=0).stop 2024-03-24T10:49:56.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:56.500+0000 7f53286c2640 1 -- 172.21.15.5:0/3734992574 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53200ccb50 msgr2=0x7f5320159790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:56.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:56.500+0000 7f53286c2640 1 --2- 172.21.15.5:0/3734992574 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53200ccb50 0x7f5320159790 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f531c00e870 tx=0x7f531c00ed30 comp rx=0 tx=0).stop 2024-03-24T10:49:56.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:56.500+0000 7f53286c2640 1 -- 172.21.15.5:0/3734992574 shutdown_connections 2024-03-24T10:49:56.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:56.500+0000 7f53286c2640 1 --2- 172.21.15.5:0/3734992574 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f530406b890 0x7f530406dd50 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:56.500+0000 7f53286c2640 1 --2- 172.21.15.5:0/3734992574 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5320150d80 0x7f532014ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:56.500+0000 7f53286c2640 1 --2- 172.21.15.5:0/3734992574 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53200ccf30 0x7f532014c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:56.500+0000 7f53286c2640 1 --2- 172.21.15.5:0/3734992574 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53200ccb50 0x7f5320159790 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:56.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:56.500+0000 7f53286c2640 1 -- 172.21.15.5:0/3734992574 >> 172.21.15.5:0/3734992574 conn(0x7f532006c240 msgr2=0x7f5320145740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:56.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:56.501+0000 7f53286c2640 1 -- 172.21.15.5:0/3734992574 shutdown_connections 2024-03-24T10:49:56.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:56.501+0000 7f53286c2640 1 -- 172.21.15.5:0/3734992574 wait complete. 2024-03-24T10:49:57.117 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:57.117 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:49:58.118 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:49:58.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.219+0000 7f4af9c6d640 1 Processor -- start 2024-03-24T10:49:58.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.219+0000 7f4af9c6d640 1 -- start start 2024-03-24T10:49:58.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.220+0000 7f4af9c6d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4af415ac00 0x7f4af40cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.221+0000 7f4af9c6d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4af40ccaa0 0x7f4af406bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.221+0000 7f4af9c6d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4af40cb2f0 con 0x7f4af41543a0 2024-03-24T10:49:58.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.221+0000 7f4af9c6d640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f4af40c6750 con 0x7f4af40ccaa0 2024-03-24T10:49:58.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.221+0000 7f4af9c6d640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4af40cc380 con 0x7f4af415ac00 2024-03-24T10:49:58.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.221+0000 7f4af37fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f4af41543a0 0x7f4af406d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:60926/0 (socket says 172.21.15.5:60926) 2024-03-24T10:49:58.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.221+0000 7f4af37fe640 1 -- 172.21.15.5:0/7022629 learned_addr learned my addr 172.21.15.5:0/7022629 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:49:58.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.221+0000 7f4af3fff640 1 --2- 172.21.15.5:0/7022629 >> v2:172.21.15.5:3300/0 conn(0x7f4af40ccaa0 0x7f4af406bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.221+0000 7f4af2ffd640 1 --2- 172.21.15.5:0/7022629 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4af415ac00 0x7f4af40cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.224+0000 7f4af3fff640 1 -- 172.21.15.5:0/7022629 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4af415ac00 msgr2=0x7f4af40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:49:58.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.224+0000 7f4af3fff640 1 --2- 172.21.15.5:0/7022629 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4af415ac00 0x7f4af40cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.224+0000 7f4af3fff640 1 -- 172.21.15.5:0/7022629 >> v1:172.21.15.57:6789/0 conn(0x7f4af41543a0 legacy=0x7f4af406d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:58.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.224+0000 7f4af3fff640 1 -- 172.21.15.5:0/7022629 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af415b070 con 0x7f4af40ccaa0 2024-03-24T10:49:58.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.226+0000 7f4af3fff640 1 --2- 172.21.15.5:0/7022629 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4af40ccaa0 0x7f4af406bf50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4aec0097a0 tx=0x7f4aec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36556dca6be11711 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:58.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.230+0000 7f4af27fc640 1 -- 172.21.15.5:0/7022629 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aec031830 con 0x7f4af40ccaa0 2024-03-24T10:49:58.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.230+0000 7f4af27fc640 1 -- 172.21.15.5:0/7022629 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4aec031990 con 0x7f4af40ccaa0 2024-03-24T10:49:58.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.230+0000 7f4af27fc640 1 -- 172.21.15.5:0/7022629 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aec031d20 con 0x7f4af40ccaa0 2024-03-24T10:49:58.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.231+0000 7f4af9c6d640 1 -- 172.21.15.5:0/7022629 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4af40ccaa0 msgr2=0x7f4af406bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:58.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.231+0000 7f4af9c6d640 1 --2- 172.21.15.5:0/7022629 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4af40ccaa0 0x7f4af406bf50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4aec0097a0 tx=0x7f4aec02b8d0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.231+0000 7f4af9c6d640 1 -- 172.21.15.5:0/7022629 shutdown_connections 2024-03-24T10:49:58.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.231+0000 7f4af9c6d640 1 --2- 172.21.15.5:0/7022629 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4af415ac00 0x7f4af40cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.231+0000 7f4af9c6d640 1 --2- 172.21.15.5:0/7022629 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4af40ccaa0 0x7f4af406bf50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.231+0000 7f4af9c6d640 1 -- 172.21.15.5:0/7022629 >> 172.21.15.5:0/7022629 conn(0x7f4af4072ef0 msgr2=0x7f4af4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:58.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.231+0000 7f4af9c6d640 1 -- 172.21.15.5:0/7022629 shutdown_connections 2024-03-24T10:49:58.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.231+0000 7f4af9c6d640 1 -- 172.21.15.5:0/7022629 wait complete. 2024-03-24T10:49:58.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.233+0000 7f4af9c6d640 1 Processor -- start 2024-03-24T10:49:58.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.233+0000 7f4af9c6d640 1 -- start start 2024-03-24T10:49:58.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.234+0000 7f4af9c6d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4af40ccaa0 0x7f4af414e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.234+0000 7f4af9c6d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4af41543a0 0x7f4af41531f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.234+0000 7f4af9c6d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4af415ac00 0x7f4af40c6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.235+0000 7f4af9c6d640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f4af40c93e0 con 0x7f4af40ccaa0 2024-03-24T10:49:58.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.235+0000 7f4af9c6d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4af4151f00 con 0x7f4af41543a0 2024-03-24T10:49:58.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.235+0000 7f4af9c6d640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4af4152d60 con 0x7f4af415ac00 2024-03-24T10:49:58.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.235+0000 7f4af2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4af41543a0 0x7f4af41531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.235+0000 7f4af37fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4af40ccaa0 0x7f4af414e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.235+0000 7f4af3fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4af415ac00 0x7f4af40c6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.235+0000 7f4af2ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4af41543a0 0x7f4af41531f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:40692/0 (socket says 172.21.15.5:40692) 2024-03-24T10:49:58.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.235+0000 7f4af2ffd640 1 -- 172.21.15.5:0/4268906302 learned_addr learned my addr 172.21.15.5:0/4268906302 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:49:58.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.239+0000 7f4af37fe640 1 -- 172.21.15.5:0/4268906302 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4af415ac00 msgr2=0x7f4af40c6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:58.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.239+0000 7f4af37fe640 1 --2- 172.21.15.5:0/4268906302 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4af415ac00 0x7f4af40c6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.239+0000 7f4af37fe640 1 -- 172.21.15.5:0/4268906302 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4af41543a0 msgr2=0x7f4af41531f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:58.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.239+0000 7f4af37fe640 1 --2- 172.21.15.5:0/4268906302 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4af41543a0 0x7f4af41531f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.239+0000 7f4af37fe640 1 -- 172.21.15.5:0/4268906302 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aec009400 con 0x7f4af40ccaa0 2024-03-24T10:49:58.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.239+0000 7f4af2ffd640 1 --2- 172.21.15.5:0/4268906302 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4af41543a0 0x7f4af41531f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:49:58.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.242+0000 7f4af37fe640 1 --2- 172.21.15.5:0/4268906302 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4af40ccaa0 0x7f4af414e530 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4ae400cb10 tx=0x7f4ae400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:58.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.246+0000 7f4adbfff640 1 -- 172.21.15.5:0/4268906302 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae400ac70 con 0x7f4af40ccaa0 2024-03-24T10:49:58.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.246+0000 7f4adbfff640 1 -- 172.21.15.5:0/4268906302 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ae400add0 con 0x7f4af40ccaa0 2024-03-24T10:49:58.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.246+0000 7f4af9c6d640 1 -- 172.21.15.5:0/4268906302 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4af4150780 con 0x7f4af40ccaa0 2024-03-24T10:49:58.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.247+0000 7f4adbfff640 1 -- 172.21.15.5:0/4268906302 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae4012600 con 0x7f4af40ccaa0 2024-03-24T10:49:58.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.247+0000 7f4af9c6d640 1 -- 172.21.15.5:0/4268906302 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4af4152430 con 0x7f4af40ccaa0 2024-03-24T10:49:58.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.248+0000 7f4af9c6d640 1 -- 172.21.15.5:0/4268906302 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4af40681d0 con 0x7f4af40ccaa0 2024-03-24T10:49:58.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.517+0000 7f4adbfff640 1 -- 172.21.15.5:0/4268906302 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4ae4007070 con 0x7f4af40ccaa0 2024-03-24T10:49:58.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.518+0000 7f4adbfff640 1 --2- 172.21.15.5:0/4268906302 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4ae006b810 0x7f4ae006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:49:58.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.518+0000 7f4af2ffd640 1 --2- 172.21.15.5:0/4268906302 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4ae006b810 0x7f4ae006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:49:58.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.518+0000 7f4af2ffd640 1 --2- 172.21.15.5:0/4268906302 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4ae006b810 0x7f4ae006dcd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4af40cc9e0 tx=0x7f4ae8009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:49:58.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.534+0000 7f4adbfff640 1 -- 172.21.15.5:0/4268906302 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f4ae4058da0 con 0x7f4af40ccaa0 2024-03-24T10:49:58.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.534+0000 7f4adbfff640 1 -- 172.21.15.5:0/4268906302 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4ae4055040 con 0x7f4af40ccaa0 2024-03-24T10:49:58.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.790+0000 7f4af9c6d640 1 -- 172.21.15.5:0/4268906302 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4af40c9fa0 con 0x7f4af40ccaa0 2024-03-24T10:49:58.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.804+0000 7f4adbfff640 1 -- 172.21.15.5:0/4268906302 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f4ae4058700 con 0x7f4af40ccaa0 2024-03-24T10:49:58.806 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:49:58.806 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:49:58.806 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:49:58.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.806+0000 7f4af9c6d640 1 -- 172.21.15.5:0/4268906302 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4ae006b810 msgr2=0x7f4ae006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:58.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.806+0000 7f4af9c6d640 1 --2- 172.21.15.5:0/4268906302 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4ae006b810 0x7f4ae006dcd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4af40cc9e0 tx=0x7f4ae8009190 comp rx=0 tx=0).stop 2024-03-24T10:49:58.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.806+0000 7f4af9c6d640 1 -- 172.21.15.5:0/4268906302 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4af40ccaa0 msgr2=0x7f4af414e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:49:58.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.806+0000 7f4af9c6d640 1 --2- 172.21.15.5:0/4268906302 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4af40ccaa0 0x7f4af414e530 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4ae400cb10 tx=0x7f4ae400a060 comp rx=0 tx=0).stop 2024-03-24T10:49:58.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.806+0000 7f4af9c6d640 1 -- 172.21.15.5:0/4268906302 shutdown_connections 2024-03-24T10:49:58.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.806+0000 7f4af9c6d640 1 --2- 172.21.15.5:0/4268906302 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4ae006b810 0x7f4ae006dcd0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.806+0000 7f4af9c6d640 1 --2- 172.21.15.5:0/4268906302 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4af415ac00 0x7f4af40c6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.806+0000 7f4af9c6d640 1 --2- 172.21.15.5:0/4268906302 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4af41543a0 0x7f4af41531f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.806+0000 7f4af9c6d640 1 --2- 172.21.15.5:0/4268906302 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4af40ccaa0 0x7f4af414e530 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:49:58.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.807+0000 7f4af9c6d640 1 -- 172.21.15.5:0/4268906302 >> 172.21.15.5:0/4268906302 conn(0x7f4af4072ef0 msgr2=0x7f4af4157f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:49:58.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.807+0000 7f4af9c6d640 1 -- 172.21.15.5:0/4268906302 shutdown_connections 2024-03-24T10:49:58.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:49:58.807+0000 7f4af9c6d640 1 -- 172.21.15.5:0/4268906302 wait complete. 2024-03-24T10:49:59.422 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:49:59.422 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:00.423 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:00.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.529+0000 7f4a7d343640 1 Processor -- start 2024-03-24T10:50:00.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.530+0000 7f4a7d343640 1 -- start start 2024-03-24T10:50:00.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.530+0000 7f4a7d343640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a780c59c0 0x7f4a780c2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:00.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.530+0000 7f4a7d343640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a780cb470 con 0x7f4a78157140 2024-03-24T10:50:00.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.530+0000 7f4a7d343640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a7806eca0 con 0x7f4a780c55e0 2024-03-24T10:50:00.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.530+0000 7f4a7d343640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4a781565e0 con 0x7f4a780c59c0 2024-03-24T10:50:00.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.531+0000 7f4a76ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f4a780c55e0 0x7f4a78156a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:60936/0 (socket says 172.21.15.5:60936) 2024-03-24T10:50:00.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.531+0000 7f4a76ffd640 1 -- 172.21.15.5:0/868687044 learned_addr learned my addr 172.21.15.5:0/868687044 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:50:00.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.531+0000 7f4a767fc640 1 --2- 172.21.15.5:0/868687044 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a780c59c0 0x7f4a780c2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:00.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.534+0000 7f4a75ffb640 1 -- 172.21.15.5:0/868687044 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1965103529 0 0) 0x7f4a780cb470 con 0x7f4a78157140 2024-03-24T10:50:00.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.534+0000 7f4a75ffb640 1 -- 172.21.15.5:0/868687044 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a60004630 con 0x7f4a78157140 2024-03-24T10:50:00.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.535+0000 7f4a75ffb640 1 -- 172.21.15.5:0/868687044 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 45586531 0 0) 0x7f4a7806eca0 con 0x7f4a780c55e0 2024-03-24T10:50:00.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.535+0000 7f4a75ffb640 1 -- 172.21.15.5:0/868687044 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a780cb470 con 0x7f4a780c55e0 2024-03-24T10:50:00.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.535+0000 7f4a767fc640 1 -- 172.21.15.5:0/868687044 >> v1:172.21.15.57:6789/0 conn(0x7f4a780c55e0 legacy=0x7f4a78156a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:00.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.535+0000 7f4a767fc640 1 -- 172.21.15.5:0/868687044 >> v1:172.21.15.5:6789/0 conn(0x7f4a78157140 legacy=0x7f4a78150250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:00.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.535+0000 7f4a767fc640 1 -- 172.21.15.5:0/868687044 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a7806ef40 con 0x7f4a780c59c0 2024-03-24T10:50:00.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.538+0000 7f4a767fc640 1 --2- 172.21.15.5:0/868687044 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a780c59c0 0x7f4a780c2ca0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c0097d0 tx=0x7f4a6c02b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f94a60bd1f04c120 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:00.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.543+0000 7f4a75ffb640 1 -- 172.21.15.5:0/868687044 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c030880 con 0x7f4a780c59c0 2024-03-24T10:50:00.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.543+0000 7f4a75ffb640 1 -- 172.21.15.5:0/868687044 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a6c0309e0 con 0x7f4a780c59c0 2024-03-24T10:50:00.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.543+0000 7f4a75ffb640 1 -- 172.21.15.5:0/868687044 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c030d70 con 0x7f4a780c59c0 2024-03-24T10:50:00.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.544+0000 7f4a7d343640 1 -- 172.21.15.5:0/868687044 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a780c59c0 msgr2=0x7f4a780c2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:00.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.544+0000 7f4a7d343640 1 --2- 172.21.15.5:0/868687044 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a780c59c0 0x7f4a780c2ca0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c0097d0 tx=0x7f4a6c02b9c0 comp rx=0 tx=0).stop 2024-03-24T10:50:00.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.544+0000 7f4a7d343640 1 -- 172.21.15.5:0/868687044 shutdown_connections 2024-03-24T10:50:00.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.544+0000 7f4a7d343640 1 --2- 172.21.15.5:0/868687044 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a780c59c0 0x7f4a780c2ca0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:00.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.544+0000 7f4a7d343640 1 -- 172.21.15.5:0/868687044 >> 172.21.15.5:0/868687044 conn(0x7f4a7806b130 msgr2=0x7f4a7806b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:00.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.544+0000 7f4a7d343640 1 -- 172.21.15.5:0/868687044 shutdown_connections 2024-03-24T10:50:00.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.544+0000 7f4a7d343640 1 -- 172.21.15.5:0/868687044 wait complete. 2024-03-24T10:50:00.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.545+0000 7f4a7d343640 1 Processor -- start 2024-03-24T10:50:00.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.545+0000 7f4a7d343640 1 -- start start 2024-03-24T10:50:00.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.546+0000 7f4a7d343640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a780c55e0 0x7f4a781e30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:00.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.546+0000 7f4a7d343640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a780c59c0 0x7f4a781ede70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:00.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.546+0000 7f4a7d343640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a78157140 0x7f4a781e08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:00.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.546+0000 7f4a7d343640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f4a78153b00 con 0x7f4a78157140 2024-03-24T10:50:00.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.546+0000 7f4a7d343640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4a781ee3b0 con 0x7f4a780c59c0 2024-03-24T10:50:00.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.546+0000 7f4a7d343640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4a781e37c0 con 0x7f4a780c55e0 2024-03-24T10:50:00.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.547+0000 7f4a767fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a780c59c0 0x7f4a781ede70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:00.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.547+0000 7f4a767fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a780c59c0 0x7f4a781ede70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:40702/0 (socket says 172.21.15.5:40702) 2024-03-24T10:50:00.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.547+0000 7f4a767fc640 1 -- 172.21.15.5:0/4174251449 learned_addr learned my addr 172.21.15.5:0/4174251449 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:00.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.547+0000 7f4a777fe640 1 --2- 172.21.15.5:0/4174251449 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a78157140 0x7f4a781e08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:00.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.548+0000 7f4a76ffd640 1 --2- 172.21.15.5:0/4174251449 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a780c55e0 0x7f4a781e30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:00.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.551+0000 7f4a767fc640 1 -- 172.21.15.5:0/4174251449 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a780c55e0 msgr2=0x7f4a781e30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:00.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.551+0000 7f4a767fc640 1 --2- 172.21.15.5:0/4174251449 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a780c55e0 0x7f4a781e30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:00.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.551+0000 7f4a767fc640 1 -- 172.21.15.5:0/4174251449 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a78157140 msgr2=0x7f4a781e08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:00.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.551+0000 7f4a767fc640 1 --2- 172.21.15.5:0/4174251449 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a78157140 0x7f4a781e08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:00.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.551+0000 7f4a767fc640 1 -- 172.21.15.5:0/4174251449 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a6c009400 con 0x7f4a780c59c0 2024-03-24T10:50:00.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.554+0000 7f4a767fc640 1 --2- 172.21.15.5:0/4174251449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a780c59c0 0x7f4a781ede70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c0098f0 tx=0x7f4a6c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:00.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.558+0000 7f4a5f7fe640 1 -- 172.21.15.5:0/4174251449 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c03e040 con 0x7f4a780c59c0 2024-03-24T10:50:00.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.558+0000 7f4a5f7fe640 1 -- 172.21.15.5:0/4174251449 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a6c039cd0 con 0x7f4a780c59c0 2024-03-24T10:50:00.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.558+0000 7f4a7d343640 1 -- 172.21.15.5:0/4174251449 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a78152e80 con 0x7f4a780c59c0 2024-03-24T10:50:00.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.558+0000 7f4a5f7fe640 1 -- 172.21.15.5:0/4174251449 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c044630 con 0x7f4a780c59c0 2024-03-24T10:50:00.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.559+0000 7f4a7d343640 1 -- 172.21.15.5:0/4174251449 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a78153440 con 0x7f4a780c59c0 2024-03-24T10:50:00.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.560+0000 7f4a7d343640 1 -- 172.21.15.5:0/4174251449 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a78062a00 con 0x7f4a780c59c0 2024-03-24T10:50:00.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.829+0000 7f4a5f7fe640 1 -- 172.21.15.5:0/4174251449 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4a6c04d3f0 con 0x7f4a780c59c0 2024-03-24T10:50:00.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.829+0000 7f4a5f7fe640 1 --2- 172.21.15.5:0/4174251449 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4a6406b840 0x7f4a6406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:00.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.829+0000 7f4a76ffd640 1 --2- 172.21.15.5:0/4174251449 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4a6406b840 0x7f4a6406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:00.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.830+0000 7f4a76ffd640 1 --2- 172.21.15.5:0/4174251449 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4a6406b840 0x7f4a6406dd00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4a68005880 tx=0x7f4a68009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:00.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.832+0000 7f4a5f7fe640 1 -- 172.21.15.5:0/4174251449 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f4a6c039e40 con 0x7f4a780c59c0 2024-03-24T10:50:00.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:00.843+0000 7f4a5f7fe640 1 -- 172.21.15.5:0/4174251449 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4a6c077060 con 0x7f4a780c59c0 2024-03-24T10:50:01.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:01.086+0000 7f4a7d343640 1 -- 172.21.15.5:0/4174251449 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a78153c40 con 0x7f4a780c59c0 2024-03-24T10:50:01.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:01.108+0000 7f4a5f7fe640 1 -- 172.21.15.5:0/4174251449 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f4a78153c40 con 0x7f4a780c59c0 2024-03-24T10:50:01.108 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:01.108 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:01.109 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:01.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:01.110+0000 7f4a7d343640 1 -- 172.21.15.5:0/4174251449 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4a6406b840 msgr2=0x7f4a6406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:01.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:01.110+0000 7f4a7d343640 1 --2- 172.21.15.5:0/4174251449 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4a6406b840 0x7f4a6406dd00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4a68005880 tx=0x7f4a68009040 comp rx=0 tx=0).stop 2024-03-24T10:50:01.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:01.110+0000 7f4a7d343640 1 -- 172.21.15.5:0/4174251449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a780c59c0 msgr2=0x7f4a781ede70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:01.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:01.110+0000 7f4a7d343640 1 --2- 172.21.15.5:0/4174251449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a780c59c0 0x7f4a781ede70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c0098f0 tx=0x7f4a6c004730 comp rx=0 tx=0).stop 2024-03-24T10:50:01.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:01.110+0000 7f4a7d343640 1 -- 172.21.15.5:0/4174251449 shutdown_connections 2024-03-24T10:50:01.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:01.110+0000 7f4a7d343640 1 --2- 172.21.15.5:0/4174251449 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4a6406b840 0x7f4a6406dd00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:01.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:01.110+0000 7f4a7d343640 1 --2- 172.21.15.5:0/4174251449 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a78157140 0x7f4a781e08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:01.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:01.110+0000 7f4a7d343640 1 --2- 172.21.15.5:0/4174251449 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a780c59c0 0x7f4a781ede70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:01.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:01.110+0000 7f4a7d343640 1 --2- 172.21.15.5:0/4174251449 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a780c55e0 0x7f4a781e30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:01.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:01.110+0000 7f4a7d343640 1 -- 172.21.15.5:0/4174251449 >> 172.21.15.5:0/4174251449 conn(0x7f4a7806b130 msgr2=0x7f4a781460b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:01.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:01.110+0000 7f4a7d343640 1 -- 172.21.15.5:0/4174251449 shutdown_connections 2024-03-24T10:50:01.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:01.111+0000 7f4a7d343640 1 -- 172.21.15.5:0/4174251449 wait complete. 2024-03-24T10:50:01.726 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:01.726 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:01.789 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:50:01.791 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:50:02.726 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:02.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.829+0000 7f8725df4640 1 Processor -- start 2024-03-24T10:50:02.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.829+0000 7f8725df4640 1 -- start start 2024-03-24T10:50:02.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.829+0000 7f8725df4640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f87200cc8f0 0x7f872015b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:02.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.829+0000 7f8725df4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f87200cf110 0x7f8720153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:02.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.829+0000 7f8725df4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87200cd890 con 0x7f872006c580 2024-03-24T10:50:02.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.829+0000 7f8725df4640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f872006cae0 con 0x7f87200cc8f0 2024-03-24T10:50:02.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.829+0000 7f8725df4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f87200cdb50 con 0x7f87200cf110 2024-03-24T10:50:02.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.830+0000 7f871f7fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f87200cc8f0 0x7f872015b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:02.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.830+0000 7f871ffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f87200cf110 0x7f8720153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:02.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.830+0000 7f871effd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f872006c580 0x7f87201538b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:50242/0 (socket says 172.21.15.5:50242) 2024-03-24T10:50:02.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.830+0000 7f871effd640 1 -- 172.21.15.5:0/3738756277 learned_addr learned my addr 172.21.15.5:0/3738756277 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:50:02.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.834+0000 7f871ffff640 1 -- 172.21.15.5:0/3738756277 >> v2:172.21.15.5:3300/0 conn(0x7f87200cc8f0 msgr2=0x7f872015b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:02.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.834+0000 7f871ffff640 1 --2- 172.21.15.5:0/3738756277 >> v2:172.21.15.5:3300/0 conn(0x7f87200cc8f0 0x7f872015b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:02.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.834+0000 7f871ffff640 1 -- 172.21.15.5:0/3738756277 >> v1:172.21.15.5:6789/0 conn(0x7f872006c580 legacy=0x7f87201538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:02.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.834+0000 7f871ffff640 1 -- 172.21.15.5:0/3738756277 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87200cde30 con 0x7f87200cf110 2024-03-24T10:50:02.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.837+0000 7f871ffff640 1 --2- 172.21.15.5:0/3738756277 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87200cf110 0x7f8720153fc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f872006e100 tx=0x7f871802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=438297bdd767211f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:02.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.842+0000 7f871e7fc640 1 -- 172.21.15.5:0/3738756277 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718030610 con 0x7f87200cf110 2024-03-24T10:50:02.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.842+0000 7f871e7fc640 1 -- 172.21.15.5:0/3738756277 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8718030770 con 0x7f87200cf110 2024-03-24T10:50:02.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.842+0000 7f871e7fc640 1 -- 172.21.15.5:0/3738756277 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8718030b00 con 0x7f87200cf110 2024-03-24T10:50:02.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.843+0000 7f8725df4640 1 -- 172.21.15.5:0/3738756277 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87200cf110 msgr2=0x7f8720153fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:02.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.843+0000 7f8725df4640 1 --2- 172.21.15.5:0/3738756277 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87200cf110 0x7f8720153fc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f872006e100 tx=0x7f871802b960 comp rx=0 tx=0).stop 2024-03-24T10:50:02.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.843+0000 7f8725df4640 1 -- 172.21.15.5:0/3738756277 shutdown_connections 2024-03-24T10:50:02.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.843+0000 7f8725df4640 1 --2- 172.21.15.5:0/3738756277 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87200cf110 0x7f8720153fc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:02.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.843+0000 7f8725df4640 1 --2- 172.21.15.5:0/3738756277 >> v2:172.21.15.5:3300/0 conn(0x7f87200cc8f0 0x7f872015b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:02.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.843+0000 7f8725df4640 1 -- 172.21.15.5:0/3738756277 >> 172.21.15.5:0/3738756277 conn(0x7f87200b1820 msgr2=0x7f87200b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:02.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.843+0000 7f8725df4640 1 -- 172.21.15.5:0/3738756277 shutdown_connections 2024-03-24T10:50:02.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.843+0000 7f8725df4640 1 -- 172.21.15.5:0/3738756277 wait complete. 2024-03-24T10:50:02.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.844+0000 7f8725df4640 1 Processor -- start 2024-03-24T10:50:02.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.844+0000 7f8725df4640 1 -- start start 2024-03-24T10:50:02.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.845+0000 7f8725df4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f872006c580 0x7f872014bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:02.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.845+0000 7f8725df4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87200cc8f0 0x7f87201f0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:02.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.845+0000 7f8725df4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87200cf110 0x7f87201510f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:02.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.845+0000 7f8725df4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f872014c280 con 0x7f87200cf110 2024-03-24T10:50:02.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.845+0000 7f8725df4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f872014cb20 con 0x7f87200cc8f0 2024-03-24T10:50:02.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.845+0000 7f8725df4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f87201f0e40 con 0x7f872006c580 2024-03-24T10:50:02.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.845+0000 7f871ffff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87200cf110 0x7f87201510f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:02.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.846+0000 7f871f7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f872006c580 0x7f872014bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:02.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.846+0000 7f871effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87200cc8f0 0x7f87201f0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:02.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.846+0000 7f871effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87200cc8f0 0x7f87201f0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:47464/0 (socket says 172.21.15.5:47464) 2024-03-24T10:50:02.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.846+0000 7f871effd640 1 -- 172.21.15.5:0/432938596 learned_addr learned my addr 172.21.15.5:0/432938596 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:02.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.850+0000 7f871f7fe640 1 -- 172.21.15.5:0/432938596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87200cc8f0 msgr2=0x7f87201f0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:02.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.850+0000 7f871f7fe640 1 --2- 172.21.15.5:0/432938596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87200cc8f0 0x7f87201f0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:02.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.850+0000 7f871f7fe640 1 -- 172.21.15.5:0/432938596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87200cf110 msgr2=0x7f87201510f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:02.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.850+0000 7f871f7fe640 1 --2- 172.21.15.5:0/432938596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87200cf110 0x7f87201510f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:02.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.850+0000 7f871f7fe640 1 -- 172.21.15.5:0/432938596 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8718009400 con 0x7f872006c580 2024-03-24T10:50:02.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.853+0000 7f871f7fe640 1 --2- 172.21.15.5:0/432938596 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f872006c580 0x7f872014bb00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f871000efc0 tx=0x7f871000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:02.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.858+0000 7f8707fff640 1 -- 172.21.15.5:0/432938596 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8710014480 con 0x7f872006c580 2024-03-24T10:50:02.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.858+0000 7f8707fff640 1 -- 172.21.15.5:0/432938596 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f871000f040 con 0x7f872006c580 2024-03-24T10:50:02.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.858+0000 7f8707fff640 1 -- 172.21.15.5:0/432938596 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8710014800 con 0x7f872006c580 2024-03-24T10:50:02.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.858+0000 7f8725df4640 1 -- 172.21.15.5:0/432938596 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f872014e640 con 0x7f872006c580 2024-03-24T10:50:02.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.858+0000 7f8725df4640 1 -- 172.21.15.5:0/432938596 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f872014c770 con 0x7f872006c580 2024-03-24T10:50:02.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:02.859+0000 7f8725df4640 1 -- 172.21.15.5:0/432938596 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f872014c930 con 0x7f872006c580 2024-03-24T10:50:03.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:03.132+0000 7f8707fff640 1 -- 172.21.15.5:0/432938596 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8710014960 con 0x7f872006c580 2024-03-24T10:50:03.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:03.132+0000 7f8707fff640 1 --2- 172.21.15.5:0/432938596 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f870c06b860 0x7f870c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:03.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:03.133+0000 7f871effd640 1 --2- 172.21.15.5:0/432938596 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f870c06b860 0x7f870c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:03.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:03.133+0000 7f871effd640 1 --2- 172.21.15.5:0/432938596 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f870c06b860 0x7f870c06dd20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f87200cbd40 tx=0x7f8714007f10 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:03.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:03.136+0000 7f8707fff640 1 -- 172.21.15.5:0/432938596 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f8710059e10 con 0x7f872006c580 2024-03-24T10:50:03.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:03.148+0000 7f8707fff640 1 -- 172.21.15.5:0/432938596 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f87100560b0 con 0x7f872006c580 2024-03-24T10:50:03.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:03.382+0000 7f8725df4640 1 -- 172.21.15.5:0/432938596 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87200681d0 con 0x7f872006c580 2024-03-24T10:50:03.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:03.407+0000 7f8707fff640 1 -- 172.21.15.5:0/432938596 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f8710059770 con 0x7f872006c580 2024-03-24T10:50:03.407 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:03.407 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:03.408 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:03.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:03.409+0000 7f8725df4640 1 -- 172.21.15.5:0/432938596 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f870c06b860 msgr2=0x7f870c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:03.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:03.409+0000 7f8725df4640 1 --2- 172.21.15.5:0/432938596 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f870c06b860 0x7f870c06dd20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f87200cbd40 tx=0x7f8714007f10 comp rx=0 tx=0).stop 2024-03-24T10:50:03.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:03.409+0000 7f8725df4640 1 -- 172.21.15.5:0/432938596 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f872006c580 msgr2=0x7f872014bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:03.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:03.409+0000 7f8725df4640 1 --2- 172.21.15.5:0/432938596 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f872006c580 0x7f872014bb00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f871000efc0 tx=0x7f871000c490 comp rx=0 tx=0).stop 2024-03-24T10:50:03.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:03.409+0000 7f8725df4640 1 -- 172.21.15.5:0/432938596 shutdown_connections 2024-03-24T10:50:03.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:03.409+0000 7f8725df4640 1 --2- 172.21.15.5:0/432938596 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f870c06b860 0x7f870c06dd20 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:03.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:03.409+0000 7f8725df4640 1 --2- 172.21.15.5:0/432938596 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f87200cf110 0x7f87201510f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:03.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:03.409+0000 7f8725df4640 1 --2- 172.21.15.5:0/432938596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87200cc8f0 0x7f87201f0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:03.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:03.409+0000 7f8725df4640 1 --2- 172.21.15.5:0/432938596 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f872006c580 0x7f872014bb00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:03.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:03.409+0000 7f8725df4640 1 -- 172.21.15.5:0/432938596 >> 172.21.15.5:0/432938596 conn(0x7f87200b1820 msgr2=0x7f87200b2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:03.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:03.410+0000 7f8725df4640 1 -- 172.21.15.5:0/432938596 shutdown_connections 2024-03-24T10:50:03.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:03.410+0000 7f8725df4640 1 -- 172.21.15.5:0/432938596 wait complete. 2024-03-24T10:50:04.024 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:04.025 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:05.026 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:05.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.130+0000 7f60fb0d4640 1 Processor -- start 2024-03-24T10:50:05.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.130+0000 7f60fb0d4640 1 -- start start 2024-03-24T10:50:05.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.130+0000 7f60fb0d4640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f60f40c2ea0 0x7f60f40caab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:05.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.130+0000 7f60fb0d4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f60f40c9610 0x7f60f40c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:05.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.130+0000 7f60fb0d4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f40ca000 con 0x7f60f40c58c0 2024-03-24T10:50:05.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.130+0000 7f60fb0d4640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f60f40cb210 con 0x7f60f40c2ea0 2024-03-24T10:50:05.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.130+0000 7f60fb0d4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f60f4156db0 con 0x7f60f40c9610 2024-03-24T10:50:05.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.131+0000 7f60f8e70640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f60f40c2ea0 0x7f60f40caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:05.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.131+0000 7f60f9671640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f60f40c9610 0x7f60f40c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:05.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.131+0000 7f60f9671640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f60f40c9610 0x7f60f40c9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:47478/0 (socket says 172.21.15.5:47478) 2024-03-24T10:50:05.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.131+0000 7f60f9671640 1 -- 172.21.15.5:0/53356216 learned_addr learned my addr 172.21.15.5:0/53356216 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:05.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.134+0000 7f60f9671640 1 -- 172.21.15.5:0/53356216 >> v2:172.21.15.5:3300/0 conn(0x7f60f40c2ea0 msgr2=0x7f60f40caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:05.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.134+0000 7f60f9671640 1 --2- 172.21.15.5:0/53356216 >> v2:172.21.15.5:3300/0 conn(0x7f60f40c2ea0 0x7f60f40caab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:05.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.134+0000 7f60f9671640 1 -- 172.21.15.5:0/53356216 >> v1:172.21.15.5:6789/0 conn(0x7f60f40c58c0 legacy=0x7f60f40c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:05.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.134+0000 7f60f9671640 1 -- 172.21.15.5:0/53356216 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f4156fc0 con 0x7f60f40c9610 2024-03-24T10:50:05.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.136+0000 7f60f9671640 1 --2- 172.21.15.5:0/53356216 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60f40c9610 0x7f60f40c9ac0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f60f40c69a0 tx=0x7f60ec02e210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5e3cadec4abdb4a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:05.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.140+0000 7f60f37fe640 1 -- 172.21.15.5:0/53356216 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ec00b0d0 con 0x7f60f40c9610 2024-03-24T10:50:05.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.140+0000 7f60f37fe640 1 -- 172.21.15.5:0/53356216 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60ec00b230 con 0x7f60f40c9610 2024-03-24T10:50:05.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.140+0000 7f60f37fe640 1 -- 172.21.15.5:0/53356216 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ec009600 con 0x7f60f40c9610 2024-03-24T10:50:05.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.141+0000 7f60fb0d4640 1 -- 172.21.15.5:0/53356216 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60f40c9610 msgr2=0x7f60f40c9ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:05.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.141+0000 7f60fb0d4640 1 --2- 172.21.15.5:0/53356216 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60f40c9610 0x7f60f40c9ac0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f60f40c69a0 tx=0x7f60ec02e210 comp rx=0 tx=0).stop 2024-03-24T10:50:05.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.141+0000 7f60fb0d4640 1 -- 172.21.15.5:0/53356216 shutdown_connections 2024-03-24T10:50:05.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.141+0000 7f60fb0d4640 1 --2- 172.21.15.5:0/53356216 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60f40c9610 0x7f60f40c9ac0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:05.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.141+0000 7f60fb0d4640 1 --2- 172.21.15.5:0/53356216 >> v2:172.21.15.5:3300/0 conn(0x7f60f40c2ea0 0x7f60f40caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:05.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.141+0000 7f60fb0d4640 1 -- 172.21.15.5:0/53356216 >> 172.21.15.5:0/53356216 conn(0x7f60f406b130 msgr2=0x7f60f406b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:05.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.141+0000 7f60fb0d4640 1 -- 172.21.15.5:0/53356216 shutdown_connections 2024-03-24T10:50:05.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.141+0000 7f60fb0d4640 1 -- 172.21.15.5:0/53356216 wait complete. 2024-03-24T10:50:05.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.142+0000 7f60fb0d4640 1 Processor -- start 2024-03-24T10:50:05.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.142+0000 7f60fb0d4640 1 -- start start 2024-03-24T10:50:05.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.143+0000 7f60fb0d4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f60f40c2ea0 0x7f60f41581e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:05.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.143+0000 7f60fb0d4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60f40c58c0 0x7f60f41e1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:05.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.143+0000 7f60fb0d4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60f40c9610 0x7f60f41e2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:05.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.143+0000 7f60fb0d4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f60f41e1b60 con 0x7f60f40c9610 2024-03-24T10:50:05.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.143+0000 7f60fb0d4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f60f4146980 con 0x7f60f40c58c0 2024-03-24T10:50:05.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.143+0000 7f60fb0d4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f60f4146c60 con 0x7f60f40c2ea0 2024-03-24T10:50:05.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.144+0000 7f60f9671640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60f40c9610 0x7f60f41e2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:05.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.144+0000 7f60f3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60f40c58c0 0x7f60f41e1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:05.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.144+0000 7f60f8e70640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f60f40c2ea0 0x7f60f41581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:05.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.144+0000 7f60f9671640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60f40c9610 0x7f60f41e2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51202/0 (socket says 172.21.15.5:51202) 2024-03-24T10:50:05.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.144+0000 7f60f9671640 1 -- 172.21.15.5:0/2107411371 learned_addr learned my addr 172.21.15.5:0/2107411371 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:05.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.148+0000 7f60f9671640 1 -- 172.21.15.5:0/2107411371 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f60f40c2ea0 msgr2=0x7f60f41581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:05.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.148+0000 7f60f9671640 1 --2- 172.21.15.5:0/2107411371 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f60f40c2ea0 0x7f60f41581e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:05.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.148+0000 7f60f9671640 1 -- 172.21.15.5:0/2107411371 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60f40c58c0 msgr2=0x7f60f41e1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:05.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.148+0000 7f60f9671640 1 --2- 172.21.15.5:0/2107411371 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60f40c58c0 0x7f60f41e1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:05.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.148+0000 7f60f9671640 1 -- 172.21.15.5:0/2107411371 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60ec00bd40 con 0x7f60f40c9610 2024-03-24T10:50:05.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.151+0000 7f60f9671640 1 --2- 172.21.15.5:0/2107411371 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60f40c9610 0x7f60f41e2530 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f60ec038040 tx=0x7f60ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:05.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.155+0000 7f60f17fa640 1 -- 172.21.15.5:0/2107411371 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ec009e30 con 0x7f60f40c9610 2024-03-24T10:50:05.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.155+0000 7f60f17fa640 1 -- 172.21.15.5:0/2107411371 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60ec040660 con 0x7f60f40c9610 2024-03-24T10:50:05.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.155+0000 7f60f17fa640 1 -- 172.21.15.5:0/2107411371 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ec040a10 con 0x7f60f40c9610 2024-03-24T10:50:05.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.156+0000 7f60fb0d4640 1 -- 172.21.15.5:0/2107411371 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60f41e1230 con 0x7f60f40c9610 2024-03-24T10:50:05.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.156+0000 7f60fb0d4640 1 -- 172.21.15.5:0/2107411371 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60f41e17f0 con 0x7f60f40c9610 2024-03-24T10:50:05.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.157+0000 7f60fb0d4640 1 -- 172.21.15.5:0/2107411371 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60f4062a00 con 0x7f60f40c9610 2024-03-24T10:50:05.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.421+0000 7f60f17fa640 1 -- 172.21.15.5:0/2107411371 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f60ec040b70 con 0x7f60f40c9610 2024-03-24T10:50:05.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.421+0000 7f60f17fa640 1 --2- 172.21.15.5:0/2107411371 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f60d806b840 0x7f60d806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:05.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.421+0000 7f60f8e70640 1 --2- 172.21.15.5:0/2107411371 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f60d806b840 0x7f60d806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:05.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.422+0000 7f60f8e70640 1 --2- 172.21.15.5:0/2107411371 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f60d806b840 0x7f60d806dd00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f60e8009aa0 tx=0x7f60e8009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:05.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.434+0000 7f60f17fa640 1 -- 172.21.15.5:0/2107411371 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f60ec07df50 con 0x7f60f40c9610 2024-03-24T10:50:05.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.435+0000 7f60f17fa640 1 -- 172.21.15.5:0/2107411371 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f60ec07a1f0 con 0x7f60f40c9610 2024-03-24T10:50:05.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.674+0000 7f60fb0d4640 1 -- 172.21.15.5:0/2107411371 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60f40c2500 con 0x7f60f40c9610 2024-03-24T10:50:05.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.690+0000 7f60f17fa640 1 -- 172.21.15.5:0/2107411371 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f60ec07d8b0 con 0x7f60f40c9610 2024-03-24T10:50:05.691 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:05.691 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:05.691 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:05.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.692+0000 7f60fb0d4640 1 -- 172.21.15.5:0/2107411371 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f60d806b840 msgr2=0x7f60d806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:05.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.692+0000 7f60fb0d4640 1 --2- 172.21.15.5:0/2107411371 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f60d806b840 0x7f60d806dd00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f60e8009aa0 tx=0x7f60e8009190 comp rx=0 tx=0).stop 2024-03-24T10:50:05.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.692+0000 7f60fb0d4640 1 -- 172.21.15.5:0/2107411371 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60f40c9610 msgr2=0x7f60f41e2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:05.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.692+0000 7f60fb0d4640 1 --2- 172.21.15.5:0/2107411371 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60f40c9610 0x7f60f41e2530 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f60ec038040 tx=0x7f60ec004a40 comp rx=0 tx=0).stop 2024-03-24T10:50:05.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.692+0000 7f60fb0d4640 1 -- 172.21.15.5:0/2107411371 shutdown_connections 2024-03-24T10:50:05.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.693+0000 7f60fb0d4640 1 --2- 172.21.15.5:0/2107411371 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f60d806b840 0x7f60d806dd00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:05.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.693+0000 7f60fb0d4640 1 --2- 172.21.15.5:0/2107411371 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f60f40c9610 0x7f60f41e2530 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:05.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.693+0000 7f60fb0d4640 1 --2- 172.21.15.5:0/2107411371 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f60f40c58c0 0x7f60f41e1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:05.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.693+0000 7f60fb0d4640 1 --2- 172.21.15.5:0/2107411371 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f60f40c2ea0 0x7f60f41581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:05.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.693+0000 7f60fb0d4640 1 -- 172.21.15.5:0/2107411371 >> 172.21.15.5:0/2107411371 conn(0x7f60f406b130 msgr2=0x7f60f406b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:05.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.693+0000 7f60fb0d4640 1 -- 172.21.15.5:0/2107411371 shutdown_connections 2024-03-24T10:50:05.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:05.693+0000 7f60fb0d4640 1 -- 172.21.15.5:0/2107411371 wait complete. 2024-03-24T10:50:06.306 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:06.307 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:07.308 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:07.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.410+0000 7f9857719640 1 Processor -- start 2024-03-24T10:50:07.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.410+0000 7f9857719640 1 -- start start 2024-03-24T10:50:07.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.411+0000 7f9857719640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f985015ac00 0x7f98500cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:07.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.411+0000 7f9857719640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98500cb2f0 con 0x7f98501543a0 2024-03-24T10:50:07.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.411+0000 7f9857719640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f985015b070 con 0x7f98500ccaa0 2024-03-24T10:50:07.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.411+0000 7f9857719640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f98500c6750 con 0x7f985015ac00 2024-03-24T10:50:07.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.412+0000 7f9855cb6640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f98500ccaa0 0x7f985006bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:54072/0 (socket says 172.21.15.5:54072) 2024-03-24T10:50:07.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.412+0000 7f9855cb6640 1 -- 172.21.15.5:0/2573587903 learned_addr learned my addr 172.21.15.5:0/2573587903 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:50:07.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.412+0000 7f9854cb4640 1 --2- 172.21.15.5:0/2573587903 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f985015ac00 0x7f98500cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:07.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.414+0000 7f9847fff640 1 -- 172.21.15.5:0/2573587903 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2068807453 0 0) 0x7f98500cb2f0 con 0x7f98501543a0 2024-03-24T10:50:07.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.414+0000 7f9847fff640 1 -- 172.21.15.5:0/2573587903 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f983c003520 con 0x7f98501543a0 2024-03-24T10:50:07.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.415+0000 7f9847fff640 1 -- 172.21.15.5:0/2573587903 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4234348222 0 0) 0x7f985015b070 con 0x7f98500ccaa0 2024-03-24T10:50:07.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.415+0000 7f9847fff640 1 -- 172.21.15.5:0/2573587903 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98500cb2f0 con 0x7f98500ccaa0 2024-03-24T10:50:07.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.416+0000 7f9854cb4640 1 -- 172.21.15.5:0/2573587903 >> v1:172.21.15.57:6789/0 conn(0x7f98500ccaa0 legacy=0x7f985006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:07.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.416+0000 7f9854cb4640 1 -- 172.21.15.5:0/2573587903 >> v1:172.21.15.5:6789/0 conn(0x7f98501543a0 legacy=0x7f985006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:07.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.416+0000 7f9854cb4640 1 -- 172.21.15.5:0/2573587903 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98500ca7e0 con 0x7f985015ac00 2024-03-24T10:50:07.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.419+0000 7f9854cb4640 1 --2- 172.21.15.5:0/2573587903 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f985015ac00 0x7f98500cc560 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f98400097d0 tx=0x7f984002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cc98f9bb6c118bf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:07.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.424+0000 7f9847fff640 1 -- 172.21.15.5:0/2573587903 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9840031890 con 0x7f985015ac00 2024-03-24T10:50:07.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.424+0000 7f9847fff640 1 -- 172.21.15.5:0/2573587903 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98400319f0 con 0x7f985015ac00 2024-03-24T10:50:07.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.424+0000 7f9847fff640 1 -- 172.21.15.5:0/2573587903 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9840031d80 con 0x7f985015ac00 2024-03-24T10:50:07.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.425+0000 7f9857719640 1 -- 172.21.15.5:0/2573587903 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f985015ac00 msgr2=0x7f98500cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:07.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.425+0000 7f9857719640 1 --2- 172.21.15.5:0/2573587903 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f985015ac00 0x7f98500cc560 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f98400097d0 tx=0x7f984002b930 comp rx=0 tx=0).stop 2024-03-24T10:50:07.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.425+0000 7f9857719640 1 -- 172.21.15.5:0/2573587903 shutdown_connections 2024-03-24T10:50:07.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.425+0000 7f9857719640 1 --2- 172.21.15.5:0/2573587903 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f985015ac00 0x7f98500cc560 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:07.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.425+0000 7f9857719640 1 -- 172.21.15.5:0/2573587903 >> 172.21.15.5:0/2573587903 conn(0x7f9850072ef0 msgr2=0x7f9850061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:07.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.425+0000 7f9857719640 1 -- 172.21.15.5:0/2573587903 shutdown_connections 2024-03-24T10:50:07.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.425+0000 7f9857719640 1 -- 172.21.15.5:0/2573587903 wait complete. 2024-03-24T10:50:07.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.426+0000 7f9857719640 1 Processor -- start 2024-03-24T10:50:07.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.426+0000 7f9857719640 1 -- start start 2024-03-24T10:50:07.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.427+0000 7f9857719640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f98500ccaa0 0x7f985014cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:07.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.427+0000 7f9857719640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98501543a0 0x7f98501eff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:07.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.427+0000 7f9857719640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f985015ac00 0x7f985014a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:07.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.427+0000 7f9857719640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f985014be50 con 0x7f985015ac00 2024-03-24T10:50:07.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.427+0000 7f9857719640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9850157f60 con 0x7f98501543a0 2024-03-24T10:50:07.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.427+0000 7f9857719640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f98501ef7d0 con 0x7f98500ccaa0 2024-03-24T10:50:07.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.428+0000 7f9855cb6640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f985015ac00 0x7f985014a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:07.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.428+0000 7f9854cb4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98501543a0 0x7f98501eff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:07.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.428+0000 7f9854cb4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98501543a0 0x7f98501eff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:47488/0 (socket says 172.21.15.5:47488) 2024-03-24T10:50:07.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.428+0000 7f9854cb4640 1 -- 172.21.15.5:0/2431690818 learned_addr learned my addr 172.21.15.5:0/2431690818 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:07.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.428+0000 7f98554b5640 1 --2- 172.21.15.5:0/2431690818 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f98500ccaa0 0x7f985014cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:07.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.432+0000 7f9854cb4640 1 -- 172.21.15.5:0/2431690818 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f98500ccaa0 msgr2=0x7f985014cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:07.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.432+0000 7f9854cb4640 1 --2- 172.21.15.5:0/2431690818 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f98500ccaa0 0x7f985014cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:07.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.432+0000 7f9854cb4640 1 -- 172.21.15.5:0/2431690818 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f985015ac00 msgr2=0x7f985014a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:07.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.432+0000 7f9854cb4640 1 --2- 172.21.15.5:0/2431690818 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f985015ac00 0x7f985014a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:07.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.432+0000 7f9854cb4640 1 -- 172.21.15.5:0/2431690818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9840009400 con 0x7f98501543a0 2024-03-24T10:50:07.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.435+0000 7f9854cb4640 1 --2- 172.21.15.5:0/2431690818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98501543a0 0x7f98501eff00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f98400097d0 tx=0x7f98400046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:07.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.439+0000 7f9845ffb640 1 -- 172.21.15.5:0/2431690818 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984003abe0 con 0x7f98501543a0 2024-03-24T10:50:07.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.439+0000 7f9845ffb640 1 -- 172.21.15.5:0/2431690818 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f984003ad60 con 0x7f98501543a0 2024-03-24T10:50:07.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.439+0000 7f9845ffb640 1 -- 172.21.15.5:0/2431690818 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9840044630 con 0x7f98501543a0 2024-03-24T10:50:07.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.439+0000 7f9857719640 1 -- 172.21.15.5:0/2431690818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f985014e0b0 con 0x7f98501543a0 2024-03-24T10:50:07.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.439+0000 7f9857719640 1 -- 172.21.15.5:0/2431690818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f985014b1b0 con 0x7f98501543a0 2024-03-24T10:50:07.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.440+0000 7f9857719640 1 -- 172.21.15.5:0/2431690818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98500681d0 con 0x7f98501543a0 2024-03-24T10:50:07.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.460+0000 7f9845ffb640 1 -- 172.21.15.5:0/2431690818 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f984004d450 con 0x7f98501543a0 2024-03-24T10:50:07.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.460+0000 7f9845ffb640 1 --2- 172.21.15.5:0/2431690818 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f983406b810 0x7f983406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:07.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.460+0000 7f98554b5640 1 --2- 172.21.15.5:0/2431690818 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f983406b810 0x7f983406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:07.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.461+0000 7f98554b5640 1 --2- 172.21.15.5:0/2431690818 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f983406b810 0x7f983406dcd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f984c007fd0 tx=0x7f984c009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:07.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.464+0000 7f9845ffb640 1 -- 172.21.15.5:0/2431690818 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f984007b290 con 0x7f98501543a0 2024-03-24T10:50:07.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.741+0000 7f9845ffb640 1 -- 172.21.15.5:0/2431690818 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9840077530 con 0x7f98501543a0 2024-03-24T10:50:07.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.965+0000 7f9857719640 1 -- 172.21.15.5:0/2431690818 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98500c5c80 con 0x7f98501543a0 2024-03-24T10:50:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.979+0000 7f9845ffb640 1 -- 172.21.15.5:0/2431690818 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f9840035020 con 0x7f98501543a0 2024-03-24T10:50:07.979 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:07.979 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:07.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.981+0000 7f9857719640 1 -- 172.21.15.5:0/2431690818 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f983406b810 msgr2=0x7f983406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:07.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.981+0000 7f9857719640 1 --2- 172.21.15.5:0/2431690818 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f983406b810 0x7f983406dcd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f984c007fd0 tx=0x7f984c009040 comp rx=0 tx=0).stop 2024-03-24T10:50:07.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.981+0000 7f9857719640 1 -- 172.21.15.5:0/2431690818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98501543a0 msgr2=0x7f98501eff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:07.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.981+0000 7f9857719640 1 --2- 172.21.15.5:0/2431690818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98501543a0 0x7f98501eff00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f98400097d0 tx=0x7f98400046c0 comp rx=0 tx=0).stop 2024-03-24T10:50:07.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.981+0000 7f9857719640 1 -- 172.21.15.5:0/2431690818 shutdown_connections 2024-03-24T10:50:07.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.981+0000 7f9857719640 1 --2- 172.21.15.5:0/2431690818 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f983406b810 0x7f983406dcd0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:07.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.981+0000 7f9857719640 1 --2- 172.21.15.5:0/2431690818 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f985015ac00 0x7f985014a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:07.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.981+0000 7f9857719640 1 --2- 172.21.15.5:0/2431690818 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98501543a0 0x7f98501eff00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:07.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.981+0000 7f9857719640 1 --2- 172.21.15.5:0/2431690818 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f98500ccaa0 0x7f985014cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:07.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.981+0000 7f9857719640 1 -- 172.21.15.5:0/2431690818 >> 172.21.15.5:0/2431690818 conn(0x7f9850072ef0 msgr2=0x7f98501527f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:07.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.981+0000 7f9857719640 1 -- 172.21.15.5:0/2431690818 shutdown_connections 2024-03-24T10:50:07.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:07.981+0000 7f9857719640 1 -- 172.21.15.5:0/2431690818 wait complete. 2024-03-24T10:50:08.596 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:08.596 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:09.597 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:09.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.698+0000 7fa7dd223640 1 Processor -- start 2024-03-24T10:50:09.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.698+0000 7fa7dd223640 1 -- start start 2024-03-24T10:50:09.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.699+0000 7fa7dd223640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa7d80cae90 0x7fa7d80cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:09.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.699+0000 7fa7dd223640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7d80ccba0 con 0x7fa7d80cd960 2024-03-24T10:50:09.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.699+0000 7fa7dd223640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7d80cce40 con 0x7fa7d80cb270 2024-03-24T10:50:09.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.699+0000 7fa7dd223640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fa7d80cc9b0 con 0x7fa7d80cae90 2024-03-24T10:50:09.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.700+0000 7fa7d759e640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fa7d80cd960 0x7fa7d80cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:50258/0 (socket says 172.21.15.5:50258) 2024-03-24T10:50:09.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.700+0000 7fa7d759e640 1 -- 172.21.15.5:0/2658612653 learned_addr learned my addr 172.21.15.5:0/2658612653 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:50:09.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.700+0000 7fa7d6d9d640 1 --2- 172.21.15.5:0/2658612653 >> v2:172.21.15.57:3300/0 conn(0x7fa7d80cae90 0x7fa7d80cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:09.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.703+0000 7fa7d5d9b640 1 -- 172.21.15.5:0/2658612653 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2598553065 0 0) 0x7fa7d80ccba0 con 0x7fa7d80cd960 2024-03-24T10:50:09.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.703+0000 7fa7d5d9b640 1 -- 172.21.15.5:0/2658612653 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7c0003520 con 0x7fa7d80cd960 2024-03-24T10:50:09.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.706+0000 7fa7d6d9d640 1 -- 172.21.15.5:0/2658612653 >> v1:172.21.15.57:6789/0 conn(0x7fa7d80cb270 legacy=0x7fa7d80cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:09.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.706+0000 7fa7d6d9d640 1 -- 172.21.15.5:0/2658612653 >> v1:172.21.15.5:6789/0 conn(0x7fa7d80cd960 legacy=0x7fa7d80cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:09.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.706+0000 7fa7d6d9d640 1 -- 172.21.15.5:0/2658612653 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7d815a590 con 0x7fa7d80cae90 2024-03-24T10:50:09.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.710+0000 7fa7d6d9d640 1 --2- 172.21.15.5:0/2658612653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7d80cae90 0x7fa7d80cf530 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa7c80097a0 tx=0x7fa7c802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a655e0d806e7521 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:09.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.716+0000 7fa7d5d9b640 1 -- 172.21.15.5:0/2658612653 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c802f810 con 0x7fa7d80cae90 2024-03-24T10:50:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.716+0000 7fa7d5d9b640 1 -- 172.21.15.5:0/2658612653 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7c802f970 con 0x7fa7d80cae90 2024-03-24T10:50:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.716+0000 7fa7d5d9b640 1 -- 172.21.15.5:0/2658612653 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7c802fd00 con 0x7fa7d80cae90 2024-03-24T10:50:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.717+0000 7fa7dd223640 1 -- 172.21.15.5:0/2658612653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7d80cae90 msgr2=0x7fa7d80cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.717+0000 7fa7dd223640 1 --2- 172.21.15.5:0/2658612653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7d80cae90 0x7fa7d80cf530 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa7c80097a0 tx=0x7fa7c802b9f0 comp rx=0 tx=0).stop 2024-03-24T10:50:09.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.717+0000 7fa7dd223640 1 -- 172.21.15.5:0/2658612653 shutdown_connections 2024-03-24T10:50:09.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.717+0000 7fa7dd223640 1 --2- 172.21.15.5:0/2658612653 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7d80cae90 0x7fa7d80cf530 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:09.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.717+0000 7fa7dd223640 1 -- 172.21.15.5:0/2658612653 >> 172.21.15.5:0/2658612653 conn(0x7fa7d80cfb80 msgr2=0x7fa7d80cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:09.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.717+0000 7fa7dd223640 1 -- 172.21.15.5:0/2658612653 shutdown_connections 2024-03-24T10:50:09.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.717+0000 7fa7dd223640 1 -- 172.21.15.5:0/2658612653 wait complete. 2024-03-24T10:50:09.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.719+0000 7fa7dd223640 1 Processor -- start 2024-03-24T10:50:09.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.719+0000 7fa7dd223640 1 -- start start 2024-03-24T10:50:09.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.719+0000 7fa7dd223640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa7d80cae90 0x7fa7d80c46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:09.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.719+0000 7fa7dd223640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7d80cb270 0x7fa7d80b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:09.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.719+0000 7fa7dd223640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa7d80cd960 0x7fa7d80c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:09.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.719+0000 7fa7dd223640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d80c4be0 con 0x7fa7d80cae90 2024-03-24T10:50:09.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.719+0000 7fa7dd223640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa7d80b3d80 con 0x7fa7d80cb270 2024-03-24T10:50:09.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.719+0000 7fa7dd223640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fa7d80c37d0 con 0x7fa7d80cd960 2024-03-24T10:50:09.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.720+0000 7fa7d6d9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa7d80cae90 0x7fa7d80c46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:09.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.720+0000 7fa7d659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7d80cb270 0x7fa7d80b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:09.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.720+0000 7fa7d759e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa7d80cd960 0x7fa7d80c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:09.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.720+0000 7fa7d659c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7d80cb270 0x7fa7d80b50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:47510/0 (socket says 172.21.15.5:47510) 2024-03-24T10:50:09.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.720+0000 7fa7d659c640 1 -- 172.21.15.5:0/2136834511 learned_addr learned my addr 172.21.15.5:0/2136834511 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:09.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.725+0000 7fa7d759e640 1 -- 172.21.15.5:0/2136834511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7d80cb270 msgr2=0x7fa7d80b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:09.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.725+0000 7fa7d759e640 1 --2- 172.21.15.5:0/2136834511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7d80cb270 0x7fa7d80b50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:09.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.725+0000 7fa7d759e640 1 -- 172.21.15.5:0/2136834511 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa7d80cae90 msgr2=0x7fa7d80c46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:09.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.725+0000 7fa7d759e640 1 --2- 172.21.15.5:0/2136834511 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa7d80cae90 0x7fa7d80c46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:09.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.725+0000 7fa7d759e640 1 -- 172.21.15.5:0/2136834511 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7c8009400 con 0x7fa7d80cd960 2024-03-24T10:50:09.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.727+0000 7fa7d759e640 1 --2- 172.21.15.5:0/2136834511 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa7d80cd960 0x7fa7d80c5760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa7d000caa0 tx=0x7fa7d000cea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:09.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.732+0000 7fa7bf7fe640 1 -- 172.21.15.5:0/2136834511 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d000ab40 con 0x7fa7d80cd960 2024-03-24T10:50:09.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.732+0000 7fa7bf7fe640 1 -- 172.21.15.5:0/2136834511 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7d000aca0 con 0x7fa7d80cd960 2024-03-24T10:50:09.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.732+0000 7fa7bf7fe640 1 -- 172.21.15.5:0/2136834511 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7d0012600 con 0x7fa7d80cd960 2024-03-24T10:50:09.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.732+0000 7fa7dd223640 1 -- 172.21.15.5:0/2136834511 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7d80c3a50 con 0x7fa7d80cd960 2024-03-24T10:50:09.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.732+0000 7fa7dd223640 1 -- 172.21.15.5:0/2136834511 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7d80b42b0 con 0x7fa7d80cd960 2024-03-24T10:50:09.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.733+0000 7fa7dd223640 1 -- 172.21.15.5:0/2136834511 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7d80c5480 con 0x7fa7d80cd960 2024-03-24T10:50:09.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.752+0000 7fa7bf7fe640 1 -- 172.21.15.5:0/2136834511 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa7d0007070 con 0x7fa7d80cd960 2024-03-24T10:50:09.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.752+0000 7fa7bf7fe640 1 --2- 172.21.15.5:0/2136834511 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa7c406b890 0x7fa7c406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:09.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.752+0000 7fa7d6d9d640 1 --2- 172.21.15.5:0/2136834511 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa7c406b890 0x7fa7c406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:09.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:09.752+0000 7fa7d6d9d640 1 --2- 172.21.15.5:0/2136834511 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa7c406b890 0x7fa7c406dd50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa7c8036000 tx=0x7fa7c8008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:10.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:10.006+0000 7fa7bf7fe640 1 -- 172.21.15.5:0/2136834511 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fa7d0058b20 con 0x7fa7d80cd960 2024-03-24T10:50:10.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:10.017+0000 7fa7bf7fe640 1 -- 172.21.15.5:0/2136834511 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa7d0054dc0 con 0x7fa7d80cd960 2024-03-24T10:50:10.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:10.267+0000 7fa7dd223640 1 -- 172.21.15.5:0/2136834511 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7d80c5480 con 0x7fa7d80cd960 2024-03-24T10:50:10.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:10.287+0000 7fa7bf7fe640 1 -- 172.21.15.5:0/2136834511 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fa7d80c5480 con 0x7fa7d80cd960 2024-03-24T10:50:10.288 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:10.288 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:10.288 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:10.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:10.290+0000 7fa7dd223640 1 -- 172.21.15.5:0/2136834511 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa7c406b890 msgr2=0x7fa7c406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:10.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:10.290+0000 7fa7dd223640 1 --2- 172.21.15.5:0/2136834511 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa7c406b890 0x7fa7c406dd50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa7c8036000 tx=0x7fa7c8008030 comp rx=0 tx=0).stop 2024-03-24T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:10.290+0000 7fa7dd223640 1 -- 172.21.15.5:0/2136834511 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa7d80cd960 msgr2=0x7fa7d80c5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:10.290+0000 7fa7dd223640 1 --2- 172.21.15.5:0/2136834511 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa7d80cd960 0x7fa7d80c5760 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa7d000caa0 tx=0x7fa7d000cea0 comp rx=0 tx=0).stop 2024-03-24T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:10.290+0000 7fa7dd223640 1 -- 172.21.15.5:0/2136834511 shutdown_connections 2024-03-24T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:10.290+0000 7fa7dd223640 1 --2- 172.21.15.5:0/2136834511 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa7c406b890 0x7fa7c406dd50 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:10.290+0000 7fa7dd223640 1 --2- 172.21.15.5:0/2136834511 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa7d80cd960 0x7fa7d80c5760 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:10.290+0000 7fa7dd223640 1 --2- 172.21.15.5:0/2136834511 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa7d80cb270 0x7fa7d80b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:10.290+0000 7fa7dd223640 1 --2- 172.21.15.5:0/2136834511 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa7d80cae90 0x7fa7d80c46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:10.290+0000 7fa7dd223640 1 -- 172.21.15.5:0/2136834511 >> 172.21.15.5:0/2136834511 conn(0x7fa7d80cfb80 msgr2=0x7fa7d80c7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:10.290+0000 7fa7dd223640 1 -- 172.21.15.5:0/2136834511 shutdown_connections 2024-03-24T10:50:10.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:10.290+0000 7fa7dd223640 1 -- 172.21.15.5:0/2136834511 wait complete. 2024-03-24T10:50:10.905 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:10.906 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:11.907 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:12.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.009+0000 7fb2e88df640 1 Processor -- start 2024-03-24T10:50:12.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.009+0000 7fb2e88df640 1 -- start start 2024-03-24T10:50:12.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.010+0000 7fb2e88df640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb2e40ca5a0 0x7fb2e40caa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:12.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.010+0000 7fb2e88df640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2e406cd50 con 0x7fb2e415aea0 2024-03-24T10:50:12.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.010+0000 7fb2e88df640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2e40caf50 con 0x7fb2e415aac0 2024-03-24T10:50:12.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.010+0000 7fb2e88df640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fb2e40ce790 con 0x7fb2e40ca5a0 2024-03-24T10:50:12.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.011+0000 7fb2e2d9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb2e40ca5a0 0x7fb2e40caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:12.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.011+0000 7fb2e259c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb2e415aac0 0x7fb2e406c230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:54086/0 (socket says 172.21.15.5:54086) 2024-03-24T10:50:12.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.011+0000 7fb2e259c640 1 -- 172.21.15.5:0/474016702 learned_addr learned my addr 172.21.15.5:0/474016702 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:50:12.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.014+0000 7fb2e2d9d640 1 -- 172.21.15.5:0/474016702 >> v1:172.21.15.57:6789/0 conn(0x7fb2e415aac0 legacy=0x7fb2e406c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:12.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.014+0000 7fb2e2d9d640 1 -- 172.21.15.5:0/474016702 >> v1:172.21.15.5:6789/0 conn(0x7fb2e415aea0 legacy=0x7fb2e40cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:12.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.014+0000 7fb2e159a640 1 -- 172.21.15.5:0/474016702 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1216855789 0 0) 0x7fb2e40caf50 con 0x7fb2e415aac0 2024-03-24T10:50:12.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.014+0000 7fb2e2d9d640 1 -- 172.21.15.5:0/474016702 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2e40cdb10 con 0x7fb2e40ca5a0 2024-03-24T10:50:12.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.017+0000 7fb2e2d9d640 1 --2- 172.21.15.5:0/474016702 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb2e40ca5a0 0x7fb2e40caa10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc009d90 tx=0x7fb2dc02bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa047c2ef0daae2c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:12.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.021+0000 7fb2e159a640 1 -- 172.21.15.5:0/474016702 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc02cd80 con 0x7fb2e40ca5a0 2024-03-24T10:50:12.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.021+0000 7fb2e159a640 1 -- 172.21.15.5:0/474016702 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2dc034440 con 0x7fb2e40ca5a0 2024-03-24T10:50:12.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.021+0000 7fb2e159a640 1 -- 172.21.15.5:0/474016702 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2dc0347d0 con 0x7fb2e40ca5a0 2024-03-24T10:50:12.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.022+0000 7fb2e88df640 1 -- 172.21.15.5:0/474016702 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb2e40ca5a0 msgr2=0x7fb2e40caa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:12.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.022+0000 7fb2e88df640 1 --2- 172.21.15.5:0/474016702 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb2e40ca5a0 0x7fb2e40caa10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc009d90 tx=0x7fb2dc02bfd0 comp rx=0 tx=0).stop 2024-03-24T10:50:12.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.022+0000 7fb2e88df640 1 -- 172.21.15.5:0/474016702 shutdown_connections 2024-03-24T10:50:12.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.022+0000 7fb2e88df640 1 --2- 172.21.15.5:0/474016702 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb2e40ca5a0 0x7fb2e40caa10 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:12.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.022+0000 7fb2e88df640 1 -- 172.21.15.5:0/474016702 >> 172.21.15.5:0/474016702 conn(0x7fb2e40cfb90 msgr2=0x7fb2e40cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:12.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.022+0000 7fb2e88df640 1 -- 172.21.15.5:0/474016702 shutdown_connections 2024-03-24T10:50:12.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.022+0000 7fb2e88df640 1 -- 172.21.15.5:0/474016702 wait complete. 2024-03-24T10:50:12.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.024+0000 7fb2e88df640 1 Processor -- start 2024-03-24T10:50:12.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.024+0000 7fb2e88df640 1 -- start start 2024-03-24T10:50:12.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.025+0000 7fb2e88df640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb2e40ca5a0 0x7fb2e40b0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:12.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.025+0000 7fb2e88df640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb2e415aac0 0x7fb2e415d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:12.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.025+0000 7fb2e88df640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2e415aea0 0x7fb2e415dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:12.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.025+0000 7fb2e88df640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e41e6530 con 0x7fb2e415aac0 2024-03-24T10:50:12.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.025+0000 7fb2e88df640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb2e41eff10 con 0x7fb2e415aea0 2024-03-24T10:50:12.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.025+0000 7fb2e88df640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fb2e41f0190 con 0x7fb2e40ca5a0 2024-03-24T10:50:12.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.025+0000 7fb2e1d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb2e415aac0 0x7fb2e415d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:12.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.025+0000 7fb2e259c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb2e40ca5a0 0x7fb2e40b0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:12.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.025+0000 7fb2e2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2e415aea0 0x7fb2e415dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:12.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.025+0000 7fb2e1d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb2e415aac0 0x7fb2e415d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51230/0 (socket says 172.21.15.5:51230) 2024-03-24T10:50:12.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.025+0000 7fb2e1d9b640 1 -- 172.21.15.5:0/2710258678 learned_addr learned my addr 172.21.15.5:0/2710258678 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:12.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.029+0000 7fb2e1d9b640 1 -- 172.21.15.5:0/2710258678 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb2e40ca5a0 msgr2=0x7fb2e40b0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:12.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.029+0000 7fb2e1d9b640 1 --2- 172.21.15.5:0/2710258678 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb2e40ca5a0 0x7fb2e40b0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:12.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.029+0000 7fb2e1d9b640 1 -- 172.21.15.5:0/2710258678 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2e415aea0 msgr2=0x7fb2e415dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:12.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.029+0000 7fb2e1d9b640 1 --2- 172.21.15.5:0/2710258678 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2e415aea0 0x7fb2e415dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:12.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.029+0000 7fb2e1d9b640 1 -- 172.21.15.5:0/2710258678 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2dc009a70 con 0x7fb2e415aac0 2024-03-24T10:50:12.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.031+0000 7fb2e1d9b640 1 --2- 172.21.15.5:0/2710258678 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb2e415aac0 0x7fb2e415d7b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800c480 tx=0x7fb2d800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:12.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.035+0000 7fb2d2ffd640 1 -- 172.21.15.5:0/2710258678 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d80052d0 con 0x7fb2e415aac0 2024-03-24T10:50:12.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.035+0000 7fb2d2ffd640 1 -- 172.21.15.5:0/2710258678 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2d8005430 con 0x7fb2e415aac0 2024-03-24T10:50:12.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.035+0000 7fb2d2ffd640 1 -- 172.21.15.5:0/2710258678 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d8012870 con 0x7fb2e415aac0 2024-03-24T10:50:12.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.035+0000 7fb2e88df640 1 -- 172.21.15.5:0/2710258678 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2e41f0470 con 0x7fb2e415aac0 2024-03-24T10:50:12.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.035+0000 7fb2e88df640 1 -- 172.21.15.5:0/2710258678 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2e41f0a30 con 0x7fb2e415aac0 2024-03-24T10:50:12.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.036+0000 7fb2e88df640 1 -- 172.21.15.5:0/2710258678 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2e40681d0 con 0x7fb2e415aac0 2024-03-24T10:50:12.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.305+0000 7fb2d2ffd640 1 -- 172.21.15.5:0/2710258678 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb2d80129d0 con 0x7fb2e415aac0 2024-03-24T10:50:12.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.305+0000 7fb2d2ffd640 1 --2- 172.21.15.5:0/2710258678 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb2c406b840 0x7fb2c406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:12.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.305+0000 7fb2e259c640 1 --2- 172.21.15.5:0/2710258678 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb2c406b840 0x7fb2c406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:12.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.306+0000 7fb2e259c640 1 --2- 172.21.15.5:0/2710258678 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb2c406b840 0x7fb2c406dd00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4000f60 tx=0x7fb2d4007e40 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:12.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.322+0000 7fb2d2ffd640 1 -- 172.21.15.5:0/2710258678 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fb2d8058f60 con 0x7fb2e415aac0 2024-03-24T10:50:12.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.322+0000 7fb2d2ffd640 1 -- 172.21.15.5:0/2710258678 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb2d8055200 con 0x7fb2e415aac0 2024-03-24T10:50:12.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.579+0000 7fb2e88df640 1 -- 172.21.15.5:0/2710258678 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb2e40cf770 con 0x7fb2e415aac0 2024-03-24T10:50:12.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.592+0000 7fb2d2ffd640 1 -- 172.21.15.5:0/2710258678 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fb2d80588c0 con 0x7fb2e415aac0 2024-03-24T10:50:12.592 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:12.592 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:12.592 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.594+0000 7fb2e88df640 1 -- 172.21.15.5:0/2710258678 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb2c406b840 msgr2=0x7fb2c406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.594+0000 7fb2e88df640 1 --2- 172.21.15.5:0/2710258678 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb2c406b840 0x7fb2c406dd00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4000f60 tx=0x7fb2d4007e40 comp rx=0 tx=0).stop 2024-03-24T10:50:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.594+0000 7fb2e88df640 1 -- 172.21.15.5:0/2710258678 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb2e415aac0 msgr2=0x7fb2e415d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.594+0000 7fb2e88df640 1 --2- 172.21.15.5:0/2710258678 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb2e415aac0 0x7fb2e415d7b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800c480 tx=0x7fb2d800c940 comp rx=0 tx=0).stop 2024-03-24T10:50:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.594+0000 7fb2e88df640 1 -- 172.21.15.5:0/2710258678 shutdown_connections 2024-03-24T10:50:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.594+0000 7fb2e88df640 1 --2- 172.21.15.5:0/2710258678 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb2c406b840 0x7fb2c406dd00 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.594+0000 7fb2e88df640 1 --2- 172.21.15.5:0/2710258678 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb2e415aac0 0x7fb2e415d7b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.594+0000 7fb2e88df640 1 --2- 172.21.15.5:0/2710258678 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb2e415aea0 0x7fb2e415dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.594+0000 7fb2e88df640 1 --2- 172.21.15.5:0/2710258678 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb2e40ca5a0 0x7fb2e40b0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.594+0000 7fb2e88df640 1 -- 172.21.15.5:0/2710258678 >> 172.21.15.5:0/2710258678 conn(0x7fb2e40cfb90 msgr2=0x7fb2e40c6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.594+0000 7fb2e88df640 1 -- 172.21.15.5:0/2710258678 shutdown_connections 2024-03-24T10:50:12.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:12.594+0000 7fb2e88df640 1 -- 172.21.15.5:0/2710258678 wait complete. 2024-03-24T10:50:13.209 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:13.209 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:14.210 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:14.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.312+0000 7f6b2724e640 1 Processor -- start 2024-03-24T10:50:14.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.312+0000 7f6b2724e640 1 -- start start 2024-03-24T10:50:14.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.313+0000 7f6b2724e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6b201543a0 0x7f6b2015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:14.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.313+0000 7f6b2724e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6b2006bf50 0x7f6b2006c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:14.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.314+0000 7f6b2724e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b200cb2f0 con 0x7f6b2006d4f0 2024-03-24T10:50:14.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.314+0000 7f6b2724e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f6b200c6750 con 0x7f6b2006bf50 2024-03-24T10:50:14.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.314+0000 7f6b2724e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f6b200cc380 con 0x7f6b201543a0 2024-03-24T10:50:14.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.314+0000 7f6b257eb640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6b2006bf50 0x7f6b2006c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:14.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.315+0000 7f6b24fea640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6b201543a0 0x7f6b2015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:14.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.315+0000 7f6b1ffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f6b2006d4f0 0x7f6b200cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:57332/0 (socket says 172.21.15.5:57332) 2024-03-24T10:50:14.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.315+0000 7f6b1ffff640 1 -- 172.21.15.5:0/3061494283 learned_addr learned my addr 172.21.15.5:0/3061494283 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:50:14.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.318+0000 7f6b257eb640 1 -- 172.21.15.5:0/3061494283 >> v2:172.21.15.57:3300/0 conn(0x7f6b201543a0 msgr2=0x7f6b2015ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:14.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.318+0000 7f6b257eb640 1 --2- 172.21.15.5:0/3061494283 >> v2:172.21.15.57:3300/0 conn(0x7f6b201543a0 0x7f6b2015ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:14.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.318+0000 7f6b257eb640 1 -- 172.21.15.5:0/3061494283 >> v1:172.21.15.57:6789/0 conn(0x7f6b2006d4f0 legacy=0x7f6b200cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:14.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.318+0000 7f6b257eb640 1 -- 172.21.15.5:0/3061494283 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b200ca7e0 con 0x7f6b2006bf50 2024-03-24T10:50:14.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.321+0000 7f6b257eb640 1 --2- 172.21.15.5:0/3061494283 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b2006bf50 0x7f6b2006c330 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6b200cd600 tx=0x7f6b1802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad4ab995d494264e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:14.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.323+0000 7f6b1f7fe640 1 -- 172.21.15.5:0/3061494283 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b1802cd00 con 0x7f6b2006bf50 2024-03-24T10:50:14.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.326+0000 7f6b1f7fe640 1 -- 172.21.15.5:0/3061494283 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b1802ce60 con 0x7f6b2006bf50 2024-03-24T10:50:14.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.326+0000 7f6b1f7fe640 1 -- 172.21.15.5:0/3061494283 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b18034660 con 0x7f6b2006bf50 2024-03-24T10:50:14.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.326+0000 7f6b2724e640 1 -- 172.21.15.5:0/3061494283 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b2006bf50 msgr2=0x7f6b2006c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:14.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.326+0000 7f6b2724e640 1 --2- 172.21.15.5:0/3061494283 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b2006bf50 0x7f6b2006c330 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6b200cd600 tx=0x7f6b1802bf50 comp rx=0 tx=0).stop 2024-03-24T10:50:14.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.326+0000 7f6b2724e640 1 -- 172.21.15.5:0/3061494283 shutdown_connections 2024-03-24T10:50:14.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.326+0000 7f6b2724e640 1 --2- 172.21.15.5:0/3061494283 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b2006bf50 0x7f6b2006c330 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:14.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.326+0000 7f6b2724e640 1 --2- 172.21.15.5:0/3061494283 >> v2:172.21.15.57:3300/0 conn(0x7f6b201543a0 0x7f6b2015ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:14.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.326+0000 7f6b2724e640 1 -- 172.21.15.5:0/3061494283 >> 172.21.15.5:0/3061494283 conn(0x7f6b20072ef0 msgr2=0x7f6b20061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:14.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.326+0000 7f6b2724e640 1 -- 172.21.15.5:0/3061494283 shutdown_connections 2024-03-24T10:50:14.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.326+0000 7f6b2724e640 1 -- 172.21.15.5:0/3061494283 wait complete. 2024-03-24T10:50:14.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.328+0000 7f6b2724e640 1 Processor -- start 2024-03-24T10:50:14.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.328+0000 7f6b2724e640 1 -- start start 2024-03-24T10:50:14.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.329+0000 7f6b2724e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6b2006bf50 0x7f6b200b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:14.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.329+0000 7f6b2724e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b2006d4f0 0x7f6b200b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:14.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.329+0000 7f6b2724e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b201543a0 0x7f6b20146320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:14.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.329+0000 7f6b2724e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f6b20158970 con 0x7f6b201543a0 2024-03-24T10:50:14.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.329+0000 7f6b2724e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6b20155470 con 0x7f6b2006d4f0 2024-03-24T10:50:14.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.329+0000 7f6b2724e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f6b20150230 con 0x7f6b2006bf50 2024-03-24T10:50:14.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.329+0000 7f6b257eb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b201543a0 0x7f6b20146320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:14.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.329+0000 7f6b1ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b2006d4f0 0x7f6b200b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:14.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.329+0000 7f6b257eb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b201543a0 0x7f6b20146320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:40448/0 (socket says 172.21.15.5:40448) 2024-03-24T10:50:14.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.329+0000 7f6b257eb640 1 -- 172.21.15.5:0/4099026747 learned_addr learned my addr 172.21.15.5:0/4099026747 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:14.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.329+0000 7f6b24fea640 1 --2- 172.21.15.5:0/4099026747 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6b2006bf50 0x7f6b200b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:14.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.333+0000 7f6b1ffff640 1 -- 172.21.15.5:0/4099026747 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6b2006bf50 msgr2=0x7f6b200b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:14.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.333+0000 7f6b1ffff640 1 --2- 172.21.15.5:0/4099026747 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6b2006bf50 0x7f6b200b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:14.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.333+0000 7f6b1ffff640 1 -- 172.21.15.5:0/4099026747 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b201543a0 msgr2=0x7f6b20146320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:14.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.333+0000 7f6b1ffff640 1 --2- 172.21.15.5:0/4099026747 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b201543a0 0x7f6b20146320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:14.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.333+0000 7f6b1ffff640 1 -- 172.21.15.5:0/4099026747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b18009a70 con 0x7f6b2006d4f0 2024-03-24T10:50:14.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.336+0000 7f6b1ffff640 1 --2- 172.21.15.5:0/4099026747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b2006d4f0 0x7f6b200b3d10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c000f60 tx=0x7f6b0c00c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:14.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.341+0000 7f6b1d7fa640 1 -- 172.21.15.5:0/4099026747 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c005ce0 con 0x7f6b2006d4f0 2024-03-24T10:50:14.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.341+0000 7f6b1d7fa640 1 -- 172.21.15.5:0/4099026747 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b0c005e40 con 0x7f6b2006d4f0 2024-03-24T10:50:14.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.341+0000 7f6b2724e640 1 -- 172.21.15.5:0/4099026747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b2014e950 con 0x7f6b2006d4f0 2024-03-24T10:50:14.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.341+0000 7f6b1d7fa640 1 -- 172.21.15.5:0/4099026747 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c007310 con 0x7f6b2006d4f0 2024-03-24T10:50:14.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.341+0000 7f6b2724e640 1 -- 172.21.15.5:0/4099026747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b20158ea0 con 0x7f6b2006d4f0 2024-03-24T10:50:14.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.342+0000 7f6b2724e640 1 -- 172.21.15.5:0/4099026747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b200681d0 con 0x7f6b2006d4f0 2024-03-24T10:50:14.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.367+0000 7f6b1d7fa640 1 -- 172.21.15.5:0/4099026747 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6b0c007470 con 0x7f6b2006d4f0 2024-03-24T10:50:14.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.368+0000 7f6b1d7fa640 1 --2- 172.21.15.5:0/4099026747 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6b0406b810 0x7f6b0406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:14.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.368+0000 7f6b24fea640 1 --2- 172.21.15.5:0/4099026747 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6b0406b810 0x7f6b0406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:14.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.368+0000 7f6b24fea640 1 --2- 172.21.15.5:0/4099026747 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6b0406b810 0x7f6b0406dcd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6b14009aa0 tx=0x7f6b14009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:14.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.661+0000 7f6b1d7fa640 1 -- 172.21.15.5:0/4099026747 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f6b0c007720 con 0x7f6b2006d4f0 2024-03-24T10:50:14.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.677+0000 7f6b1d7fa640 1 -- 172.21.15.5:0/4099026747 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6b0c0545b0 con 0x7f6b2006d4f0 2024-03-24T10:50:14.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.900+0000 7f6b2724e640 1 -- 172.21.15.5:0/4099026747 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b200b2620 con 0x7f6b2006d4f0 2024-03-24T10:50:14.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.922+0000 7f6b1d7fa640 1 -- 172.21.15.5:0/4099026747 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f6b0c017070 con 0x7f6b2006d4f0 2024-03-24T10:50:14.922 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:14.923 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:14.923 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:14.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.925+0000 7f6b2724e640 1 -- 172.21.15.5:0/4099026747 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6b0406b810 msgr2=0x7f6b0406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:14.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.925+0000 7f6b2724e640 1 --2- 172.21.15.5:0/4099026747 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6b0406b810 0x7f6b0406dcd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6b14009aa0 tx=0x7f6b14009190 comp rx=0 tx=0).stop 2024-03-24T10:50:14.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.925+0000 7f6b2724e640 1 -- 172.21.15.5:0/4099026747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b2006d4f0 msgr2=0x7f6b200b3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:14.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.925+0000 7f6b2724e640 1 --2- 172.21.15.5:0/4099026747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b2006d4f0 0x7f6b200b3d10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c000f60 tx=0x7f6b0c00c800 comp rx=0 tx=0).stop 2024-03-24T10:50:14.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.925+0000 7f6b2724e640 1 -- 172.21.15.5:0/4099026747 shutdown_connections 2024-03-24T10:50:14.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.925+0000 7f6b2724e640 1 --2- 172.21.15.5:0/4099026747 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6b0406b810 0x7f6b0406dcd0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:14.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.925+0000 7f6b2724e640 1 --2- 172.21.15.5:0/4099026747 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b201543a0 0x7f6b20146320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:14.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.925+0000 7f6b2724e640 1 --2- 172.21.15.5:0/4099026747 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b2006d4f0 0x7f6b200b3d10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:14.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.925+0000 7f6b2724e640 1 --2- 172.21.15.5:0/4099026747 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6b2006bf50 0x7f6b200b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:14.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.925+0000 7f6b2724e640 1 -- 172.21.15.5:0/4099026747 >> 172.21.15.5:0/4099026747 conn(0x7f6b20072ef0 msgr2=0x7f6b200ca9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:14.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.925+0000 7f6b2724e640 1 -- 172.21.15.5:0/4099026747 shutdown_connections 2024-03-24T10:50:14.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:14.925+0000 7f6b2724e640 1 -- 172.21.15.5:0/4099026747 wait complete. 2024-03-24T10:50:15.539 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:15.540 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:16.541 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:16.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.645+0000 7f910c264640 1 Processor -- start 2024-03-24T10:50:16.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.645+0000 7f910c264640 1 -- start start 2024-03-24T10:50:16.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.646+0000 7f910c264640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f91041543a0 0x7f910415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:16.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.646+0000 7f910c264640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91040cb2f0 con 0x7f910406d4f0 2024-03-24T10:50:16.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.646+0000 7f910c264640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91040ca7e0 con 0x7f910406bf50 2024-03-24T10:50:16.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.646+0000 7f910c264640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f91040c6750 con 0x7f91041543a0 2024-03-24T10:50:16.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.647+0000 7f91097ff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f910406d4f0 0x7f91040cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36538/0 (socket says 172.21.15.5:36538) 2024-03-24T10:50:16.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.647+0000 7f91097ff640 1 -- 172.21.15.5:0/3481093635 learned_addr learned my addr 172.21.15.5:0/3481093635 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:50:16.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.647+0000 7f910a000640 1 --2- 172.21.15.5:0/3481093635 >> v2:172.21.15.57:3300/0 conn(0x7f91041543a0 0x7f910415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:16.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.649+0000 7f9108ffe640 1 -- 172.21.15.5:0/3481093635 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 29388793 0 0) 0x7f91040cb2f0 con 0x7f910406d4f0 2024-03-24T10:50:16.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.649+0000 7f9108ffe640 1 -- 172.21.15.5:0/3481093635 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90f0003520 con 0x7f910406d4f0 2024-03-24T10:50:16.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.652+0000 7f9108ffe640 1 -- 172.21.15.5:0/3481093635 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 816907601 0 0) 0x7f91040ca7e0 con 0x7f910406bf50 2024-03-24T10:50:16.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.652+0000 7f9108ffe640 1 -- 172.21.15.5:0/3481093635 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91040cb2f0 con 0x7f910406bf50 2024-03-24T10:50:16.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.653+0000 7f9108ffe640 1 -- 172.21.15.5:0/3481093635 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 631511659 0 0) 0x7f90f0003520 con 0x7f910406d4f0 2024-03-24T10:50:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.653+0000 7f9108ffe640 1 -- 172.21.15.5:0/3481093635 >> v2:172.21.15.57:3300/0 conn(0x7f91041543a0 msgr2=0x7f910415ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.653+0000 7f9108ffe640 1 --2- 172.21.15.5:0/3481093635 >> v2:172.21.15.57:3300/0 conn(0x7f91041543a0 0x7f910415ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.653+0000 7f9108ffe640 1 -- 172.21.15.5:0/3481093635 >> v1:172.21.15.57:6789/0 conn(0x7f910406bf50 legacy=0x7f910406c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.653+0000 7f9108ffe640 1 -- 172.21.15.5:0/3481093635 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91040cb520 con 0x7f910406d4f0 2024-03-24T10:50:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.653+0000 7f9108ffe640 1 -- 172.21.15.5:0/3481093635 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 100681612 0 0) 0x7f90f4001670 con 0x7f910406d4f0 2024-03-24T10:50:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.653+0000 7f9108ffe640 1 -- 172.21.15.5:0/3481093635 >> v1:172.21.15.5:6789/0 conn(0x7f910406d4f0 legacy=0x7f91040cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.653+0000 7f9108ffe640 1 --2- 172.21.15.5:0/3481093635 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90f0003b50 0x7f90f0003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.653+0000 7f9108ffe640 1 --2- 172.21.15.5:0/3481093635 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90f00047c0 0x7f90f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.653+0000 7f9108ffe640 1 --2- 172.21.15.5:0/3481093635 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90f0005180 0x7f90f000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.653+0000 7f9108ffe640 1 -- 172.21.15.5:0/3481093635 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f91040c6750 con 0x7f90f0003b50 2024-03-24T10:50:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.653+0000 7f9108ffe640 1 -- 172.21.15.5:0/3481093635 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f90f000bfa0 con 0x7f90f00047c0 2024-03-24T10:50:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.653+0000 7f9108ffe640 1 -- 172.21.15.5:0/3481093635 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f90f000c160 con 0x7f90f0005180 2024-03-24T10:50:16.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.654+0000 7f910a801640 1 --2- 172.21.15.5:0/3481093635 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90f0005180 0x7f90f000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:16.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.656+0000 7f910a000640 1 --2- 172.21.15.5:0/3481093635 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90f0003b50 0x7f90f0003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:16.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.658+0000 7f910a801640 1 -- 172.21.15.5:0/3481093635 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90f00047c0 msgr2=0x7f90f0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:16.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.658+0000 7f910a801640 1 --2- 172.21.15.5:0/3481093635 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90f00047c0 0x7f90f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:16.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.658+0000 7f910a801640 1 -- 172.21.15.5:0/3481093635 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90f0003b50 msgr2=0x7f90f0003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:16.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.658+0000 7f910a801640 1 --2- 172.21.15.5:0/3481093635 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90f0003b50 0x7f90f0003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:16.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.658+0000 7f910a801640 1 -- 172.21.15.5:0/3481093635 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90f0003520 con 0x7f90f0005180 2024-03-24T10:50:16.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.659+0000 7f910a801640 1 -- 172.21.15.5:0/3481093635 reap_dead start 2024-03-24T10:50:16.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.660+0000 7f910a801640 1 --2- 172.21.15.5:0/3481093635 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90f0005180 0x7f90f000b980 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f90fc001fd0 tx=0x7f90f00057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91d744cbd070a81c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:16.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.665+0000 7f9108ffe640 1 -- 172.21.15.5:0/3481093635 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90fc003b70 con 0x7f90f0005180 2024-03-24T10:50:16.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.665+0000 7f9108ffe640 1 -- 172.21.15.5:0/3481093635 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90fc003cd0 con 0x7f90f0005180 2024-03-24T10:50:16.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.665+0000 7f9108ffe640 1 -- 172.21.15.5:0/3481093635 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90fc005310 con 0x7f90f0005180 2024-03-24T10:50:16.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.666+0000 7f910c264640 1 -- 172.21.15.5:0/3481093635 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90f0005180 msgr2=0x7f90f000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:16.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.666+0000 7f910c264640 1 --2- 172.21.15.5:0/3481093635 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90f0005180 0x7f90f000b980 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f90fc001fd0 tx=0x7f90f00057f0 comp rx=0 tx=0).stop 2024-03-24T10:50:16.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.666+0000 7f910c264640 1 -- 172.21.15.5:0/3481093635 shutdown_connections 2024-03-24T10:50:16.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.666+0000 7f910c264640 1 --2- 172.21.15.5:0/3481093635 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90f0005180 0x7f90f000b980 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:16.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.666+0000 7f910c264640 1 -- 172.21.15.5:0/3481093635 >> 172.21.15.5:0/3481093635 conn(0x7f9104072ef0 msgr2=0x7f9104061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:16.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.666+0000 7f910c264640 1 -- 172.21.15.5:0/3481093635 shutdown_connections 2024-03-24T10:50:16.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.666+0000 7f910c264640 1 -- 172.21.15.5:0/3481093635 wait complete. 2024-03-24T10:50:16.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.667+0000 7f910c264640 1 Processor -- start 2024-03-24T10:50:16.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.667+0000 7f910c264640 1 -- start start 2024-03-24T10:50:16.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.668+0000 7f910c264640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90f0005180 0x7f91040ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:16.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.668+0000 7f910c264640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91040c9b70 0x7f91040cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:16.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.668+0000 7f910c264640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9104151bc0 0x7f9104152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:16.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.668+0000 7f910c264640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f9104148d90 con 0x7f90f0005180 2024-03-24T10:50:16.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.668+0000 7f910c264640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9104157a20 con 0x7f91040c9b70 2024-03-24T10:50:16.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.668+0000 7f910c264640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f9104152580 con 0x7f9104151bc0 2024-03-24T10:50:16.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.669+0000 7f910a000640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90f0005180 0x7f91040ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:16.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.669+0000 7f910a000640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90f0005180 0x7f91040ceb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:40462/0 (socket says 172.21.15.5:40462) 2024-03-24T10:50:16.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.669+0000 7f910a000640 1 -- 172.21.15.5:0/1236025541 learned_addr learned my addr 172.21.15.5:0/1236025541 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:16.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.669+0000 7f910a801640 1 --2- 172.21.15.5:0/1236025541 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9104151bc0 0x7f9104152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:16.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.670+0000 7f91097ff640 1 --2- 172.21.15.5:0/1236025541 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91040c9b70 0x7f91040cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:16.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.672+0000 7f910a000640 1 -- 172.21.15.5:0/1236025541 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9104151bc0 msgr2=0x7f9104152040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:16.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.672+0000 7f910a000640 1 --2- 172.21.15.5:0/1236025541 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9104151bc0 0x7f9104152040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:16.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.672+0000 7f910a000640 1 -- 172.21.15.5:0/1236025541 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91040c9b70 msgr2=0x7f91040cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:16.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.672+0000 7f910a000640 1 --2- 172.21.15.5:0/1236025541 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91040c9b70 0x7f91040cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:16.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.672+0000 7f910a000640 1 -- 172.21.15.5:0/1236025541 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9104153d10 con 0x7f90f0005180 2024-03-24T10:50:16.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.675+0000 7f910a000640 1 --2- 172.21.15.5:0/1236025541 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90f0005180 0x7f91040ceb90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f91000118e0 tx=0x7f9100011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:16.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.680+0000 7f90fa7fc640 1 -- 172.21.15.5:0/1236025541 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f910000fa70 con 0x7f90f0005180 2024-03-24T10:50:16.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.680+0000 7f90fa7fc640 1 -- 172.21.15.5:0/1236025541 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f910000fbd0 con 0x7f90f0005180 2024-03-24T10:50:16.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.680+0000 7f910c264640 1 -- 172.21.15.5:0/1236025541 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91041f0140 con 0x7f90f0005180 2024-03-24T10:50:16.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.680+0000 7f90fa7fc640 1 -- 172.21.15.5:0/1236025541 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9100014440 con 0x7f90f0005180 2024-03-24T10:50:16.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.680+0000 7f910c264640 1 -- 172.21.15.5:0/1236025541 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9104148a20 con 0x7f90f0005180 2024-03-24T10:50:16.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.681+0000 7f910c264640 1 -- 172.21.15.5:0/1236025541 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91041f0660 con 0x7f90f0005180 2024-03-24T10:50:16.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.706+0000 7f90fa7fc640 1 -- 172.21.15.5:0/1236025541 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f910001d400 con 0x7f90f0005180 2024-03-24T10:50:16.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.706+0000 7f90fa7fc640 1 --2- 172.21.15.5:0/1236025541 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f90e806b890 0x7f90e806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:16.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.707+0000 7f91097ff640 1 --2- 172.21.15.5:0/1236025541 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f90e806b890 0x7f90e806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:16.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.707+0000 7f91097ff640 1 --2- 172.21.15.5:0/1236025541 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f90e806b890 0x7f90e806dd50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f91040621a0 tx=0x7f90f4001930 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:16.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.710+0000 7f90fa7fc640 1 -- 172.21.15.5:0/1236025541 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f9100059610 con 0x7f90f0005180 2024-03-24T10:50:16.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:16.998+0000 7f90fa7fc640 1 -- 172.21.15.5:0/1236025541 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f91000558b0 con 0x7f90f0005180 2024-03-24T10:50:17.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:17.262+0000 7f910c264640 1 -- 172.21.15.5:0/1236025541 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91040681d0 con 0x7f90f0005180 2024-03-24T10:50:17.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:17.275+0000 7f90fa7fc640 1 -- 172.21.15.5:0/1236025541 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f9100058f70 con 0x7f90f0005180 2024-03-24T10:50:17.276 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:17.276 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:17.276 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:17.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:17.277+0000 7f910c264640 1 -- 172.21.15.5:0/1236025541 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f90e806b890 msgr2=0x7f90e806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:17.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:17.277+0000 7f910c264640 1 --2- 172.21.15.5:0/1236025541 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f90e806b890 0x7f90e806dd50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f91040621a0 tx=0x7f90f4001930 comp rx=0 tx=0).stop 2024-03-24T10:50:17.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:17.277+0000 7f910c264640 1 -- 172.21.15.5:0/1236025541 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90f0005180 msgr2=0x7f91040ceb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:17.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:17.277+0000 7f910c264640 1 --2- 172.21.15.5:0/1236025541 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90f0005180 0x7f91040ceb90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f91000118e0 tx=0x7f9100011da0 comp rx=0 tx=0).stop 2024-03-24T10:50:17.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:17.278+0000 7f910c264640 1 -- 172.21.15.5:0/1236025541 shutdown_connections 2024-03-24T10:50:17.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:17.278+0000 7f910c264640 1 --2- 172.21.15.5:0/1236025541 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f90e806b890 0x7f90e806dd50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:17.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:17.278+0000 7f910c264640 1 --2- 172.21.15.5:0/1236025541 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9104151bc0 0x7f9104152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:17.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:17.278+0000 7f910c264640 1 --2- 172.21.15.5:0/1236025541 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f91040c9b70 0x7f91040cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:17.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:17.278+0000 7f910c264640 1 --2- 172.21.15.5:0/1236025541 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90f0005180 0x7f91040ceb90 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:17.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:17.278+0000 7f910c264640 1 -- 172.21.15.5:0/1236025541 >> 172.21.15.5:0/1236025541 conn(0x7f9104072ef0 msgr2=0x7f9104158080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:17.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:17.278+0000 7f910c264640 1 -- 172.21.15.5:0/1236025541 shutdown_connections 2024-03-24T10:50:17.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:17.278+0000 7f910c264640 1 -- 172.21.15.5:0/1236025541 wait complete. 2024-03-24T10:50:17.893 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:17.893 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:18.894 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:18.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:18.996+0000 7f68a751e640 1 Processor -- start 2024-03-24T10:50:18.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:18.996+0000 7f68a751e640 1 -- start start 2024-03-24T10:50:18.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:18.997+0000 7f68a751e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f68a00cae90 0x7f68a00cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:18.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:18.997+0000 7f68a751e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68a00ccba0 con 0x7f68a00cb270 2024-03-24T10:50:18.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:18.997+0000 7f68a751e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68a00cce40 con 0x7f68a00cd960 2024-03-24T10:50:18.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:18.997+0000 7f68a751e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f68a00cc9b0 con 0x7f68a00cae90 2024-03-24T10:50:18.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:18.998+0000 7f68a5abb640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f68a00cd960 0x7f68a00cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:57356/0 (socket says 172.21.15.5:57356) 2024-03-24T10:50:18.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:18.998+0000 7f68a5abb640 1 -- 172.21.15.5:0/1450036921 learned_addr learned my addr 172.21.15.5:0/1450036921 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:50:18.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:18.998+0000 7f68a52ba640 1 --2- 172.21.15.5:0/1450036921 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f68a00cae90 0x7f68a00cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:19.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.001+0000 7f6897fff640 1 -- 172.21.15.5:0/1450036921 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2512347814 0 0) 0x7f68a00ccba0 con 0x7f68a00cb270 2024-03-24T10:50:19.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.001+0000 7f6897fff640 1 -- 172.21.15.5:0/1450036921 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f688c003520 con 0x7f68a00cb270 2024-03-24T10:50:19.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.001+0000 7f6897fff640 1 -- 172.21.15.5:0/1450036921 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3049251938 0 0) 0x7f68a00cce40 con 0x7f68a00cd960 2024-03-24T10:50:19.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.001+0000 7f6897fff640 1 -- 172.21.15.5:0/1450036921 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68a00ccba0 con 0x7f68a00cd960 2024-03-24T10:50:19.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.002+0000 7f68a52ba640 1 -- 172.21.15.5:0/1450036921 >> v1:172.21.15.57:6789/0 conn(0x7f68a00cd960 legacy=0x7f68a00cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:19.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.002+0000 7f68a52ba640 1 -- 172.21.15.5:0/1450036921 >> v1:172.21.15.5:6789/0 conn(0x7f68a00cb270 legacy=0x7f68a00cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:19.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.002+0000 7f68a52ba640 1 -- 172.21.15.5:0/1450036921 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a015a590 con 0x7f68a00cae90 2024-03-24T10:50:19.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.005+0000 7f68a52ba640 1 --2- 172.21.15.5:0/1450036921 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f68a00cae90 0x7f68a00cf530 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f689c0097d0 tx=0x7f689c02b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf32fde35cab9f93 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:19.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.008+0000 7f6897fff640 1 -- 172.21.15.5:0/1450036921 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c030880 con 0x7f68a00cae90 2024-03-24T10:50:19.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.010+0000 7f6897fff640 1 -- 172.21.15.5:0/1450036921 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f689c0309e0 con 0x7f68a00cae90 2024-03-24T10:50:19.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.010+0000 7f6897fff640 1 -- 172.21.15.5:0/1450036921 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c030d50 con 0x7f68a00cae90 2024-03-24T10:50:19.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.011+0000 7f68a751e640 1 -- 172.21.15.5:0/1450036921 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f68a00cae90 msgr2=0x7f68a00cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:19.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.011+0000 7f68a751e640 1 --2- 172.21.15.5:0/1450036921 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f68a00cae90 0x7f68a00cf530 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f689c0097d0 tx=0x7f689c02b9c0 comp rx=0 tx=0).stop 2024-03-24T10:50:19.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.011+0000 7f68a751e640 1 -- 172.21.15.5:0/1450036921 shutdown_connections 2024-03-24T10:50:19.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.011+0000 7f68a751e640 1 --2- 172.21.15.5:0/1450036921 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f68a00cae90 0x7f68a00cf530 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:19.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.011+0000 7f68a751e640 1 -- 172.21.15.5:0/1450036921 >> 172.21.15.5:0/1450036921 conn(0x7f68a00cfb80 msgr2=0x7f68a00cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:19.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.011+0000 7f68a751e640 1 -- 172.21.15.5:0/1450036921 shutdown_connections 2024-03-24T10:50:19.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.011+0000 7f68a751e640 1 -- 172.21.15.5:0/1450036921 wait complete. 2024-03-24T10:50:19.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.012+0000 7f68a751e640 1 Processor -- start 2024-03-24T10:50:19.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.012+0000 7f68a751e640 1 -- start start 2024-03-24T10:50:19.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.013+0000 7f68a751e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f68a00cae90 0x7f68a01f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:19.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.013+0000 7f68a751e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f68a00cb270 0x7f68a00ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:19.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.013+0000 7f68a751e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68a00cd960 0x7f68a01f0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:19.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.013+0000 7f68a751e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f68a00b29d0 con 0x7f68a00cae90 2024-03-24T10:50:19.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.013+0000 7f68a751e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f68a015db50 con 0x7f68a00cd960 2024-03-24T10:50:19.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.013+0000 7f68a751e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f68a015be00 con 0x7f68a00cb270 2024-03-24T10:50:19.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.013+0000 7f68a5abb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68a00cd960 0x7f68a01f0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:19.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.013+0000 7f68a4ab9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f68a00cb270 0x7f68a00ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:19.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.014+0000 7f68a52ba640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f68a00cae90 0x7f68a01f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:19.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.014+0000 7f68a5abb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68a00cd960 0x7f68a01f0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:39590/0 (socket says 172.21.15.5:39590) 2024-03-24T10:50:19.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.014+0000 7f68a5abb640 1 -- 172.21.15.5:0/1331618064 learned_addr learned my addr 172.21.15.5:0/1331618064 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:19.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.018+0000 7f68a5abb640 1 -- 172.21.15.5:0/1331618064 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f68a00cb270 msgr2=0x7f68a00ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:19.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.018+0000 7f68a5abb640 1 --2- 172.21.15.5:0/1331618064 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f68a00cb270 0x7f68a00ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:19.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.018+0000 7f68a5abb640 1 -- 172.21.15.5:0/1331618064 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f68a00cae90 msgr2=0x7f68a01f3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:19.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.018+0000 7f68a5abb640 1 --2- 172.21.15.5:0/1331618064 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f68a00cae90 0x7f68a01f3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:19.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.018+0000 7f68a5abb640 1 -- 172.21.15.5:0/1331618064 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f689c009400 con 0x7f68a00cd960 2024-03-24T10:50:19.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.021+0000 7f68a5abb640 1 --2- 172.21.15.5:0/1331618064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68a00cd960 0x7f68a01f0520 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f689800cbf0 tx=0x7f689800a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:19.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.025+0000 7f6895ffb640 1 -- 172.21.15.5:0/1331618064 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689800ad40 con 0x7f68a00cd960 2024-03-24T10:50:19.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.026+0000 7f6895ffb640 1 -- 172.21.15.5:0/1331618064 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6898012400 con 0x7f68a00cd960 2024-03-24T10:50:19.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.026+0000 7f68a751e640 1 -- 172.21.15.5:0/1331618064 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68a01efd80 con 0x7f68a00cd960 2024-03-24T10:50:19.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.026+0000 7f6895ffb640 1 -- 172.21.15.5:0/1331618064 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689800aea0 con 0x7f68a00cd960 2024-03-24T10:50:19.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.026+0000 7f68a751e640 1 -- 172.21.15.5:0/1331618064 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68a0157170 con 0x7f68a00cd960 2024-03-24T10:50:19.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.027+0000 7f68a751e640 1 -- 172.21.15.5:0/1331618064 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68a00681d0 con 0x7f68a00cd960 2024-03-24T10:50:19.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.296+0000 7f6895ffb640 1 -- 172.21.15.5:0/1331618064 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6898007070 con 0x7f68a00cd960 2024-03-24T10:50:19.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.296+0000 7f6895ffb640 1 --2- 172.21.15.5:0/1331618064 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f688406b810 0x7f688406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:19.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.297+0000 7f68a52ba640 1 --2- 172.21.15.5:0/1331618064 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f688406b810 0x7f688406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:19.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.297+0000 7f68a52ba640 1 --2- 172.21.15.5:0/1331618064 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f688406b810 0x7f688406dcd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f689c036000 tx=0x7f689c008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:19.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.301+0000 7f6895ffb640 1 -- 172.21.15.5:0/1331618064 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f6898058e50 con 0x7f68a00cd960 2024-03-24T10:50:19.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.322+0000 7f6895ffb640 1 -- 172.21.15.5:0/1331618064 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f68980550f0 con 0x7f68a00cd960 2024-03-24T10:50:19.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.547+0000 7f68a751e640 1 -- 172.21.15.5:0/1331618064 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68a00c9d50 con 0x7f68a00cd960 2024-03-24T10:50:19.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.622+0000 7f6895ffb640 1 -- 172.21.15.5:0/1331618064 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f6898017070 con 0x7f68a00cd960 2024-03-24T10:50:19.623 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:19.623 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:19.623 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:19.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.625+0000 7f68a751e640 1 -- 172.21.15.5:0/1331618064 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f688406b810 msgr2=0x7f688406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:19.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.625+0000 7f68a751e640 1 --2- 172.21.15.5:0/1331618064 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f688406b810 0x7f688406dcd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f689c036000 tx=0x7f689c008030 comp rx=0 tx=0).stop 2024-03-24T10:50:19.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.625+0000 7f68a751e640 1 -- 172.21.15.5:0/1331618064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68a00cd960 msgr2=0x7f68a01f0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:19.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.625+0000 7f68a751e640 1 --2- 172.21.15.5:0/1331618064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68a00cd960 0x7f68a01f0520 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f689800cbf0 tx=0x7f689800a430 comp rx=0 tx=0).stop 2024-03-24T10:50:19.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.625+0000 7f68a751e640 1 -- 172.21.15.5:0/1331618064 shutdown_connections 2024-03-24T10:50:19.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.625+0000 7f68a751e640 1 --2- 172.21.15.5:0/1331618064 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f688406b810 0x7f688406dcd0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:19.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.625+0000 7f68a751e640 1 --2- 172.21.15.5:0/1331618064 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f68a00cd960 0x7f68a01f0520 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:19.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.625+0000 7f68a751e640 1 --2- 172.21.15.5:0/1331618064 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f68a00cb270 0x7f68a00ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:19.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.625+0000 7f68a751e640 1 --2- 172.21.15.5:0/1331618064 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f68a00cae90 0x7f68a01f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:19.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.625+0000 7f68a751e640 1 -- 172.21.15.5:0/1331618064 >> 172.21.15.5:0/1331618064 conn(0x7f68a00cfb80 msgr2=0x7f68a006c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:19.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.625+0000 7f68a751e640 1 -- 172.21.15.5:0/1331618064 shutdown_connections 2024-03-24T10:50:19.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:19.625+0000 7f68a751e640 1 -- 172.21.15.5:0/1331618064 wait complete. 2024-03-24T10:50:20.240 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:20.240 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:21.241 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:21.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.343+0000 7f0f9607c640 1 Processor -- start 2024-03-24T10:50:21.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.344+0000 7f0f9607c640 1 -- start start 2024-03-24T10:50:21.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.344+0000 7f0f9607c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f9015ac00 0x7f0f900cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:21.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.344+0000 7f0f9607c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0f900ccaa0 0x7f0f9006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:21.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.344+0000 7f0f9607c640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f900cb2f0 con 0x7f0f901543a0 2024-03-24T10:50:21.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.344+0000 7f0f9607c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f0f900c6750 con 0x7f0f900ccaa0 2024-03-24T10:50:21.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.344+0000 7f0f9607c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0f900cc380 con 0x7f0f9015ac00 2024-03-24T10:50:21.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.345+0000 7f0f8f7fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f0f901543a0 0x7f0f9006d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36560/0 (socket says 172.21.15.5:36560) 2024-03-24T10:50:21.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.345+0000 7f0f8f7fe640 1 -- 172.21.15.5:0/3342040019 learned_addr learned my addr 172.21.15.5:0/3342040019 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:50:21.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.345+0000 7f0f8effd640 1 --2- 172.21.15.5:0/3342040019 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f9015ac00 0x7f0f900cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:21.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.345+0000 7f0f8ffff640 1 --2- 172.21.15.5:0/3342040019 >> v2:172.21.15.57:3300/0 conn(0x7f0f900ccaa0 0x7f0f9006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:21.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.348+0000 7f0f8e7fc640 1 -- 172.21.15.5:0/3342040019 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 490028175 0 0) 0x7f0f900cb2f0 con 0x7f0f901543a0 2024-03-24T10:50:21.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.348+0000 7f0f8e7fc640 1 -- 172.21.15.5:0/3342040019 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f78003520 con 0x7f0f901543a0 2024-03-24T10:50:21.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.349+0000 7f0f8ffff640 1 -- 172.21.15.5:0/3342040019 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f9015ac00 msgr2=0x7f0f900cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:21.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.349+0000 7f0f8ffff640 1 --2- 172.21.15.5:0/3342040019 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f9015ac00 0x7f0f900cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:21.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.349+0000 7f0f8ffff640 1 -- 172.21.15.5:0/3342040019 >> v1:172.21.15.5:6789/0 conn(0x7f0f901543a0 legacy=0x7f0f9006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:21.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.349+0000 7f0f8ffff640 1 -- 172.21.15.5:0/3342040019 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f9015b070 con 0x7f0f900ccaa0 2024-03-24T10:50:21.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.352+0000 7f0f8ffff640 1 --2- 172.21.15.5:0/3342040019 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f900ccaa0 0x7f0f9006bf50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009d90 tx=0x7f0f8802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4563f4e6356906d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:21.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.357+0000 7f0f8e7fc640 1 -- 172.21.15.5:0/3342040019 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f8802cc50 con 0x7f0f900ccaa0 2024-03-24T10:50:21.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.357+0000 7f0f8e7fc640 1 -- 172.21.15.5:0/3342040019 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f8802cdb0 con 0x7f0f900ccaa0 2024-03-24T10:50:21.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.357+0000 7f0f8e7fc640 1 -- 172.21.15.5:0/3342040019 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f88034660 con 0x7f0f900ccaa0 2024-03-24T10:50:21.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.357+0000 7f0f9607c640 1 -- 172.21.15.5:0/3342040019 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f900ccaa0 msgr2=0x7f0f9006bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:21.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.357+0000 7f0f9607c640 1 --2- 172.21.15.5:0/3342040019 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f900ccaa0 0x7f0f9006bf50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009d90 tx=0x7f0f8802bea0 comp rx=0 tx=0).stop 2024-03-24T10:50:21.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.357+0000 7f0f9607c640 1 -- 172.21.15.5:0/3342040019 shutdown_connections 2024-03-24T10:50:21.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.357+0000 7f0f9607c640 1 --2- 172.21.15.5:0/3342040019 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f900ccaa0 0x7f0f9006bf50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:21.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.357+0000 7f0f9607c640 1 --2- 172.21.15.5:0/3342040019 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f9015ac00 0x7f0f900cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:21.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.357+0000 7f0f9607c640 1 -- 172.21.15.5:0/3342040019 >> 172.21.15.5:0/3342040019 conn(0x7f0f90072ef0 msgr2=0x7f0f90061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:21.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.357+0000 7f0f9607c640 1 -- 172.21.15.5:0/3342040019 shutdown_connections 2024-03-24T10:50:21.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.357+0000 7f0f9607c640 1 -- 172.21.15.5:0/3342040019 wait complete. 2024-03-24T10:50:21.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.359+0000 7f0f9607c640 1 Processor -- start 2024-03-24T10:50:21.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.359+0000 7f0f9607c640 1 -- start start 2024-03-24T10:50:21.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.359+0000 7f0f9607c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f900ccaa0 0x7f0f9014e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:21.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.359+0000 7f0f9607c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f901543a0 0x7f0f90149680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:21.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.359+0000 7f0f9607c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f9015ac00 0x7f0f90149bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:21.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.359+0000 7f0f9607c640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f0f90155a60 con 0x7f0f901543a0 2024-03-24T10:50:21.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.359+0000 7f0f9607c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0f90155470 con 0x7f0f9015ac00 2024-03-24T10:50:21.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.359+0000 7f0f9607c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0f90150bd0 con 0x7f0f900ccaa0 2024-03-24T10:50:21.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.360+0000 7f0f8effd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f901543a0 0x7f0f90149680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:21.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.360+0000 7f0f8ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f9015ac00 0x7f0f90149bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:21.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.360+0000 7f0f8f7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f900ccaa0 0x7f0f9014e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:21.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.360+0000 7f0f8ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f9015ac00 0x7f0f90149bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:39614/0 (socket says 172.21.15.5:39614) 2024-03-24T10:50:21.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.360+0000 7f0f8ffff640 1 -- 172.21.15.5:0/2792474759 learned_addr learned my addr 172.21.15.5:0/2792474759 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:21.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.366+0000 7f0f8ffff640 1 -- 172.21.15.5:0/2792474759 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f900ccaa0 msgr2=0x7f0f9014e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:21.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.366+0000 7f0f8ffff640 1 --2- 172.21.15.5:0/2792474759 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f900ccaa0 0x7f0f9014e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:21.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.366+0000 7f0f8ffff640 1 -- 172.21.15.5:0/2792474759 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f901543a0 msgr2=0x7f0f90149680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:21.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.366+0000 7f0f8ffff640 1 --2- 172.21.15.5:0/2792474759 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f901543a0 0x7f0f90149680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:21.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.366+0000 7f0f8ffff640 1 -- 172.21.15.5:0/2792474759 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f88009a70 con 0x7f0f9015ac00 2024-03-24T10:50:21.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.369+0000 7f0f8ffff640 1 --2- 172.21.15.5:0/2792474759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f9015ac00 0x7f0f90149bc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009eb0 tx=0x7f0f88004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:21.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.374+0000 7f0f77fff640 1 -- 172.21.15.5:0/2792474759 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f88040650 con 0x7f0f9015ac00 2024-03-24T10:50:21.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.374+0000 7f0f9607c640 1 -- 172.21.15.5:0/2792474759 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f900c77f0 con 0x7f0f9015ac00 2024-03-24T10:50:21.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.374+0000 7f0f9607c640 1 -- 172.21.15.5:0/2792474759 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f90150fc0 con 0x7f0f9015ac00 2024-03-24T10:50:21.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.374+0000 7f0f77fff640 1 -- 172.21.15.5:0/2792474759 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f88009450 con 0x7f0f9015ac00 2024-03-24T10:50:21.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.374+0000 7f0f77fff640 1 -- 172.21.15.5:0/2792474759 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f880409b0 con 0x7f0f9015ac00 2024-03-24T10:50:21.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.375+0000 7f0f9607c640 1 -- 172.21.15.5:0/2792474759 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f70005290 con 0x7f0f9015ac00 2024-03-24T10:50:21.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.408+0000 7f0f77fff640 1 -- 172.21.15.5:0/2792474759 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0f88040b10 con 0x7f0f9015ac00 2024-03-24T10:50:21.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.408+0000 7f0f77fff640 1 --2- 172.21.15.5:0/2792474759 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0f7c06b810 0x7f0f7c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:21.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.408+0000 7f0f8f7fe640 1 --2- 172.21.15.5:0/2792474759 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0f7c06b810 0x7f0f7c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:21.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.408+0000 7f0f8f7fe640 1 --2- 172.21.15.5:0/2792474759 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0f7c06b810 0x7f0f7c06dcd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0f80005880 tx=0x7f0f80009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:21.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.414+0000 7f0f77fff640 1 -- 172.21.15.5:0/2792474759 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f0f8807bc30 con 0x7f0f9015ac00 2024-03-24T10:50:21.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.683+0000 7f0f77fff640 1 -- 172.21.15.5:0/2792474759 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0f88077ed0 con 0x7f0f9015ac00 2024-03-24T10:50:21.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.909+0000 7f0f9607c640 1 -- 172.21.15.5:0/2792474759 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f700058f0 con 0x7f0f9015ac00 2024-03-24T10:50:21.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.931+0000 7f0f77fff640 1 -- 172.21.15.5:0/2792474759 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f0f700058f0 con 0x7f0f9015ac00 2024-03-24T10:50:21.931 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:21.932 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:21.932 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:21.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.933+0000 7f0f9607c640 1 -- 172.21.15.5:0/2792474759 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0f7c06b810 msgr2=0x7f0f7c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:21.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.933+0000 7f0f9607c640 1 --2- 172.21.15.5:0/2792474759 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0f7c06b810 0x7f0f7c06dcd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0f80005880 tx=0x7f0f80009040 comp rx=0 tx=0).stop 2024-03-24T10:50:21.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.933+0000 7f0f9607c640 1 -- 172.21.15.5:0/2792474759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f9015ac00 msgr2=0x7f0f90149bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:21.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.933+0000 7f0f9607c640 1 --2- 172.21.15.5:0/2792474759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f9015ac00 0x7f0f90149bc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0f88009eb0 tx=0x7f0f88004a40 comp rx=0 tx=0).stop 2024-03-24T10:50:21.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.933+0000 7f0f9607c640 1 -- 172.21.15.5:0/2792474759 shutdown_connections 2024-03-24T10:50:21.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.933+0000 7f0f9607c640 1 --2- 172.21.15.5:0/2792474759 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0f7c06b810 0x7f0f7c06dcd0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:21.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.933+0000 7f0f9607c640 1 --2- 172.21.15.5:0/2792474759 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f901543a0 0x7f0f90149680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:21.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.933+0000 7f0f9607c640 1 --2- 172.21.15.5:0/2792474759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f9015ac00 0x7f0f90149bc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:21.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.933+0000 7f0f9607c640 1 --2- 172.21.15.5:0/2792474759 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f900ccaa0 0x7f0f9014e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:21.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.933+0000 7f0f9607c640 1 -- 172.21.15.5:0/2792474759 >> 172.21.15.5:0/2792474759 conn(0x7f0f90072ef0 msgr2=0x7f0f9015a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:21.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.933+0000 7f0f9607c640 1 -- 172.21.15.5:0/2792474759 shutdown_connections 2024-03-24T10:50:21.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:21.934+0000 7f0f9607c640 1 -- 172.21.15.5:0/2792474759 wait complete. 2024-03-24T10:50:22.552 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:22.552 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:23.553 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:23.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.656+0000 7fd0601c7640 1 Processor -- start 2024-03-24T10:50:23.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.656+0000 7fd0601c7640 1 -- start start 2024-03-24T10:50:23.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.657+0000 7fd0601c7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd0580cae90 0x7fd0580cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:23.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.657+0000 7fd0601c7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd0580cd960 0x7fd0580cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:23.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.657+0000 7fd0601c7640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0580ccba0 con 0x7fd0580cb270 2024-03-24T10:50:23.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.657+0000 7fd0601c7640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fd0580cc9b0 con 0x7fd0580cd960 2024-03-24T10:50:23.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.657+0000 7fd0601c7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd0580cce40 con 0x7fd0580cae90 2024-03-24T10:50:23.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.658+0000 7fd05df63640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd0580cae90 0x7fd0580cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:23.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.658+0000 7fd05e764640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd0580cd960 0x7fd0580cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:23.658 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.658+0000 7fd05d762640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fd0580cb270 0x7fd0580cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:53044/0 (socket says 172.21.15.5:53044) 2024-03-24T10:50:23.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.658+0000 7fd05d762640 1 -- 172.21.15.5:0/1952752957 learned_addr learned my addr 172.21.15.5:0/1952752957 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:50:23.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.660+0000 7fd05df63640 1 -- 172.21.15.5:0/1952752957 >> v2:172.21.15.5:3300/0 conn(0x7fd0580cd960 msgr2=0x7fd0580cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:23.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.660+0000 7fd05df63640 1 --2- 172.21.15.5:0/1952752957 >> v2:172.21.15.5:3300/0 conn(0x7fd0580cd960 0x7fd0580cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:23.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.660+0000 7fd05df63640 1 -- 172.21.15.5:0/1952752957 >> v1:172.21.15.57:6789/0 conn(0x7fd0580cb270 legacy=0x7fd0580cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:23.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.660+0000 7fd05df63640 1 -- 172.21.15.5:0/1952752957 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd05815a590 con 0x7fd0580cae90 2024-03-24T10:50:23.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.663+0000 7fd05df63640 1 --2- 172.21.15.5:0/1952752957 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd0580cae90 0x7fd0580cf530 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd0540097a0 tx=0x7fd05402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53cac6dffc63794c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:23.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.667+0000 7fd05cf61640 1 -- 172.21.15.5:0/1952752957 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054030610 con 0x7fd0580cae90 2024-03-24T10:50:23.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.667+0000 7fd05cf61640 1 -- 172.21.15.5:0/1952752957 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd054030770 con 0x7fd0580cae90 2024-03-24T10:50:23.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.667+0000 7fd05cf61640 1 -- 172.21.15.5:0/1952752957 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054030b00 con 0x7fd0580cae90 2024-03-24T10:50:23.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.667+0000 7fd0601c7640 1 -- 172.21.15.5:0/1952752957 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd0580cae90 msgr2=0x7fd0580cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:23.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.668+0000 7fd0601c7640 1 --2- 172.21.15.5:0/1952752957 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd0580cae90 0x7fd0580cf530 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd0540097a0 tx=0x7fd05402b960 comp rx=0 tx=0).stop 2024-03-24T10:50:23.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.668+0000 7fd0601c7640 1 -- 172.21.15.5:0/1952752957 shutdown_connections 2024-03-24T10:50:23.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.668+0000 7fd0601c7640 1 --2- 172.21.15.5:0/1952752957 >> v2:172.21.15.5:3300/0 conn(0x7fd0580cd960 0x7fd0580cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:23.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.668+0000 7fd0601c7640 1 --2- 172.21.15.5:0/1952752957 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd0580cae90 0x7fd0580cf530 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:23.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.668+0000 7fd0601c7640 1 -- 172.21.15.5:0/1952752957 >> 172.21.15.5:0/1952752957 conn(0x7fd0580cfb80 msgr2=0x7fd0580cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:23.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.668+0000 7fd0601c7640 1 -- 172.21.15.5:0/1952752957 shutdown_connections 2024-03-24T10:50:23.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.668+0000 7fd0601c7640 1 -- 172.21.15.5:0/1952752957 wait complete. 2024-03-24T10:50:23.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.669+0000 7fd0601c7640 1 Processor -- start 2024-03-24T10:50:23.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.669+0000 7fd0601c7640 1 -- start start 2024-03-24T10:50:23.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.669+0000 7fd0601c7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd0580cae90 0x7fd0580c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:23.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.669+0000 7fd0601c7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0580cb270 0x7fd05814f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:23.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.669+0000 7fd0601c7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0580cd960 0x7fd0580c37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:23.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.669+0000 7fd0601c7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd0580c54e0 con 0x7fd0580cb270 2024-03-24T10:50:23.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.670+0000 7fd0601c7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd0581f0e10 con 0x7fd0580cd960 2024-03-24T10:50:23.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.670+0000 7fd0601c7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd0581f0200 con 0x7fd0580cae90 2024-03-24T10:50:23.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.670+0000 7fd05d762640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0580cb270 0x7fd05814f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:23.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.670+0000 7fd05df63640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd0580cae90 0x7fd0580c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:23.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.670+0000 7fd05e764640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0580cd960 0x7fd0580c37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:23.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.670+0000 7fd05df63640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd0580cae90 0x7fd0580c4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:41660/0 (socket says 172.21.15.5:41660) 2024-03-24T10:50:23.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.670+0000 7fd05df63640 1 -- 172.21.15.5:0/1567831043 learned_addr learned my addr 172.21.15.5:0/1567831043 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:23.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.674+0000 7fd05df63640 1 -- 172.21.15.5:0/1567831043 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0580cd960 msgr2=0x7fd0580c37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:23.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.674+0000 7fd05df63640 1 --2- 172.21.15.5:0/1567831043 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0580cd960 0x7fd0580c37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:23.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.674+0000 7fd05df63640 1 -- 172.21.15.5:0/1567831043 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0580cb270 msgr2=0x7fd05814f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:23.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.674+0000 7fd05df63640 1 --2- 172.21.15.5:0/1567831043 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0580cb270 0x7fd05814f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:23.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.674+0000 7fd05df63640 1 -- 172.21.15.5:0/1567831043 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd054009400 con 0x7fd0580cae90 2024-03-24T10:50:23.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.677+0000 7fd05df63640 1 --2- 172.21.15.5:0/1567831043 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd0580cae90 0x7fd0580c4fa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd05402bf70 tx=0x7fd054004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:23.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.684+0000 7fd04e7fc640 1 -- 172.21.15.5:0/1567831043 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05403e040 con 0x7fd0580cae90 2024-03-24T10:50:23.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.684+0000 7fd04e7fc640 1 -- 172.21.15.5:0/1567831043 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd054039a40 con 0x7fd0580cae90 2024-03-24T10:50:23.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.684+0000 7fd04e7fc640 1 -- 172.21.15.5:0/1567831043 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054039df0 con 0x7fd0580cae90 2024-03-24T10:50:23.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.684+0000 7fd0601c7640 1 -- 172.21.15.5:0/1567831043 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0581f04e0 con 0x7fd0580cae90 2024-03-24T10:50:23.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.684+0000 7fd0601c7640 1 -- 172.21.15.5:0/1567831043 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0581f0aa0 con 0x7fd0580cae90 2024-03-24T10:50:23.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.685+0000 7fd0601c7640 1 -- 172.21.15.5:0/1567831043 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0580cefd0 con 0x7fd0580cae90 2024-03-24T10:50:23.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.956+0000 7fd04e7fc640 1 -- 172.21.15.5:0/1567831043 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd054043390 con 0x7fd0580cae90 2024-03-24T10:50:23.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.957+0000 7fd04e7fc640 1 --2- 172.21.15.5:0/1567831043 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd03c06b840 0x7fd03c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:23.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.957+0000 7fd05d762640 1 --2- 172.21.15.5:0/1567831043 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd03c06b840 0x7fd03c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:23.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.957+0000 7fd05d762640 1 --2- 172.21.15.5:0/1567831043 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd03c06b840 0x7fd03c06dd00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd058154570 tx=0x7fd048007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:23.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.970+0000 7fd04e7fc640 1 -- 172.21.15.5:0/1567831043 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fd05407b100 con 0x7fd0580cae90 2024-03-24T10:50:23.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:23.991+0000 7fd04e7fc640 1 -- 172.21.15.5:0/1567831043 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd0540773a0 con 0x7fd0580cae90 2024-03-24T10:50:24.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:24.214+0000 7fd0601c7640 1 -- 172.21.15.5:0/1567831043 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0581547e0 con 0x7fd0580cae90 2024-03-24T10:50:24.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:24.230+0000 7fd04e7fc640 1 -- 172.21.15.5:0/1567831043 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fd054034090 con 0x7fd0580cae90 2024-03-24T10:50:24.231 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:24.231 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:24.231 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:24.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:24.232+0000 7fd0601c7640 1 -- 172.21.15.5:0/1567831043 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd03c06b840 msgr2=0x7fd03c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:24.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:24.232+0000 7fd0601c7640 1 --2- 172.21.15.5:0/1567831043 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd03c06b840 0x7fd03c06dd00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd058154570 tx=0x7fd048007e80 comp rx=0 tx=0).stop 2024-03-24T10:50:24.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:24.232+0000 7fd0601c7640 1 -- 172.21.15.5:0/1567831043 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd0580cae90 msgr2=0x7fd0580c4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:24.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:24.232+0000 7fd0601c7640 1 --2- 172.21.15.5:0/1567831043 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd0580cae90 0x7fd0580c4fa0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd05402bf70 tx=0x7fd054004730 comp rx=0 tx=0).stop 2024-03-24T10:50:24.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:24.232+0000 7fd0601c7640 1 -- 172.21.15.5:0/1567831043 shutdown_connections 2024-03-24T10:50:24.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:24.232+0000 7fd0601c7640 1 --2- 172.21.15.5:0/1567831043 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd03c06b840 0x7fd03c06dd00 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:24.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:24.232+0000 7fd0601c7640 1 --2- 172.21.15.5:0/1567831043 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd0580cb270 0x7fd05814f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:24.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:24.232+0000 7fd0601c7640 1 --2- 172.21.15.5:0/1567831043 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd0580cd960 0x7fd0580c37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:24.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:24.232+0000 7fd0601c7640 1 --2- 172.21.15.5:0/1567831043 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd0580cae90 0x7fd0580c4fa0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:24.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:24.232+0000 7fd0601c7640 1 -- 172.21.15.5:0/1567831043 >> 172.21.15.5:0/1567831043 conn(0x7fd0580cfb80 msgr2=0x7fd058159e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:24.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:24.233+0000 7fd0601c7640 1 -- 172.21.15.5:0/1567831043 shutdown_connections 2024-03-24T10:50:24.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:24.233+0000 7fd0601c7640 1 -- 172.21.15.5:0/1567831043 wait complete. 2024-03-24T10:50:24.851 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:24.851 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:25.852 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:25.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.954+0000 7f30fcec0640 1 Processor -- start 2024-03-24T10:50:25.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.954+0000 7f30fcec0640 1 -- start start 2024-03-24T10:50:25.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.956+0000 7f30fcec0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f30f80cae90 0x7f30f80cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:25.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.956+0000 7f30fcec0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f30f80cb270 0x7f30f80cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:25.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.956+0000 7f30fcec0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f30f80cd960 0x7f30f80cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:25.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.956+0000 7f30fcec0640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f30f80ccba0 con 0x7f30f80cae90 2024-03-24T10:50:25.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.956+0000 7f30fcec0640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f30f80cce40 con 0x7f30f80cb270 2024-03-24T10:50:25.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.956+0000 7f30fcec0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f30f815a590 con 0x7f30f80cd960 2024-03-24T10:50:25.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.957+0000 7f30f659c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f30f80cae90 0x7f30f80cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:25.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.957+0000 7f30f6d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f30f80cd960 0x7f30f80cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:25.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.957+0000 7f30f5d9b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f30f80cb270 0x7f30f80cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:25.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.957+0000 7f30f659c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f30f80cae90 0x7f30f80cf530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51166/0 (socket says 172.21.15.5:51166) 2024-03-24T10:50:25.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.957+0000 7f30f659c640 1 -- 172.21.15.5:0/2495188687 learned_addr learned my addr 172.21.15.5:0/2495188687 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:25.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.960+0000 7f30f659c640 1 -- 172.21.15.5:0/2495188687 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f30f80cd960 msgr2=0x7f30f80cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:25.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.960+0000 7f30f659c640 1 --2- 172.21.15.5:0/2495188687 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f30f80cd960 0x7f30f80cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:25.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.960+0000 7f30f659c640 1 -- 172.21.15.5:0/2495188687 >> v2:172.21.15.57:3300/0 conn(0x7f30f80cb270 msgr2=0x7f30f80cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:25.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.960+0000 7f30f659c640 1 --2- 172.21.15.5:0/2495188687 >> v2:172.21.15.57:3300/0 conn(0x7f30f80cb270 0x7f30f80cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:25.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.960+0000 7f30f659c640 1 -- 172.21.15.5:0/2495188687 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30f815a870 con 0x7f30f80cae90 2024-03-24T10:50:25.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.960+0000 7f30f5d9b640 1 --2- 172.21.15.5:0/2495188687 >> v2:172.21.15.57:3300/0 conn(0x7f30f80cb270 0x7f30f80cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:50:25.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.963+0000 7f30f659c640 1 --2- 172.21.15.5:0/2495188687 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f30f80cae90 0x7f30f80cf530 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f30f80cd680 tx=0x7f30e802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fc9debbfd54f89a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:25.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.967+0000 7f30f559a640 1 -- 172.21.15.5:0/2495188687 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e8030860 con 0x7f30f80cae90 2024-03-24T10:50:25.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.968+0000 7f30f559a640 1 -- 172.21.15.5:0/2495188687 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30e80309c0 con 0x7f30f80cae90 2024-03-24T10:50:25.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.968+0000 7f30f559a640 1 -- 172.21.15.5:0/2495188687 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e8030d50 con 0x7f30f80cae90 2024-03-24T10:50:25.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.969+0000 7f30fcec0640 1 -- 172.21.15.5:0/2495188687 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f30f80cae90 msgr2=0x7f30f80cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:25.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.969+0000 7f30fcec0640 1 --2- 172.21.15.5:0/2495188687 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f30f80cae90 0x7f30f80cf530 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f30f80cd680 tx=0x7f30e802b9a0 comp rx=0 tx=0).stop 2024-03-24T10:50:25.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.969+0000 7f30fcec0640 1 -- 172.21.15.5:0/2495188687 shutdown_connections 2024-03-24T10:50:25.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.969+0000 7f30fcec0640 1 --2- 172.21.15.5:0/2495188687 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f30f80cd960 0x7f30f80cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:25.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.969+0000 7f30fcec0640 1 --2- 172.21.15.5:0/2495188687 >> v2:172.21.15.57:3300/0 conn(0x7f30f80cb270 0x7f30f80cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:25.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.969+0000 7f30fcec0640 1 --2- 172.21.15.5:0/2495188687 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f30f80cae90 0x7f30f80cf530 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:25.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.969+0000 7f30fcec0640 1 -- 172.21.15.5:0/2495188687 >> 172.21.15.5:0/2495188687 conn(0x7f30f80cfb80 msgr2=0x7f30f80cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:25.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.969+0000 7f30fcec0640 1 -- 172.21.15.5:0/2495188687 shutdown_connections 2024-03-24T10:50:25.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.969+0000 7f30fcec0640 1 -- 172.21.15.5:0/2495188687 wait complete. 2024-03-24T10:50:25.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.970+0000 7f30fcec0640 1 Processor -- start 2024-03-24T10:50:25.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.970+0000 7f30fcec0640 1 -- start start 2024-03-24T10:50:25.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.971+0000 7f30fcec0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f30f80cae90 0x7f30f815d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:25.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.971+0000 7f30fcec0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30f80cb270 0x7f30f81f0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:25.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.971+0000 7f30fcec0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f30f80cd960 0x7f30f8155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:25.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.971+0000 7f30fcec0640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f30f81eff90 con 0x7f30f80cae90 2024-03-24T10:50:25.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.971+0000 7f30fcec0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f30f815d8f0 con 0x7f30f80cb270 2024-03-24T10:50:25.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.971+0000 7f30fcec0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f30f815dbd0 con 0x7f30f80cd960 2024-03-24T10:50:25.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.972+0000 7f30f659c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f30f80cae90 0x7f30f815d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:25.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.972+0000 7f30f5d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30f80cb270 0x7f30f81f0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:25.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.972+0000 7f30f6d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f30f80cd960 0x7f30f8155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:25.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.972+0000 7f30f659c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f30f80cae90 0x7f30f815d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51174/0 (socket says 172.21.15.5:51174) 2024-03-24T10:50:25.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.972+0000 7f30f659c640 1 -- 172.21.15.5:0/3197172937 learned_addr learned my addr 172.21.15.5:0/3197172937 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:25.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.977+0000 7f30f659c640 1 -- 172.21.15.5:0/3197172937 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f30f80cd960 msgr2=0x7f30f8155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:25.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.977+0000 7f30f659c640 1 --2- 172.21.15.5:0/3197172937 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f30f80cd960 0x7f30f8155d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:25.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.977+0000 7f30f659c640 1 -- 172.21.15.5:0/3197172937 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30f80cb270 msgr2=0x7f30f81f0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:25.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.977+0000 7f30f659c640 1 --2- 172.21.15.5:0/3197172937 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30f80cb270 0x7f30f81f0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:25.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.977+0000 7f30f659c640 1 -- 172.21.15.5:0/3197172937 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30ec009400 con 0x7f30f80cae90 2024-03-24T10:50:25.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.980+0000 7f30f659c640 1 --2- 172.21.15.5:0/3197172937 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f30f80cae90 0x7f30f815d2b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f30e802b9e0 tx=0x7f30e8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:25.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.986+0000 7f30e6ffd640 1 -- 172.21.15.5:0/3197172937 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e8039c90 con 0x7f30f80cae90 2024-03-24T10:50:25.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.986+0000 7f30e6ffd640 1 -- 172.21.15.5:0/3197172937 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30e8039e10 con 0x7f30f80cae90 2024-03-24T10:50:25.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.986+0000 7f30e6ffd640 1 -- 172.21.15.5:0/3197172937 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30e8043640 con 0x7f30f80cae90 2024-03-24T10:50:25.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.986+0000 7f30fcec0640 1 -- 172.21.15.5:0/3197172937 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30e80094d0 con 0x7f30f80cae90 2024-03-24T10:50:25.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.986+0000 7f30fcec0640 1 -- 172.21.15.5:0/3197172937 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30f81f04e0 con 0x7f30f80cae90 2024-03-24T10:50:25.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:25.987+0000 7f30fcec0640 1 -- 172.21.15.5:0/3197172937 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30f81f0700 con 0x7f30f80cae90 2024-03-24T10:50:26.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:26.022+0000 7f30e6ffd640 1 -- 172.21.15.5:0/3197172937 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f30e80437a0 con 0x7f30f80cae90 2024-03-24T10:50:26.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:26.022+0000 7f30e6ffd640 1 --2- 172.21.15.5:0/3197172937 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f30d806b860 0x7f30d806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:26.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:26.023+0000 7f30f5d9b640 1 --2- 172.21.15.5:0/3197172937 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f30d806b860 0x7f30d806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:26.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:26.023+0000 7f30f5d9b640 1 --2- 172.21.15.5:0/3197172937 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f30d806b860 0x7f30d806dd20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f30f806d010 tx=0x7f30ec009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:26.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:26.027+0000 7f30e6ffd640 1 -- 172.21.15.5:0/3197172937 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f30e803e070 con 0x7f30f80cae90 2024-03-24T10:50:26.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:26.313+0000 7f30e6ffd640 1 -- 172.21.15.5:0/3197172937 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f30e80785e0 con 0x7f30f80cae90 2024-03-24T10:50:26.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:26.541+0000 7f30fcec0640 1 -- 172.21.15.5:0/3197172937 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f30f80681d0 con 0x7f30f80cae90 2024-03-24T10:50:26.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:26.558+0000 7f30e6ffd640 1 -- 172.21.15.5:0/3197172937 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f30e8034070 con 0x7f30f80cae90 2024-03-24T10:50:26.558 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:26.558 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:26.559 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:26.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:26.560+0000 7f30fcec0640 1 -- 172.21.15.5:0/3197172937 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f30d806b860 msgr2=0x7f30d806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:26.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:26.560+0000 7f30fcec0640 1 --2- 172.21.15.5:0/3197172937 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f30d806b860 0x7f30d806dd20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f30f806d010 tx=0x7f30ec009190 comp rx=0 tx=0).stop 2024-03-24T10:50:26.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:26.560+0000 7f30fcec0640 1 -- 172.21.15.5:0/3197172937 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f30f80cae90 msgr2=0x7f30f815d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:26.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:26.560+0000 7f30fcec0640 1 --2- 172.21.15.5:0/3197172937 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f30f80cae90 0x7f30f815d2b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f30e802b9e0 tx=0x7f30e8004730 comp rx=0 tx=0).stop 2024-03-24T10:50:26.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:26.560+0000 7f30fcec0640 1 -- 172.21.15.5:0/3197172937 shutdown_connections 2024-03-24T10:50:26.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:26.560+0000 7f30fcec0640 1 --2- 172.21.15.5:0/3197172937 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f30d806b860 0x7f30d806dd20 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:26.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:26.560+0000 7f30fcec0640 1 --2- 172.21.15.5:0/3197172937 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f30f80cd960 0x7f30f8155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:26.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:26.560+0000 7f30fcec0640 1 --2- 172.21.15.5:0/3197172937 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f30f80cb270 0x7f30f81f0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:26.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:26.560+0000 7f30fcec0640 1 --2- 172.21.15.5:0/3197172937 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f30f80cae90 0x7f30f815d2b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:26.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:26.560+0000 7f30fcec0640 1 -- 172.21.15.5:0/3197172937 >> 172.21.15.5:0/3197172937 conn(0x7f30f80cfb80 msgr2=0x7f30f8158040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:26.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:26.560+0000 7f30fcec0640 1 -- 172.21.15.5:0/3197172937 shutdown_connections 2024-03-24T10:50:26.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:26.560+0000 7f30fcec0640 1 -- 172.21.15.5:0/3197172937 wait complete. 2024-03-24T10:50:27.176 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:27.176 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:28.177 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:28.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.280+0000 7f07b4cd4640 1 Processor -- start 2024-03-24T10:50:28.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.280+0000 7f07b4cd4640 1 -- start start 2024-03-24T10:50:28.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.281+0000 7f07b4cd4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f07b015ac00 0x7f07b00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.281+0000 7f07b4cd4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07b00cb2f0 con 0x7f07b00ccaa0 2024-03-24T10:50:28.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.281+0000 7f07b4cd4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07b015b070 con 0x7f07b01543a0 2024-03-24T10:50:28.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.281+0000 7f07b4cd4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f07b00c6750 con 0x7f07b015ac00 2024-03-24T10:50:28.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.281+0000 7f07ae59c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f07b01543a0 0x7f07b006d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:53060/0 (socket says 172.21.15.5:53060) 2024-03-24T10:50:28.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.281+0000 7f07ae59c640 1 -- 172.21.15.5:0/1288853205 learned_addr learned my addr 172.21.15.5:0/1288853205 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:50:28.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.282+0000 7f07add9b640 1 --2- 172.21.15.5:0/1288853205 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f07b015ac00 0x7f07b00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.285+0000 7f07ad59a640 1 -- 172.21.15.5:0/1288853205 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2706450114 0 0) 0x7f07b015b070 con 0x7f07b01543a0 2024-03-24T10:50:28.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.285+0000 7f07ad59a640 1 -- 172.21.15.5:0/1288853205 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0798003520 con 0x7f07b01543a0 2024-03-24T10:50:28.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.285+0000 7f07add9b640 1 -- 172.21.15.5:0/1288853205 >> v1:172.21.15.57:6789/0 conn(0x7f07b01543a0 legacy=0x7f07b006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.285+0000 7f07add9b640 1 -- 172.21.15.5:0/1288853205 >> v1:172.21.15.5:6789/0 conn(0x7f07b00ccaa0 legacy=0x7f07b006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.285+0000 7f07add9b640 1 -- 172.21.15.5:0/1288853205 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b00ca7e0 con 0x7f07b015ac00 2024-03-24T10:50:28.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.289+0000 7f07add9b640 1 --2- 172.21.15.5:0/1288853205 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f07b015ac00 0x7f07b00cc560 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f07a40097d0 tx=0x7f07a402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98c55af4a69a1212 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.294+0000 7f07ad59a640 1 -- 172.21.15.5:0/1288853205 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a4031890 con 0x7f07b015ac00 2024-03-24T10:50:28.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.294+0000 7f07ad59a640 1 -- 172.21.15.5:0/1288853205 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07a40319f0 con 0x7f07b015ac00 2024-03-24T10:50:28.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.294+0000 7f07ad59a640 1 -- 172.21.15.5:0/1288853205 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a4031d80 con 0x7f07b015ac00 2024-03-24T10:50:28.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.295+0000 7f07b4cd4640 1 -- 172.21.15.5:0/1288853205 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f07b015ac00 msgr2=0x7f07b00cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.295+0000 7f07b4cd4640 1 --2- 172.21.15.5:0/1288853205 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f07b015ac00 0x7f07b00cc560 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f07a40097d0 tx=0x7f07a402b930 comp rx=0 tx=0).stop 2024-03-24T10:50:28.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.295+0000 7f07b4cd4640 1 -- 172.21.15.5:0/1288853205 shutdown_connections 2024-03-24T10:50:28.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.295+0000 7f07b4cd4640 1 --2- 172.21.15.5:0/1288853205 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f07b015ac00 0x7f07b00cc560 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.295+0000 7f07b4cd4640 1 -- 172.21.15.5:0/1288853205 >> 172.21.15.5:0/1288853205 conn(0x7f07b0072ef0 msgr2=0x7f07b0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.295+0000 7f07b4cd4640 1 -- 172.21.15.5:0/1288853205 shutdown_connections 2024-03-24T10:50:28.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.295+0000 7f07b4cd4640 1 -- 172.21.15.5:0/1288853205 wait complete. 2024-03-24T10:50:28.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.296+0000 7f07b4cd4640 1 Processor -- start 2024-03-24T10:50:28.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.296+0000 7f07b4cd4640 1 -- start start 2024-03-24T10:50:28.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.297+0000 7f07b4cd4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f07b00ccaa0 0x7f07b00c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.297+0000 7f07b4cd4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b01543a0 0x7f07b0157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.297+0000 7f07b4cd4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07b015ac00 0x7f07b01499d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.297+0000 7f07b4cd4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f07b00c9430 con 0x7f07b015ac00 2024-03-24T10:50:28.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.297+0000 7f07b4cd4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f07b0061dc0 con 0x7f07b01543a0 2024-03-24T10:50:28.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.297+0000 7f07b4cd4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f07b015b830 con 0x7f07b00ccaa0 2024-03-24T10:50:28.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.298+0000 7f07aed9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07b015ac00 0x7f07b01499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.298+0000 7f07ae59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f07b00ccaa0 0x7f07b00c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.298+0000 7f07add9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b01543a0 0x7f07b0157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.298+0000 7f07ae59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f07b00ccaa0 0x7f07b00c9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:41706/0 (socket says 172.21.15.5:41706) 2024-03-24T10:50:28.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.298+0000 7f07ae59c640 1 -- 172.21.15.5:0/3682753147 learned_addr learned my addr 172.21.15.5:0/3682753147 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:28.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.302+0000 7f07aed9d640 1 -- 172.21.15.5:0/3682753147 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f07b00ccaa0 msgr2=0x7f07b00c9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.302+0000 7f07aed9d640 1 --2- 172.21.15.5:0/3682753147 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f07b00ccaa0 0x7f07b00c9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.302+0000 7f07aed9d640 1 -- 172.21.15.5:0/3682753147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b01543a0 msgr2=0x7f07b0157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.302+0000 7f07aed9d640 1 --2- 172.21.15.5:0/3682753147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b01543a0 0x7f07b0157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.302+0000 7f07aed9d640 1 -- 172.21.15.5:0/3682753147 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a4009400 con 0x7f07b015ac00 2024-03-24T10:50:28.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.305+0000 7f07aed9d640 1 --2- 172.21.15.5:0/3682753147 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07b015ac00 0x7f07b01499d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f07a800c690 tx=0x7f07a800cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.309+0000 7f079effd640 1 -- 172.21.15.5:0/3682753147 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a8003b70 con 0x7f07b015ac00 2024-03-24T10:50:28.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.309+0000 7f079effd640 1 -- 172.21.15.5:0/3682753147 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07a8003cd0 con 0x7f07b015ac00 2024-03-24T10:50:28.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.309+0000 7f07b4cd4640 1 -- 172.21.15.5:0/3682753147 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07b015b2b0 con 0x7f07b015ac00 2024-03-24T10:50:28.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.309+0000 7f079effd640 1 -- 172.21.15.5:0/3682753147 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a8005310 con 0x7f07b015ac00 2024-03-24T10:50:28.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.309+0000 7f07b4cd4640 1 -- 172.21.15.5:0/3682753147 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07b00c8d70 con 0x7f07b015ac00 2024-03-24T10:50:28.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.310+0000 7f07b4cd4640 1 -- 172.21.15.5:0/3682753147 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07b00c9570 con 0x7f07b015ac00 2024-03-24T10:50:28.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.578+0000 7f079effd640 1 -- 172.21.15.5:0/3682753147 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f07a8005470 con 0x7f07b015ac00 2024-03-24T10:50:28.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.579+0000 7f079effd640 1 --2- 172.21.15.5:0/3682753147 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f079006b860 0x7f079006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:28.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.579+0000 7f07ae59c640 1 --2- 172.21.15.5:0/3682753147 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f079006b860 0x7f079006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:28.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.579+0000 7f07ae59c640 1 --2- 172.21.15.5:0/3682753147 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f079006b860 0x7f079006dd20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f07a0005880 tx=0x7f07a0009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:28.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.593+0000 7f079effd640 1 -- 172.21.15.5:0/3682753147 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f07a8058fb0 con 0x7f07b015ac00 2024-03-24T10:50:28.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.594+0000 7f079effd640 1 -- 172.21.15.5:0/3682753147 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f07a8055250 con 0x7f07b015ac00 2024-03-24T10:50:28.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.821+0000 7f07b4cd4640 1 -- 172.21.15.5:0/3682753147 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07b00c9570 con 0x7f07b015ac00 2024-03-24T10:50:28.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.835+0000 7f079effd640 1 -- 172.21.15.5:0/3682753147 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f07b00c9570 con 0x7f07b015ac00 2024-03-24T10:50:28.879 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:28.879 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:28.881 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:28.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.838+0000 7f07b4cd4640 1 -- 172.21.15.5:0/3682753147 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f079006b860 msgr2=0x7f079006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.838+0000 7f07b4cd4640 1 --2- 172.21.15.5:0/3682753147 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f079006b860 0x7f079006dd20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f07a0005880 tx=0x7f07a0009040 comp rx=0 tx=0).stop 2024-03-24T10:50:28.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.838+0000 7f07b4cd4640 1 -- 172.21.15.5:0/3682753147 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07b015ac00 msgr2=0x7f07b01499d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:28.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.838+0000 7f07b4cd4640 1 --2- 172.21.15.5:0/3682753147 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07b015ac00 0x7f07b01499d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f07a800c690 tx=0x7f07a800cb50 comp rx=0 tx=0).stop 2024-03-24T10:50:28.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.838+0000 7f07b4cd4640 1 -- 172.21.15.5:0/3682753147 shutdown_connections 2024-03-24T10:50:28.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.838+0000 7f07b4cd4640 1 --2- 172.21.15.5:0/3682753147 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f079006b860 0x7f079006dd20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.838+0000 7f07b4cd4640 1 --2- 172.21.15.5:0/3682753147 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07b015ac00 0x7f07b01499d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.838+0000 7f07b4cd4640 1 --2- 172.21.15.5:0/3682753147 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07b01543a0 0x7f07b0157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.838+0000 7f07b4cd4640 1 --2- 172.21.15.5:0/3682753147 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f07b00ccaa0 0x7f07b00c9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:28.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.838+0000 7f07b4cd4640 1 -- 172.21.15.5:0/3682753147 >> 172.21.15.5:0/3682753147 conn(0x7f07b0072ef0 msgr2=0x7f07b014f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:28.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.838+0000 7f07b4cd4640 1 -- 172.21.15.5:0/3682753147 shutdown_connections 2024-03-24T10:50:28.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:28.838+0000 7f07b4cd4640 1 -- 172.21.15.5:0/3682753147 wait complete. 2024-03-24T10:50:29.487 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:29.487 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:30.488 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:30.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.592+0000 7f00a8d25640 1 Processor -- start 2024-03-24T10:50:30.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.592+0000 7f00a8d25640 1 -- start start 2024-03-24T10:50:30.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.593+0000 7f00a8d25640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f00a41543a0 0x7f00a415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:30.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.593+0000 7f00a8d25640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f00a406bf50 0x7f00a406c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:30.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.593+0000 7f00a8d25640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00a40cb2f0 con 0x7f00a406d4f0 2024-03-24T10:50:30.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.594+0000 7f00a8d25640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f00a40c6750 con 0x7f00a406bf50 2024-03-24T10:50:30.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.594+0000 7f00a8d25640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f00a40cc380 con 0x7f00a41543a0 2024-03-24T10:50:30.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.594+0000 7f00a2d9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f00a406bf50 0x7f00a406c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:30.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.595+0000 7f00a2d9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f00a406bf50 0x7f00a406c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51190/0 (socket says 172.21.15.5:51190) 2024-03-24T10:50:30.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.595+0000 7f00a2d9d640 1 -- 172.21.15.5:0/1128891770 learned_addr learned my addr 172.21.15.5:0/1128891770 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:30.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.595+0000 7f00a259c640 1 --2- 172.21.15.5:0/1128891770 >> v2:172.21.15.57:3300/0 conn(0x7f00a41543a0 0x7f00a415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:30.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.597+0000 7f00a2d9d640 1 -- 172.21.15.5:0/1128891770 >> v2:172.21.15.57:3300/0 conn(0x7f00a41543a0 msgr2=0x7f00a415ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:30.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.597+0000 7f00a2d9d640 1 --2- 172.21.15.5:0/1128891770 >> v2:172.21.15.57:3300/0 conn(0x7f00a41543a0 0x7f00a415ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.597+0000 7f00a2d9d640 1 -- 172.21.15.5:0/1128891770 >> v1:172.21.15.57:6789/0 conn(0x7f00a406d4f0 legacy=0x7f00a40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:30.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.597+0000 7f00a2d9d640 1 -- 172.21.15.5:0/1128891770 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a40ca7e0 con 0x7f00a406bf50 2024-03-24T10:50:30.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.601+0000 7f00a2d9d640 1 --2- 172.21.15.5:0/1128891770 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f00a406bf50 0x7f00a406c330 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f00a40cd600 tx=0x7f009c02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c21ff239f632c8bb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:30.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.607+0000 7f00a159a640 1 -- 172.21.15.5:0/1128891770 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c02cbb0 con 0x7f00a406bf50 2024-03-24T10:50:30.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.607+0000 7f00a159a640 1 -- 172.21.15.5:0/1128891770 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f009c02cd10 con 0x7f00a406bf50 2024-03-24T10:50:30.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.607+0000 7f00a159a640 1 -- 172.21.15.5:0/1128891770 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c034610 con 0x7f00a406bf50 2024-03-24T10:50:30.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.608+0000 7f00a8d25640 1 -- 172.21.15.5:0/1128891770 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f00a406bf50 msgr2=0x7f00a406c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:30.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.608+0000 7f00a8d25640 1 --2- 172.21.15.5:0/1128891770 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f00a406bf50 0x7f00a406c330 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f00a40cd600 tx=0x7f009c02c010 comp rx=0 tx=0).stop 2024-03-24T10:50:30.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.608+0000 7f00a8d25640 1 -- 172.21.15.5:0/1128891770 shutdown_connections 2024-03-24T10:50:30.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.608+0000 7f00a8d25640 1 --2- 172.21.15.5:0/1128891770 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f00a406bf50 0x7f00a406c330 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.608+0000 7f00a8d25640 1 --2- 172.21.15.5:0/1128891770 >> v2:172.21.15.57:3300/0 conn(0x7f00a41543a0 0x7f00a415ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.608+0000 7f00a8d25640 1 -- 172.21.15.5:0/1128891770 >> 172.21.15.5:0/1128891770 conn(0x7f00a4072ef0 msgr2=0x7f00a4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:30.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.608+0000 7f00a8d25640 1 -- 172.21.15.5:0/1128891770 shutdown_connections 2024-03-24T10:50:30.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.608+0000 7f00a8d25640 1 -- 172.21.15.5:0/1128891770 wait complete. 2024-03-24T10:50:30.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.610+0000 7f00a8d25640 1 Processor -- start 2024-03-24T10:50:30.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.610+0000 7f00a8d25640 1 -- start start 2024-03-24T10:50:30.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.611+0000 7f00a8d25640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f00a406bf50 0x7f00a40b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:30.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.611+0000 7f00a8d25640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f00a406d4f0 0x7f00a40cf060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:30.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.611+0000 7f00a8d25640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00a41543a0 0x7f00a4159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:30.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.611+0000 7f00a8d25640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f00a4150530 con 0x7f00a406d4f0 2024-03-24T10:50:30.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.611+0000 7f00a8d25640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f00a4155290 con 0x7f00a41543a0 2024-03-24T10:50:30.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.611+0000 7f00a8d25640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f00a40b39e0 con 0x7f00a406bf50 2024-03-24T10:50:30.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.611+0000 7f00a2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00a41543a0 0x7f00a4159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:30.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.611+0000 7f00a259c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f00a406bf50 0x7f00a40b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:30.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.611+0000 7f00a2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00a41543a0 0x7f00a4159220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:34566/0 (socket says 172.21.15.5:34566) 2024-03-24T10:50:30.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.611+0000 7f00a2d9d640 1 -- 172.21.15.5:0/2325640410 learned_addr learned my addr 172.21.15.5:0/2325640410 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:30.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.612+0000 7f00a1d9b640 1 --2- 172.21.15.5:0/2325640410 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f00a406d4f0 0x7f00a40cf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:30.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.615+0000 7f00a2d9d640 1 -- 172.21.15.5:0/2325640410 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f00a406bf50 msgr2=0x7f00a40b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:30.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.615+0000 7f00a2d9d640 1 --2- 172.21.15.5:0/2325640410 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f00a406bf50 0x7f00a40b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.615+0000 7f00a2d9d640 1 -- 172.21.15.5:0/2325640410 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f00a406d4f0 msgr2=0x7f00a40cf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:30.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.615+0000 7f00a2d9d640 1 --2- 172.21.15.5:0/2325640410 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f00a406d4f0 0x7f00a40cf060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:30.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.615+0000 7f00a2d9d640 1 -- 172.21.15.5:0/2325640410 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009c009b30 con 0x7f00a41543a0 2024-03-24T10:50:30.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.618+0000 7f00a2d9d640 1 --2- 172.21.15.5:0/2325640410 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00a41543a0 0x7f00a4159220 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f009c036040 tx=0x7f009c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:30.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.622+0000 7f0092ffd640 1 -- 172.21.15.5:0/2325640410 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c009510 con 0x7f00a41543a0 2024-03-24T10:50:30.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.622+0000 7f0092ffd640 1 -- 172.21.15.5:0/2325640410 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f009c03e450 con 0x7f00a41543a0 2024-03-24T10:50:30.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.622+0000 7f00a8d25640 1 -- 172.21.15.5:0/2325640410 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00a40cf670 con 0x7f00a41543a0 2024-03-24T10:50:30.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.622+0000 7f0092ffd640 1 -- 172.21.15.5:0/2325640410 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c03e800 con 0x7f00a41543a0 2024-03-24T10:50:30.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.622+0000 7f00a8d25640 1 -- 172.21.15.5:0/2325640410 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00a40b3f10 con 0x7f00a41543a0 2024-03-24T10:50:30.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.623+0000 7f00a8d25640 1 -- 172.21.15.5:0/2325640410 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00a40681d0 con 0x7f00a41543a0 2024-03-24T10:50:30.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.642+0000 7f0092ffd640 1 -- 172.21.15.5:0/2325640410 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f009c03e960 con 0x7f00a41543a0 2024-03-24T10:50:30.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.642+0000 7f0092ffd640 1 --2- 172.21.15.5:0/2325640410 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f008406b810 0x7f008406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:30.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.642+0000 7f00a259c640 1 --2- 172.21.15.5:0/2325640410 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f008406b810 0x7f008406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:30.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.643+0000 7f00a259c640 1 --2- 172.21.15.5:0/2325640410 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f008406b810 0x7f008406dcd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0094009aa0 tx=0x7f0094009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:30.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.894+0000 7f0092ffd640 1 -- 172.21.15.5:0/2325640410 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f009c07bc60 con 0x7f00a41543a0 2024-03-24T10:50:30.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:30.907+0000 7f0092ffd640 1 -- 172.21.15.5:0/2325640410 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f009c077f00 con 0x7f00a41543a0 2024-03-24T10:50:31.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:31.136+0000 7f00a8d25640 1 -- 172.21.15.5:0/2325640410 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00a4151d60 con 0x7f00a41543a0 2024-03-24T10:50:31.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:31.150+0000 7f0092ffd640 1 -- 172.21.15.5:0/2325640410 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f009c07b5c0 con 0x7f00a41543a0 2024-03-24T10:50:31.151 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:31.151 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:31.151 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:31.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:31.153+0000 7f00a8d25640 1 -- 172.21.15.5:0/2325640410 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f008406b810 msgr2=0x7f008406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:31.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:31.153+0000 7f00a8d25640 1 --2- 172.21.15.5:0/2325640410 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f008406b810 0x7f008406dcd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0094009aa0 tx=0x7f0094009190 comp rx=0 tx=0).stop 2024-03-24T10:50:31.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:31.153+0000 7f00a8d25640 1 -- 172.21.15.5:0/2325640410 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00a41543a0 msgr2=0x7f00a4159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:31.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:31.153+0000 7f00a8d25640 1 --2- 172.21.15.5:0/2325640410 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00a41543a0 0x7f00a4159220 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f009c036040 tx=0x7f009c004a40 comp rx=0 tx=0).stop 2024-03-24T10:50:31.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:31.153+0000 7f00a8d25640 1 -- 172.21.15.5:0/2325640410 shutdown_connections 2024-03-24T10:50:31.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:31.153+0000 7f00a8d25640 1 --2- 172.21.15.5:0/2325640410 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f008406b810 0x7f008406dcd0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:31.153+0000 7f00a8d25640 1 --2- 172.21.15.5:0/2325640410 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f00a406d4f0 0x7f00a40cf060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:31.153+0000 7f00a8d25640 1 --2- 172.21.15.5:0/2325640410 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f00a41543a0 0x7f00a4159220 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:31.153+0000 7f00a8d25640 1 --2- 172.21.15.5:0/2325640410 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f00a406bf50 0x7f00a40b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:31.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:31.153+0000 7f00a8d25640 1 -- 172.21.15.5:0/2325640410 >> 172.21.15.5:0/2325640410 conn(0x7f00a4072ef0 msgr2=0x7f00a40cc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:31.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:31.153+0000 7f00a8d25640 1 -- 172.21.15.5:0/2325640410 shutdown_connections 2024-03-24T10:50:31.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:31.153+0000 7f00a8d25640 1 -- 172.21.15.5:0/2325640410 wait complete. 2024-03-24T10:50:31.771 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:31.771 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:31.847 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:50:31.849 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:50:32.772 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:32.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.874+0000 7f4288083640 1 Processor -- start 2024-03-24T10:50:32.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.875+0000 7f4288083640 1 -- start start 2024-03-24T10:50:32.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.875+0000 7f4288083640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f428015a310 0x7f428006c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:32.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.875+0000 7f4288083640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42800cb0c0 con 0x7f42800cace0 2024-03-24T10:50:32.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.875+0000 7f4288083640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f428006cf70 con 0x7f428006c780 2024-03-24T10:50:32.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.875+0000 7f4288083640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f428006f110 con 0x7f428015a310 2024-03-24T10:50:32.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.876+0000 7f4285e1f640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f42800cace0 0x7f42801432d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59018/0 (socket says 172.21.15.5:59018) 2024-03-24T10:50:32.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.876+0000 7f4285e1f640 1 -- 172.21.15.5:0/2877462485 learned_addr learned my addr 172.21.15.5:0/2877462485 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:50:32.877 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.877+0000 7f428561e640 1 --2- 172.21.15.5:0/2877462485 >> v2:172.21.15.57:3300/0 conn(0x7f428015a310 0x7f428006c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:32.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.879+0000 7f4284e1d640 1 -- 172.21.15.5:0/2877462485 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3258151384 0 0) 0x7f42800cb0c0 con 0x7f42800cace0 2024-03-24T10:50:32.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.879+0000 7f4284e1d640 1 -- 172.21.15.5:0/2877462485 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f426c003520 con 0x7f42800cace0 2024-03-24T10:50:32.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.881+0000 7f4284e1d640 1 -- 172.21.15.5:0/2877462485 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 914505278 0 0) 0x7f428006cf70 con 0x7f428006c780 2024-03-24T10:50:32.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.881+0000 7f4284e1d640 1 -- 172.21.15.5:0/2877462485 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42800cb0c0 con 0x7f428006c780 2024-03-24T10:50:32.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.882+0000 7f428561e640 1 -- 172.21.15.5:0/2877462485 >> v1:172.21.15.57:6789/0 conn(0x7f428006c780 legacy=0x7f428006f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:32.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.882+0000 7f428561e640 1 -- 172.21.15.5:0/2877462485 >> v1:172.21.15.5:6789/0 conn(0x7f42800cace0 legacy=0x7f42801432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:32.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.882+0000 7f428561e640 1 -- 172.21.15.5:0/2877462485 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f428015a7a0 con 0x7f428015a310 2024-03-24T10:50:32.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.890+0000 7f428561e640 1 --2- 172.21.15.5:0/2877462485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f428015a310 0x7f428006c240 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f42700097a0 tx=0x7f427002b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf1a6c65631eaac4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:32.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.895+0000 7f4284e1d640 1 -- 172.21.15.5:0/2877462485 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427002f810 con 0x7f428015a310 2024-03-24T10:50:32.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.895+0000 7f4284e1d640 1 -- 172.21.15.5:0/2877462485 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f427002f970 con 0x7f428015a310 2024-03-24T10:50:32.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.895+0000 7f4284e1d640 1 -- 172.21.15.5:0/2877462485 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427002fd00 con 0x7f428015a310 2024-03-24T10:50:32.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.896+0000 7f4288083640 1 -- 172.21.15.5:0/2877462485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f428015a310 msgr2=0x7f428006c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:32.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.896+0000 7f4288083640 1 --2- 172.21.15.5:0/2877462485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f428015a310 0x7f428006c240 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f42700097a0 tx=0x7f427002b9f0 comp rx=0 tx=0).stop 2024-03-24T10:50:32.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.896+0000 7f4288083640 1 -- 172.21.15.5:0/2877462485 shutdown_connections 2024-03-24T10:50:32.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.896+0000 7f4288083640 1 --2- 172.21.15.5:0/2877462485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f428015a310 0x7f428006c240 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:32.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.896+0000 7f4288083640 1 -- 172.21.15.5:0/2877462485 >> 172.21.15.5:0/2877462485 conn(0x7f4280071240 msgr2=0x7f4280071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:32.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.897+0000 7f4288083640 1 -- 172.21.15.5:0/2877462485 shutdown_connections 2024-03-24T10:50:32.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.897+0000 7f4288083640 1 -- 172.21.15.5:0/2877462485 wait complete. 2024-03-24T10:50:32.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.898+0000 7f4288083640 1 Processor -- start 2024-03-24T10:50:32.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.898+0000 7f4288083640 1 -- start start 2024-03-24T10:50:32.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.900+0000 7f4288083640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f428006c780 0x7f42801e3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:32.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.900+0000 7f4288083640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f42800cace0 0x7f42800b27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:32.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.900+0000 7f4288083640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42801e4100 0x7f42801f05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:32.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.900+0000 7f4288083640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f42801f43c0 con 0x7f428006c780 2024-03-24T10:50:32.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.900+0000 7f4288083640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f42801f3840 con 0x7f42801e4100 2024-03-24T10:50:32.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.900+0000 7f4288083640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f42801f3b20 con 0x7f42800cace0 2024-03-24T10:50:32.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.900+0000 7f4285e1f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f428006c780 0x7f42801e3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:32.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.900+0000 7f428561e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f42800cace0 0x7f42800b27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:32.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.900+0000 7f4286620640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42801e4100 0x7f42801f05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:32.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.901+0000 7f4285e1f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f428006c780 0x7f42801e3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43428/0 (socket says 172.21.15.5:43428) 2024-03-24T10:50:32.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.901+0000 7f4285e1f640 1 -- 172.21.15.5:0/1082485210 learned_addr learned my addr 172.21.15.5:0/1082485210 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:32.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.904+0000 7f4285e1f640 1 -- 172.21.15.5:0/1082485210 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f42800cace0 msgr2=0x7f42800b27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:32.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.904+0000 7f4285e1f640 1 --2- 172.21.15.5:0/1082485210 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f42800cace0 0x7f42800b27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:32.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.904+0000 7f4285e1f640 1 -- 172.21.15.5:0/1082485210 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42801e4100 msgr2=0x7f42801f05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:32.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.904+0000 7f4285e1f640 1 --2- 172.21.15.5:0/1082485210 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42801e4100 0x7f42801f05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:32.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.904+0000 7f4285e1f640 1 -- 172.21.15.5:0/1082485210 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4270009400 con 0x7f428006c780 2024-03-24T10:50:32.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.907+0000 7f4285e1f640 1 --2- 172.21.15.5:0/1082485210 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f428006c780 0x7f42801e3980 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f427c00c450 tx=0x7f427c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:32.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.911+0000 7f42767fc640 1 -- 172.21.15.5:0/1082485210 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427c005ce0 con 0x7f428006c780 2024-03-24T10:50:32.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.911+0000 7f42767fc640 1 -- 172.21.15.5:0/1082485210 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f427c005e40 con 0x7f428006c780 2024-03-24T10:50:32.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.911+0000 7f4288083640 1 -- 172.21.15.5:0/1082485210 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42801f3e00 con 0x7f428006c780 2024-03-24T10:50:32.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.911+0000 7f42767fc640 1 -- 172.21.15.5:0/1082485210 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427c0072b0 con 0x7f428006c780 2024-03-24T10:50:32.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.911+0000 7f4288083640 1 -- 172.21.15.5:0/1082485210 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f428006fd20 con 0x7f428006c780 2024-03-24T10:50:32.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.913+0000 7f4288083640 1 -- 172.21.15.5:0/1082485210 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42801eff90 con 0x7f428006c780 2024-03-24T10:50:32.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.933+0000 7f42767fc640 1 -- 172.21.15.5:0/1082485210 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f427c007410 con 0x7f428006c780 2024-03-24T10:50:32.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.933+0000 7f42767fc640 1 --2- 172.21.15.5:0/1082485210 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f426406b810 0x7f426406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:32.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.933+0000 7f428561e640 1 --2- 172.21.15.5:0/1082485210 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f426406b810 0x7f426406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:32.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:32.934+0000 7f428561e640 1 --2- 172.21.15.5:0/1082485210 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f426406b810 0x7f426406dcd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4270036000 tx=0x7f4270008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:33.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:33.196+0000 7f42767fc640 1 -- 172.21.15.5:0/1082485210 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f427c0582b0 con 0x7f428006c780 2024-03-24T10:50:33.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:33.196+0000 7f42767fc640 1 -- 172.21.15.5:0/1082485210 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f427c054550 con 0x7f428006c780 2024-03-24T10:50:33.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:33.465+0000 7f4288083640 1 -- 172.21.15.5:0/1082485210 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42801f0410 con 0x7f428006c780 2024-03-24T10:50:33.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:33.482+0000 7f42767fc640 1 -- 172.21.15.5:0/1082485210 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f427c017070 con 0x7f428006c780 2024-03-24T10:50:33.483 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:33.483 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:33.483 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:33.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:33.484+0000 7f4288083640 1 -- 172.21.15.5:0/1082485210 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f426406b810 msgr2=0x7f426406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:33.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:33.484+0000 7f4288083640 1 --2- 172.21.15.5:0/1082485210 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f426406b810 0x7f426406dcd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4270036000 tx=0x7f4270008030 comp rx=0 tx=0).stop 2024-03-24T10:50:33.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:33.484+0000 7f4288083640 1 -- 172.21.15.5:0/1082485210 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f428006c780 msgr2=0x7f42801e3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:33.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:33.484+0000 7f4288083640 1 --2- 172.21.15.5:0/1082485210 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f428006c780 0x7f42801e3980 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f427c00c450 tx=0x7f427c00c910 comp rx=0 tx=0).stop 2024-03-24T10:50:33.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:33.484+0000 7f4288083640 1 -- 172.21.15.5:0/1082485210 shutdown_connections 2024-03-24T10:50:33.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:33.484+0000 7f4288083640 1 --2- 172.21.15.5:0/1082485210 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f426406b810 0x7f426406dcd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:33.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:33.484+0000 7f4288083640 1 --2- 172.21.15.5:0/1082485210 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42801e4100 0x7f42801f05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:33.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:33.484+0000 7f4288083640 1 --2- 172.21.15.5:0/1082485210 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f42800cace0 0x7f42800b27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:33.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:33.484+0000 7f4288083640 1 --2- 172.21.15.5:0/1082485210 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f428006c780 0x7f42801e3980 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:33.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:33.484+0000 7f4288083640 1 -- 172.21.15.5:0/1082485210 >> 172.21.15.5:0/1082485210 conn(0x7f4280071240 msgr2=0x7f4280159be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:33.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:33.485+0000 7f4288083640 1 -- 172.21.15.5:0/1082485210 shutdown_connections 2024-03-24T10:50:33.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:33.485+0000 7f4288083640 1 -- 172.21.15.5:0/1082485210 wait complete. 2024-03-24T10:50:34.099 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:34.099 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:35.101 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:35.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.213+0000 7f8134c6f640 1 Processor -- start 2024-03-24T10:50:35.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.214+0000 7f8134c6f640 1 -- start start 2024-03-24T10:50:35.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.215+0000 7f8134c6f640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f813015aac0 0x7f813015aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:35.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.215+0000 7f8134c6f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f813006c230 0x7f813006c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:35.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.215+0000 7f8134c6f640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f813006cd50 con 0x7f81300cebc0 2024-03-24T10:50:35.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.215+0000 7f8134c6f640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f81300ce790 con 0x7f813015aac0 2024-03-24T10:50:35.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.215+0000 7f8134c6f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f813015b3e0 con 0x7f813006c230 2024-03-24T10:50:35.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.215+0000 7f812e59c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f813015aac0 0x7f813015aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:35.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.216+0000 7f812dd9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f813006c230 0x7f813006c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:35.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.216+0000 7f812dd9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f813006c230 0x7f813006c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:50854/0 (socket says 172.21.15.5:50854) 2024-03-24T10:50:35.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.216+0000 7f812dd9b640 1 -- 172.21.15.5:0/3633261082 learned_addr learned my addr 172.21.15.5:0/3633261082 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:35.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.218+0000 7f812dd9b640 1 -- 172.21.15.5:0/3633261082 >> v2:172.21.15.5:3300/0 conn(0x7f813015aac0 msgr2=0x7f813015aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:35.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.218+0000 7f812dd9b640 1 --2- 172.21.15.5:0/3633261082 >> v2:172.21.15.5:3300/0 conn(0x7f813015aac0 0x7f813015aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:35.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.218+0000 7f812dd9b640 1 -- 172.21.15.5:0/3633261082 >> v1:172.21.15.5:6789/0 conn(0x7f81300cebc0 legacy=0x7f81300ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:35.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.218+0000 7f812dd9b640 1 -- 172.21.15.5:0/3633261082 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81300cdb10 con 0x7f813006c230 2024-03-24T10:50:35.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.221+0000 7f812dd9b640 1 --2- 172.21.15.5:0/3633261082 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f813006c230 0x7f813006c670 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8124009870 tx=0x7f812402ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2acdb93d254271d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:35.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.225+0000 7f812d59a640 1 -- 172.21.15.5:0/3633261082 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812402f840 con 0x7f813006c230 2024-03-24T10:50:35.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.225+0000 7f812d59a640 1 -- 172.21.15.5:0/3633261082 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f812402f9a0 con 0x7f813006c230 2024-03-24T10:50:35.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.225+0000 7f812d59a640 1 -- 172.21.15.5:0/3633261082 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812402fd30 con 0x7f813006c230 2024-03-24T10:50:35.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.226+0000 7f8134c6f640 1 -- 172.21.15.5:0/3633261082 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f813006c230 msgr2=0x7f813006c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:35.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.226+0000 7f8134c6f640 1 --2- 172.21.15.5:0/3633261082 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f813006c230 0x7f813006c670 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8124009870 tx=0x7f812402ba20 comp rx=0 tx=0).stop 2024-03-24T10:50:35.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.226+0000 7f8134c6f640 1 -- 172.21.15.5:0/3633261082 shutdown_connections 2024-03-24T10:50:35.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.226+0000 7f8134c6f640 1 --2- 172.21.15.5:0/3633261082 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f813006c230 0x7f813006c670 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:35.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.226+0000 7f8134c6f640 1 --2- 172.21.15.5:0/3633261082 >> v2:172.21.15.5:3300/0 conn(0x7f813015aac0 0x7f813015aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:35.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.226+0000 7f8134c6f640 1 -- 172.21.15.5:0/3633261082 >> 172.21.15.5:0/3633261082 conn(0x7f81300cfb90 msgr2=0x7f81300cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:35.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.226+0000 7f8134c6f640 1 -- 172.21.15.5:0/3633261082 shutdown_connections 2024-03-24T10:50:35.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.226+0000 7f8134c6f640 1 -- 172.21.15.5:0/3633261082 wait complete. 2024-03-24T10:50:35.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.228+0000 7f8134c6f640 1 Processor -- start 2024-03-24T10:50:35.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.228+0000 7f8134c6f640 1 -- start start 2024-03-24T10:50:35.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.228+0000 7f8134c6f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f813006c230 0x7f8130154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:35.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.228+0000 7f8134c6f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81300cebc0 0x7f8130156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:35.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.228+0000 7f8134c6f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f813015aac0 0x7f81300b0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:35.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.228+0000 7f8134c6f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f8130156ce0 con 0x7f81300cebc0 2024-03-24T10:50:35.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.228+0000 7f8134c6f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f813015e100 con 0x7f813015aac0 2024-03-24T10:50:35.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.228+0000 7f8134c6f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f8130154f10 con 0x7f813006c230 2024-03-24T10:50:35.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.229+0000 7f812ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f813015aac0 0x7f81300b0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:35.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.229+0000 7f812ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f813015aac0 0x7f81300b0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:50654/0 (socket says 172.21.15.5:50654) 2024-03-24T10:50:35.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.230+0000 7f812ed9d640 1 -- 172.21.15.5:0/416714442 learned_addr learned my addr 172.21.15.5:0/416714442 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:35.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.230+0000 7f812e59c640 1 --2- 172.21.15.5:0/416714442 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f813006c230 0x7f8130154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:35.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.235+0000 7f812ed9d640 1 -- 172.21.15.5:0/416714442 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f813006c230 msgr2=0x7f8130154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:35.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.235+0000 7f812ed9d640 1 --2- 172.21.15.5:0/416714442 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f813006c230 0x7f8130154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:35.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.235+0000 7f812ed9d640 1 -- 172.21.15.5:0/416714442 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81300cebc0 msgr2=0x7f8130156310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:35.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.235+0000 7f812ed9d640 1 --2- 172.21.15.5:0/416714442 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81300cebc0 0x7f8130156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:35.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.235+0000 7f812ed9d640 1 -- 172.21.15.5:0/416714442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81240094d0 con 0x7f813015aac0 2024-03-24T10:50:35.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.238+0000 7f812ed9d640 1 --2- 172.21.15.5:0/416714442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f813015aac0 0x7f81300b0920 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f812800c7a0 tx=0x7f812800cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:35.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.244+0000 7f811effd640 1 -- 172.21.15.5:0/416714442 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8128003b70 con 0x7f813015aac0 2024-03-24T10:50:35.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.244+0000 7f811effd640 1 -- 172.21.15.5:0/416714442 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8128003cd0 con 0x7f813015aac0 2024-03-24T10:50:35.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.244+0000 7f8134c6f640 1 -- 172.21.15.5:0/416714442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81301551f0 con 0x7f813015aac0 2024-03-24T10:50:35.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.244+0000 7f811effd640 1 -- 172.21.15.5:0/416714442 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812800aa90 con 0x7f813015aac0 2024-03-24T10:50:35.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.244+0000 7f8134c6f640 1 -- 172.21.15.5:0/416714442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81301557b0 con 0x7f813015aac0 2024-03-24T10:50:35.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.245+0000 7f8134c6f640 1 -- 172.21.15.5:0/416714442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81300681d0 con 0x7f813015aac0 2024-03-24T10:50:35.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.280+0000 7f811effd640 1 -- 172.21.15.5:0/416714442 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f812800abf0 con 0x7f813015aac0 2024-03-24T10:50:35.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.280+0000 7f811effd640 1 --2- 172.21.15.5:0/416714442 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f811006b810 0x7f811006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:35.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.280+0000 7f812e59c640 1 --2- 172.21.15.5:0/416714442 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f811006b810 0x7f811006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:35.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.281+0000 7f812e59c640 1 --2- 172.21.15.5:0/416714442 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f811006b810 0x7f811006dcd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8120009aa0 tx=0x7f8120009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:35.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.566+0000 7f811effd640 1 -- 172.21.15.5:0/416714442 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f8128059910 con 0x7f813015aac0 2024-03-24T10:50:35.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.585+0000 7f811effd640 1 -- 172.21.15.5:0/416714442 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f81280554b0 con 0x7f813015aac0 2024-03-24T10:50:35.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.810+0000 7f8134c6f640 1 -- 172.21.15.5:0/416714442 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f813006c610 con 0x7f813015aac0 2024-03-24T10:50:35.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.962+0000 7f811effd640 1 -- 172.21.15.5:0/416714442 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f8128058b70 con 0x7f813015aac0 2024-03-24T10:50:35.962 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:35.962 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:35.963 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:35.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.964+0000 7f8134c6f640 1 -- 172.21.15.5:0/416714442 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f811006b810 msgr2=0x7f811006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:35.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.964+0000 7f8134c6f640 1 --2- 172.21.15.5:0/416714442 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f811006b810 0x7f811006dcd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8120009aa0 tx=0x7f8120009190 comp rx=0 tx=0).stop 2024-03-24T10:50:35.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.964+0000 7f8134c6f640 1 -- 172.21.15.5:0/416714442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f813015aac0 msgr2=0x7f81300b0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:35.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.964+0000 7f8134c6f640 1 --2- 172.21.15.5:0/416714442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f813015aac0 0x7f81300b0920 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f812800c7a0 tx=0x7f812800cc60 comp rx=0 tx=0).stop 2024-03-24T10:50:35.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.964+0000 7f8134c6f640 1 -- 172.21.15.5:0/416714442 shutdown_connections 2024-03-24T10:50:35.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.964+0000 7f8134c6f640 1 --2- 172.21.15.5:0/416714442 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f811006b810 0x7f811006dcd0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:35.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.964+0000 7f8134c6f640 1 --2- 172.21.15.5:0/416714442 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81300cebc0 0x7f8130156310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:35.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.964+0000 7f8134c6f640 1 --2- 172.21.15.5:0/416714442 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f813015aac0 0x7f81300b0920 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:35.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.964+0000 7f8134c6f640 1 --2- 172.21.15.5:0/416714442 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f813006c230 0x7f8130154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:35.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.964+0000 7f8134c6f640 1 -- 172.21.15.5:0/416714442 >> 172.21.15.5:0/416714442 conn(0x7f81300cfb90 msgr2=0x7f81300c8390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:35.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.964+0000 7f8134c6f640 1 -- 172.21.15.5:0/416714442 shutdown_connections 2024-03-24T10:50:35.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:35.964+0000 7f8134c6f640 1 -- 172.21.15.5:0/416714442 wait complete. 2024-03-24T10:50:36.579 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:36.579 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:37.580 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:37.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.683+0000 7f4d4eceb640 1 Processor -- start 2024-03-24T10:50:37.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.683+0000 7f4d4eceb640 1 -- start start 2024-03-24T10:50:37.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.684+0000 7f4d4eceb640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4d4815ac00 0x7f4d480cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:37.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.684+0000 7f4d4eceb640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4d480ccaa0 0x7f4d4806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:37.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.684+0000 7f4d4eceb640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d480cb2f0 con 0x7f4d481543a0 2024-03-24T10:50:37.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.684+0000 7f4d4eceb640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f4d480c6750 con 0x7f4d480ccaa0 2024-03-24T10:50:37.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.684+0000 7f4d4eceb640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f4d480cc380 con 0x7f4d4815ac00 2024-03-24T10:50:37.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.685+0000 7f4d47fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4d4815ac00 0x7f4d480cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:37.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.685+0000 7f4d4d288640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4d480ccaa0 0x7f4d4806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:37.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.685+0000 7f4d47fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4d4815ac00 0x7f4d480cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:50664/0 (socket says 172.21.15.5:50664) 2024-03-24T10:50:37.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.685+0000 7f4d47fff640 1 -- 172.21.15.5:0/636496843 learned_addr learned my addr 172.21.15.5:0/636496843 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:37.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.688+0000 7f4d4d288640 1 -- 172.21.15.5:0/636496843 >> v2:172.21.15.57:3300/0 conn(0x7f4d4815ac00 msgr2=0x7f4d480cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:37.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.688+0000 7f4d4d288640 1 --2- 172.21.15.5:0/636496843 >> v2:172.21.15.57:3300/0 conn(0x7f4d4815ac00 0x7f4d480cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:37.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.688+0000 7f4d4d288640 1 -- 172.21.15.5:0/636496843 >> v1:172.21.15.5:6789/0 conn(0x7f4d481543a0 legacy=0x7f4d4806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:37.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.688+0000 7f4d4d288640 1 -- 172.21.15.5:0/636496843 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d4815b070 con 0x7f4d480ccaa0 2024-03-24T10:50:37.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.691+0000 7f4d4d288640 1 --2- 172.21.15.5:0/636496843 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d480ccaa0 0x7f4d4806bf50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4d480c6990 tx=0x7f4d4002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d710079dde8b7ac3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:37.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.695+0000 7f4d477fe640 1 -- 172.21.15.5:0/636496843 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4002cbf0 con 0x7f4d480ccaa0 2024-03-24T10:50:37.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.697+0000 7f4d477fe640 1 -- 172.21.15.5:0/636496843 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d4002cd50 con 0x7f4d480ccaa0 2024-03-24T10:50:37.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.697+0000 7f4d477fe640 1 -- 172.21.15.5:0/636496843 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d40034660 con 0x7f4d480ccaa0 2024-03-24T10:50:37.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.697+0000 7f4d4eceb640 1 -- 172.21.15.5:0/636496843 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d480ccaa0 msgr2=0x7f4d4806bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:37.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.697+0000 7f4d4eceb640 1 --2- 172.21.15.5:0/636496843 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d480ccaa0 0x7f4d4806bf50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4d480c6990 tx=0x7f4d4002be40 comp rx=0 tx=0).stop 2024-03-24T10:50:37.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.698+0000 7f4d4eceb640 1 -- 172.21.15.5:0/636496843 shutdown_connections 2024-03-24T10:50:37.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.698+0000 7f4d4eceb640 1 --2- 172.21.15.5:0/636496843 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d480ccaa0 0x7f4d4806bf50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:37.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.698+0000 7f4d4eceb640 1 --2- 172.21.15.5:0/636496843 >> v2:172.21.15.57:3300/0 conn(0x7f4d4815ac00 0x7f4d480cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:37.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.698+0000 7f4d4eceb640 1 -- 172.21.15.5:0/636496843 >> 172.21.15.5:0/636496843 conn(0x7f4d48072ef0 msgr2=0x7f4d48061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:37.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.698+0000 7f4d4eceb640 1 -- 172.21.15.5:0/636496843 shutdown_connections 2024-03-24T10:50:37.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.698+0000 7f4d4eceb640 1 -- 172.21.15.5:0/636496843 wait complete. 2024-03-24T10:50:37.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.699+0000 7f4d4eceb640 1 Processor -- start 2024-03-24T10:50:37.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.699+0000 7f4d4eceb640 1 -- start start 2024-03-24T10:50:37.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.700+0000 7f4d4eceb640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4d480ccaa0 0x7f4d481e5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:37.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.700+0000 7f4d4eceb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d481543a0 0x7f4d480b3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:37.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.700+0000 7f4d4eceb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d4815ac00 0x7f4d481f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:37.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.700+0000 7f4d4eceb640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f4d480b3ea0 con 0x7f4d4815ac00 2024-03-24T10:50:37.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.700+0000 7f4d4eceb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4d481e4990 con 0x7f4d481543a0 2024-03-24T10:50:37.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.700+0000 7f4d4eceb640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4d481e5b50 con 0x7f4d480ccaa0 2024-03-24T10:50:37.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.701+0000 7f4d47fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d481543a0 0x7f4d480b3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:37.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.701+0000 7f4d4d288640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d4815ac00 0x7f4d481f2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:37.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.701+0000 7f4d4ca87640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4d480ccaa0 0x7f4d481e5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:37.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.701+0000 7f4d47fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d481543a0 0x7f4d480b3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:50666/0 (socket says 172.21.15.5:50666) 2024-03-24T10:50:37.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.701+0000 7f4d47fff640 1 -- 172.21.15.5:0/3382465552 learned_addr learned my addr 172.21.15.5:0/3382465552 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:37.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.704+0000 7f4d47fff640 1 -- 172.21.15.5:0/3382465552 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4d480ccaa0 msgr2=0x7f4d481e5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:37.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.704+0000 7f4d47fff640 1 --2- 172.21.15.5:0/3382465552 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4d480ccaa0 0x7f4d481e5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:37.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.704+0000 7f4d47fff640 1 -- 172.21.15.5:0/3382465552 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d4815ac00 msgr2=0x7f4d481f2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:37.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.704+0000 7f4d47fff640 1 --2- 172.21.15.5:0/3382465552 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d4815ac00 0x7f4d481f2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:37.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.705+0000 7f4d47fff640 1 -- 172.21.15.5:0/3382465552 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d40009a70 con 0x7f4d481543a0 2024-03-24T10:50:37.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.707+0000 7f4d47fff640 1 --2- 172.21.15.5:0/3382465552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d481543a0 0x7f4d480b3960 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4d3400e840 tx=0x7f4d3400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:37.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.711+0000 7f4d457fa640 1 -- 172.21.15.5:0/3382465552 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d3400c8e0 con 0x7f4d481543a0 2024-03-24T10:50:37.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.711+0000 7f4d457fa640 1 -- 172.21.15.5:0/3382465552 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d3400ca40 con 0x7f4d481543a0 2024-03-24T10:50:37.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.711+0000 7f4d4eceb640 1 -- 172.21.15.5:0/3382465552 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d480b33c0 con 0x7f4d481543a0 2024-03-24T10:50:37.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.711+0000 7f4d457fa640 1 -- 172.21.15.5:0/3382465552 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d3400cdf0 con 0x7f4d481543a0 2024-03-24T10:50:37.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.712+0000 7f4d4eceb640 1 -- 172.21.15.5:0/3382465552 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d481e46b0 con 0x7f4d481543a0 2024-03-24T10:50:37.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:37.713+0000 7f4d4eceb640 1 -- 172.21.15.5:0/3382465552 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d480681d0 con 0x7f4d481543a0 2024-03-24T10:50:38.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:38.012+0000 7f4d457fa640 1 -- 172.21.15.5:0/3382465552 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4d34013470 con 0x7f4d481543a0 2024-03-24T10:50:38.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:38.013+0000 7f4d457fa640 1 --2- 172.21.15.5:0/3382465552 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4d2c06b890 0x7f4d2c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:38.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:38.013+0000 7f4d4ca87640 1 --2- 172.21.15.5:0/3382465552 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4d2c06b890 0x7f4d2c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:38.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:38.013+0000 7f4d4ca87640 1 --2- 172.21.15.5:0/3382465552 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4d2c06b890 0x7f4d2c06dd50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c004f60 tx=0x7f4d3c007e30 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:38.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:38.019+0000 7f4d457fa640 1 -- 172.21.15.5:0/3382465552 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f4d34059200 con 0x7f4d481543a0 2024-03-24T10:50:38.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:38.031+0000 7f4d457fa640 1 -- 172.21.15.5:0/3382465552 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4d340554a0 con 0x7f4d481543a0 2024-03-24T10:50:38.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:38.303+0000 7f4d4eceb640 1 -- 172.21.15.5:0/3382465552 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d4815e580 con 0x7f4d481543a0 2024-03-24T10:50:38.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:38.324+0000 7f4d457fa640 1 -- 172.21.15.5:0/3382465552 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f4d34017090 con 0x7f4d481543a0 2024-03-24T10:50:38.324 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:38.325 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:38.325 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:38.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:38.326+0000 7f4d4eceb640 1 -- 172.21.15.5:0/3382465552 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4d2c06b890 msgr2=0x7f4d2c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:38.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:38.326+0000 7f4d4eceb640 1 --2- 172.21.15.5:0/3382465552 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4d2c06b890 0x7f4d2c06dd50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c004f60 tx=0x7f4d3c007e30 comp rx=0 tx=0).stop 2024-03-24T10:50:38.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:38.326+0000 7f4d4eceb640 1 -- 172.21.15.5:0/3382465552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d481543a0 msgr2=0x7f4d480b3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:38.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:38.326+0000 7f4d4eceb640 1 --2- 172.21.15.5:0/3382465552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d481543a0 0x7f4d480b3960 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4d3400e840 tx=0x7f4d3400ed00 comp rx=0 tx=0).stop 2024-03-24T10:50:38.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:38.326+0000 7f4d4eceb640 1 -- 172.21.15.5:0/3382465552 shutdown_connections 2024-03-24T10:50:38.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:38.326+0000 7f4d4eceb640 1 --2- 172.21.15.5:0/3382465552 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4d2c06b890 0x7f4d2c06dd50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:38.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:38.326+0000 7f4d4eceb640 1 --2- 172.21.15.5:0/3382465552 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4d4815ac00 0x7f4d481f2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:38.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:38.326+0000 7f4d4eceb640 1 --2- 172.21.15.5:0/3382465552 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4d481543a0 0x7f4d480b3960 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:38.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:38.326+0000 7f4d4eceb640 1 --2- 172.21.15.5:0/3382465552 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4d480ccaa0 0x7f4d481e5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:38.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:38.326+0000 7f4d4eceb640 1 -- 172.21.15.5:0/3382465552 >> 172.21.15.5:0/3382465552 conn(0x7f4d48072ef0 msgr2=0x7f4d481530e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:38.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:38.326+0000 7f4d4eceb640 1 -- 172.21.15.5:0/3382465552 shutdown_connections 2024-03-24T10:50:38.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:38.326+0000 7f4d4eceb640 1 -- 172.21.15.5:0/3382465552 wait complete. 2024-03-24T10:50:38.942 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:38.942 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:39.943 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:40.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.044+0000 7fcc63fb6640 1 Processor -- start 2024-03-24T10:50:40.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.045+0000 7fcc63fb6640 1 -- start start 2024-03-24T10:50:40.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.046+0000 7fcc63fb6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcc5c1543a0 0x7fcc5c15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:40.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.046+0000 7fcc63fb6640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcc5c06d4f0 0x7fcc5c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:40.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.046+0000 7fcc63fb6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc5c0ccaa0 0x7fcc5c06bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:40.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.046+0000 7fcc63fb6640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fcc5c0c6750 con 0x7fcc5c06d4f0 2024-03-24T10:50:40.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.046+0000 7fcc63fb6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fcc5c0cceb0 con 0x7fcc5c1543a0 2024-03-24T10:50:40.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.046+0000 7fcc63fb6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fcc5c0cc380 con 0x7fcc5c0ccaa0 2024-03-24T10:50:40.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.047+0000 7fcc61551640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcc5c06d4f0 0x7fcc5c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:40.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.047+0000 7fcc62553640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc5c0ccaa0 0x7fcc5c06bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:40.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.047+0000 7fcc61d52640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcc5c1543a0 0x7fcc5c15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:40.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.047+0000 7fcc61d52640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcc5c1543a0 0x7fcc5c15ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:50682/0 (socket says 172.21.15.5:50682) 2024-03-24T10:50:40.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.047+0000 7fcc61d52640 1 -- 172.21.15.5:0/2031758095 learned_addr learned my addr 172.21.15.5:0/2031758095 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:40.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.049+0000 7fcc61d52640 1 -- 172.21.15.5:0/2031758095 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc5c0ccaa0 msgr2=0x7fcc5c06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:40.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.049+0000 7fcc61d52640 1 --2- 172.21.15.5:0/2031758095 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc5c0ccaa0 0x7fcc5c06bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:40.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.049+0000 7fcc61d52640 1 -- 172.21.15.5:0/2031758095 >> v2:172.21.15.5:3300/0 conn(0x7fcc5c06d4f0 msgr2=0x7fcc5c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:40.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.049+0000 7fcc61d52640 1 --2- 172.21.15.5:0/2031758095 >> v2:172.21.15.5:3300/0 conn(0x7fcc5c06d4f0 0x7fcc5c0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:40.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.049+0000 7fcc61d52640 1 -- 172.21.15.5:0/2031758095 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc5c0cb2f0 con 0x7fcc5c1543a0 2024-03-24T10:50:40.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.051+0000 7fcc61d52640 1 --2- 172.21.15.5:0/2031758095 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc5c1543a0 0x7fcc5c15ac00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c0cd600 tx=0x7fcc5802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a85cec3637e59f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:40.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.056+0000 7fcc60d50640 1 -- 172.21.15.5:0/2031758095 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc58030860 con 0x7fcc5c1543a0 2024-03-24T10:50:40.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.056+0000 7fcc60d50640 1 -- 172.21.15.5:0/2031758095 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc580309c0 con 0x7fcc5c1543a0 2024-03-24T10:50:40.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.056+0000 7fcc60d50640 1 -- 172.21.15.5:0/2031758095 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc58030d50 con 0x7fcc5c1543a0 2024-03-24T10:50:40.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.056+0000 7fcc63fb6640 1 -- 172.21.15.5:0/2031758095 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc5c1543a0 msgr2=0x7fcc5c15ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:40.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.056+0000 7fcc63fb6640 1 --2- 172.21.15.5:0/2031758095 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc5c1543a0 0x7fcc5c15ac00 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c0cd600 tx=0x7fcc5802b9a0 comp rx=0 tx=0).stop 2024-03-24T10:50:40.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.057+0000 7fcc63fb6640 1 -- 172.21.15.5:0/2031758095 shutdown_connections 2024-03-24T10:50:40.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.057+0000 7fcc63fb6640 1 --2- 172.21.15.5:0/2031758095 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc5c0ccaa0 0x7fcc5c06bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:40.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.057+0000 7fcc63fb6640 1 --2- 172.21.15.5:0/2031758095 >> v2:172.21.15.5:3300/0 conn(0x7fcc5c06d4f0 0x7fcc5c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:40.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.057+0000 7fcc63fb6640 1 --2- 172.21.15.5:0/2031758095 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc5c1543a0 0x7fcc5c15ac00 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:40.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.057+0000 7fcc63fb6640 1 -- 172.21.15.5:0/2031758095 >> 172.21.15.5:0/2031758095 conn(0x7fcc5c072ef0 msgr2=0x7fcc5c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:40.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.057+0000 7fcc63fb6640 1 -- 172.21.15.5:0/2031758095 shutdown_connections 2024-03-24T10:50:40.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.057+0000 7fcc63fb6640 1 -- 172.21.15.5:0/2031758095 wait complete. 2024-03-24T10:50:40.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.058+0000 7fcc63fb6640 1 Processor -- start 2024-03-24T10:50:40.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.058+0000 7fcc63fb6640 1 -- start start 2024-03-24T10:50:40.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.058+0000 7fcc63fb6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc5c06d4f0 0x7fcc5c0b3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:40.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.059+0000 7fcc63fb6640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc5c0ccaa0 0x7fcc5c0c6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:40.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.059+0000 7fcc63fb6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc5c1543a0 0x7fcc5c0ce920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:40.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.059+0000 7fcc63fb6640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c0c7980 con 0x7fcc5c0ccaa0 2024-03-24T10:50:40.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.059+0000 7fcc63fb6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c0c8810 con 0x7fcc5c06d4f0 2024-03-24T10:50:40.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.059+0000 7fcc63fb6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fcc5c158860 con 0x7fcc5c1543a0 2024-03-24T10:50:40.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.059+0000 7fcc61551640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc5c0ccaa0 0x7fcc5c0c6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:40.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.059+0000 7fcc61d52640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc5c06d4f0 0x7fcc5c0b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:40.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.059+0000 7fcc62553640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc5c1543a0 0x7fcc5c0ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:40.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.059+0000 7fcc61d52640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc5c06d4f0 0x7fcc5c0b3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:50698/0 (socket says 172.21.15.5:50698) 2024-03-24T10:50:40.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.059+0000 7fcc61d52640 1 -- 172.21.15.5:0/1038590275 learned_addr learned my addr 172.21.15.5:0/1038590275 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:40.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.063+0000 7fcc61d52640 1 -- 172.21.15.5:0/1038590275 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc5c1543a0 msgr2=0x7fcc5c0ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:40.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.063+0000 7fcc61d52640 1 --2- 172.21.15.5:0/1038590275 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc5c1543a0 0x7fcc5c0ce920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:40.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.063+0000 7fcc61d52640 1 -- 172.21.15.5:0/1038590275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc5c0ccaa0 msgr2=0x7fcc5c0c6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:40.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.063+0000 7fcc61d52640 1 --2- 172.21.15.5:0/1038590275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc5c0ccaa0 0x7fcc5c0c6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:40.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.063+0000 7fcc61d52640 1 -- 172.21.15.5:0/1038590275 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc580094d0 con 0x7fcc5c06d4f0 2024-03-24T10:50:40.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.063+0000 7fcc61551640 1 --2- 172.21.15.5:0/1038590275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc5c0ccaa0 0x7fcc5c0c6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:50:40.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.066+0000 7fcc61d52640 1 --2- 172.21.15.5:0/1038590275 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc5c06d4f0 0x7fcc5c0b3080 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcc5802b9e0 tx=0x7fcc58004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:40.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.070+0000 7fcc527fc640 1 -- 172.21.15.5:0/1038590275 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc58039a90 con 0x7fcc5c06d4f0 2024-03-24T10:50:40.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.070+0000 7fcc527fc640 1 -- 172.21.15.5:0/1038590275 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc58039c10 con 0x7fcc5c06d4f0 2024-03-24T10:50:40.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.070+0000 7fcc63fb6640 1 -- 172.21.15.5:0/1038590275 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc5c157f60 con 0x7fcc5c06d4f0 2024-03-24T10:50:40.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.070+0000 7fcc527fc640 1 -- 172.21.15.5:0/1038590275 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc58043410 con 0x7fcc5c06d4f0 2024-03-24T10:50:40.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.070+0000 7fcc63fb6640 1 -- 172.21.15.5:0/1038590275 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc5c0b4e20 con 0x7fcc5c06d4f0 2024-03-24T10:50:40.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.071+0000 7fcc63fb6640 1 -- 172.21.15.5:0/1038590275 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc5c0681d0 con 0x7fcc5c06d4f0 2024-03-24T10:50:40.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.095+0000 7fcc527fc640 1 -- 172.21.15.5:0/1038590275 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcc5804c3b0 con 0x7fcc5c06d4f0 2024-03-24T10:50:40.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.096+0000 7fcc527fc640 1 --2- 172.21.15.5:0/1038590275 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcc4006b890 0x7fcc4006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:40.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.096+0000 7fcc61551640 1 --2- 172.21.15.5:0/1038590275 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcc4006b890 0x7fcc4006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:40.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.096+0000 7fcc61551640 1 --2- 172.21.15.5:0/1038590275 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcc4006b890 0x7fcc4006dd50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c1f4080 tx=0x7fcc4c009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:40.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.473+0000 7fcc527fc640 1 -- 172.21.15.5:0/1038590275 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fcc5807b5c0 con 0x7fcc5c06d4f0 2024-03-24T10:50:40.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.492+0000 7fcc527fc640 1 -- 172.21.15.5:0/1038590275 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcc58077860 con 0x7fcc5c06d4f0 2024-03-24T10:50:40.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.738+0000 7fcc63fb6640 1 -- 172.21.15.5:0/1038590275 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc5c14e420 con 0x7fcc5c06d4f0 2024-03-24T10:50:40.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.752+0000 7fcc527fc640 1 -- 172.21.15.5:0/1038590275 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fcc58034020 con 0x7fcc5c06d4f0 2024-03-24T10:50:40.753 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:40.753 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:40.753 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:40.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.754+0000 7fcc63fb6640 1 -- 172.21.15.5:0/1038590275 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcc4006b890 msgr2=0x7fcc4006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:40.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.754+0000 7fcc63fb6640 1 --2- 172.21.15.5:0/1038590275 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcc4006b890 0x7fcc4006dd50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c1f4080 tx=0x7fcc4c009190 comp rx=0 tx=0).stop 2024-03-24T10:50:40.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.754+0000 7fcc63fb6640 1 -- 172.21.15.5:0/1038590275 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc5c06d4f0 msgr2=0x7fcc5c0b3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:40.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.754+0000 7fcc63fb6640 1 --2- 172.21.15.5:0/1038590275 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc5c06d4f0 0x7fcc5c0b3080 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcc5802b9e0 tx=0x7fcc58004730 comp rx=0 tx=0).stop 2024-03-24T10:50:40.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.754+0000 7fcc63fb6640 1 -- 172.21.15.5:0/1038590275 shutdown_connections 2024-03-24T10:50:40.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.754+0000 7fcc63fb6640 1 --2- 172.21.15.5:0/1038590275 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcc4006b890 0x7fcc4006dd50 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:40.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.754+0000 7fcc63fb6640 1 --2- 172.21.15.5:0/1038590275 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc5c0ccaa0 0x7fcc5c0c6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:40.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.754+0000 7fcc63fb6640 1 --2- 172.21.15.5:0/1038590275 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc5c1543a0 0x7fcc5c0ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:40.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.755+0000 7fcc63fb6640 1 --2- 172.21.15.5:0/1038590275 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc5c06d4f0 0x7fcc5c0b3080 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:40.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.755+0000 7fcc63fb6640 1 -- 172.21.15.5:0/1038590275 >> 172.21.15.5:0/1038590275 conn(0x7fcc5c072ef0 msgr2=0x7fcc5c1525d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:40.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.755+0000 7fcc63fb6640 1 -- 172.21.15.5:0/1038590275 shutdown_connections 2024-03-24T10:50:40.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:40.755+0000 7fcc63fb6640 1 -- 172.21.15.5:0/1038590275 wait complete. 2024-03-24T10:50:41.370 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:41.370 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:42.370 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:42.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.472+0000 7fa023d4b640 1 Processor -- start 2024-03-24T10:50:42.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.472+0000 7fa023d4b640 1 -- start start 2024-03-24T10:50:42.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.473+0000 7fa023d4b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa01c06c580 0x7fa01c1538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:42.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.473+0000 7fa023d4b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa01c0cf110 0x7fa01c153df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:42.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.473+0000 7fa023d4b640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa01c15bce0 con 0x7fa01c0cc8f0 2024-03-24T10:50:42.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.473+0000 7fa023d4b640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fa01c0cd890 con 0x7fa01c0cf110 2024-03-24T10:50:42.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.473+0000 7fa023d4b640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fa01c06cae0 con 0x7fa01c06c580 2024-03-24T10:50:42.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.473+0000 7fa021ae7640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fa01c0cc8f0 0x7fa01c15b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59046/0 (socket says 172.21.15.5:59046) 2024-03-24T10:50:42.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.473+0000 7fa021ae7640 1 -- 172.21.15.5:0/3329481403 learned_addr learned my addr 172.21.15.5:0/3329481403 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:50:42.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.474+0000 7fa0222e8640 1 --2- 172.21.15.5:0/3329481403 >> v2:172.21.15.57:3300/0 conn(0x7fa01c0cf110 0x7fa01c153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:42.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.474+0000 7fa0212e6640 1 --2- 172.21.15.5:0/3329481403 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa01c06c580 0x7fa01c1538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:42.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.476+0000 7fa020ae5640 1 -- 172.21.15.5:0/3329481403 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3085725973 0 0) 0x7fa01c15bce0 con 0x7fa01c0cc8f0 2024-03-24T10:50:42.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.476+0000 7fa020ae5640 1 -- 172.21.15.5:0/3329481403 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa008003520 con 0x7fa01c0cc8f0 2024-03-24T10:50:42.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.476+0000 7fa0222e8640 1 -- 172.21.15.5:0/3329481403 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa01c06c580 msgr2=0x7fa01c1538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:42.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.476+0000 7fa0222e8640 1 --2- 172.21.15.5:0/3329481403 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa01c06c580 0x7fa01c1538b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:42.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.476+0000 7fa0222e8640 1 -- 172.21.15.5:0/3329481403 >> v1:172.21.15.5:6789/0 conn(0x7fa01c0cc8f0 legacy=0x7fa01c15b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:42.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.476+0000 7fa0222e8640 1 -- 172.21.15.5:0/3329481403 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa01c0cdb50 con 0x7fa01c0cf110 2024-03-24T10:50:42.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.479+0000 7fa0222e8640 1 --2- 172.21.15.5:0/3329481403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c0cf110 0x7fa01c153df0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa014009d90 tx=0x7fa01402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11b95539762f62f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:42.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.483+0000 7fa020ae5640 1 -- 172.21.15.5:0/3329481403 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01402cc50 con 0x7fa01c0cf110 2024-03-24T10:50:42.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.483+0000 7fa020ae5640 1 -- 172.21.15.5:0/3329481403 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa01402cdb0 con 0x7fa01c0cf110 2024-03-24T10:50:42.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.483+0000 7fa020ae5640 1 -- 172.21.15.5:0/3329481403 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa014034660 con 0x7fa01c0cf110 2024-03-24T10:50:42.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.483+0000 7fa023d4b640 1 -- 172.21.15.5:0/3329481403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c0cf110 msgr2=0x7fa01c153df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:42.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.483+0000 7fa023d4b640 1 --2- 172.21.15.5:0/3329481403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c0cf110 0x7fa01c153df0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa014009d90 tx=0x7fa01402bea0 comp rx=0 tx=0).stop 2024-03-24T10:50:42.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.483+0000 7fa023d4b640 1 -- 172.21.15.5:0/3329481403 shutdown_connections 2024-03-24T10:50:42.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.483+0000 7fa023d4b640 1 --2- 172.21.15.5:0/3329481403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c0cf110 0x7fa01c153df0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:42.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.483+0000 7fa023d4b640 1 --2- 172.21.15.5:0/3329481403 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa01c06c580 0x7fa01c1538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:42.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.483+0000 7fa023d4b640 1 -- 172.21.15.5:0/3329481403 >> 172.21.15.5:0/3329481403 conn(0x7fa01c0b1820 msgr2=0x7fa01c0b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:42.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.484+0000 7fa023d4b640 1 -- 172.21.15.5:0/3329481403 shutdown_connections 2024-03-24T10:50:42.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.484+0000 7fa023d4b640 1 -- 172.21.15.5:0/3329481403 wait complete. 2024-03-24T10:50:42.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.485+0000 7fa023d4b640 1 Processor -- start 2024-03-24T10:50:42.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.485+0000 7fa023d4b640 1 -- start start 2024-03-24T10:50:42.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.485+0000 7fa023d4b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c06c580 0x7fa01c0cbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:42.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.485+0000 7fa023d4b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa01c0cc8f0 0x7fa01c14c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:42.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.485+0000 7fa023d4b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c0cf110 0x7fa01c1e4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:42.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.485+0000 7fa023d4b640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c0cc3e0 con 0x7fa01c0cf110 2024-03-24T10:50:42.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.485+0000 7fa023d4b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c1e4390 con 0x7fa01c06c580 2024-03-24T10:50:42.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.485+0000 7fa023d4b640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fa01c1e4950 con 0x7fa01c0cc8f0 2024-03-24T10:50:42.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.486+0000 7fa021ae7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c06c580 0x7fa01c0cbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:42.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.486+0000 7fa0222e8640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c0cf110 0x7fa01c1e4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:42.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.486+0000 7fa0212e6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa01c0cc8f0 0x7fa01c14c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:42.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.486+0000 7fa021ae7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c06c580 0x7fa01c0cbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:50706/0 (socket says 172.21.15.5:50706) 2024-03-24T10:50:42.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.486+0000 7fa021ae7640 1 -- 172.21.15.5:0/4175841498 learned_addr learned my addr 172.21.15.5:0/4175841498 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:42.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.490+0000 7fa021ae7640 1 -- 172.21.15.5:0/4175841498 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa01c0cc8f0 msgr2=0x7fa01c14c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:42.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.490+0000 7fa021ae7640 1 --2- 172.21.15.5:0/4175841498 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa01c0cc8f0 0x7fa01c14c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:42.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.490+0000 7fa021ae7640 1 -- 172.21.15.5:0/4175841498 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c0cf110 msgr2=0x7fa01c1e4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:42.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.490+0000 7fa021ae7640 1 --2- 172.21.15.5:0/4175841498 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c0cf110 0x7fa01c1e4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:42.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.490+0000 7fa021ae7640 1 -- 172.21.15.5:0/4175841498 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa014009a70 con 0x7fa01c06c580 2024-03-24T10:50:42.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.493+0000 7fa021ae7640 1 --2- 172.21.15.5:0/4175841498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c06c580 0x7fa01c0cbd50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa01800c490 tx=0x7fa01800c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:42.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.497+0000 7fa0127fc640 1 -- 172.21.15.5:0/4175841498 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa018005ce0 con 0x7fa01c06c580 2024-03-24T10:50:42.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.497+0000 7fa0127fc640 1 -- 172.21.15.5:0/4175841498 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa018005e40 con 0x7fa01c06c580 2024-03-24T10:50:42.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.497+0000 7fa0127fc640 1 -- 172.21.15.5:0/4175841498 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa018007290 con 0x7fa01c06c580 2024-03-24T10:50:42.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.497+0000 7fa023d4b640 1 -- 172.21.15.5:0/4175841498 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa01c0c94b0 con 0x7fa01c06c580 2024-03-24T10:50:42.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.497+0000 7fa023d4b640 1 -- 172.21.15.5:0/4175841498 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa01c0cb550 con 0x7fa01c06c580 2024-03-24T10:50:42.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.499+0000 7fa023d4b640 1 -- 172.21.15.5:0/4175841498 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa01c0cb770 con 0x7fa01c06c580 2024-03-24T10:50:42.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.769+0000 7fa0127fc640 1 -- 172.21.15.5:0/4175841498 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa0180073f0 con 0x7fa01c06c580 2024-03-24T10:50:42.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.769+0000 7fa0127fc640 1 --2- 172.21.15.5:0/4175841498 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa00006b810 0x7fa00006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:42.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.770+0000 7fa0212e6640 1 --2- 172.21.15.5:0/4175841498 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa00006b810 0x7fa00006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:42.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.770+0000 7fa0212e6640 1 --2- 172.21.15.5:0/4175841498 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa00006b810 0x7fa00006dcd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0c9eb0 tx=0x7fa00c009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:42.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.776+0000 7fa0127fc640 1 -- 172.21.15.5:0/4175841498 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fa018058260 con 0x7fa01c06c580 2024-03-24T10:50:42.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:42.795+0000 7fa0127fc640 1 -- 172.21.15.5:0/4175841498 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa018054500 con 0x7fa01c06c580 2024-03-24T10:50:43.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:43.018+0000 7fa023d4b640 1 -- 172.21.15.5:0/4175841498 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa01c0cd3b0 con 0x7fa01c06c580 2024-03-24T10:50:43.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:43.043+0000 7fa0127fc640 1 -- 172.21.15.5:0/4175841498 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fa018017070 con 0x7fa01c06c580 2024-03-24T10:50:43.044 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:43.044 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:43.044 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:43.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:43.046+0000 7fa023d4b640 1 -- 172.21.15.5:0/4175841498 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa00006b810 msgr2=0x7fa00006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:43.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:43.046+0000 7fa023d4b640 1 --2- 172.21.15.5:0/4175841498 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa00006b810 0x7fa00006dcd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0c9eb0 tx=0x7fa00c009190 comp rx=0 tx=0).stop 2024-03-24T10:50:43.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:43.046+0000 7fa023d4b640 1 -- 172.21.15.5:0/4175841498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c06c580 msgr2=0x7fa01c0cbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:43.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:43.046+0000 7fa023d4b640 1 --2- 172.21.15.5:0/4175841498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c06c580 0x7fa01c0cbd50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa01800c490 tx=0x7fa01800c950 comp rx=0 tx=0).stop 2024-03-24T10:50:43.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:43.046+0000 7fa023d4b640 1 -- 172.21.15.5:0/4175841498 shutdown_connections 2024-03-24T10:50:43.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:43.046+0000 7fa023d4b640 1 --2- 172.21.15.5:0/4175841498 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa00006b810 0x7fa00006dcd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:43.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:43.046+0000 7fa023d4b640 1 --2- 172.21.15.5:0/4175841498 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa01c0cf110 0x7fa01c1e4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:43.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:43.046+0000 7fa023d4b640 1 --2- 172.21.15.5:0/4175841498 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa01c0cc8f0 0x7fa01c14c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:43.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:43.046+0000 7fa023d4b640 1 --2- 172.21.15.5:0/4175841498 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa01c06c580 0x7fa01c0cbd50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:43.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:43.046+0000 7fa023d4b640 1 -- 172.21.15.5:0/4175841498 >> 172.21.15.5:0/4175841498 conn(0x7fa01c0b1820 msgr2=0x7fa01c0b3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:43.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:43.046+0000 7fa023d4b640 1 -- 172.21.15.5:0/4175841498 shutdown_connections 2024-03-24T10:50:43.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:43.046+0000 7fa023d4b640 1 -- 172.21.15.5:0/4175841498 wait complete. 2024-03-24T10:50:43.661 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:43.662 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:44.662 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:44.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.763+0000 7f4ceb9e2640 1 Processor -- start 2024-03-24T10:50:44.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.763+0000 7f4ceb9e2640 1 -- start start 2024-03-24T10:50:44.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.765+0000 7f4ceb9e2640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4ce41543a0 0x7f4ce415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:44.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.765+0000 7f4ceb9e2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ce406d4f0 0x7f4ce40cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:44.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.765+0000 7f4ceb9e2640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ce40cb2f0 con 0x7f4ce40ccaa0 2024-03-24T10:50:44.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.765+0000 7f4ceb9e2640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f4ce40c6750 con 0x7f4ce41543a0 2024-03-24T10:50:44.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.765+0000 7f4ceb9e2640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4ce40cc380 con 0x7f4ce406d4f0 2024-03-24T10:50:44.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.765+0000 7f4ce8f7d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ce406d4f0 0x7f4ce40cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:44.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.766+0000 7f4ce977e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4ce41543a0 0x7f4ce415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:44.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.766+0000 7f4ce8f7d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ce406d4f0 0x7f4ce40cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:55576/0 (socket says 172.21.15.5:55576) 2024-03-24T10:50:44.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.766+0000 7f4ce8f7d640 1 -- 172.21.15.5:0/3269740385 learned_addr learned my addr 172.21.15.5:0/3269740385 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:44.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.768+0000 7f4ce8f7d640 1 -- 172.21.15.5:0/3269740385 >> v2:172.21.15.5:3300/0 conn(0x7f4ce41543a0 msgr2=0x7f4ce415ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:44.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.768+0000 7f4ce8f7d640 1 --2- 172.21.15.5:0/3269740385 >> v2:172.21.15.5:3300/0 conn(0x7f4ce41543a0 0x7f4ce415ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:44.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.768+0000 7f4ce8f7d640 1 -- 172.21.15.5:0/3269740385 >> v1:172.21.15.5:6789/0 conn(0x7f4ce40ccaa0 legacy=0x7f4ce406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:44.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.768+0000 7f4ce8f7d640 1 -- 172.21.15.5:0/3269740385 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ce40ca7e0 con 0x7f4ce406d4f0 2024-03-24T10:50:44.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.770+0000 7f4ce8f7d640 1 --2- 172.21.15.5:0/3269740385 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ce406d4f0 0x7f4ce40cc560 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4009870 tx=0x7f4cd402ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0c51bcd5637a8f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:44.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.774+0000 7f4cdbfff640 1 -- 172.21.15.5:0/3269740385 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd402f840 con 0x7f4ce406d4f0 2024-03-24T10:50:44.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.774+0000 7f4cdbfff640 1 -- 172.21.15.5:0/3269740385 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cd402f9a0 con 0x7f4ce406d4f0 2024-03-24T10:50:44.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.774+0000 7f4cdbfff640 1 -- 172.21.15.5:0/3269740385 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cd402fd30 con 0x7f4ce406d4f0 2024-03-24T10:50:44.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.775+0000 7f4ceb9e2640 1 -- 172.21.15.5:0/3269740385 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ce406d4f0 msgr2=0x7f4ce40cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:44.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.775+0000 7f4ceb9e2640 1 --2- 172.21.15.5:0/3269740385 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ce406d4f0 0x7f4ce40cc560 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4009870 tx=0x7f4cd402ba20 comp rx=0 tx=0).stop 2024-03-24T10:50:44.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.775+0000 7f4ceb9e2640 1 -- 172.21.15.5:0/3269740385 shutdown_connections 2024-03-24T10:50:44.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.775+0000 7f4ceb9e2640 1 --2- 172.21.15.5:0/3269740385 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ce406d4f0 0x7f4ce40cc560 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:44.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.775+0000 7f4ceb9e2640 1 --2- 172.21.15.5:0/3269740385 >> v2:172.21.15.5:3300/0 conn(0x7f4ce41543a0 0x7f4ce415ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:44.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.775+0000 7f4ceb9e2640 1 -- 172.21.15.5:0/3269740385 >> 172.21.15.5:0/3269740385 conn(0x7f4ce4072ef0 msgr2=0x7f4ce4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:44.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.776+0000 7f4ceb9e2640 1 -- 172.21.15.5:0/3269740385 shutdown_connections 2024-03-24T10:50:44.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.776+0000 7f4ceb9e2640 1 -- 172.21.15.5:0/3269740385 wait complete. 2024-03-24T10:50:44.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.777+0000 7f4ceb9e2640 1 Processor -- start 2024-03-24T10:50:44.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.777+0000 7f4ceb9e2640 1 -- start start 2024-03-24T10:50:44.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.779+0000 7f4ceb9e2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ce406d4f0 0x7f4ce41f0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:44.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.779+0000 7f4ceb9e2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ce40ccaa0 0x7f4ce414a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:44.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.779+0000 7f4ceb9e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ce41543a0 0x7f4ce41e4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:44.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.779+0000 7f4ceb9e2640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce41e5c10 con 0x7f4ce406d4f0 2024-03-24T10:50:44.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.779+0000 7f4ceb9e2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4ce41f0a90 con 0x7f4ce41543a0 2024-03-24T10:50:44.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.779+0000 7f4ceb9e2640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4ce41e3a60 con 0x7f4ce40ccaa0 2024-03-24T10:50:44.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.779+0000 7f4ce977e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ce406d4f0 0x7f4ce41f0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:44.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.779+0000 7f4ce8f7d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ce40ccaa0 0x7f4ce414a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:44.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.779+0000 7f4ce9f7f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ce41543a0 0x7f4ce41e4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:44.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.779+0000 7f4ce977e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ce406d4f0 0x7f4ce41f0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:45408/0 (socket says 172.21.15.5:45408) 2024-03-24T10:50:44.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.779+0000 7f4ce977e640 1 -- 172.21.15.5:0/2270307713 learned_addr learned my addr 172.21.15.5:0/2270307713 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:44.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.783+0000 7f4ce977e640 1 -- 172.21.15.5:0/2270307713 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ce40ccaa0 msgr2=0x7f4ce414a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:44.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.783+0000 7f4ce977e640 1 --2- 172.21.15.5:0/2270307713 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ce40ccaa0 0x7f4ce414a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:44.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.783+0000 7f4ce977e640 1 -- 172.21.15.5:0/2270307713 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ce41543a0 msgr2=0x7f4ce41e4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:44.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.783+0000 7f4ce977e640 1 --2- 172.21.15.5:0/2270307713 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ce41543a0 0x7f4ce41e4780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:44.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.783+0000 7f4ce977e640 1 -- 172.21.15.5:0/2270307713 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd40094d0 con 0x7f4ce406d4f0 2024-03-24T10:50:44.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.786+0000 7f4ce977e640 1 --2- 172.21.15.5:0/2270307713 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ce406d4f0 0x7f4ce41f0550 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4ce000e870 tx=0x7f4ce000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:44.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.790+0000 7f4cd9ffb640 1 -- 172.21.15.5:0/2270307713 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce00094f0 con 0x7f4ce406d4f0 2024-03-24T10:50:44.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.790+0000 7f4cd9ffb640 1 -- 172.21.15.5:0/2270307713 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ce000c940 con 0x7f4ce406d4f0 2024-03-24T10:50:44.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.790+0000 7f4cd9ffb640 1 -- 172.21.15.5:0/2270307713 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ce000ccf0 con 0x7f4ce406d4f0 2024-03-24T10:50:44.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.790+0000 7f4ceb9e2640 1 -- 172.21.15.5:0/2270307713 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ce41e3d40 con 0x7f4ce406d4f0 2024-03-24T10:50:44.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.790+0000 7f4ceb9e2640 1 -- 172.21.15.5:0/2270307713 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ce4158240 con 0x7f4ce406d4f0 2024-03-24T10:50:44.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:44.791+0000 7f4ceb9e2640 1 -- 172.21.15.5:0/2270307713 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ce40681d0 con 0x7f4ce406d4f0 2024-03-24T10:50:45.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:45.071+0000 7f4cd9ffb640 1 -- 172.21.15.5:0/2270307713 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4ce0013470 con 0x7f4ce406d4f0 2024-03-24T10:50:45.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:45.071+0000 7f4cd9ffb640 1 --2- 172.21.15.5:0/2270307713 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4cc806b840 0x7f4cc806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:45.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:45.072+0000 7f4ce8f7d640 1 --2- 172.21.15.5:0/2270307713 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4cc806b840 0x7f4cc806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:45.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:45.072+0000 7f4ce8f7d640 1 --2- 172.21.15.5:0/2270307713 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4cc806b840 0x7f4cc806dd00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4036000 tx=0x7f4cd4008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:45.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:45.095+0000 7f4cd9ffb640 1 -- 172.21.15.5:0/2270307713 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f4ce0059210 con 0x7f4ce406d4f0 2024-03-24T10:50:45.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:45.095+0000 7f4cd9ffb640 1 -- 172.21.15.5:0/2270307713 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4ce00554b0 con 0x7f4ce406d4f0 2024-03-24T10:50:45.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:45.315+0000 7f4ceb9e2640 1 -- 172.21.15.5:0/2270307713 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ce40ca1b0 con 0x7f4ce406d4f0 2024-03-24T10:50:45.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:45.337+0000 7f4cd9ffb640 1 -- 172.21.15.5:0/2270307713 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f4ce0021020 con 0x7f4ce406d4f0 2024-03-24T10:50:45.338 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:45.338 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:45.338 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:45.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:45.339+0000 7f4ceb9e2640 1 -- 172.21.15.5:0/2270307713 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4cc806b840 msgr2=0x7f4cc806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:45.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:45.339+0000 7f4ceb9e2640 1 --2- 172.21.15.5:0/2270307713 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4cc806b840 0x7f4cc806dd00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4cd4036000 tx=0x7f4cd4008030 comp rx=0 tx=0).stop 2024-03-24T10:50:45.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:45.339+0000 7f4ceb9e2640 1 -- 172.21.15.5:0/2270307713 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ce406d4f0 msgr2=0x7f4ce41f0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:45.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:45.339+0000 7f4ceb9e2640 1 --2- 172.21.15.5:0/2270307713 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ce406d4f0 0x7f4ce41f0550 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4ce000e870 tx=0x7f4ce000ed30 comp rx=0 tx=0).stop 2024-03-24T10:50:45.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:45.339+0000 7f4ceb9e2640 1 -- 172.21.15.5:0/2270307713 shutdown_connections 2024-03-24T10:50:45.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:45.339+0000 7f4ceb9e2640 1 --2- 172.21.15.5:0/2270307713 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4cc806b840 0x7f4cc806dd00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:45.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:45.339+0000 7f4ceb9e2640 1 --2- 172.21.15.5:0/2270307713 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ce41543a0 0x7f4ce41e4780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:45.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:45.339+0000 7f4ceb9e2640 1 --2- 172.21.15.5:0/2270307713 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ce40ccaa0 0x7f4ce414a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:45.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:45.339+0000 7f4ceb9e2640 1 --2- 172.21.15.5:0/2270307713 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ce406d4f0 0x7f4ce41f0550 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:45.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:45.339+0000 7f4ceb9e2640 1 -- 172.21.15.5:0/2270307713 >> 172.21.15.5:0/2270307713 conn(0x7f4ce4072ef0 msgr2=0x7f4ce414f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:45.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:45.339+0000 7f4ceb9e2640 1 -- 172.21.15.5:0/2270307713 shutdown_connections 2024-03-24T10:50:45.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:45.339+0000 7f4ceb9e2640 1 -- 172.21.15.5:0/2270307713 wait complete. 2024-03-24T10:50:45.954 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:45.954 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:46.955 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:47.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.056+0000 7f62e0fb5640 1 Processor -- start 2024-03-24T10:50:47.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.056+0000 7f62e0fb5640 1 -- start start 2024-03-24T10:50:47.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.057+0000 7f62e0fb5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f62dc1543a0 0x7f62dc15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:47.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.057+0000 7f62e0fb5640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f62dc06bf50 0x7f62dc06c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:47.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.057+0000 7f62e0fb5640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62dc0cb2f0 con 0x7f62dc06d4f0 2024-03-24T10:50:47.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.057+0000 7f62e0fb5640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f62dc0c6750 con 0x7f62dc06bf50 2024-03-24T10:50:47.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.057+0000 7f62e0fb5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f62dc0cc380 con 0x7f62dc1543a0 2024-03-24T10:50:47.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.058+0000 7f62da59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f62dc1543a0 0x7f62dc15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:47.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.058+0000 7f62dad9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f62dc06bf50 0x7f62dc06c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:47.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.058+0000 7f62da59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f62dc1543a0 0x7f62dc15ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:60254/0 (socket says 172.21.15.5:60254) 2024-03-24T10:50:47.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.058+0000 7f62da59c640 1 -- 172.21.15.5:0/756138567 learned_addr learned my addr 172.21.15.5:0/756138567 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:47.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.060+0000 7f62da59c640 1 -- 172.21.15.5:0/756138567 >> v2:172.21.15.5:3300/0 conn(0x7f62dc06bf50 msgr2=0x7f62dc06c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:47.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.060+0000 7f62da59c640 1 --2- 172.21.15.5:0/756138567 >> v2:172.21.15.5:3300/0 conn(0x7f62dc06bf50 0x7f62dc06c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:47.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.060+0000 7f62da59c640 1 -- 172.21.15.5:0/756138567 >> v1:172.21.15.5:6789/0 conn(0x7f62dc06d4f0 legacy=0x7f62dc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:47.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.060+0000 7f62da59c640 1 -- 172.21.15.5:0/756138567 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62dc0ca7e0 con 0x7f62dc1543a0 2024-03-24T10:50:47.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.062+0000 7f62da59c640 1 --2- 172.21.15.5:0/756138567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62dc1543a0 0x7f62dc15ac00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0cd600 tx=0x7f62cc02ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fde55ec043cdbbb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:47.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.066+0000 7f62d9d9b640 1 -- 172.21.15.5:0/756138567 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc02f850 con 0x7f62dc1543a0 2024-03-24T10:50:47.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.067+0000 7f62d9d9b640 1 -- 172.21.15.5:0/756138567 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62cc02f9b0 con 0x7f62dc1543a0 2024-03-24T10:50:47.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.067+0000 7f62d9d9b640 1 -- 172.21.15.5:0/756138567 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc02fd40 con 0x7f62dc1543a0 2024-03-24T10:50:47.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.067+0000 7f62e0fb5640 1 -- 172.21.15.5:0/756138567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62dc1543a0 msgr2=0x7f62dc15ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:47.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.067+0000 7f62e0fb5640 1 --2- 172.21.15.5:0/756138567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62dc1543a0 0x7f62dc15ac00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0cd600 tx=0x7f62cc02ba30 comp rx=0 tx=0).stop 2024-03-24T10:50:47.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.067+0000 7f62e0fb5640 1 -- 172.21.15.5:0/756138567 shutdown_connections 2024-03-24T10:50:47.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.067+0000 7f62e0fb5640 1 --2- 172.21.15.5:0/756138567 >> v2:172.21.15.5:3300/0 conn(0x7f62dc06bf50 0x7f62dc06c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:47.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.067+0000 7f62e0fb5640 1 --2- 172.21.15.5:0/756138567 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62dc1543a0 0x7f62dc15ac00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:47.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.067+0000 7f62e0fb5640 1 -- 172.21.15.5:0/756138567 >> 172.21.15.5:0/756138567 conn(0x7f62dc072ef0 msgr2=0x7f62dc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:47.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.067+0000 7f62e0fb5640 1 -- 172.21.15.5:0/756138567 shutdown_connections 2024-03-24T10:50:47.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.067+0000 7f62e0fb5640 1 -- 172.21.15.5:0/756138567 wait complete. 2024-03-24T10:50:47.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.069+0000 7f62e0fb5640 1 Processor -- start 2024-03-24T10:50:47.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.069+0000 7f62e0fb5640 1 -- start start 2024-03-24T10:50:47.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.069+0000 7f62e0fb5640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62dc06bf50 0x7f62dc156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:47.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.069+0000 7f62e0fb5640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f62dc06d4f0 0x7f62dc14ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:47.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.069+0000 7f62e0fb5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62dc1543a0 0x7f62dc1f3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:47.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.069+0000 7f62e0fb5640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc154d40 con 0x7f62dc06bf50 2024-03-24T10:50:47.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.069+0000 7f62e0fb5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f62dc156560 con 0x7f62dc1543a0 2024-03-24T10:50:47.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.069+0000 7f62e0fb5640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f62dc14f390 con 0x7f62dc06d4f0 2024-03-24T10:50:47.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.070+0000 7f62dad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62dc1543a0 0x7f62dc1f3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:47.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.070+0000 7f62da59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62dc06bf50 0x7f62dc156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:47.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.070+0000 7f62d3fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f62dc06d4f0 0x7f62dc14ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:47.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.070+0000 7f62dad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62dc1543a0 0x7f62dc1f3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:60262/0 (socket says 172.21.15.5:60262) 2024-03-24T10:50:47.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.070+0000 7f62dad9d640 1 -- 172.21.15.5:0/4257697931 learned_addr learned my addr 172.21.15.5:0/4257697931 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:47.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.074+0000 7f62dad9d640 1 -- 172.21.15.5:0/4257697931 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f62dc06d4f0 msgr2=0x7f62dc14ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:47.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.074+0000 7f62dad9d640 1 --2- 172.21.15.5:0/4257697931 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f62dc06d4f0 0x7f62dc14ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:47.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.074+0000 7f62dad9d640 1 -- 172.21.15.5:0/4257697931 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62dc06bf50 msgr2=0x7f62dc156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:47.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.074+0000 7f62dad9d640 1 --2- 172.21.15.5:0/4257697931 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62dc06bf50 0x7f62dc156ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:47.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.074+0000 7f62dad9d640 1 -- 172.21.15.5:0/4257697931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62cc0094d0 con 0x7f62dc1543a0 2024-03-24T10:50:47.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.077+0000 7f62dad9d640 1 --2- 172.21.15.5:0/4257697931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62dc1543a0 0x7f62dc1f3130 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009dd0 tx=0x7f62d400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:47.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.081+0000 7f62d2ffd640 1 -- 172.21.15.5:0/4257697931 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d4014480 con 0x7f62dc1543a0 2024-03-24T10:50:47.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.081+0000 7f62d2ffd640 1 -- 172.21.15.5:0/4257697931 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62d400f040 con 0x7f62dc1543a0 2024-03-24T10:50:47.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.081+0000 7f62d2ffd640 1 -- 172.21.15.5:0/4257697931 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d4014840 con 0x7f62dc1543a0 2024-03-24T10:50:47.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.081+0000 7f62e0fb5640 1 -- 172.21.15.5:0/4257697931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62dc154a60 con 0x7f62dc1543a0 2024-03-24T10:50:47.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.081+0000 7f62e0fb5640 1 -- 172.21.15.5:0/4257697931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62dc157890 con 0x7f62dc1543a0 2024-03-24T10:50:47.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.082+0000 7f62e0fb5640 1 -- 172.21.15.5:0/4257697931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62dc154ef0 con 0x7f62dc1543a0 2024-03-24T10:50:47.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.349+0000 7f62d2ffd640 1 -- 172.21.15.5:0/4257697931 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f62d40149a0 con 0x7f62dc1543a0 2024-03-24T10:50:47.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.349+0000 7f62d2ffd640 1 --2- 172.21.15.5:0/4257697931 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f62bc06b810 0x7f62bc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:47.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.349+0000 7f62da59c640 1 --2- 172.21.15.5:0/4257697931 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f62bc06b810 0x7f62bc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:47.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.350+0000 7f62da59c640 1 --2- 172.21.15.5:0/4257697931 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f62bc06b810 0x7f62bc06dcd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f62cc036000 tx=0x7f62cc008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:47.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.353+0000 7f62d2ffd640 1 -- 172.21.15.5:0/4257697931 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f62d4059ea0 con 0x7f62dc1543a0 2024-03-24T10:50:47.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.366+0000 7f62d2ffd640 1 -- 172.21.15.5:0/4257697931 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f62d4056140 con 0x7f62dc1543a0 2024-03-24T10:50:47.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.613+0000 7f62e0fb5640 1 -- 172.21.15.5:0/4257697931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f62dc154ef0 con 0x7f62dc1543a0 2024-03-24T10:50:47.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.668+0000 7f62d2ffd640 1 -- 172.21.15.5:0/4257697931 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f62dc154ef0 con 0x7f62dc1543a0 2024-03-24T10:50:47.670 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:47.670 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:47.670 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:47.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.671+0000 7f62e0fb5640 1 -- 172.21.15.5:0/4257697931 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f62bc06b810 msgr2=0x7f62bc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:47.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.671+0000 7f62e0fb5640 1 --2- 172.21.15.5:0/4257697931 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f62bc06b810 0x7f62bc06dcd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f62cc036000 tx=0x7f62cc008030 comp rx=0 tx=0).stop 2024-03-24T10:50:47.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.671+0000 7f62e0fb5640 1 -- 172.21.15.5:0/4257697931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62dc1543a0 msgr2=0x7f62dc1f3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:47.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.671+0000 7f62e0fb5640 1 --2- 172.21.15.5:0/4257697931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62dc1543a0 0x7f62dc1f3130 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f62d4009dd0 tx=0x7f62d400c490 comp rx=0 tx=0).stop 2024-03-24T10:50:47.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.671+0000 7f62e0fb5640 1 -- 172.21.15.5:0/4257697931 shutdown_connections 2024-03-24T10:50:47.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.671+0000 7f62e0fb5640 1 --2- 172.21.15.5:0/4257697931 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f62bc06b810 0x7f62bc06dcd0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:47.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.671+0000 7f62e0fb5640 1 --2- 172.21.15.5:0/4257697931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f62dc1543a0 0x7f62dc1f3130 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:47.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.671+0000 7f62e0fb5640 1 --2- 172.21.15.5:0/4257697931 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f62dc06d4f0 0x7f62dc14ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:47.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.671+0000 7f62e0fb5640 1 --2- 172.21.15.5:0/4257697931 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f62dc06bf50 0x7f62dc156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:47.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.671+0000 7f62e0fb5640 1 -- 172.21.15.5:0/4257697931 >> 172.21.15.5:0/4257697931 conn(0x7f62dc072ef0 msgr2=0x7f62dc1598d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:47.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.671+0000 7f62e0fb5640 1 -- 172.21.15.5:0/4257697931 shutdown_connections 2024-03-24T10:50:47.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:47.671+0000 7f62e0fb5640 1 -- 172.21.15.5:0/4257697931 wait complete. 2024-03-24T10:50:48.287 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:48.287 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:49.288 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:49.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.390+0000 7f9e4722e640 1 Processor -- start 2024-03-24T10:50:49.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.390+0000 7f9e4722e640 1 -- start start 2024-03-24T10:50:49.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.391+0000 7f9e4722e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9e400cb270 0x7f9e400cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:49.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.391+0000 7f9e4722e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e400ccba0 con 0x7f9e400cae90 2024-03-24T10:50:49.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.391+0000 7f9e4722e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e400cce40 con 0x7f9e400cd960 2024-03-24T10:50:49.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.391+0000 7f9e4722e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f9e400cc9b0 con 0x7f9e400cb270 2024-03-24T10:50:49.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.392+0000 7f9e44fca640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f9e400cae90 0x7f9e400cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:38390/0 (socket says 172.21.15.5:38390) 2024-03-24T10:50:49.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.392+0000 7f9e44fca640 1 -- 172.21.15.5:0/2262813414 learned_addr learned my addr 172.21.15.5:0/2262813414 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:50:49.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.392+0000 7f9e3ffff640 1 --2- 172.21.15.5:0/2262813414 >> v2:172.21.15.57:3300/0 conn(0x7f9e400cb270 0x7f9e400cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.394+0000 7f9e3f7fe640 1 -- 172.21.15.5:0/2262813414 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2603060929 0 0) 0x7f9e400ccba0 con 0x7f9e400cae90 2024-03-24T10:50:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.394+0000 7f9e3f7fe640 1 -- 172.21.15.5:0/2262813414 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e30003520 con 0x7f9e400cae90 2024-03-24T10:50:49.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.397+0000 7f9e3ffff640 1 -- 172.21.15.5:0/2262813414 >> v1:172.21.15.57:6789/0 conn(0x7f9e400cd960 legacy=0x7f9e400cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:49.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.397+0000 7f9e3ffff640 1 -- 172.21.15.5:0/2262813414 >> v1:172.21.15.5:6789/0 conn(0x7f9e400cae90 legacy=0x7f9e400cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:49.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.397+0000 7f9e3ffff640 1 -- 172.21.15.5:0/2262813414 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e4015a590 con 0x7f9e400cb270 2024-03-24T10:50:49.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.403+0000 7f9e3ffff640 1 --2- 172.21.15.5:0/2262813414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e400cb270 0x7f9e400cf530 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c0097a0 tx=0x7f9e2c02b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54226b136ebf4b09 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:49.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.408+0000 7f9e3f7fe640 1 -- 172.21.15.5:0/2262813414 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c02f810 con 0x7f9e400cb270 2024-03-24T10:50:49.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.408+0000 7f9e3f7fe640 1 -- 172.21.15.5:0/2262813414 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e2c02f970 con 0x7f9e400cb270 2024-03-24T10:50:49.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.408+0000 7f9e3f7fe640 1 -- 172.21.15.5:0/2262813414 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e2c02fd00 con 0x7f9e400cb270 2024-03-24T10:50:49.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.409+0000 7f9e4722e640 1 -- 172.21.15.5:0/2262813414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e400cb270 msgr2=0x7f9e400cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:49.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.409+0000 7f9e4722e640 1 --2- 172.21.15.5:0/2262813414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e400cb270 0x7f9e400cf530 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9e2c0097a0 tx=0x7f9e2c02b9f0 comp rx=0 tx=0).stop 2024-03-24T10:50:49.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.409+0000 7f9e4722e640 1 -- 172.21.15.5:0/2262813414 shutdown_connections 2024-03-24T10:50:49.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.409+0000 7f9e4722e640 1 --2- 172.21.15.5:0/2262813414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e400cb270 0x7f9e400cf530 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:49.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.409+0000 7f9e4722e640 1 -- 172.21.15.5:0/2262813414 >> 172.21.15.5:0/2262813414 conn(0x7f9e400cfb80 msgr2=0x7f9e400cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:49.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.409+0000 7f9e4722e640 1 -- 172.21.15.5:0/2262813414 shutdown_connections 2024-03-24T10:50:49.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.409+0000 7f9e4722e640 1 -- 172.21.15.5:0/2262813414 wait complete. 2024-03-24T10:50:49.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.410+0000 7f9e4722e640 1 Processor -- start 2024-03-24T10:50:49.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.410+0000 7f9e4722e640 1 -- start start 2024-03-24T10:50:49.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.411+0000 7f9e4722e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e400cae90 0x7f9e40156610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:49.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.411+0000 7f9e4722e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e400cb270 0x7f9e4014f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:49.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.411+0000 7f9e4722e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9e400cd960 0x7f9e401f3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:49.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.411+0000 7f9e4722e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f9e400b5aa0 con 0x7f9e400cae90 2024-03-24T10:50:49.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.411+0000 7f9e4722e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9e401561f0 con 0x7f9e400cb270 2024-03-24T10:50:49.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.411+0000 7f9e4722e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f9e401570b0 con 0x7f9e400cd960 2024-03-24T10:50:49.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.412+0000 7f9e44fca640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e400cae90 0x7f9e40156610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:49.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.412+0000 7f9e3ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e400cb270 0x7f9e4014f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:49.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.412+0000 7f9e457cb640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9e400cd960 0x7f9e401f3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:49.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.412+0000 7f9e3ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e400cb270 0x7f9e4014f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:60286/0 (socket says 172.21.15.5:60286) 2024-03-24T10:50:49.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.412+0000 7f9e3ffff640 1 -- 172.21.15.5:0/518622604 learned_addr learned my addr 172.21.15.5:0/518622604 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:49.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.416+0000 7f9e44fca640 1 -- 172.21.15.5:0/518622604 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9e400cd960 msgr2=0x7f9e401f3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:49.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.416+0000 7f9e44fca640 1 --2- 172.21.15.5:0/518622604 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9e400cd960 0x7f9e401f3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:49.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.416+0000 7f9e44fca640 1 -- 172.21.15.5:0/518622604 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e400cb270 msgr2=0x7f9e4014f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:49.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.416+0000 7f9e44fca640 1 --2- 172.21.15.5:0/518622604 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e400cb270 0x7f9e4014f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:49.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.416+0000 7f9e44fca640 1 -- 172.21.15.5:0/518622604 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e2c009400 con 0x7f9e400cae90 2024-03-24T10:50:49.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.419+0000 7f9e44fca640 1 --2- 172.21.15.5:0/518622604 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e400cae90 0x7f9e40156610 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400c430 tx=0x7f9e3400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:49.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.423+0000 7f9e3d7fa640 1 -- 172.21.15.5:0/518622604 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e34005ce0 con 0x7f9e400cae90 2024-03-24T10:50:49.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.423+0000 7f9e3d7fa640 1 -- 172.21.15.5:0/518622604 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e34005e40 con 0x7f9e400cae90 2024-03-24T10:50:49.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.423+0000 7f9e4722e640 1 -- 172.21.15.5:0/518622604 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e401f0bd0 con 0x7f9e400cae90 2024-03-24T10:50:49.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.424+0000 7f9e3d7fa640 1 -- 172.21.15.5:0/518622604 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e340072f0 con 0x7f9e400cae90 2024-03-24T10:50:49.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.424+0000 7f9e4722e640 1 -- 172.21.15.5:0/518622604 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e400b57c0 con 0x7f9e400cae90 2024-03-24T10:50:49.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.424+0000 7f9e4722e640 1 -- 172.21.15.5:0/518622604 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e40159430 con 0x7f9e400cae90 2024-03-24T10:50:49.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.444+0000 7f9e3d7fa640 1 -- 172.21.15.5:0/518622604 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9e34007450 con 0x7f9e400cae90 2024-03-24T10:50:49.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.445+0000 7f9e3d7fa640 1 --2- 172.21.15.5:0/518622604 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9e2406b860 0x7f9e2406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:49.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.445+0000 7f9e3ffff640 1 --2- 172.21.15.5:0/518622604 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9e2406b860 0x7f9e2406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:49.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.446+0000 7f9e3ffff640 1 --2- 172.21.15.5:0/518622604 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9e2406b860 0x7f9e2406dd20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9e4015e7b0 tx=0x7f9e2c008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:49.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.448+0000 7f9e3d7fa640 1 -- 172.21.15.5:0/518622604 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f9e34007e40 con 0x7f9e400cae90 2024-03-24T10:50:49.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.727+0000 7f9e3d7fa640 1 -- 172.21.15.5:0/518622604 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9e340b5e10 con 0x7f9e400cae90 2024-03-24T10:50:49.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.954+0000 7f9e4722e640 1 -- 172.21.15.5:0/518622604 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e400c7a70 con 0x7f9e400cae90 2024-03-24T10:50:49.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:49.998+0000 7f9e3d7fa640 1 -- 172.21.15.5:0/518622604 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f9e34017070 con 0x7f9e400cae90 2024-03-24T10:50:50.000 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:50.000 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:50.000 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:50.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:50.000+0000 7f9e4722e640 1 -- 172.21.15.5:0/518622604 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9e2406b860 msgr2=0x7f9e2406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:50.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:50.000+0000 7f9e4722e640 1 --2- 172.21.15.5:0/518622604 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9e2406b860 0x7f9e2406dd20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9e4015e7b0 tx=0x7f9e2c008030 comp rx=0 tx=0).stop 2024-03-24T10:50:50.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:50.000+0000 7f9e4722e640 1 -- 172.21.15.5:0/518622604 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e400cae90 msgr2=0x7f9e40156610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:50.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:50.000+0000 7f9e4722e640 1 --2- 172.21.15.5:0/518622604 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e400cae90 0x7f9e40156610 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400c430 tx=0x7f9e3400c8f0 comp rx=0 tx=0).stop 2024-03-24T10:50:50.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:50.001+0000 7f9e4722e640 1 -- 172.21.15.5:0/518622604 shutdown_connections 2024-03-24T10:50:50.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:50.001+0000 7f9e4722e640 1 --2- 172.21.15.5:0/518622604 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9e2406b860 0x7f9e2406dd20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:50.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:50.001+0000 7f9e4722e640 1 --2- 172.21.15.5:0/518622604 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9e400cd960 0x7f9e401f3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:50.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:50.001+0000 7f9e4722e640 1 --2- 172.21.15.5:0/518622604 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9e400cb270 0x7f9e4014f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:50.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:50.001+0000 7f9e4722e640 1 --2- 172.21.15.5:0/518622604 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9e400cae90 0x7f9e40156610 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:50.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:50.001+0000 7f9e4722e640 1 -- 172.21.15.5:0/518622604 >> 172.21.15.5:0/518622604 conn(0x7f9e400cfb80 msgr2=0x7f9e400cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:50.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:50.001+0000 7f9e4722e640 1 -- 172.21.15.5:0/518622604 shutdown_connections 2024-03-24T10:50:50.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:50.001+0000 7f9e4722e640 1 -- 172.21.15.5:0/518622604 wait complete. 2024-03-24T10:50:50.617 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:50.617 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:51.618 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:51.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.721+0000 7f61337ef640 1 Processor -- start 2024-03-24T10:50:51.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.722+0000 7f61337ef640 1 -- start start 2024-03-24T10:50:51.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.722+0000 7f61337ef640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f612c0cae90 0x7f612c0cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:51.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.722+0000 7f61337ef640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f612c0cb270 0x7f612c0cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:51.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.722+0000 7f61337ef640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f612c0ccba0 con 0x7f612c0cd960 2024-03-24T10:50:51.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.722+0000 7f61337ef640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f612c0cce40 con 0x7f612c0cae90 2024-03-24T10:50:51.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.722+0000 7f61337ef640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f612c15a590 con 0x7f612c0cb270 2024-03-24T10:50:51.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.723+0000 7f613158b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f612c0cae90 0x7f612c0cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:51.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.723+0000 7f6130d8a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f612c0cb270 0x7f612c0cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:51.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.724+0000 7f613158b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f612c0cae90 0x7f612c0cf530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:45450/0 (socket says 172.21.15.5:45450) 2024-03-24T10:50:51.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.724+0000 7f613158b640 1 -- 172.21.15.5:0/2004044769 learned_addr learned my addr 172.21.15.5:0/2004044769 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:51.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.726+0000 7f613158b640 1 -- 172.21.15.5:0/2004044769 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f612c0cb270 msgr2=0x7f612c0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:51.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.726+0000 7f613158b640 1 --2- 172.21.15.5:0/2004044769 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f612c0cb270 0x7f612c0cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:51.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.726+0000 7f613158b640 1 -- 172.21.15.5:0/2004044769 >> v1:172.21.15.5:6789/0 conn(0x7f612c0cd960 legacy=0x7f612c0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:51.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.726+0000 7f613158b640 1 -- 172.21.15.5:0/2004044769 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f612c15a870 con 0x7f612c0cae90 2024-03-24T10:50:51.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.729+0000 7f613158b640 1 --2- 172.21.15.5:0/2004044769 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f612c0cae90 0x7f612c0cf530 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6128009870 tx=0x7f612802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbcc42fccf56a88d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:51.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.734+0000 7f6123fff640 1 -- 172.21.15.5:0/2004044769 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128030840 con 0x7f612c0cae90 2024-03-24T10:50:51.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.734+0000 7f6123fff640 1 -- 172.21.15.5:0/2004044769 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61280309a0 con 0x7f612c0cae90 2024-03-24T10:50:51.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.734+0000 7f6123fff640 1 -- 172.21.15.5:0/2004044769 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128030d30 con 0x7f612c0cae90 2024-03-24T10:50:51.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.735+0000 7f61337ef640 1 -- 172.21.15.5:0/2004044769 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f612c0cae90 msgr2=0x7f612c0cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:51.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.735+0000 7f61337ef640 1 --2- 172.21.15.5:0/2004044769 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f612c0cae90 0x7f612c0cf530 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6128009870 tx=0x7f612802b980 comp rx=0 tx=0).stop 2024-03-24T10:50:51.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.735+0000 7f61337ef640 1 -- 172.21.15.5:0/2004044769 shutdown_connections 2024-03-24T10:50:51.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.735+0000 7f61337ef640 1 --2- 172.21.15.5:0/2004044769 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f612c0cb270 0x7f612c0cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:51.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.735+0000 7f61337ef640 1 --2- 172.21.15.5:0/2004044769 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f612c0cae90 0x7f612c0cf530 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:51.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.735+0000 7f61337ef640 1 -- 172.21.15.5:0/2004044769 >> 172.21.15.5:0/2004044769 conn(0x7f612c0cfb80 msgr2=0x7f612c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:51.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.735+0000 7f61337ef640 1 -- 172.21.15.5:0/2004044769 shutdown_connections 2024-03-24T10:50:51.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.735+0000 7f61337ef640 1 -- 172.21.15.5:0/2004044769 wait complete. 2024-03-24T10:50:51.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.737+0000 7f61337ef640 1 Processor -- start 2024-03-24T10:50:51.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.737+0000 7f61337ef640 1 -- start start 2024-03-24T10:50:51.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.737+0000 7f61337ef640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f612c0cae90 0x7f612c0b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:51.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.737+0000 7f61337ef640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f612c0cb270 0x7f612c1f03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:51.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.737+0000 7f61337ef640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f612c0cd960 0x7f612c153f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:51.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.737+0000 7f61337ef640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f612c1f3b40 con 0x7f612c0cd960 2024-03-24T10:50:51.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.737+0000 7f61337ef640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f612c1f4070 con 0x7f612c0cb270 2024-03-24T10:50:51.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.737+0000 7f61337ef640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f612c15c840 con 0x7f612c0cae90 2024-03-24T10:50:51.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.738+0000 7f613158b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f612c0cae90 0x7f612c0b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:51.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.739+0000 7f613158b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f612c0cae90 0x7f612c0b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:55632/0 (socket says 172.21.15.5:55632) 2024-03-24T10:50:51.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.739+0000 7f613158b640 1 -- 172.21.15.5:0/2376620090 learned_addr learned my addr 172.21.15.5:0/2376620090 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:51.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.739+0000 7f6131d8c640 1 --2- 172.21.15.5:0/2376620090 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f612c0cd960 0x7f612c153f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:51.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.741+0000 7f6130d8a640 1 --2- 172.21.15.5:0/2376620090 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f612c0cb270 0x7f612c1f03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:51.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.744+0000 7f613158b640 1 -- 172.21.15.5:0/2376620090 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f612c0cb270 msgr2=0x7f612c1f03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:51.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.744+0000 7f613158b640 1 --2- 172.21.15.5:0/2376620090 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f612c0cb270 0x7f612c1f03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:51.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.744+0000 7f613158b640 1 -- 172.21.15.5:0/2376620090 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f612c0cd960 msgr2=0x7f612c153f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:51.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.744+0000 7f613158b640 1 --2- 172.21.15.5:0/2376620090 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f612c0cd960 0x7f612c153f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:51.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.744+0000 7f613158b640 1 -- 172.21.15.5:0/2376620090 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61280094d0 con 0x7f612c0cae90 2024-03-24T10:50:51.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.747+0000 7f613158b640 1 --2- 172.21.15.5:0/2376620090 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f612c0cae90 0x7f612c0b3f80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6128009870 tx=0x7f6128004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:51.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.754+0000 7f6121ffb640 1 -- 172.21.15.5:0/2376620090 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128039c90 con 0x7f612c0cae90 2024-03-24T10:50:51.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.754+0000 7f6121ffb640 1 -- 172.21.15.5:0/2376620090 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6128039df0 con 0x7f612c0cae90 2024-03-24T10:50:51.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.754+0000 7f6121ffb640 1 -- 172.21.15.5:0/2376620090 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128043620 con 0x7f612c0cae90 2024-03-24T10:50:51.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.754+0000 7f61337ef640 1 -- 172.21.15.5:0/2376620090 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f612c1efbc0 con 0x7f612c0cae90 2024-03-24T10:50:51.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.755+0000 7f61337ef640 1 -- 172.21.15.5:0/2376620090 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f612c0b2fa0 con 0x7f612c0cae90 2024-03-24T10:50:51.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:51.756+0000 7f61337ef640 1 -- 172.21.15.5:0/2376620090 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f612c0681d0 con 0x7f612c0cae90 2024-03-24T10:50:52.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:52.055+0000 7f6121ffb640 1 -- 172.21.15.5:0/2376620090 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6128043780 con 0x7f612c0cae90 2024-03-24T10:50:52.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:52.055+0000 7f6121ffb640 1 --2- 172.21.15.5:0/2376620090 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f611006b890 0x7f611006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:52.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:52.056+0000 7f6130d8a640 1 --2- 172.21.15.5:0/2376620090 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f611006b890 0x7f611006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:52.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:52.056+0000 7f6130d8a640 1 --2- 172.21.15.5:0/2376620090 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f611006b890 0x7f611006dd50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f612c0c32a0 tx=0x7f611c009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:52.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:52.061+0000 7f6121ffb640 1 -- 172.21.15.5:0/2376620090 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f612803e070 con 0x7f612c0cae90 2024-03-24T10:50:52.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:52.078+0000 7f6121ffb640 1 -- 172.21.15.5:0/2376620090 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6128078600 con 0x7f612c0cae90 2024-03-24T10:50:52.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:52.318+0000 7f61337ef640 1 -- 172.21.15.5:0/2376620090 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f612c0c9070 con 0x7f612c0cae90 2024-03-24T10:50:52.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:52.347+0000 7f6121ffb640 1 -- 172.21.15.5:0/2376620090 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f6128034020 con 0x7f612c0cae90 2024-03-24T10:50:52.348 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:52.348 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:52.348 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:52.350+0000 7f61337ef640 1 -- 172.21.15.5:0/2376620090 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f611006b890 msgr2=0x7f611006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:52.350+0000 7f61337ef640 1 --2- 172.21.15.5:0/2376620090 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f611006b890 0x7f611006dd50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f612c0c32a0 tx=0x7f611c009190 comp rx=0 tx=0).stop 2024-03-24T10:50:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:52.350+0000 7f61337ef640 1 -- 172.21.15.5:0/2376620090 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f612c0cae90 msgr2=0x7f612c0b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:52.350+0000 7f61337ef640 1 --2- 172.21.15.5:0/2376620090 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f612c0cae90 0x7f612c0b3f80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6128009870 tx=0x7f6128004730 comp rx=0 tx=0).stop 2024-03-24T10:50:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:52.350+0000 7f61337ef640 1 -- 172.21.15.5:0/2376620090 shutdown_connections 2024-03-24T10:50:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:52.350+0000 7f61337ef640 1 --2- 172.21.15.5:0/2376620090 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f611006b890 0x7f611006dd50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:52.350+0000 7f61337ef640 1 --2- 172.21.15.5:0/2376620090 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f612c0cd960 0x7f612c153f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:52.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:52.350+0000 7f61337ef640 1 --2- 172.21.15.5:0/2376620090 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f612c0cb270 0x7f612c1f03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:52.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:52.350+0000 7f61337ef640 1 --2- 172.21.15.5:0/2376620090 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f612c0cae90 0x7f612c0b3f80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:52.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:52.350+0000 7f61337ef640 1 -- 172.21.15.5:0/2376620090 >> 172.21.15.5:0/2376620090 conn(0x7f612c0cfb80 msgr2=0x7f612c0c6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:52.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:52.350+0000 7f61337ef640 1 -- 172.21.15.5:0/2376620090 shutdown_connections 2024-03-24T10:50:52.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:52.350+0000 7f61337ef640 1 -- 172.21.15.5:0/2376620090 wait complete. 2024-03-24T10:50:52.965 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:52.965 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:53.966 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:54.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.068+0000 7f05afe4e640 1 Processor -- start 2024-03-24T10:50:54.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.068+0000 7f05afe4e640 1 -- start start 2024-03-24T10:50:54.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.069+0000 7f05afe4e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f05a81543a0 0x7f05a815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:54.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.069+0000 7f05afe4e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f05a806d4f0 0x7f05a80cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:54.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.069+0000 7f05afe4e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f05a80ccaa0 0x7f05a806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:54.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.069+0000 7f05afe4e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f05a80c6750 con 0x7f05a806d4f0 2024-03-24T10:50:54.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.069+0000 7f05afe4e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f05a80cc380 con 0x7f05a80ccaa0 2024-03-24T10:50:54.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.069+0000 7f05afe4e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f05a80cb2f0 con 0x7f05a81543a0 2024-03-24T10:50:54.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.069+0000 7f05ae3eb640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f05a80ccaa0 0x7f05a806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:54.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.070+0000 7f05adbea640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f05a81543a0 0x7f05a815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:54.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.070+0000 7f05ae3eb640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f05a80ccaa0 0x7f05a806bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:49248/0 (socket says 172.21.15.5:49248) 2024-03-24T10:50:54.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.070+0000 7f05ae3eb640 1 -- 172.21.15.5:0/7661532 learned_addr learned my addr 172.21.15.5:0/7661532 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:54.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.071+0000 7f05ad3e9640 1 --2- 172.21.15.5:0/7661532 >> v2:172.21.15.5:3300/0 conn(0x7f05a806d4f0 0x7f05a80cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:54.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.072+0000 7f05adbea640 1 -- 172.21.15.5:0/7661532 >> v2:172.21.15.57:3300/0 conn(0x7f05a80ccaa0 msgr2=0x7f05a806bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:54.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.072+0000 7f05adbea640 1 --2- 172.21.15.5:0/7661532 >> v2:172.21.15.57:3300/0 conn(0x7f05a80ccaa0 0x7f05a806bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:54.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.072+0000 7f05adbea640 1 -- 172.21.15.5:0/7661532 >> v2:172.21.15.5:3300/0 conn(0x7f05a806d4f0 msgr2=0x7f05a80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:54.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.072+0000 7f05adbea640 1 --2- 172.21.15.5:0/7661532 >> v2:172.21.15.5:3300/0 conn(0x7f05a806d4f0 0x7f05a80cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:54.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.072+0000 7f05adbea640 1 -- 172.21.15.5:0/7661532 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a80ca7e0 con 0x7f05a81543a0 2024-03-24T10:50:54.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.074+0000 7f05adbea640 1 --2- 172.21.15.5:0/7661532 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f05a81543a0 0x7f05a815ac00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f05a806de60 tx=0x7f05a402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9b7c9d89b6cdd5c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:54.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.079+0000 7f05acbe8640 1 -- 172.21.15.5:0/7661532 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a4031830 con 0x7f05a81543a0 2024-03-24T10:50:54.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.079+0000 7f05acbe8640 1 -- 172.21.15.5:0/7661532 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05a4031990 con 0x7f05a81543a0 2024-03-24T10:50:54.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.079+0000 7f05acbe8640 1 -- 172.21.15.5:0/7661532 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a4031d20 con 0x7f05a81543a0 2024-03-24T10:50:54.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.079+0000 7f05afe4e640 1 -- 172.21.15.5:0/7661532 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f05a81543a0 msgr2=0x7f05a815ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:54.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.079+0000 7f05afe4e640 1 --2- 172.21.15.5:0/7661532 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f05a81543a0 0x7f05a815ac00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f05a806de60 tx=0x7f05a402b8d0 comp rx=0 tx=0).stop 2024-03-24T10:50:54.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.079+0000 7f05afe4e640 1 -- 172.21.15.5:0/7661532 shutdown_connections 2024-03-24T10:50:54.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.079+0000 7f05afe4e640 1 --2- 172.21.15.5:0/7661532 >> v2:172.21.15.57:3300/0 conn(0x7f05a80ccaa0 0x7f05a806bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:54.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.079+0000 7f05afe4e640 1 --2- 172.21.15.5:0/7661532 >> v2:172.21.15.5:3300/0 conn(0x7f05a806d4f0 0x7f05a80cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:54.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.079+0000 7f05afe4e640 1 --2- 172.21.15.5:0/7661532 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f05a81543a0 0x7f05a815ac00 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:54.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.079+0000 7f05afe4e640 1 -- 172.21.15.5:0/7661532 >> 172.21.15.5:0/7661532 conn(0x7f05a8072ef0 msgr2=0x7f05a8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:54.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.079+0000 7f05afe4e640 1 -- 172.21.15.5:0/7661532 shutdown_connections 2024-03-24T10:50:54.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.079+0000 7f05afe4e640 1 -- 172.21.15.5:0/7661532 wait complete. 2024-03-24T10:50:54.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.081+0000 7f05afe4e640 1 Processor -- start 2024-03-24T10:50:54.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.081+0000 7f05afe4e640 1 -- start start 2024-03-24T10:50:54.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.081+0000 7f05afe4e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f05a806d4f0 0x7f05a81f2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:54.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.081+0000 7f05afe4e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05a80ccaa0 0x7f05a8159730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:54.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.081+0000 7f05afe4e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f05a81543a0 0x7f05a8147540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:54.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.081+0000 7f05afe4e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f05a8149710 con 0x7f05a81543a0 2024-03-24T10:50:54.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.081+0000 7f05afe4e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f05a80b3740 con 0x7f05a80ccaa0 2024-03-24T10:50:54.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.081+0000 7f05afe4e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f05a80b3a20 con 0x7f05a806d4f0 2024-03-24T10:50:54.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.082+0000 7f05ae3eb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f05a81543a0 0x7f05a8147540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:54.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.082+0000 7f05adbea640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f05a806d4f0 0x7f05a81f2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:54.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.082+0000 7f05ad3e9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05a80ccaa0 0x7f05a8159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:54.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.082+0000 7f05ae3eb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f05a81543a0 0x7f05a8147540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36788/0 (socket says 172.21.15.5:36788) 2024-03-24T10:50:54.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.082+0000 7f05ae3eb640 1 -- 172.21.15.5:0/1287979598 learned_addr learned my addr 172.21.15.5:0/1287979598 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:54.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.086+0000 7f05ae3eb640 1 -- 172.21.15.5:0/1287979598 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f05a806d4f0 msgr2=0x7f05a81f2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:54.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.086+0000 7f05ae3eb640 1 --2- 172.21.15.5:0/1287979598 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f05a806d4f0 0x7f05a81f2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:54.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.086+0000 7f05ae3eb640 1 -- 172.21.15.5:0/1287979598 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05a80ccaa0 msgr2=0x7f05a8159730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:54.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.086+0000 7f05ae3eb640 1 --2- 172.21.15.5:0/1287979598 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05a80ccaa0 0x7f05a8159730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:54.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.086+0000 7f05ae3eb640 1 -- 172.21.15.5:0/1287979598 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a4009400 con 0x7f05a81543a0 2024-03-24T10:50:54.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.089+0000 7f05ae3eb640 1 --2- 172.21.15.5:0/1287979598 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f05a81543a0 0x7f05a8147540 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f05a000efc0 tx=0x7f05a000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:54.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.094+0000 7f059e7fc640 1 -- 172.21.15.5:0/1287979598 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a000cd20 con 0x7f05a81543a0 2024-03-24T10:50:54.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.094+0000 7f059e7fc640 1 -- 172.21.15.5:0/1287979598 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05a000f040 con 0x7f05a81543a0 2024-03-24T10:50:54.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.094+0000 7f05afe4e640 1 -- 172.21.15.5:0/1287979598 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05a80b3d00 con 0x7f05a81543a0 2024-03-24T10:50:54.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.094+0000 7f059e7fc640 1 -- 172.21.15.5:0/1287979598 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a0003bd0 con 0x7f05a81543a0 2024-03-24T10:50:54.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.094+0000 7f05afe4e640 1 -- 172.21.15.5:0/1287979598 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05a8149c40 con 0x7f05a81543a0 2024-03-24T10:50:54.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.095+0000 7f05afe4e640 1 -- 172.21.15.5:0/1287979598 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05a8149e20 con 0x7f05a81543a0 2024-03-24T10:50:54.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.128+0000 7f059e7fc640 1 -- 172.21.15.5:0/1287979598 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f05a0003d30 con 0x7f05a81543a0 2024-03-24T10:50:54.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.129+0000 7f059e7fc640 1 --2- 172.21.15.5:0/1287979598 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f058c06b810 0x7f058c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:54.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.129+0000 7f05adbea640 1 --2- 172.21.15.5:0/1287979598 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f058c06b810 0x7f058c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:54.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.129+0000 7f05adbea640 1 --2- 172.21.15.5:0/1287979598 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f058c06b810 0x7f058c06dcd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f05a4036000 tx=0x7f05a403c040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:54.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.134+0000 7f059e7fc640 1 -- 172.21.15.5:0/1287979598 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f05a0059d30 con 0x7f05a81543a0 2024-03-24T10:50:54.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.410+0000 7f059e7fc640 1 -- 172.21.15.5:0/1287979598 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f05a0055fd0 con 0x7f05a81543a0 2024-03-24T10:50:54.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.681+0000 7f05afe4e640 1 -- 172.21.15.5:0/1287979598 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05a80681d0 con 0x7f05a81543a0 2024-03-24T10:50:54.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.702+0000 7f059e7fc640 1 -- 172.21.15.5:0/1287979598 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f05a0059690 con 0x7f05a81543a0 2024-03-24T10:50:54.702 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:54.703 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:54.703 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.704+0000 7f05afe4e640 1 -- 172.21.15.5:0/1287979598 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f058c06b810 msgr2=0x7f058c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.704+0000 7f05afe4e640 1 --2- 172.21.15.5:0/1287979598 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f058c06b810 0x7f058c06dcd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f05a4036000 tx=0x7f05a403c040 comp rx=0 tx=0).stop 2024-03-24T10:50:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.704+0000 7f05afe4e640 1 -- 172.21.15.5:0/1287979598 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f05a81543a0 msgr2=0x7f05a8147540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.704+0000 7f05afe4e640 1 --2- 172.21.15.5:0/1287979598 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f05a81543a0 0x7f05a8147540 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f05a000efc0 tx=0x7f05a000c490 comp rx=0 tx=0).stop 2024-03-24T10:50:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.704+0000 7f05afe4e640 1 -- 172.21.15.5:0/1287979598 shutdown_connections 2024-03-24T10:50:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.704+0000 7f05afe4e640 1 --2- 172.21.15.5:0/1287979598 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f058c06b810 0x7f058c06dcd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.704+0000 7f05afe4e640 1 --2- 172.21.15.5:0/1287979598 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f05a81543a0 0x7f05a8147540 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.704+0000 7f05afe4e640 1 --2- 172.21.15.5:0/1287979598 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f05a80ccaa0 0x7f05a8159730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.704+0000 7f05afe4e640 1 --2- 172.21.15.5:0/1287979598 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f05a806d4f0 0x7f05a81f2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:54.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.704+0000 7f05afe4e640 1 -- 172.21.15.5:0/1287979598 >> 172.21.15.5:0/1287979598 conn(0x7f05a8072ef0 msgr2=0x7f05a8061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.704+0000 7f05afe4e640 1 -- 172.21.15.5:0/1287979598 shutdown_connections 2024-03-24T10:50:54.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:54.705+0000 7f05afe4e640 1 -- 172.21.15.5:0/1287979598 wait complete. 2024-03-24T10:50:55.319 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:55.319 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:56.320 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:56.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.422+0000 7f5a0d981640 1 Processor -- start 2024-03-24T10:50:56.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.422+0000 7f5a0d981640 1 -- start start 2024-03-24T10:50:56.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.423+0000 7f5a0d981640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a081543a0 0x7f5a0815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:56.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.423+0000 7f5a0d981640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5a0806bf50 0x7f5a0806c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:56.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.424+0000 7f5a0d981640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a080cb2f0 con 0x7f5a0806d4f0 2024-03-24T10:50:56.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.424+0000 7f5a0d981640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f5a080c6750 con 0x7f5a0806bf50 2024-03-24T10:50:56.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.424+0000 7f5a0d981640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f5a080cc380 con 0x7f5a081543a0 2024-03-24T10:50:56.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.424+0000 7f5a06ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a081543a0 0x7f5a0815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:56.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.424+0000 7f5a077fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5a0806bf50 0x7f5a0806c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:56.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.425+0000 7f59fffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f5a0806d4f0 0x7f5a080cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:59280/0 (socket says 172.21.15.5:59280) 2024-03-24T10:50:56.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.425+0000 7f59fffff640 1 -- 172.21.15.5:0/3543891757 learned_addr learned my addr 172.21.15.5:0/3543891757 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:50:56.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.428+0000 7f5a06ffd640 1 -- 172.21.15.5:0/3543891757 >> v2:172.21.15.57:3300/0 conn(0x7f5a0806bf50 msgr2=0x7f5a0806c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:56.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.428+0000 7f5a06ffd640 1 --2- 172.21.15.5:0/3543891757 >> v2:172.21.15.57:3300/0 conn(0x7f5a0806bf50 0x7f5a0806c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:56.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.428+0000 7f5a06ffd640 1 -- 172.21.15.5:0/3543891757 >> v1:172.21.15.57:6789/0 conn(0x7f5a0806d4f0 legacy=0x7f5a080cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:56.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.428+0000 7f5a06ffd640 1 -- 172.21.15.5:0/3543891757 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a080ca7e0 con 0x7f5a081543a0 2024-03-24T10:50:56.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.431+0000 7f5a06ffd640 1 --2- 172.21.15.5:0/3543891757 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a081543a0 0x7f5a0815ac00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f59f80097a0 tx=0x7f59f802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=237195385e3171d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:56.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.437+0000 7f5a067fc640 1 -- 172.21.15.5:0/3543891757 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8030610 con 0x7f5a081543a0 2024-03-24T10:50:56.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.437+0000 7f5a067fc640 1 -- 172.21.15.5:0/3543891757 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59f8030770 con 0x7f5a081543a0 2024-03-24T10:50:56.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.437+0000 7f5a067fc640 1 -- 172.21.15.5:0/3543891757 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8030b00 con 0x7f5a081543a0 2024-03-24T10:50:56.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.437+0000 7f5a0d981640 1 -- 172.21.15.5:0/3543891757 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a081543a0 msgr2=0x7f5a0815ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:56.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.437+0000 7f5a0d981640 1 --2- 172.21.15.5:0/3543891757 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a081543a0 0x7f5a0815ac00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f59f80097a0 tx=0x7f59f802b960 comp rx=0 tx=0).stop 2024-03-24T10:50:56.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.437+0000 7f5a0d981640 1 -- 172.21.15.5:0/3543891757 shutdown_connections 2024-03-24T10:50:56.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.437+0000 7f5a0d981640 1 --2- 172.21.15.5:0/3543891757 >> v2:172.21.15.57:3300/0 conn(0x7f5a0806bf50 0x7f5a0806c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:56.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.437+0000 7f5a0d981640 1 --2- 172.21.15.5:0/3543891757 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a081543a0 0x7f5a0815ac00 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:56.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.437+0000 7f5a0d981640 1 -- 172.21.15.5:0/3543891757 >> 172.21.15.5:0/3543891757 conn(0x7f5a08072ef0 msgr2=0x7f5a08061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:56.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.437+0000 7f5a0d981640 1 -- 172.21.15.5:0/3543891757 shutdown_connections 2024-03-24T10:50:56.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.437+0000 7f5a0d981640 1 -- 172.21.15.5:0/3543891757 wait complete. 2024-03-24T10:50:56.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.438+0000 7f5a0d981640 1 Processor -- start 2024-03-24T10:50:56.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.438+0000 7f5a0d981640 1 -- start start 2024-03-24T10:50:56.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.439+0000 7f5a0d981640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a0806bf50 0x7f5a080ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:56.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.439+0000 7f5a0d981640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a0806d4f0 0x7f5a081e4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:56.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.439+0000 7f5a0d981640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a081543a0 0x7f5a0814a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:56.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.439+0000 7f5a0d981640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f5a081e5b60 con 0x7f5a081543a0 2024-03-24T10:50:56.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.439+0000 7f5a0d981640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0814f7e0 con 0x7f5a0806d4f0 2024-03-24T10:50:56.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.439+0000 7f5a0d981640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f5a08150bf0 con 0x7f5a0806bf50 2024-03-24T10:50:56.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.440+0000 7f5a06ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a0806bf50 0x7f5a080ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:56.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.440+0000 7f59fffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a0806d4f0 0x7f5a081e4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:56.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.440+0000 7f5a077fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a081543a0 0x7f5a0814a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:56.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.440+0000 7f5a06ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a0806bf50 0x7f5a080ceb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:38022/0 (socket says 172.21.15.5:38022) 2024-03-24T10:50:56.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.440+0000 7f5a06ffd640 1 -- 172.21.15.5:0/1300282966 learned_addr learned my addr 172.21.15.5:0/1300282966 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:56.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.444+0000 7f5a077fe640 1 -- 172.21.15.5:0/1300282966 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a0806bf50 msgr2=0x7f5a080ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:56.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.444+0000 7f5a077fe640 1 --2- 172.21.15.5:0/1300282966 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a0806bf50 0x7f5a080ceb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:56.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.444+0000 7f5a077fe640 1 -- 172.21.15.5:0/1300282966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a0806d4f0 msgr2=0x7f5a081e4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:56.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.444+0000 7f5a077fe640 1 --2- 172.21.15.5:0/1300282966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a0806d4f0 0x7f5a081e4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:56.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.444+0000 7f5a077fe640 1 -- 172.21.15.5:0/1300282966 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f8009400 con 0x7f5a081543a0 2024-03-24T10:50:56.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.446+0000 7f5a077fe640 1 --2- 172.21.15.5:0/1300282966 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a081543a0 0x7f5a0814a340 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5a0000eba0 tx=0x7f5a0000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:56.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.450+0000 7f5a04ff9640 1 -- 172.21.15.5:0/1300282966 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0000cd20 con 0x7f5a081543a0 2024-03-24T10:50:56.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.450+0000 7f5a04ff9640 1 -- 172.21.15.5:0/1300282966 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a0000f040 con 0x7f5a081543a0 2024-03-24T10:50:56.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.450+0000 7f5a04ff9640 1 -- 172.21.15.5:0/1300282966 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a000146c0 con 0x7f5a081543a0 2024-03-24T10:50:56.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.450+0000 7f5a0d981640 1 -- 172.21.15.5:0/1300282966 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a0814fc00 con 0x7f5a081543a0 2024-03-24T10:50:56.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.450+0000 7f5a0d981640 1 -- 172.21.15.5:0/1300282966 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a0814ee40 con 0x7f5a081543a0 2024-03-24T10:50:56.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.451+0000 7f5a0d981640 1 -- 172.21.15.5:0/1300282966 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a0814f920 con 0x7f5a081543a0 2024-03-24T10:50:56.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.717+0000 7f5a04ff9640 1 -- 172.21.15.5:0/1300282966 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5a00014820 con 0x7f5a081543a0 2024-03-24T10:50:56.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.717+0000 7f5a04ff9640 1 --2- 172.21.15.5:0/1300282966 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f59e806b860 0x7f59e806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:56.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.717+0000 7f5a06ffd640 1 --2- 172.21.15.5:0/1300282966 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f59e806b860 0x7f59e806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:56.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.718+0000 7f5a06ffd640 1 --2- 172.21.15.5:0/1300282966 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f59e806b860 0x7f59e806dd20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f59f802b9a0 tx=0x7f59f8008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:56.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.732+0000 7f5a04ff9640 1 -- 172.21.15.5:0/1300282966 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f5a00059860 con 0x7f5a081543a0 2024-03-24T10:50:56.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.732+0000 7f5a04ff9640 1 -- 172.21.15.5:0/1300282966 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5a00055b00 con 0x7f5a081543a0 2024-03-24T10:50:56.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.963+0000 7f5a0d981640 1 -- 172.21.15.5:0/1300282966 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a0814f920 con 0x7f5a081543a0 2024-03-24T10:50:56.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.976+0000 7f5a04ff9640 1 -- 172.21.15.5:0/1300282966 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f5a0814f920 con 0x7f5a081543a0 2024-03-24T10:50:56.976 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:56.976 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:56.977 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:56.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.979+0000 7f5a0d981640 1 -- 172.21.15.5:0/1300282966 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f59e806b860 msgr2=0x7f59e806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:56.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.979+0000 7f5a0d981640 1 --2- 172.21.15.5:0/1300282966 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f59e806b860 0x7f59e806dd20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f59f802b9a0 tx=0x7f59f8008030 comp rx=0 tx=0).stop 2024-03-24T10:50:56.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.979+0000 7f5a0d981640 1 -- 172.21.15.5:0/1300282966 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a081543a0 msgr2=0x7f5a0814a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:56.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.979+0000 7f5a0d981640 1 --2- 172.21.15.5:0/1300282966 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a081543a0 0x7f5a0814a340 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5a0000eba0 tx=0x7f5a0000c490 comp rx=0 tx=0).stop 2024-03-24T10:50:56.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.979+0000 7f5a0d981640 1 -- 172.21.15.5:0/1300282966 shutdown_connections 2024-03-24T10:50:56.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.979+0000 7f5a0d981640 1 --2- 172.21.15.5:0/1300282966 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f59e806b860 0x7f59e806dd20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:56.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.979+0000 7f5a0d981640 1 --2- 172.21.15.5:0/1300282966 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a081543a0 0x7f5a0814a340 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:56.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.979+0000 7f5a0d981640 1 --2- 172.21.15.5:0/1300282966 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a0806d4f0 0x7f5a081e4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:56.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.979+0000 7f5a0d981640 1 --2- 172.21.15.5:0/1300282966 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a0806bf50 0x7f5a080ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:56.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.979+0000 7f5a0d981640 1 -- 172.21.15.5:0/1300282966 >> 172.21.15.5:0/1300282966 conn(0x7f5a08072ef0 msgr2=0x7f5a08061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:56.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.979+0000 7f5a0d981640 1 -- 172.21.15.5:0/1300282966 shutdown_connections 2024-03-24T10:50:56.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:56.979+0000 7f5a0d981640 1 -- 172.21.15.5:0/1300282966 wait complete. 2024-03-24T10:50:57.595 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:57.595 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:50:58.595 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:50:58.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.697+0000 7ff4f6eda640 1 Processor -- start 2024-03-24T10:50:58.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.697+0000 7ff4f6eda640 1 -- start start 2024-03-24T10:50:58.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.698+0000 7ff4f6eda640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4f015aea0 0x7ff4f00cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:58.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.699+0000 7ff4f6eda640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff4f00cf100 0x7ff4f00ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:58.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.699+0000 7ff4f6eda640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4f006cd50 con 0x7ff4f015aac0 2024-03-24T10:50:58.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.699+0000 7ff4f6eda640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7ff4f00ce790 con 0x7ff4f00cf100 2024-03-24T10:50:58.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.699+0000 7ff4f6eda640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff4f00cdb10 con 0x7ff4f015aea0 2024-03-24T10:50:58.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.699+0000 7ff4effff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4f015aea0 0x7ff4f00cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:58.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.700+0000 7ff4f4c76640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff4f015aac0 0x7ff4f006c230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:59294/0 (socket says 172.21.15.5:59294) 2024-03-24T10:50:58.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.700+0000 7ff4f4c76640 1 -- 172.21.15.5:0/3279509861 learned_addr learned my addr 172.21.15.5:0/3279509861 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:50:58.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.700+0000 7ff4f5477640 1 --2- 172.21.15.5:0/3279509861 >> v2:172.21.15.57:3300/0 conn(0x7ff4f00cf100 0x7ff4f00ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:58.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.703+0000 7ff4effff640 1 -- 172.21.15.5:0/3279509861 >> v2:172.21.15.57:3300/0 conn(0x7ff4f00cf100 msgr2=0x7ff4f00ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:50:58.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.703+0000 7ff4effff640 1 --2- 172.21.15.5:0/3279509861 >> v2:172.21.15.57:3300/0 conn(0x7ff4f00cf100 0x7ff4f00ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:58.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.703+0000 7ff4effff640 1 -- 172.21.15.5:0/3279509861 >> v1:172.21.15.57:6789/0 conn(0x7ff4f015aac0 legacy=0x7ff4f006c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:58.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.703+0000 7ff4effff640 1 -- 172.21.15.5:0/3279509861 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f00cddf0 con 0x7ff4f015aea0 2024-03-24T10:50:58.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.706+0000 7ff4effff640 1 --2- 172.21.15.5:0/3279509861 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4f015aea0 0x7ff4f00cebc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0097a0 tx=0x7ff4dc02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d523f1b381162381 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:58.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.712+0000 7ff4ef7fe640 1 -- 172.21.15.5:0/3279509861 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc030610 con 0x7ff4f015aea0 2024-03-24T10:50:58.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.712+0000 7ff4ef7fe640 1 -- 172.21.15.5:0/3279509861 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4dc030770 con 0x7ff4f015aea0 2024-03-24T10:50:58.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.712+0000 7ff4ef7fe640 1 -- 172.21.15.5:0/3279509861 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc030b00 con 0x7ff4f015aea0 2024-03-24T10:50:58.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.713+0000 7ff4f6eda640 1 -- 172.21.15.5:0/3279509861 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4f015aea0 msgr2=0x7ff4f00cebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:58.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.713+0000 7ff4f6eda640 1 --2- 172.21.15.5:0/3279509861 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4f015aea0 0x7ff4f00cebc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0097a0 tx=0x7ff4dc02b960 comp rx=0 tx=0).stop 2024-03-24T10:50:58.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.713+0000 7ff4f6eda640 1 -- 172.21.15.5:0/3279509861 shutdown_connections 2024-03-24T10:50:58.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.713+0000 7ff4f6eda640 1 --2- 172.21.15.5:0/3279509861 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4f015aea0 0x7ff4f00cebc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:58.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.713+0000 7ff4f6eda640 1 --2- 172.21.15.5:0/3279509861 >> v2:172.21.15.57:3300/0 conn(0x7ff4f00cf100 0x7ff4f00ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:58.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.713+0000 7ff4f6eda640 1 -- 172.21.15.5:0/3279509861 >> 172.21.15.5:0/3279509861 conn(0x7ff4f00cfb90 msgr2=0x7ff4f00cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:58.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.713+0000 7ff4f6eda640 1 -- 172.21.15.5:0/3279509861 shutdown_connections 2024-03-24T10:50:58.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.713+0000 7ff4f6eda640 1 -- 172.21.15.5:0/3279509861 wait complete. 2024-03-24T10:50:58.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.715+0000 7ff4f6eda640 1 Processor -- start 2024-03-24T10:50:58.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.715+0000 7ff4f6eda640 1 -- start start 2024-03-24T10:50:58.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.716+0000 7ff4f6eda640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4f00cf100 0x7ff4f014c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:58.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.716+0000 7ff4f6eda640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4f015aac0 0x7ff4f014e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:58.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.716+0000 7ff4f6eda640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4f015aea0 0x7ff4f0146a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:58.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.717+0000 7ff4f6eda640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f01f3d50 con 0x7ff4f015aac0 2024-03-24T10:50:58.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.717+0000 7ff4f6eda640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f014e030 con 0x7ff4f00cf100 2024-03-24T10:50:58.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.717+0000 7ff4f6eda640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff4f014dd50 con 0x7ff4f015aea0 2024-03-24T10:50:58.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.717+0000 7ff4f4c76640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4f00cf100 0x7ff4f014c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:58.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.717+0000 7ff4f5477640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4f015aea0 0x7ff4f0146a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:58.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.717+0000 7ff4effff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4f015aac0 0x7ff4f014e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:58.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.717+0000 7ff4f4c76640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4f00cf100 0x7ff4f014c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:49294/0 (socket says 172.21.15.5:49294) 2024-03-24T10:50:58.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.717+0000 7ff4f4c76640 1 -- 172.21.15.5:0/3975067587 learned_addr learned my addr 172.21.15.5:0/3975067587 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:50:58.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.721+0000 7ff4f4c76640 1 -- 172.21.15.5:0/3975067587 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4f015aea0 msgr2=0x7ff4f0146a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:58.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.721+0000 7ff4f4c76640 1 --2- 172.21.15.5:0/3975067587 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4f015aea0 0x7ff4f0146a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:58.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.721+0000 7ff4f4c76640 1 -- 172.21.15.5:0/3975067587 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4f015aac0 msgr2=0x7ff4f014e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:58.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.721+0000 7ff4f4c76640 1 --2- 172.21.15.5:0/3975067587 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4f015aac0 0x7ff4f014e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:58.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.721+0000 7ff4f4c76640 1 -- 172.21.15.5:0/3975067587 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4dc009400 con 0x7ff4f00cf100 2024-03-24T10:50:58.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.724+0000 7ff4f4c76640 1 --2- 172.21.15.5:0/3975067587 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4f00cf100 0x7ff4f014c910 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff4e400c4a0 tx=0x7ff4e400c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:58.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.728+0000 7ff4ed7fa640 1 -- 172.21.15.5:0/3975067587 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e4005ce0 con 0x7ff4f00cf100 2024-03-24T10:50:58.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.729+0000 7ff4ed7fa640 1 -- 172.21.15.5:0/3975067587 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4e4005e40 con 0x7ff4f00cf100 2024-03-24T10:50:58.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.729+0000 7ff4ed7fa640 1 -- 172.21.15.5:0/3975067587 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e4007270 con 0x7ff4f00cf100 2024-03-24T10:50:58.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.729+0000 7ff4f6eda640 1 -- 172.21.15.5:0/3975067587 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4f01f33b0 con 0x7ff4f00cf100 2024-03-24T10:50:58.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.729+0000 7ff4f6eda640 1 -- 172.21.15.5:0/3975067587 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4f01f3970 con 0x7ff4f00cf100 2024-03-24T10:50:58.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.730+0000 7ff4f6eda640 1 -- 172.21.15.5:0/3975067587 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4f01f3e90 con 0x7ff4f00cf100 2024-03-24T10:50:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.754+0000 7ff4ed7fa640 1 -- 172.21.15.5:0/3975067587 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff4e4007490 con 0x7ff4f00cf100 2024-03-24T10:50:58.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.754+0000 7ff4ed7fa640 1 --2- 172.21.15.5:0/3975067587 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff4d406b810 0x7ff4d406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:50:58.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.754+0000 7ff4effff640 1 --2- 172.21.15.5:0/3975067587 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff4d406b810 0x7ff4d406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:50:58.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.755+0000 7ff4effff640 1 --2- 172.21.15.5:0/3975067587 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff4d406b810 0x7ff4d406dcd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc036000 tx=0x7ff4dc03b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:50:58.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:58.758+0000 7ff4ed7fa640 1 -- 172.21.15.5:0/3975067587 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7ff4e4058220 con 0x7ff4f00cf100 2024-03-24T10:50:59.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:59.072+0000 7ff4ed7fa640 1 -- 172.21.15.5:0/3975067587 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff4e40544c0 con 0x7ff4f00cf100 2024-03-24T10:50:59.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:59.293+0000 7ff4f6eda640 1 -- 172.21.15.5:0/3975067587 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4f01f3e90 con 0x7ff4f00cf100 2024-03-24T10:50:59.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:59.318+0000 7ff4ed7fa640 1 -- 172.21.15.5:0/3975067587 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7ff4f01f3e90 con 0x7ff4f00cf100 2024-03-24T10:50:59.319 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:50:59.319 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:50:59.319 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:50:59.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:59.320+0000 7ff4f6eda640 1 -- 172.21.15.5:0/3975067587 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff4d406b810 msgr2=0x7ff4d406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:59.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:59.320+0000 7ff4f6eda640 1 --2- 172.21.15.5:0/3975067587 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff4d406b810 0x7ff4d406dcd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc036000 tx=0x7ff4dc03b040 comp rx=0 tx=0).stop 2024-03-24T10:50:59.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:59.320+0000 7ff4f6eda640 1 -- 172.21.15.5:0/3975067587 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4f00cf100 msgr2=0x7ff4f014c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:50:59.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:59.320+0000 7ff4f6eda640 1 --2- 172.21.15.5:0/3975067587 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4f00cf100 0x7ff4f014c910 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff4e400c4a0 tx=0x7ff4e400c960 comp rx=0 tx=0).stop 2024-03-24T10:50:59.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:59.320+0000 7ff4f6eda640 1 -- 172.21.15.5:0/3975067587 shutdown_connections 2024-03-24T10:50:59.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:59.320+0000 7ff4f6eda640 1 --2- 172.21.15.5:0/3975067587 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff4d406b810 0x7ff4d406dcd0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:59.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:59.320+0000 7ff4f6eda640 1 --2- 172.21.15.5:0/3975067587 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff4f015aac0 0x7ff4f014e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:59.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:59.320+0000 7ff4f6eda640 1 --2- 172.21.15.5:0/3975067587 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff4f015aea0 0x7ff4f0146a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:59.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:59.320+0000 7ff4f6eda640 1 --2- 172.21.15.5:0/3975067587 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4f00cf100 0x7ff4f014c910 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:50:59.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:59.321+0000 7ff4f6eda640 1 -- 172.21.15.5:0/3975067587 >> 172.21.15.5:0/3975067587 conn(0x7ff4f00cfb90 msgr2=0x7ff4f00cff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:50:59.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:59.321+0000 7ff4f6eda640 1 -- 172.21.15.5:0/3975067587 shutdown_connections 2024-03-24T10:50:59.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:50:59.321+0000 7ff4f6eda640 1 -- 172.21.15.5:0/3975067587 wait complete. 2024-03-24T10:50:59.936 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:50:59.936 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:00.937 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:01.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.040+0000 7fd4c4d8a640 1 Processor -- start 2024-03-24T10:51:01.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.040+0000 7fd4c4d8a640 1 -- start start 2024-03-24T10:51:01.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.041+0000 7fd4c4d8a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd4c01543a0 0x7fd4c015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:01.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.041+0000 7fd4c4d8a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4c006d4f0 0x7fd4c00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:01.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.041+0000 7fd4c4d8a640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd4c00ccaa0 0x7fd4c006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:01.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.041+0000 7fd4c4d8a640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fd4c00c6750 con 0x7fd4c01543a0 2024-03-24T10:51:01.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.041+0000 7fd4c4d8a640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fd4c00cc380 con 0x7fd4c00ccaa0 2024-03-24T10:51:01.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.041+0000 7fd4c4d8a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd4c00cb2f0 con 0x7fd4c006d4f0 2024-03-24T10:51:01.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.042+0000 7fd4be59c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd4c01543a0 0x7fd4c015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:01.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.042+0000 7fd4b7fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4c006d4f0 0x7fd4c00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:01.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.042+0000 7fd4bed9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd4c00ccaa0 0x7fd4c006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:01.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.042+0000 7fd4be59c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd4c01543a0 0x7fd4c015ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36818/0 (socket says 172.21.15.5:36818) 2024-03-24T10:51:01.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.042+0000 7fd4be59c640 1 -- 172.21.15.5:0/3588711752 learned_addr learned my addr 172.21.15.5:0/3588711752 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:01.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.044+0000 7fd4be59c640 1 -- 172.21.15.5:0/3588711752 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4c006d4f0 msgr2=0x7fd4c00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:01.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.044+0000 7fd4be59c640 1 --2- 172.21.15.5:0/3588711752 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4c006d4f0 0x7fd4c00cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:01.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.044+0000 7fd4be59c640 1 -- 172.21.15.5:0/3588711752 >> v2:172.21.15.57:3300/0 conn(0x7fd4c00ccaa0 msgr2=0x7fd4c006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:01.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.044+0000 7fd4be59c640 1 --2- 172.21.15.5:0/3588711752 >> v2:172.21.15.57:3300/0 conn(0x7fd4c00ccaa0 0x7fd4c006bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:01.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.044+0000 7fd4be59c640 1 -- 172.21.15.5:0/3588711752 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c00ca7e0 con 0x7fd4c01543a0 2024-03-24T10:51:01.046 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.046+0000 7fd4be59c640 1 --2- 172.21.15.5:0/3588711752 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4c01543a0 0x7fd4c015ac00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd4c00628b0 tx=0x7fd4b002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e142ef5bddd8bc25 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:01.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.051+0000 7fd4bdd9b640 1 -- 172.21.15.5:0/3588711752 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0030860 con 0x7fd4c01543a0 2024-03-24T10:51:01.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.051+0000 7fd4bdd9b640 1 -- 172.21.15.5:0/3588711752 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4b00309c0 con 0x7fd4c01543a0 2024-03-24T10:51:01.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.051+0000 7fd4bdd9b640 1 -- 172.21.15.5:0/3588711752 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0030d50 con 0x7fd4c01543a0 2024-03-24T10:51:01.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.052+0000 7fd4c4d8a640 1 -- 172.21.15.5:0/3588711752 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4c01543a0 msgr2=0x7fd4c015ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:01.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.052+0000 7fd4c4d8a640 1 --2- 172.21.15.5:0/3588711752 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4c01543a0 0x7fd4c015ac00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd4c00628b0 tx=0x7fd4b002b9a0 comp rx=0 tx=0).stop 2024-03-24T10:51:01.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.052+0000 7fd4c4d8a640 1 -- 172.21.15.5:0/3588711752 shutdown_connections 2024-03-24T10:51:01.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.052+0000 7fd4c4d8a640 1 --2- 172.21.15.5:0/3588711752 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4c006d4f0 0x7fd4c00cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:01.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.052+0000 7fd4c4d8a640 1 --2- 172.21.15.5:0/3588711752 >> v2:172.21.15.57:3300/0 conn(0x7fd4c00ccaa0 0x7fd4c006bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:01.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.052+0000 7fd4c4d8a640 1 --2- 172.21.15.5:0/3588711752 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4c01543a0 0x7fd4c015ac00 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:01.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.052+0000 7fd4c4d8a640 1 -- 172.21.15.5:0/3588711752 >> 172.21.15.5:0/3588711752 conn(0x7fd4c0072ef0 msgr2=0x7fd4c0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:01.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.052+0000 7fd4c4d8a640 1 -- 172.21.15.5:0/3588711752 shutdown_connections 2024-03-24T10:51:01.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.052+0000 7fd4c4d8a640 1 -- 172.21.15.5:0/3588711752 wait complete. 2024-03-24T10:51:01.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.053+0000 7fd4c4d8a640 1 Processor -- start 2024-03-24T10:51:01.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.053+0000 7fd4c4d8a640 1 -- start start 2024-03-24T10:51:01.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.054+0000 7fd4c4d8a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4c006d4f0 0x7fd4c01f00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:01.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.054+0000 7fd4c4d8a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4c00ccaa0 0x7fd4c00c5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:01.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.054+0000 7fd4c4d8a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4c01543a0 0x7fd4c01f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:01.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.054+0000 7fd4c4d8a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c015bc30 con 0x7fd4c006d4f0 2024-03-24T10:51:01.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.054+0000 7fd4c4d8a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c01efc50 con 0x7fd4c01543a0 2024-03-24T10:51:01.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.054+0000 7fd4c4d8a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd4c00cdd20 con 0x7fd4c00ccaa0 2024-03-24T10:51:01.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.054+0000 7fd4be59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4c006d4f0 0x7fd4c01f00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:01.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.054+0000 7fd4b7fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4c00ccaa0 0x7fd4c00c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:01.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.055+0000 7fd4bed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4c01543a0 0x7fd4c01f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:01.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.055+0000 7fd4be59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4c006d4f0 0x7fd4c01f00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36834/0 (socket says 172.21.15.5:36834) 2024-03-24T10:51:01.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.055+0000 7fd4be59c640 1 -- 172.21.15.5:0/3974040473 learned_addr learned my addr 172.21.15.5:0/3974040473 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:01.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.058+0000 7fd4be59c640 1 -- 172.21.15.5:0/3974040473 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4c00ccaa0 msgr2=0x7fd4c00c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:01.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.058+0000 7fd4be59c640 1 --2- 172.21.15.5:0/3974040473 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4c00ccaa0 0x7fd4c00c5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:01.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.058+0000 7fd4be59c640 1 -- 172.21.15.5:0/3974040473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4c01543a0 msgr2=0x7fd4c01f2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:01.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.058+0000 7fd4be59c640 1 --2- 172.21.15.5:0/3974040473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4c01543a0 0x7fd4c01f2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:01.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.058+0000 7fd4be59c640 1 -- 172.21.15.5:0/3974040473 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4b00094d0 con 0x7fd4c006d4f0 2024-03-24T10:51:01.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.061+0000 7fd4be59c640 1 --2- 172.21.15.5:0/3974040473 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4c006d4f0 0x7fd4c01f00e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0000c00 tx=0x7fd4b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:01.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.066+0000 7fd4b6ffd640 1 -- 172.21.15.5:0/3974040473 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b0039a90 con 0x7fd4c006d4f0 2024-03-24T10:51:01.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.066+0000 7fd4b6ffd640 1 -- 172.21.15.5:0/3974040473 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4b0039bf0 con 0x7fd4c006d4f0 2024-03-24T10:51:01.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.066+0000 7fd4c4d8a640 1 -- 172.21.15.5:0/3974040473 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c00cdfc0 con 0x7fd4c006d4f0 2024-03-24T10:51:01.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.066+0000 7fd4b6ffd640 1 -- 172.21.15.5:0/3974040473 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b00433f0 con 0x7fd4c006d4f0 2024-03-24T10:51:01.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.066+0000 7fd4c4d8a640 1 -- 172.21.15.5:0/3974040473 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c01ef8e0 con 0x7fd4c006d4f0 2024-03-24T10:51:01.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.067+0000 7fd4c4d8a640 1 -- 172.21.15.5:0/3974040473 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4c00681d0 con 0x7fd4c006d4f0 2024-03-24T10:51:01.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.087+0000 7fd4b6ffd640 1 -- 172.21.15.5:0/3974040473 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd4b004c3b0 con 0x7fd4c006d4f0 2024-03-24T10:51:01.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.087+0000 7fd4b6ffd640 1 --2- 172.21.15.5:0/3974040473 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd4a006b860 0x7fd4a006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:01.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.087+0000 7fd4b7fff640 1 --2- 172.21.15.5:0/3974040473 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd4a006b860 0x7fd4a006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:01.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.088+0000 7fd4b7fff640 1 --2- 172.21.15.5:0/3974040473 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd4a006b860 0x7fd4a006dd20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd4c00b23d0 tx=0x7fd4a8009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:01.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.090+0000 7fd4b6ffd640 1 -- 172.21.15.5:0/3974040473 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fd4b007b270 con 0x7fd4c006d4f0 2024-03-24T10:51:01.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.367+0000 7fd4b6ffd640 1 -- 172.21.15.5:0/3974040473 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd4b0077570 con 0x7fd4c006d4f0 2024-03-24T10:51:01.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.607+0000 7fd4c4d8a640 1 -- 172.21.15.5:0/3974040473 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd4c0152e50 con 0x7fd4c006d4f0 2024-03-24T10:51:01.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.627+0000 7fd4b6ffd640 1 -- 172.21.15.5:0/3974040473 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fd4b0048070 con 0x7fd4c006d4f0 2024-03-24T10:51:01.627 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:01.627 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:01.628 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:01.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.630+0000 7fd4c4d8a640 1 -- 172.21.15.5:0/3974040473 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd4a006b860 msgr2=0x7fd4a006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:01.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.630+0000 7fd4c4d8a640 1 --2- 172.21.15.5:0/3974040473 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd4a006b860 0x7fd4a006dd20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd4c00b23d0 tx=0x7fd4a8009190 comp rx=0 tx=0).stop 2024-03-24T10:51:01.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.630+0000 7fd4c4d8a640 1 -- 172.21.15.5:0/3974040473 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4c006d4f0 msgr2=0x7fd4c01f00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:01.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.630+0000 7fd4c4d8a640 1 --2- 172.21.15.5:0/3974040473 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4c006d4f0 0x7fd4c01f00e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd4b0000c00 tx=0x7fd4b0004730 comp rx=0 tx=0).stop 2024-03-24T10:51:01.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.630+0000 7fd4c4d8a640 1 -- 172.21.15.5:0/3974040473 shutdown_connections 2024-03-24T10:51:01.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.630+0000 7fd4c4d8a640 1 --2- 172.21.15.5:0/3974040473 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd4a006b860 0x7fd4a006dd20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:01.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.630+0000 7fd4c4d8a640 1 --2- 172.21.15.5:0/3974040473 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd4c01543a0 0x7fd4c01f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:01.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.630+0000 7fd4c4d8a640 1 --2- 172.21.15.5:0/3974040473 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd4c00ccaa0 0x7fd4c00c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:01.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.630+0000 7fd4c4d8a640 1 --2- 172.21.15.5:0/3974040473 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd4c006d4f0 0x7fd4c01f00e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:01.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.630+0000 7fd4c4d8a640 1 -- 172.21.15.5:0/3974040473 >> 172.21.15.5:0/3974040473 conn(0x7fd4c0072ef0 msgr2=0x7fd4c01532f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:01.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.630+0000 7fd4c4d8a640 1 -- 172.21.15.5:0/3974040473 shutdown_connections 2024-03-24T10:51:01.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:01.630+0000 7fd4c4d8a640 1 -- 172.21.15.5:0/3974040473 wait complete. 2024-03-24T10:51:01.913 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:51:01.916 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:51:02.245 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:02.245 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:03.246 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:03.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.350+0000 7ff9e0bc7640 1 Processor -- start 2024-03-24T10:51:03.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.350+0000 7ff9e0bc7640 1 -- start start 2024-03-24T10:51:03.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.351+0000 7ff9e0bc7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff9dc15ac00 0x7ff9dc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:03.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.351+0000 7ff9e0bc7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff9dc0ccaa0 0x7ff9dc06bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:03.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.351+0000 7ff9e0bc7640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9dc0cb2f0 con 0x7ff9dc1543a0 2024-03-24T10:51:03.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.351+0000 7ff9e0bc7640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7ff9dc0c6750 con 0x7ff9dc15ac00 2024-03-24T10:51:03.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.351+0000 7ff9e0bc7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff9dc0cceb0 con 0x7ff9dc0ccaa0 2024-03-24T10:51:03.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.352+0000 7ff9dad9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff9dc0ccaa0 0x7ff9dc06bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:03.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.352+0000 7ff9da59c640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7ff9dc1543a0 0x7ff9dc06d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:45520/0 (socket says 172.21.15.5:45520) 2024-03-24T10:51:03.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.352+0000 7ff9da59c640 1 -- 172.21.15.5:0/3802808344 learned_addr learned my addr 172.21.15.5:0/3802808344 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:51:03.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.352+0000 7ff9d9d9b640 1 --2- 172.21.15.5:0/3802808344 >> v2:172.21.15.5:3300/0 conn(0x7ff9dc15ac00 0x7ff9dc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:03.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.354+0000 7ff9dad9d640 1 -- 172.21.15.5:0/3802808344 >> v2:172.21.15.5:3300/0 conn(0x7ff9dc15ac00 msgr2=0x7ff9dc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:03.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.354+0000 7ff9dad9d640 1 --2- 172.21.15.5:0/3802808344 >> v2:172.21.15.5:3300/0 conn(0x7ff9dc15ac00 0x7ff9dc0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:03.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.354+0000 7ff9dad9d640 1 -- 172.21.15.5:0/3802808344 >> v1:172.21.15.5:6789/0 conn(0x7ff9dc1543a0 legacy=0x7ff9dc06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:03.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.354+0000 7ff9dad9d640 1 -- 172.21.15.5:0/3802808344 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9dc15b070 con 0x7ff9dc0ccaa0 2024-03-24T10:51:03.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.357+0000 7ff9dad9d640 1 --2- 172.21.15.5:0/3802808344 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff9dc0ccaa0 0x7ff9dc06bf50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4009d90 tx=0x7ff9d402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61ad9f6d14c688f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:03.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.364+0000 7ff9d959a640 1 -- 172.21.15.5:0/3802808344 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d402cc50 con 0x7ff9dc0ccaa0 2024-03-24T10:51:03.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.364+0000 7ff9d959a640 1 -- 172.21.15.5:0/3802808344 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9d402cdb0 con 0x7ff9dc0ccaa0 2024-03-24T10:51:03.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.364+0000 7ff9d959a640 1 -- 172.21.15.5:0/3802808344 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d4034660 con 0x7ff9dc0ccaa0 2024-03-24T10:51:03.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.365+0000 7ff9e0bc7640 1 -- 172.21.15.5:0/3802808344 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff9dc0ccaa0 msgr2=0x7ff9dc06bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:03.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.365+0000 7ff9e0bc7640 1 --2- 172.21.15.5:0/3802808344 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff9dc0ccaa0 0x7ff9dc06bf50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4009d90 tx=0x7ff9d402bea0 comp rx=0 tx=0).stop 2024-03-24T10:51:03.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.365+0000 7ff9e0bc7640 1 -- 172.21.15.5:0/3802808344 shutdown_connections 2024-03-24T10:51:03.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.365+0000 7ff9e0bc7640 1 --2- 172.21.15.5:0/3802808344 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff9dc0ccaa0 0x7ff9dc06bf50 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:03.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.365+0000 7ff9e0bc7640 1 --2- 172.21.15.5:0/3802808344 >> v2:172.21.15.5:3300/0 conn(0x7ff9dc15ac00 0x7ff9dc0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:03.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.365+0000 7ff9e0bc7640 1 -- 172.21.15.5:0/3802808344 >> 172.21.15.5:0/3802808344 conn(0x7ff9dc072ef0 msgr2=0x7ff9dc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:03.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.365+0000 7ff9e0bc7640 1 -- 172.21.15.5:0/3802808344 shutdown_connections 2024-03-24T10:51:03.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.365+0000 7ff9e0bc7640 1 -- 172.21.15.5:0/3802808344 wait complete. 2024-03-24T10:51:03.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.367+0000 7ff9e0bc7640 1 Processor -- start 2024-03-24T10:51:03.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.367+0000 7ff9e0bc7640 1 -- start start 2024-03-24T10:51:03.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.367+0000 7ff9e0bc7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff9dc0ccaa0 0x7ff9dc154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:03.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.367+0000 7ff9e0bc7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9dc1543a0 0x7ff9dc15db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:03.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.367+0000 7ff9e0bc7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9dc15ac00 0x7ff9dc14a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:03.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.367+0000 7ff9e0bc7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1550f0 con 0x7ff9dc15ac00 2024-03-24T10:51:03.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.367+0000 7ff9e0bc7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff9dc1efbc0 con 0x7ff9dc1543a0 2024-03-24T10:51:03.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.367+0000 7ff9e0bc7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff9dc1efe60 con 0x7ff9dc0ccaa0 2024-03-24T10:51:03.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.368+0000 7ff9dad9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9dc15ac00 0x7ff9dc14a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:03.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.368+0000 7ff9d9d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9dc1543a0 0x7ff9dc15db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:03.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.368+0000 7ff9da59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff9dc0ccaa0 0x7ff9dc154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:03.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.368+0000 7ff9d9d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9dc1543a0 0x7ff9dc15db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:53596/0 (socket says 172.21.15.5:53596) 2024-03-24T10:51:03.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.368+0000 7ff9d9d9b640 1 -- 172.21.15.5:0/2081423136 learned_addr learned my addr 172.21.15.5:0/2081423136 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:03.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.372+0000 7ff9dad9d640 1 -- 172.21.15.5:0/2081423136 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff9dc0ccaa0 msgr2=0x7ff9dc154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:03.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.372+0000 7ff9dad9d640 1 --2- 172.21.15.5:0/2081423136 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff9dc0ccaa0 0x7ff9dc154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:03.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.372+0000 7ff9dad9d640 1 -- 172.21.15.5:0/2081423136 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9dc1543a0 msgr2=0x7ff9dc15db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:03.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.372+0000 7ff9dad9d640 1 --2- 172.21.15.5:0/2081423136 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9dc1543a0 0x7ff9dc15db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:03.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.372+0000 7ff9dad9d640 1 -- 172.21.15.5:0/2081423136 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d4009a70 con 0x7ff9dc15ac00 2024-03-24T10:51:03.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.375+0000 7ff9dad9d640 1 --2- 172.21.15.5:0/2081423136 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9dc15ac00 0x7ff9dc14a3d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4033000 tx=0x7ff9d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:03.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.380+0000 7ff9caffd640 1 -- 172.21.15.5:0/2081423136 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d403e610 con 0x7ff9dc15ac00 2024-03-24T10:51:03.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.380+0000 7ff9caffd640 1 -- 172.21.15.5:0/2081423136 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9d403e7b0 con 0x7ff9dc15ac00 2024-03-24T10:51:03.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.380+0000 7ff9caffd640 1 -- 172.21.15.5:0/2081423136 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d403eb60 con 0x7ff9dc15ac00 2024-03-24T10:51:03.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.380+0000 7ff9e0bc7640 1 -- 172.21.15.5:0/2081423136 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9dc1f0140 con 0x7ff9dc15ac00 2024-03-24T10:51:03.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.380+0000 7ff9e0bc7640 1 -- 172.21.15.5:0/2081423136 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9dc1f0700 con 0x7ff9dc15ac00 2024-03-24T10:51:03.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.381+0000 7ff9e0bc7640 1 -- 172.21.15.5:0/2081423136 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9dc0681d0 con 0x7ff9dc15ac00 2024-03-24T10:51:03.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.405+0000 7ff9caffd640 1 -- 172.21.15.5:0/2081423136 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff9d403b050 con 0x7ff9dc15ac00 2024-03-24T10:51:03.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.406+0000 7ff9caffd640 1 --2- 172.21.15.5:0/2081423136 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff9bc06b810 0x7ff9bc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:03.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.406+0000 7ff9da59c640 1 --2- 172.21.15.5:0/2081423136 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff9bc06b810 0x7ff9bc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:03.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.407+0000 7ff9da59c640 1 --2- 172.21.15.5:0/2081423136 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff9bc06b810 0x7ff9bc06dcd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc007fd0 tx=0x7ff9cc007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:03.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.409+0000 7ff9caffd640 1 -- 172.21.15.5:0/2081423136 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7ff9d407b610 con 0x7ff9dc15ac00 2024-03-24T10:51:03.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.720+0000 7ff9caffd640 1 -- 172.21.15.5:0/2081423136 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff9d40778b0 con 0x7ff9dc15ac00 2024-03-24T10:51:03.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:03.950+0000 7ff9e0bc7640 1 -- 172.21.15.5:0/2081423136 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9dc15e580 con 0x7ff9dc15ac00 2024-03-24T10:51:04.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:04.005+0000 7ff9caffd640 1 -- 172.21.15.5:0/2081423136 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7ff9d4039070 con 0x7ff9dc15ac00 2024-03-24T10:51:04.006 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:04.006 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:04.006 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:04.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:04.008+0000 7ff9e0bc7640 1 -- 172.21.15.5:0/2081423136 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff9bc06b810 msgr2=0x7ff9bc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:04.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:04.008+0000 7ff9e0bc7640 1 --2- 172.21.15.5:0/2081423136 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff9bc06b810 0x7ff9bc06dcd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc007fd0 tx=0x7ff9cc007e80 comp rx=0 tx=0).stop 2024-03-24T10:51:04.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:04.008+0000 7ff9e0bc7640 1 -- 172.21.15.5:0/2081423136 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9dc15ac00 msgr2=0x7ff9dc14a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:04.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:04.008+0000 7ff9e0bc7640 1 --2- 172.21.15.5:0/2081423136 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9dc15ac00 0x7ff9dc14a3d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff9d4033000 tx=0x7ff9d4004a40 comp rx=0 tx=0).stop 2024-03-24T10:51:04.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:04.008+0000 7ff9e0bc7640 1 -- 172.21.15.5:0/2081423136 shutdown_connections 2024-03-24T10:51:04.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:04.008+0000 7ff9e0bc7640 1 --2- 172.21.15.5:0/2081423136 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff9bc06b810 0x7ff9bc06dcd0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:04.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:04.008+0000 7ff9e0bc7640 1 --2- 172.21.15.5:0/2081423136 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff9dc15ac00 0x7ff9dc14a3d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:04.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:04.008+0000 7ff9e0bc7640 1 --2- 172.21.15.5:0/2081423136 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff9dc1543a0 0x7ff9dc15db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:04.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:04.008+0000 7ff9e0bc7640 1 --2- 172.21.15.5:0/2081423136 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff9dc0ccaa0 0x7ff9dc154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:04.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:04.008+0000 7ff9e0bc7640 1 -- 172.21.15.5:0/2081423136 >> 172.21.15.5:0/2081423136 conn(0x7ff9dc072ef0 msgr2=0x7ff9dc152c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:04.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:04.008+0000 7ff9e0bc7640 1 -- 172.21.15.5:0/2081423136 shutdown_connections 2024-03-24T10:51:04.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:04.008+0000 7ff9e0bc7640 1 -- 172.21.15.5:0/2081423136 wait complete. 2024-03-24T10:51:04.623 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:04.624 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:05.625 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:05.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.728+0000 7fd2f8125640 1 Processor -- start 2024-03-24T10:51:05.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.728+0000 7fd2f8125640 1 -- start start 2024-03-24T10:51:05.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.729+0000 7fd2f8125640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd2f00cae90 0x7fd2f00cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:05.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.729+0000 7fd2f8125640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd2f00cb270 0x7fd2f00cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:05.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.729+0000 7fd2f8125640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2f00cd960 0x7fd2f00cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:05.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.729+0000 7fd2f8125640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fd2f00ccba0 con 0x7fd2f00cb270 2024-03-24T10:51:05.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.729+0000 7fd2f8125640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fd2f00cce40 con 0x7fd2f00cae90 2024-03-24T10:51:05.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.729+0000 7fd2f8125640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd2f015a590 con 0x7fd2f00cd960 2024-03-24T10:51:05.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.730+0000 7fd2f56c0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd2f00cb270 0x7fd2f00cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:05.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.730+0000 7fd2f66c2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2f00cd960 0x7fd2f00cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:05.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.730+0000 7fd2f5ec1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd2f00cae90 0x7fd2f00cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:05.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.730+0000 7fd2f56c0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd2f00cb270 0x7fd2f00cc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51464/0 (socket says 172.21.15.5:51464) 2024-03-24T10:51:05.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.730+0000 7fd2f56c0640 1 -- 172.21.15.5:0/306419945 learned_addr learned my addr 172.21.15.5:0/306419945 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:05.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.733+0000 7fd2f56c0640 1 -- 172.21.15.5:0/306419945 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2f00cd960 msgr2=0x7fd2f00cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:05.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.733+0000 7fd2f56c0640 1 --2- 172.21.15.5:0/306419945 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2f00cd960 0x7fd2f00cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:05.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.733+0000 7fd2f56c0640 1 -- 172.21.15.5:0/306419945 >> v2:172.21.15.57:3300/0 conn(0x7fd2f00cae90 msgr2=0x7fd2f00cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:05.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.733+0000 7fd2f56c0640 1 --2- 172.21.15.5:0/306419945 >> v2:172.21.15.57:3300/0 conn(0x7fd2f00cae90 0x7fd2f00cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:05.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.733+0000 7fd2f56c0640 1 -- 172.21.15.5:0/306419945 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f015a870 con 0x7fd2f00cb270 2024-03-24T10:51:05.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.735+0000 7fd2f56c0640 1 --2- 172.21.15.5:0/306419945 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2f00cb270 0x7fd2f00cc2a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd2f00b26a0 tx=0x7fd2e002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=579b2ef6bbe558dc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:05.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.737+0000 7fd2f4ebf640 1 -- 172.21.15.5:0/306419945 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0030860 con 0x7fd2f00cb270 2024-03-24T10:51:05.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.739+0000 7fd2f4ebf640 1 -- 172.21.15.5:0/306419945 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e00309c0 con 0x7fd2f00cb270 2024-03-24T10:51:05.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.739+0000 7fd2f4ebf640 1 -- 172.21.15.5:0/306419945 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0030d50 con 0x7fd2f00cb270 2024-03-24T10:51:05.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.740+0000 7fd2f8125640 1 -- 172.21.15.5:0/306419945 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2f00cb270 msgr2=0x7fd2f00cc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:05.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.740+0000 7fd2f8125640 1 --2- 172.21.15.5:0/306419945 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2f00cb270 0x7fd2f00cc2a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd2f00b26a0 tx=0x7fd2e002b9a0 comp rx=0 tx=0).stop 2024-03-24T10:51:05.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.740+0000 7fd2f8125640 1 -- 172.21.15.5:0/306419945 shutdown_connections 2024-03-24T10:51:05.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.740+0000 7fd2f8125640 1 --2- 172.21.15.5:0/306419945 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2f00cd960 0x7fd2f00cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:05.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.740+0000 7fd2f8125640 1 --2- 172.21.15.5:0/306419945 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2f00cb270 0x7fd2f00cc2a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:05.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.740+0000 7fd2f8125640 1 --2- 172.21.15.5:0/306419945 >> v2:172.21.15.57:3300/0 conn(0x7fd2f00cae90 0x7fd2f00cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:05.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.740+0000 7fd2f8125640 1 -- 172.21.15.5:0/306419945 >> 172.21.15.5:0/306419945 conn(0x7fd2f00cfb80 msgr2=0x7fd2f00cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:05.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.740+0000 7fd2f8125640 1 -- 172.21.15.5:0/306419945 shutdown_connections 2024-03-24T10:51:05.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.740+0000 7fd2f8125640 1 -- 172.21.15.5:0/306419945 wait complete. 2024-03-24T10:51:05.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.741+0000 7fd2f8125640 1 Processor -- start 2024-03-24T10:51:05.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.741+0000 7fd2f8125640 1 -- start start 2024-03-24T10:51:05.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.742+0000 7fd2f8125640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2f00cae90 0x7fd2f0156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:05.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.742+0000 7fd2f8125640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2f00cb270 0x7fd2f01f3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:05.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.742+0000 7fd2f8125640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2f00cd960 0x7fd2f0157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:05.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.742+0000 7fd2f8125640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f01f3770 con 0x7fd2f00cd960 2024-03-24T10:51:05.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.742+0000 7fd2f8125640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f01f0490 con 0x7fd2f00cae90 2024-03-24T10:51:05.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.742+0000 7fd2f8125640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd2f01f09f0 con 0x7fd2f00cb270 2024-03-24T10:51:05.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.742+0000 7fd2f66c2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2f00cd960 0x7fd2f0157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:05.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.742+0000 7fd2f56c0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2f00cb270 0x7fd2f01f3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:05.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.742+0000 7fd2f5ec1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2f00cae90 0x7fd2f0156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:05.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.742+0000 7fd2f66c2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2f00cd960 0x7fd2f0157a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51476/0 (socket says 172.21.15.5:51476) 2024-03-24T10:51:05.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.742+0000 7fd2f66c2640 1 -- 172.21.15.5:0/2261024544 learned_addr learned my addr 172.21.15.5:0/2261024544 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:05.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.746+0000 7fd2f66c2640 1 -- 172.21.15.5:0/2261024544 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2f00cb270 msgr2=0x7fd2f01f3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:05.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.746+0000 7fd2f66c2640 1 --2- 172.21.15.5:0/2261024544 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2f00cb270 0x7fd2f01f3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:05.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.746+0000 7fd2f66c2640 1 -- 172.21.15.5:0/2261024544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2f00cae90 msgr2=0x7fd2f0156e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:05.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.746+0000 7fd2f66c2640 1 --2- 172.21.15.5:0/2261024544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2f00cae90 0x7fd2f0156e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:05.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.746+0000 7fd2f66c2640 1 -- 172.21.15.5:0/2261024544 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2e00094d0 con 0x7fd2f00cd960 2024-03-24T10:51:05.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.748+0000 7fd2f66c2640 1 --2- 172.21.15.5:0/2261024544 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2f00cd960 0x7fd2f0157a30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009d30 tx=0x7fd2e800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:05.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.753+0000 7fd2e67fc640 1 -- 172.21.15.5:0/2261024544 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e8009ba0 con 0x7fd2f00cd960 2024-03-24T10:51:05.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.753+0000 7fd2e67fc640 1 -- 172.21.15.5:0/2261024544 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e800f040 con 0x7fd2f00cd960 2024-03-24T10:51:05.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.753+0000 7fd2e67fc640 1 -- 172.21.15.5:0/2261024544 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e80146c0 con 0x7fd2f00cd960 2024-03-24T10:51:05.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.753+0000 7fd2f8125640 1 -- 172.21.15.5:0/2261024544 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2f01f3160 con 0x7fd2f00cd960 2024-03-24T10:51:05.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.753+0000 7fd2f8125640 1 -- 172.21.15.5:0/2261024544 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2f01f3400 con 0x7fd2f00cd960 2024-03-24T10:51:05.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:05.754+0000 7fd2f8125640 1 -- 172.21.15.5:0/2261024544 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2f01f3920 con 0x7fd2f00cd960 2024-03-24T10:51:06.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:06.052+0000 7fd2e67fc640 1 -- 172.21.15.5:0/2261024544 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd2e8014820 con 0x7fd2f00cd960 2024-03-24T10:51:06.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:06.053+0000 7fd2e67fc640 1 --2- 172.21.15.5:0/2261024544 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd2d406b890 0x7fd2d406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:06.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:06.053+0000 7fd2f5ec1640 1 --2- 172.21.15.5:0/2261024544 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd2d406b890 0x7fd2d406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:06.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:06.053+0000 7fd2f5ec1640 1 --2- 172.21.15.5:0/2261024544 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd2d406b890 0x7fd2d406dd50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec009ac0 tx=0x7fd2ec009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:06.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:06.066+0000 7fd2e67fc640 1 -- 172.21.15.5:0/2261024544 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fd2e80599c0 con 0x7fd2f00cd960 2024-03-24T10:51:06.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:06.066+0000 7fd2e67fc640 1 -- 172.21.15.5:0/2261024544 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd2e8055c60 con 0x7fd2f00cd960 2024-03-24T10:51:06.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:06.293+0000 7fd2f8125640 1 -- 172.21.15.5:0/2261024544 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2f01f3920 con 0x7fd2f00cd960 2024-03-24T10:51:06.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:06.318+0000 7fd2e67fc640 1 -- 172.21.15.5:0/2261024544 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fd2f01f3920 con 0x7fd2f00cd960 2024-03-24T10:51:06.319 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:06.319 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:06.319 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:06.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:06.321+0000 7fd2f8125640 1 -- 172.21.15.5:0/2261024544 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd2d406b890 msgr2=0x7fd2d406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:06.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:06.321+0000 7fd2f8125640 1 --2- 172.21.15.5:0/2261024544 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd2d406b890 0x7fd2d406dd50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec009ac0 tx=0x7fd2ec009190 comp rx=0 tx=0).stop 2024-03-24T10:51:06.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:06.321+0000 7fd2f8125640 1 -- 172.21.15.5:0/2261024544 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2f00cd960 msgr2=0x7fd2f0157a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:06.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:06.321+0000 7fd2f8125640 1 --2- 172.21.15.5:0/2261024544 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2f00cd960 0x7fd2f0157a30 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8009d30 tx=0x7fd2e800c490 comp rx=0 tx=0).stop 2024-03-24T10:51:06.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:06.321+0000 7fd2f8125640 1 -- 172.21.15.5:0/2261024544 shutdown_connections 2024-03-24T10:51:06.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:06.321+0000 7fd2f8125640 1 --2- 172.21.15.5:0/2261024544 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd2d406b890 0x7fd2d406dd50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:06.321+0000 7fd2f8125640 1 --2- 172.21.15.5:0/2261024544 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2f00cd960 0x7fd2f0157a30 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:06.321+0000 7fd2f8125640 1 --2- 172.21.15.5:0/2261024544 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2f00cb270 0x7fd2f01f3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:06.321+0000 7fd2f8125640 1 --2- 172.21.15.5:0/2261024544 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2f00cae90 0x7fd2f0156e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:06.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:06.321+0000 7fd2f8125640 1 -- 172.21.15.5:0/2261024544 >> 172.21.15.5:0/2261024544 conn(0x7fd2f00cfb80 msgr2=0x7fd2f0159a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:06.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:06.321+0000 7fd2f8125640 1 -- 172.21.15.5:0/2261024544 shutdown_connections 2024-03-24T10:51:06.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:06.321+0000 7fd2f8125640 1 -- 172.21.15.5:0/2261024544 wait complete. 2024-03-24T10:51:06.937 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:06.937 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:07.938 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:08.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.040+0000 7fac9fa8f640 1 Processor -- start 2024-03-24T10:51:08.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.041+0000 7fac9fa8f640 1 -- start start 2024-03-24T10:51:08.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.041+0000 7fac9fa8f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fac981543a0 0x7fac9815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:08.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.041+0000 7fac9fa8f640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac980cb2f0 con 0x7fac9806bf50 2024-03-24T10:51:08.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.041+0000 7fac9fa8f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac980ca7e0 con 0x7fac9806d4f0 2024-03-24T10:51:08.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.041+0000 7fac9fa8f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fac980c6750 con 0x7fac981543a0 2024-03-24T10:51:08.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.042+0000 7fac9e02c640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fac9806bf50 0x7fac9806c330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:45526/0 (socket says 172.21.15.5:45526) 2024-03-24T10:51:08.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.042+0000 7fac9e02c640 1 -- 172.21.15.5:0/423994515 learned_addr learned my addr 172.21.15.5:0/423994515 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:51:08.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.043+0000 7fac9d82b640 1 --2- 172.21.15.5:0/423994515 >> v2:172.21.15.57:3300/0 conn(0x7fac981543a0 0x7fac9815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:08.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.044+0000 7fac9c829640 1 -- 172.21.15.5:0/423994515 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 796151945 0 0) 0x7fac980cb2f0 con 0x7fac9806bf50 2024-03-24T10:51:08.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.045+0000 7fac9c829640 1 -- 172.21.15.5:0/423994515 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac84003520 con 0x7fac9806bf50 2024-03-24T10:51:08.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.047+0000 7fac9d82b640 1 -- 172.21.15.5:0/423994515 >> v1:172.21.15.57:6789/0 conn(0x7fac9806d4f0 legacy=0x7fac980cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.047+0000 7fac9d82b640 1 -- 172.21.15.5:0/423994515 >> v1:172.21.15.5:6789/0 conn(0x7fac9806bf50 legacy=0x7fac9806c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.047+0000 7fac9d82b640 1 -- 172.21.15.5:0/423994515 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac980cb520 con 0x7fac981543a0 2024-03-24T10:51:08.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.054+0000 7fac9d82b640 1 --2- 172.21.15.5:0/423994515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac981543a0 0x7fac9815ac00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fac940097a0 tx=0x7fac9402b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5aca01e31d5d59fd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:08.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.059+0000 7fac9c829640 1 -- 172.21.15.5:0/423994515 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9402f810 con 0x7fac981543a0 2024-03-24T10:51:08.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.060+0000 7fac9c829640 1 -- 172.21.15.5:0/423994515 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac9402f970 con 0x7fac981543a0 2024-03-24T10:51:08.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.060+0000 7fac9c829640 1 -- 172.21.15.5:0/423994515 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9402fd00 con 0x7fac981543a0 2024-03-24T10:51:08.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.060+0000 7fac9fa8f640 1 -- 172.21.15.5:0/423994515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac981543a0 msgr2=0x7fac9815ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.060+0000 7fac9fa8f640 1 --2- 172.21.15.5:0/423994515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac981543a0 0x7fac9815ac00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fac940097a0 tx=0x7fac9402b9f0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.060+0000 7fac9fa8f640 1 -- 172.21.15.5:0/423994515 shutdown_connections 2024-03-24T10:51:08.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.060+0000 7fac9fa8f640 1 --2- 172.21.15.5:0/423994515 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac981543a0 0x7fac9815ac00 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.060+0000 7fac9fa8f640 1 -- 172.21.15.5:0/423994515 >> 172.21.15.5:0/423994515 conn(0x7fac98072ef0 msgr2=0x7fac98061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:08.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.060+0000 7fac9fa8f640 1 -- 172.21.15.5:0/423994515 shutdown_connections 2024-03-24T10:51:08.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.060+0000 7fac9fa8f640 1 -- 172.21.15.5:0/423994515 wait complete. 2024-03-24T10:51:08.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.062+0000 7fac9fa8f640 1 Processor -- start 2024-03-24T10:51:08.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.062+0000 7fac9fa8f640 1 -- start start 2024-03-24T10:51:08.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.062+0000 7fac9fa8f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac9806bf50 0x7fac981496f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:08.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.062+0000 7fac9fa8f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac9806d4f0 0x7fac98157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:08.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.062+0000 7fac9fa8f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac981543a0 0x7fac981f0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:08.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.062+0000 7fac9fa8f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fac981f0cd0 con 0x7fac981543a0 2024-03-24T10:51:08.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.062+0000 7fac9fa8f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fac98149260 con 0x7fac9806d4f0 2024-03-24T10:51:08.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.062+0000 7fac9fa8f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fac98157840 con 0x7fac9806bf50 2024-03-24T10:51:08.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.063+0000 7fac9e02c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac981543a0 0x7fac981f0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:08.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.063+0000 7fac9d02a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac9806d4f0 0x7fac98157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:08.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.063+0000 7fac9d82b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac9806bf50 0x7fac981496f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:08.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.063+0000 7fac9e02c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac981543a0 0x7fac981f0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51482/0 (socket says 172.21.15.5:51482) 2024-03-24T10:51:08.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.063+0000 7fac9e02c640 1 -- 172.21.15.5:0/3300741088 learned_addr learned my addr 172.21.15.5:0/3300741088 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:08.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.067+0000 7fac9e02c640 1 -- 172.21.15.5:0/3300741088 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac9806bf50 msgr2=0x7fac981496f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.067+0000 7fac9e02c640 1 --2- 172.21.15.5:0/3300741088 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac9806bf50 0x7fac981496f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.067+0000 7fac9e02c640 1 -- 172.21.15.5:0/3300741088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac9806d4f0 msgr2=0x7fac98157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.067+0000 7fac9e02c640 1 --2- 172.21.15.5:0/3300741088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac9806d4f0 0x7fac98157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.067+0000 7fac9e02c640 1 -- 172.21.15.5:0/3300741088 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac94009400 con 0x7fac981543a0 2024-03-24T10:51:08.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.069+0000 7fac9e02c640 1 --2- 172.21.15.5:0/3300741088 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac981543a0 0x7fac981f0790 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fac9000cb40 tx=0x7fac9000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.076+0000 7fac8e7fc640 1 -- 172.21.15.5:0/3300741088 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9000ac40 con 0x7fac981543a0 2024-03-24T10:51:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.076+0000 7fac8e7fc640 1 -- 172.21.15.5:0/3300741088 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac9000ada0 con 0x7fac981543a0 2024-03-24T10:51:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.076+0000 7fac9fa8f640 1 -- 172.21.15.5:0/3300741088 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac98159ec0 con 0x7fac981543a0 2024-03-24T10:51:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.076+0000 7fac8e7fc640 1 -- 172.21.15.5:0/3300741088 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac90012620 con 0x7fac981543a0 2024-03-24T10:51:08.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.076+0000 7fac9fa8f640 1 -- 172.21.15.5:0/3300741088 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac98148c10 con 0x7fac981543a0 2024-03-24T10:51:08.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.077+0000 7fac9fa8f640 1 -- 172.21.15.5:0/3300741088 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac98148a70 con 0x7fac981543a0 2024-03-24T10:51:08.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.363+0000 7fac8e7fc640 1 -- 172.21.15.5:0/3300741088 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fac90007070 con 0x7fac981543a0 2024-03-24T10:51:08.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.364+0000 7fac8e7fc640 1 --2- 172.21.15.5:0/3300741088 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac7c06b810 0x7fac7c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:08.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.364+0000 7fac9d82b640 1 --2- 172.21.15.5:0/3300741088 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac7c06b810 0x7fac7c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:08.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.364+0000 7fac9d82b640 1 --2- 172.21.15.5:0/3300741088 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac7c06b810 0x7fac7c06dcd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fac94036000 tx=0x7fac94008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:08.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.381+0000 7fac8e7fc640 1 -- 172.21.15.5:0/3300741088 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fac90058c60 con 0x7fac981543a0 2024-03-24T10:51:08.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.381+0000 7fac8e7fc640 1 -- 172.21.15.5:0/3300741088 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fac90054f00 con 0x7fac981543a0 2024-03-24T10:51:08.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.609+0000 7fac9fa8f640 1 -- 172.21.15.5:0/3300741088 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac98148a70 con 0x7fac981543a0 2024-03-24T10:51:08.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.622+0000 7fac8e7fc640 1 -- 172.21.15.5:0/3300741088 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fac98148a70 con 0x7fac981543a0 2024-03-24T10:51:08.623 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:08.623 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:08.623 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:08.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.624+0000 7fac9fa8f640 1 -- 172.21.15.5:0/3300741088 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac7c06b810 msgr2=0x7fac7c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.624+0000 7fac9fa8f640 1 --2- 172.21.15.5:0/3300741088 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac7c06b810 0x7fac7c06dcd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fac94036000 tx=0x7fac94008030 comp rx=0 tx=0).stop 2024-03-24T10:51:08.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.624+0000 7fac9fa8f640 1 -- 172.21.15.5:0/3300741088 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac981543a0 msgr2=0x7fac981f0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:08.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.624+0000 7fac9fa8f640 1 --2- 172.21.15.5:0/3300741088 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac981543a0 0x7fac981f0790 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fac9000cb40 tx=0x7fac9000a060 comp rx=0 tx=0).stop 2024-03-24T10:51:08.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.624+0000 7fac9fa8f640 1 -- 172.21.15.5:0/3300741088 shutdown_connections 2024-03-24T10:51:08.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.624+0000 7fac9fa8f640 1 --2- 172.21.15.5:0/3300741088 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac7c06b810 0x7fac7c06dcd0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.624+0000 7fac9fa8f640 1 --2- 172.21.15.5:0/3300741088 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac981543a0 0x7fac981f0790 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.624+0000 7fac9fa8f640 1 --2- 172.21.15.5:0/3300741088 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac9806d4f0 0x7fac98157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.624+0000 7fac9fa8f640 1 --2- 172.21.15.5:0/3300741088 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac9806bf50 0x7fac981496f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:08.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.624+0000 7fac9fa8f640 1 -- 172.21.15.5:0/3300741088 >> 172.21.15.5:0/3300741088 conn(0x7fac98072ef0 msgr2=0x7fac981525d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:08.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.624+0000 7fac9fa8f640 1 -- 172.21.15.5:0/3300741088 shutdown_connections 2024-03-24T10:51:08.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:08.624+0000 7fac9fa8f640 1 -- 172.21.15.5:0/3300741088 wait complete. 2024-03-24T10:51:09.240 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:09.240 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:10.240 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:10.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.343+0000 7f7108e07640 1 Processor -- start 2024-03-24T10:51:10.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.343+0000 7f7108e07640 1 -- start start 2024-03-24T10:51:10.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.344+0000 7f7108e07640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f710415aac0 0x7f710415aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:10.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.344+0000 7f7108e07640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f710406cd50 con 0x7f710406c670 2024-03-24T10:51:10.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.344+0000 7f7108e07640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f710406cb00 con 0x7f710406c230 2024-03-24T10:51:10.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.344+0000 7f7108e07640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f71040ce790 con 0x7f710415aac0 2024-03-24T10:51:10.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.344+0000 7f7102d9d640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f710406c670 0x7f71040ca5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:45534/0 (socket says 172.21.15.5:45534) 2024-03-24T10:51:10.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.344+0000 7f7102d9d640 1 -- 172.21.15.5:0/1124978515 learned_addr learned my addr 172.21.15.5:0/1124978515 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:51:10.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.344+0000 7f710259c640 1 --2- 172.21.15.5:0/1124978515 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f710415aac0 0x7f710415aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:10.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.346+0000 7f7101d9b640 1 -- 172.21.15.5:0/1124978515 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2872764781 0 0) 0x7f710406cb00 con 0x7f710406c230 2024-03-24T10:51:10.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.346+0000 7f7101d9b640 1 -- 172.21.15.5:0/1124978515 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70f0003520 con 0x7f710406c230 2024-03-24T10:51:10.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.347+0000 7f7101d9b640 1 -- 172.21.15.5:0/1124978515 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 788057350 0 0) 0x7f710406cd50 con 0x7f710406c670 2024-03-24T10:51:10.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.347+0000 7f7101d9b640 1 -- 172.21.15.5:0/1124978515 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f710406cb00 con 0x7f710406c670 2024-03-24T10:51:10.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.348+0000 7f710259c640 1 -- 172.21.15.5:0/1124978515 >> v1:172.21.15.57:6789/0 conn(0x7f710406c230 legacy=0x7f71040cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:10.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.348+0000 7f710259c640 1 -- 172.21.15.5:0/1124978515 >> v1:172.21.15.5:6789/0 conn(0x7f710406c670 legacy=0x7f71040ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:10.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.348+0000 7f710259c640 1 -- 172.21.15.5:0/1124978515 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71040cdb10 con 0x7f710415aac0 2024-03-24T10:51:10.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.352+0000 7f710259c640 1 --2- 172.21.15.5:0/1124978515 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f710415aac0 0x7f710415aea0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f70f40097d0 tx=0x7f70f402b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a44c86c4a71d176 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:10.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.356+0000 7f7101d9b640 1 -- 172.21.15.5:0/1124978515 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f4030880 con 0x7f710415aac0 2024-03-24T10:51:10.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.356+0000 7f7101d9b640 1 -- 172.21.15.5:0/1124978515 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70f40309e0 con 0x7f710415aac0 2024-03-24T10:51:10.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.356+0000 7f7101d9b640 1 -- 172.21.15.5:0/1124978515 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f4030d70 con 0x7f710415aac0 2024-03-24T10:51:10.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.357+0000 7f7108e07640 1 -- 172.21.15.5:0/1124978515 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f710415aac0 msgr2=0x7f710415aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:10.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.357+0000 7f7108e07640 1 --2- 172.21.15.5:0/1124978515 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f710415aac0 0x7f710415aea0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f70f40097d0 tx=0x7f70f402b9c0 comp rx=0 tx=0).stop 2024-03-24T10:51:10.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.357+0000 7f7108e07640 1 -- 172.21.15.5:0/1124978515 shutdown_connections 2024-03-24T10:51:10.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.357+0000 7f7108e07640 1 --2- 172.21.15.5:0/1124978515 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f710415aac0 0x7f710415aea0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:10.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.357+0000 7f7108e07640 1 -- 172.21.15.5:0/1124978515 >> 172.21.15.5:0/1124978515 conn(0x7f71040cfb90 msgr2=0x7f71040cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:10.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.358+0000 7f7108e07640 1 -- 172.21.15.5:0/1124978515 shutdown_connections 2024-03-24T10:51:10.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.358+0000 7f7108e07640 1 -- 172.21.15.5:0/1124978515 wait complete. 2024-03-24T10:51:10.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.360+0000 7f7108e07640 1 Processor -- start 2024-03-24T10:51:10.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.360+0000 7f7108e07640 1 -- start start 2024-03-24T10:51:10.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.361+0000 7f7108e07640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f710406c230 0x7f7104152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:10.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.361+0000 7f7108e07640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f710406c670 0x7f71040af4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:10.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.361+0000 7f7108e07640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f710415aac0 0x7f71040afa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:10.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.361+0000 7f7108e07640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f71041f3b60 con 0x7f710415aac0 2024-03-24T10:51:10.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.361+0000 7f7108e07640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f71040b0080 con 0x7f710406c670 2024-03-24T10:51:10.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.361+0000 7f7108e07640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f71041f3e00 con 0x7f710406c230 2024-03-24T10:51:10.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.361+0000 7f7102d9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f710415aac0 0x7f71040afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:10.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.362+0000 7f7102d9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f710415aac0 0x7f71040afa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51498/0 (socket says 172.21.15.5:51498) 2024-03-24T10:51:10.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.362+0000 7f7102d9d640 1 -- 172.21.15.5:0/1525981510 learned_addr learned my addr 172.21.15.5:0/1525981510 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:10.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.362+0000 7f70fbfff640 1 --2- 172.21.15.5:0/1525981510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f710406c670 0x7f71040af4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:10.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.364+0000 7f710259c640 1 --2- 172.21.15.5:0/1525981510 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f710406c230 0x7f7104152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:10.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.365+0000 7f7102d9d640 1 -- 172.21.15.5:0/1525981510 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f710406c230 msgr2=0x7f7104152310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:10.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.365+0000 7f7102d9d640 1 --2- 172.21.15.5:0/1525981510 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f710406c230 0x7f7104152310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:10.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.365+0000 7f7102d9d640 1 -- 172.21.15.5:0/1525981510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f710406c670 msgr2=0x7f71040af4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:10.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.365+0000 7f7102d9d640 1 --2- 172.21.15.5:0/1525981510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f710406c670 0x7f71040af4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:10.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.365+0000 7f7102d9d640 1 -- 172.21.15.5:0/1525981510 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70f4009400 con 0x7f710415aac0 2024-03-24T10:51:10.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.367+0000 7f7102d9d640 1 --2- 172.21.15.5:0/1525981510 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f710415aac0 0x7f71040afa30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f70fc00cb40 tx=0x7f70fc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:10.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.370+0000 7f70faffd640 1 -- 172.21.15.5:0/1525981510 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70fc00aca0 con 0x7f710415aac0 2024-03-24T10:51:10.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.370+0000 7f7108e07640 1 -- 172.21.15.5:0/1525981510 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71041f40e0 con 0x7f710415aac0 2024-03-24T10:51:10.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.370+0000 7f7108e07640 1 -- 172.21.15.5:0/1525981510 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71041f46a0 con 0x7f710415aac0 2024-03-24T10:51:10.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.371+0000 7f7108e07640 1 -- 172.21.15.5:0/1525981510 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71041f48e0 con 0x7f710415aac0 2024-03-24T10:51:10.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.375+0000 7f70faffd640 1 -- 172.21.15.5:0/1525981510 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70fc001c00 con 0x7f710415aac0 2024-03-24T10:51:10.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.375+0000 7f70faffd640 1 -- 172.21.15.5:0/1525981510 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70fc01c440 con 0x7f710415aac0 2024-03-24T10:51:10.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.653+0000 7f70faffd640 1 -- 172.21.15.5:0/1525981510 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f70fc012460 con 0x7f710415aac0 2024-03-24T10:51:10.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.654+0000 7f70faffd640 1 --2- 172.21.15.5:0/1525981510 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f70e406b810 0x7f70e406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:10.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.654+0000 7f710259c640 1 --2- 172.21.15.5:0/1525981510 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f70e406b810 0x7f70e406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:10.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.655+0000 7f710259c640 1 --2- 172.21.15.5:0/1525981510 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f70e406b810 0x7f70e406dcd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f70f4036000 tx=0x7f70f403b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:10.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.669+0000 7f70faffd640 1 -- 172.21.15.5:0/1525981510 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f70fc059030 con 0x7f710415aac0 2024-03-24T10:51:10.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.670+0000 7f70faffd640 1 -- 172.21.15.5:0/1525981510 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f70fc058e00 con 0x7f710415aac0 2024-03-24T10:51:10.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.918+0000 7f7108e07640 1 -- 172.21.15.5:0/1525981510 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71041f48e0 con 0x7f710415aac0 2024-03-24T10:51:10.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.932+0000 7f70faffd640 1 -- 172.21.15.5:0/1525981510 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f71041f48e0 con 0x7f710415aac0 2024-03-24T10:51:10.932 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:10.932 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:10.932 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:10.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.934+0000 7f7108e07640 1 -- 172.21.15.5:0/1525981510 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f70e406b810 msgr2=0x7f70e406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:10.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.934+0000 7f7108e07640 1 --2- 172.21.15.5:0/1525981510 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f70e406b810 0x7f70e406dcd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f70f4036000 tx=0x7f70f403b040 comp rx=0 tx=0).stop 2024-03-24T10:51:10.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.934+0000 7f7108e07640 1 -- 172.21.15.5:0/1525981510 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f710415aac0 msgr2=0x7f71040afa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:10.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.934+0000 7f7108e07640 1 --2- 172.21.15.5:0/1525981510 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f710415aac0 0x7f71040afa30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f70fc00cb40 tx=0x7f70fc00a060 comp rx=0 tx=0).stop 2024-03-24T10:51:10.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.934+0000 7f7108e07640 1 -- 172.21.15.5:0/1525981510 shutdown_connections 2024-03-24T10:51:10.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.934+0000 7f7108e07640 1 --2- 172.21.15.5:0/1525981510 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f70e406b810 0x7f70e406dcd0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:10.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.934+0000 7f7108e07640 1 --2- 172.21.15.5:0/1525981510 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f710415aac0 0x7f71040afa30 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:10.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.934+0000 7f7108e07640 1 --2- 172.21.15.5:0/1525981510 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f710406c670 0x7f71040af4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:10.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.934+0000 7f7108e07640 1 --2- 172.21.15.5:0/1525981510 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f710406c230 0x7f7104152310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:10.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.934+0000 7f7108e07640 1 -- 172.21.15.5:0/1525981510 >> 172.21.15.5:0/1525981510 conn(0x7f71040cfb90 msgr2=0x7f7104159200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:10.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.934+0000 7f7108e07640 1 -- 172.21.15.5:0/1525981510 shutdown_connections 2024-03-24T10:51:10.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:10.934+0000 7f7108e07640 1 -- 172.21.15.5:0/1525981510 wait complete. 2024-03-24T10:51:11.550 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:11.550 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:12.551 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:12.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.653+0000 7f6e0ddfd640 1 Processor -- start 2024-03-24T10:51:12.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.653+0000 7f6e0ddfd640 1 -- start start 2024-03-24T10:51:12.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.654+0000 7f6e0ddfd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6e0806bf50 0x7f6e0806c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:12.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.654+0000 7f6e0ddfd640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e080cb2f0 con 0x7f6e0815ac00 2024-03-24T10:51:12.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.654+0000 7f6e0ddfd640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e0815b070 con 0x7f6e081543a0 2024-03-24T10:51:12.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.654+0000 7f6e0ddfd640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f6e080c6750 con 0x7f6e0806bf50 2024-03-24T10:51:12.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.655+0000 7f6e06ffd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f6e0815ac00 0x7f6e080cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:52120/0 (socket says 172.21.15.5:52120) 2024-03-24T10:51:12.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.655+0000 7f6e06ffd640 1 -- 172.21.15.5:0/1264926832 learned_addr learned my addr 172.21.15.5:0/1264926832 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:51:12.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.655+0000 7f6e07fff640 1 --2- 172.21.15.5:0/1264926832 >> v2:172.21.15.57:3300/0 conn(0x7f6e0806bf50 0x7f6e0806c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:12.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.657+0000 7f6e067fc640 1 -- 172.21.15.5:0/1264926832 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1234909803 0 0) 0x7f6e080cb2f0 con 0x7f6e0815ac00 2024-03-24T10:51:12.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.657+0000 7f6e067fc640 1 -- 172.21.15.5:0/1264926832 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6df0003520 con 0x7f6e0815ac00 2024-03-24T10:51:12.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.661+0000 7f6e07fff640 1 -- 172.21.15.5:0/1264926832 >> v1:172.21.15.57:6789/0 conn(0x7f6e081543a0 legacy=0x7f6e0806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:12.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.661+0000 7f6e07fff640 1 -- 172.21.15.5:0/1264926832 >> v1:172.21.15.5:6789/0 conn(0x7f6e0815ac00 legacy=0x7f6e080cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:12.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.661+0000 7f6e07fff640 1 -- 172.21.15.5:0/1264926832 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e080ca7e0 con 0x7f6e0806bf50 2024-03-24T10:51:12.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.665+0000 7f6e07fff640 1 --2- 172.21.15.5:0/1264926832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0806bf50 0x7f6e0806c330 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6e000097a0 tx=0x7f6e0002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a0390f718385372 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:12.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.670+0000 7f6e067fc640 1 -- 172.21.15.5:0/1264926832 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e00030610 con 0x7f6e0806bf50 2024-03-24T10:51:12.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.670+0000 7f6e067fc640 1 -- 172.21.15.5:0/1264926832 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e00030770 con 0x7f6e0806bf50 2024-03-24T10:51:12.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.670+0000 7f6e067fc640 1 -- 172.21.15.5:0/1264926832 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e00030b00 con 0x7f6e0806bf50 2024-03-24T10:51:12.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.670+0000 7f6e0ddfd640 1 -- 172.21.15.5:0/1264926832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0806bf50 msgr2=0x7f6e0806c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:12.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.670+0000 7f6e0ddfd640 1 --2- 172.21.15.5:0/1264926832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0806bf50 0x7f6e0806c330 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6e000097a0 tx=0x7f6e0002b960 comp rx=0 tx=0).stop 2024-03-24T10:51:12.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.670+0000 7f6e0ddfd640 1 -- 172.21.15.5:0/1264926832 shutdown_connections 2024-03-24T10:51:12.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.670+0000 7f6e0ddfd640 1 --2- 172.21.15.5:0/1264926832 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0806bf50 0x7f6e0806c330 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:12.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.670+0000 7f6e0ddfd640 1 -- 172.21.15.5:0/1264926832 >> 172.21.15.5:0/1264926832 conn(0x7f6e08072ef0 msgr2=0x7f6e08061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:12.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.671+0000 7f6e0ddfd640 1 -- 172.21.15.5:0/1264926832 shutdown_connections 2024-03-24T10:51:12.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.671+0000 7f6e0ddfd640 1 -- 172.21.15.5:0/1264926832 wait complete. 2024-03-24T10:51:12.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.672+0000 7f6e0ddfd640 1 Processor -- start 2024-03-24T10:51:12.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.672+0000 7f6e0ddfd640 1 -- start start 2024-03-24T10:51:12.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.673+0000 7f6e0ddfd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e0806bf50 0x7f6e08154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:12.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.673+0000 7f6e0ddfd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e081543a0 0x7f6e081581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:12.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.673+0000 7f6e0ddfd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0815ac00 0x7f6e080b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:12.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.673+0000 7f6e0ddfd640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f6e08159080 con 0x7f6e081543a0 2024-03-24T10:51:12.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.673+0000 7f6e0ddfd640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6e080cf0d0 con 0x7f6e0815ac00 2024-03-24T10:51:12.673 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.673+0000 7f6e0ddfd640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f6e080ceac0 con 0x7f6e0806bf50 2024-03-24T10:51:12.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.673+0000 7f6e07fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0815ac00 0x7f6e080b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:12.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.673+0000 7f6e06ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e081543a0 0x7f6e081581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:12.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.673+0000 7f6e07fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0815ac00 0x7f6e080b45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:48588/0 (socket says 172.21.15.5:48588) 2024-03-24T10:51:12.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.673+0000 7f6e07fff640 1 -- 172.21.15.5:0/1435541403 learned_addr learned my addr 172.21.15.5:0/1435541403 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:12.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.673+0000 7f6e077fe640 1 --2- 172.21.15.5:0/1435541403 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e0806bf50 0x7f6e08154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:12.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.677+0000 7f6e07fff640 1 -- 172.21.15.5:0/1435541403 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e0806bf50 msgr2=0x7f6e08154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:12.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.677+0000 7f6e07fff640 1 --2- 172.21.15.5:0/1435541403 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e0806bf50 0x7f6e08154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:12.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.677+0000 7f6e07fff640 1 -- 172.21.15.5:0/1435541403 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e081543a0 msgr2=0x7f6e081581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:12.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.677+0000 7f6e07fff640 1 --2- 172.21.15.5:0/1435541403 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e081543a0 0x7f6e081581d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:12.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.677+0000 7f6e07fff640 1 -- 172.21.15.5:0/1435541403 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e00009400 con 0x7f6e0815ac00 2024-03-24T10:51:12.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.680+0000 7f6e07fff640 1 --2- 172.21.15.5:0/1435541403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0815ac00 0x7f6e080b45e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6e000098c0 tx=0x7f6e00004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:12.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.684+0000 7f6deffff640 1 -- 172.21.15.5:0/1435541403 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e0003e040 con 0x7f6e0815ac00 2024-03-24T10:51:12.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.684+0000 7f6deffff640 1 -- 172.21.15.5:0/1435541403 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e00039c10 con 0x7f6e0815ac00 2024-03-24T10:51:12.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.684+0000 7f6e0ddfd640 1 -- 172.21.15.5:0/1435541403 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e08158ac0 con 0x7f6e0815ac00 2024-03-24T10:51:12.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.684+0000 7f6deffff640 1 -- 172.21.15.5:0/1435541403 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e00030e10 con 0x7f6e0815ac00 2024-03-24T10:51:12.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.684+0000 7f6e0ddfd640 1 -- 172.21.15.5:0/1435541403 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e080ced60 con 0x7f6e0815ac00 2024-03-24T10:51:12.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.685+0000 7f6e0ddfd640 1 -- 172.21.15.5:0/1435541403 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e080ce920 con 0x7f6e0815ac00 2024-03-24T10:51:12.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.952+0000 7f6deffff640 1 -- 172.21.15.5:0/1435541403 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6e0004d3b0 con 0x7f6e0815ac00 2024-03-24T10:51:12.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.952+0000 7f6deffff640 1 --2- 172.21.15.5:0/1435541403 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6df406b810 0x7f6df406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:12.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.952+0000 7f6e077fe640 1 --2- 172.21.15.5:0/1435541403 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6df406b810 0x7f6df406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:12.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.953+0000 7f6e077fe640 1 --2- 172.21.15.5:0/1435541403 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6df406b810 0x7f6df406dcd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6df8001f10 tx=0x7f6df8009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:12.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.957+0000 7f6deffff640 1 -- 172.21.15.5:0/1435541403 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f6e0007b180 con 0x7f6e0815ac00 2024-03-24T10:51:12.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:12.969+0000 7f6deffff640 1 -- 172.21.15.5:0/1435541403 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6e00077420 con 0x7f6e0815ac00 2024-03-24T10:51:13.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:13.218+0000 7f6e0ddfd640 1 -- 172.21.15.5:0/1435541403 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e080ce920 con 0x7f6e0815ac00 2024-03-24T10:51:13.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:13.242+0000 7f6deffff640 1 -- 172.21.15.5:0/1435541403 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f6e080ce920 con 0x7f6e0815ac00 2024-03-24T10:51:13.242 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:13.242 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:13.243 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:13.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:13.245+0000 7f6e0ddfd640 1 -- 172.21.15.5:0/1435541403 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6df406b810 msgr2=0x7f6df406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:13.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:13.245+0000 7f6e0ddfd640 1 --2- 172.21.15.5:0/1435541403 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6df406b810 0x7f6df406dcd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6df8001f10 tx=0x7f6df8009040 comp rx=0 tx=0).stop 2024-03-24T10:51:13.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:13.245+0000 7f6e0ddfd640 1 -- 172.21.15.5:0/1435541403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0815ac00 msgr2=0x7f6e080b45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:13.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:13.245+0000 7f6e0ddfd640 1 --2- 172.21.15.5:0/1435541403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0815ac00 0x7f6e080b45e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6e000098c0 tx=0x7f6e00004730 comp rx=0 tx=0).stop 2024-03-24T10:51:13.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:13.245+0000 7f6e0ddfd640 1 -- 172.21.15.5:0/1435541403 shutdown_connections 2024-03-24T10:51:13.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:13.245+0000 7f6e0ddfd640 1 --2- 172.21.15.5:0/1435541403 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6df406b810 0x7f6df406dcd0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:13.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:13.245+0000 7f6e0ddfd640 1 --2- 172.21.15.5:0/1435541403 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6e081543a0 0x7f6e081581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:13.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:13.245+0000 7f6e0ddfd640 1 --2- 172.21.15.5:0/1435541403 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6e0815ac00 0x7f6e080b45e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:13.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:13.245+0000 7f6e0ddfd640 1 --2- 172.21.15.5:0/1435541403 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6e0806bf50 0x7f6e08154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:13.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:13.245+0000 7f6e0ddfd640 1 -- 172.21.15.5:0/1435541403 >> 172.21.15.5:0/1435541403 conn(0x7f6e08072ef0 msgr2=0x7f6e080cc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:13.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:13.245+0000 7f6e0ddfd640 1 -- 172.21.15.5:0/1435541403 shutdown_connections 2024-03-24T10:51:13.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:13.245+0000 7f6e0ddfd640 1 -- 172.21.15.5:0/1435541403 wait complete. 2024-03-24T10:51:13.860 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:13.860 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:14.861 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:14.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.962+0000 7efec0cf4640 1 Processor -- start 2024-03-24T10:51:14.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.962+0000 7efec0cf4640 1 -- start start 2024-03-24T10:51:14.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.963+0000 7efec0cf4640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7efebc06bf50 0x7efebc06c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:14.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.963+0000 7efec0cf4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efebc0cb2f0 con 0x7efebc1543a0 2024-03-24T10:51:14.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.963+0000 7efec0cf4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efebc15b070 con 0x7efebc15ac00 2024-03-24T10:51:14.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.963+0000 7efec0cf4640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7efebc0c6750 con 0x7efebc06bf50 2024-03-24T10:51:14.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.964+0000 7efebad9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7efebc06bf50 0x7efebc06c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:14.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.964+0000 7efeb9d9b640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7efebc15ac00 0x7efebc0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:50732/0 (socket says 172.21.15.5:50732) 2024-03-24T10:51:14.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.964+0000 7efeb9d9b640 1 -- 172.21.15.5:0/101843531 learned_addr learned my addr 172.21.15.5:0/101843531 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:51:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.967+0000 7efeb959a640 1 -- 172.21.15.5:0/101843531 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 417324249 0 0) 0x7efebc15b070 con 0x7efebc15ac00 2024-03-24T10:51:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.967+0000 7efeb959a640 1 -- 172.21.15.5:0/101843531 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efea4003520 con 0x7efebc15ac00 2024-03-24T10:51:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.968+0000 7efebad9d640 1 -- 172.21.15.5:0/101843531 >> v1:172.21.15.57:6789/0 conn(0x7efebc15ac00 legacy=0x7efebc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.968+0000 7efebad9d640 1 -- 172.21.15.5:0/101843531 >> v1:172.21.15.5:6789/0 conn(0x7efebc1543a0 legacy=0x7efebc06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:14.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.968+0000 7efebad9d640 1 -- 172.21.15.5:0/101843531 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efebc0ca7e0 con 0x7efebc06bf50 2024-03-24T10:51:14.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.972+0000 7efebad9d640 1 --2- 172.21.15.5:0/101843531 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efebc06bf50 0x7efebc06c330 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7efeb4009d90 tx=0x7efeb402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33208a55ee32b69a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:14.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.976+0000 7efeb959a640 1 -- 172.21.15.5:0/101843531 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb402cd00 con 0x7efebc06bf50 2024-03-24T10:51:14.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.976+0000 7efeb959a640 1 -- 172.21.15.5:0/101843531 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeb402ce60 con 0x7efebc06bf50 2024-03-24T10:51:14.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.976+0000 7efeb959a640 1 -- 172.21.15.5:0/101843531 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb4034660 con 0x7efebc06bf50 2024-03-24T10:51:14.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.977+0000 7efec0cf4640 1 -- 172.21.15.5:0/101843531 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efebc06bf50 msgr2=0x7efebc06c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:14.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.977+0000 7efec0cf4640 1 --2- 172.21.15.5:0/101843531 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efebc06bf50 0x7efebc06c330 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7efeb4009d90 tx=0x7efeb402bf50 comp rx=0 tx=0).stop 2024-03-24T10:51:14.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.977+0000 7efec0cf4640 1 -- 172.21.15.5:0/101843531 shutdown_connections 2024-03-24T10:51:14.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.977+0000 7efec0cf4640 1 --2- 172.21.15.5:0/101843531 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efebc06bf50 0x7efebc06c330 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:14.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.977+0000 7efec0cf4640 1 -- 172.21.15.5:0/101843531 >> 172.21.15.5:0/101843531 conn(0x7efebc072ef0 msgr2=0x7efebc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:14.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.977+0000 7efec0cf4640 1 -- 172.21.15.5:0/101843531 shutdown_connections 2024-03-24T10:51:14.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.977+0000 7efec0cf4640 1 -- 172.21.15.5:0/101843531 wait complete. 2024-03-24T10:51:14.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.979+0000 7efec0cf4640 1 Processor -- start 2024-03-24T10:51:14.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.979+0000 7efec0cf4640 1 -- start start 2024-03-24T10:51:14.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.980+0000 7efec0cf4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efebc06bf50 0x7efebc1446a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:14.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.980+0000 7efec0cf4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efebc1543a0 0x7efebc156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:14.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.980+0000 7efec0cf4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efebc15ac00 0x7efebc156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:14.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.980+0000 7efec0cf4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7efebc159da0 con 0x7efebc06bf50 2024-03-24T10:51:14.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.980+0000 7efec0cf4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7efebc154c00 con 0x7efebc1543a0 2024-03-24T10:51:14.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.980+0000 7efec0cf4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7efebc154ee0 con 0x7efebc15ac00 2024-03-24T10:51:14.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.981+0000 7efeba59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efebc06bf50 0x7efebc1446a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:14.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.981+0000 7efeb9d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efebc1543a0 0x7efebc156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:14.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.981+0000 7efebad9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efebc15ac00 0x7efebc156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:14.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.981+0000 7efeba59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efebc06bf50 0x7efebc1446a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36808/0 (socket says 172.21.15.5:36808) 2024-03-24T10:51:14.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.981+0000 7efeba59c640 1 -- 172.21.15.5:0/1547594875 learned_addr learned my addr 172.21.15.5:0/1547594875 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:14.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.985+0000 7efebad9d640 1 -- 172.21.15.5:0/1547594875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efebc1543a0 msgr2=0x7efebc156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:14.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.985+0000 7efebad9d640 1 --2- 172.21.15.5:0/1547594875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efebc1543a0 0x7efebc156790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:14.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.985+0000 7efebad9d640 1 -- 172.21.15.5:0/1547594875 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efebc06bf50 msgr2=0x7efebc1446a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:14.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.985+0000 7efebad9d640 1 --2- 172.21.15.5:0/1547594875 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efebc06bf50 0x7efebc1446a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:14.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.985+0000 7efebad9d640 1 -- 172.21.15.5:0/1547594875 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb4009a70 con 0x7efebc15ac00 2024-03-24T10:51:14.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.989+0000 7efebad9d640 1 --2- 172.21.15.5:0/1547594875 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efebc15ac00 0x7efebc156cd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efeb4033000 tx=0x7efeb4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:14.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.993+0000 7efeaaffd640 1 -- 172.21.15.5:0/1547594875 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb403e610 con 0x7efebc15ac00 2024-03-24T10:51:14.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.993+0000 7efeaaffd640 1 -- 172.21.15.5:0/1547594875 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeb403e770 con 0x7efebc15ac00 2024-03-24T10:51:14.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.993+0000 7efec0cf4640 1 -- 172.21.15.5:0/1547594875 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efebc15dd70 con 0x7efebc15ac00 2024-03-24T10:51:14.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.993+0000 7efeaaffd640 1 -- 172.21.15.5:0/1547594875 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb403eb20 con 0x7efebc15ac00 2024-03-24T10:51:14.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.993+0000 7efec0cf4640 1 -- 172.21.15.5:0/1547594875 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efebc144be0 con 0x7efebc15ac00 2024-03-24T10:51:14.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:14.995+0000 7efec0cf4640 1 -- 172.21.15.5:0/1547594875 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efebc15e290 con 0x7efebc15ac00 2024-03-24T10:51:15.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:15.020+0000 7efeaaffd640 1 -- 172.21.15.5:0/1547594875 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efeb403b050 con 0x7efebc15ac00 2024-03-24T10:51:15.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:15.020+0000 7efeaaffd640 1 --2- 172.21.15.5:0/1547594875 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7efe9c06b860 0x7efe9c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:15.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:15.021+0000 7efeba59c640 1 --2- 172.21.15.5:0/1547594875 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7efe9c06b860 0x7efe9c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:15.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:15.021+0000 7efeba59c640 1 --2- 172.21.15.5:0/1547594875 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7efe9c06b860 0x7efe9c06dd20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7efeac001840 tx=0x7efeac009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:15.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:15.024+0000 7efeaaffd640 1 -- 172.21.15.5:0/1547594875 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7efeb407ad30 con 0x7efebc15ac00 2024-03-24T10:51:15.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:15.299+0000 7efeaaffd640 1 -- 172.21.15.5:0/1547594875 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efeb4076fd0 con 0x7efebc15ac00 2024-03-24T10:51:15.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:15.546+0000 7efec0cf4640 1 -- 172.21.15.5:0/1547594875 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efebc15e290 con 0x7efebc15ac00 2024-03-24T10:51:15.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:15.640+0000 7efeaaffd640 1 -- 172.21.15.5:0/1547594875 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7efebc15e290 con 0x7efebc15ac00 2024-03-24T10:51:15.641 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:15.641 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:15.641 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:15.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:15.643+0000 7efec0cf4640 1 -- 172.21.15.5:0/1547594875 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7efe9c06b860 msgr2=0x7efe9c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:15.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:15.643+0000 7efec0cf4640 1 --2- 172.21.15.5:0/1547594875 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7efe9c06b860 0x7efe9c06dd20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7efeac001840 tx=0x7efeac009040 comp rx=0 tx=0).stop 2024-03-24T10:51:15.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:15.643+0000 7efec0cf4640 1 -- 172.21.15.5:0/1547594875 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efebc15ac00 msgr2=0x7efebc156cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:15.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:15.643+0000 7efec0cf4640 1 --2- 172.21.15.5:0/1547594875 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efebc15ac00 0x7efebc156cd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efeb4033000 tx=0x7efeb4004a40 comp rx=0 tx=0).stop 2024-03-24T10:51:15.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:15.643+0000 7efec0cf4640 1 -- 172.21.15.5:0/1547594875 shutdown_connections 2024-03-24T10:51:15.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:15.643+0000 7efec0cf4640 1 --2- 172.21.15.5:0/1547594875 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7efe9c06b860 0x7efe9c06dd20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:15.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:15.643+0000 7efec0cf4640 1 --2- 172.21.15.5:0/1547594875 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efebc15ac00 0x7efebc156cd0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:15.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:15.643+0000 7efec0cf4640 1 --2- 172.21.15.5:0/1547594875 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efebc1543a0 0x7efebc156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:15.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:15.643+0000 7efec0cf4640 1 --2- 172.21.15.5:0/1547594875 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efebc06bf50 0x7efebc1446a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:15.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:15.643+0000 7efec0cf4640 1 -- 172.21.15.5:0/1547594875 >> 172.21.15.5:0/1547594875 conn(0x7efebc072ef0 msgr2=0x7efebc152cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:15.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:15.643+0000 7efec0cf4640 1 -- 172.21.15.5:0/1547594875 shutdown_connections 2024-03-24T10:51:15.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:15.643+0000 7efec0cf4640 1 -- 172.21.15.5:0/1547594875 wait complete. 2024-03-24T10:51:16.258 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:16.258 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:17.260 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:17.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.362+0000 7efcc1fb5640 1 Processor -- start 2024-03-24T10:51:17.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.362+0000 7efcc1fb5640 1 -- start start 2024-03-24T10:51:17.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.362+0000 7efcc1fb5640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7efcbc154250 0x7efcbc15e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:17.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.363+0000 7efcc1fb5640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efcbc073100 0x7efcbc0cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:17.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.363+0000 7efcc1fb5640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcbc0c6e10 con 0x7efcbc0c7350 2024-03-24T10:51:17.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.363+0000 7efcc1fb5640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7efcbc0c47f0 con 0x7efcbc154250 2024-03-24T10:51:17.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.363+0000 7efcc1fb5640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7efcbc15d800 con 0x7efcbc073100 2024-03-24T10:51:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.363+0000 7efcbbfff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efcbc073100 0x7efcbc0cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.364+0000 7efcbbfff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efcbc073100 0x7efcbc0cc8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:48482/0 (socket says 172.21.15.5:48482) 2024-03-24T10:51:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.364+0000 7efcbbfff640 1 -- 172.21.15.5:0/2689209962 learned_addr learned my addr 172.21.15.5:0/2689209962 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:17.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.364+0000 7efcbb7fe640 1 --2- 172.21.15.5:0/2689209962 >> v2:172.21.15.5:3300/0 conn(0x7efcbc154250 0x7efcbc15e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:17.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.366+0000 7efcbbfff640 1 -- 172.21.15.5:0/2689209962 >> v2:172.21.15.5:3300/0 conn(0x7efcbc154250 msgr2=0x7efcbc15e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:17.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.366+0000 7efcbbfff640 1 --2- 172.21.15.5:0/2689209962 >> v2:172.21.15.5:3300/0 conn(0x7efcbc154250 0x7efcbc15e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:17.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.366+0000 7efcbbfff640 1 -- 172.21.15.5:0/2689209962 >> v1:172.21.15.5:6789/0 conn(0x7efcbc0c7350 legacy=0x7efcbc15ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:17.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.366+0000 7efcbbfff640 1 -- 172.21.15.5:0/2689209962 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcbc0726b0 con 0x7efcbc073100 2024-03-24T10:51:17.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.368+0000 7efcbbfff640 1 --2- 172.21.15.5:0/2689209962 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efcbc073100 0x7efcbc0cc8e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009e50 tx=0x7efcb402bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f02ddf3ce6a62c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:17.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.373+0000 7efcba7fc640 1 -- 172.21.15.5:0/2689209962 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb402cb90 con 0x7efcbc073100 2024-03-24T10:51:17.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.373+0000 7efcba7fc640 1 -- 172.21.15.5:0/2689209962 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcb402ccf0 con 0x7efcbc073100 2024-03-24T10:51:17.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.373+0000 7efcba7fc640 1 -- 172.21.15.5:0/2689209962 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb4034610 con 0x7efcbc073100 2024-03-24T10:51:17.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.374+0000 7efcc1fb5640 1 -- 172.21.15.5:0/2689209962 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efcbc073100 msgr2=0x7efcbc0cc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:17.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.374+0000 7efcc1fb5640 1 --2- 172.21.15.5:0/2689209962 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efcbc073100 0x7efcbc0cc8e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efcb4009e50 tx=0x7efcb402bff0 comp rx=0 tx=0).stop 2024-03-24T10:51:17.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.374+0000 7efcc1fb5640 1 -- 172.21.15.5:0/2689209962 shutdown_connections 2024-03-24T10:51:17.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.374+0000 7efcc1fb5640 1 --2- 172.21.15.5:0/2689209962 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efcbc073100 0x7efcbc0cc8e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:17.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.374+0000 7efcc1fb5640 1 --2- 172.21.15.5:0/2689209962 >> v2:172.21.15.5:3300/0 conn(0x7efcbc154250 0x7efcbc15e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:17.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.374+0000 7efcc1fb5640 1 -- 172.21.15.5:0/2689209962 >> 172.21.15.5:0/2689209962 conn(0x7efcbc06c240 msgr2=0x7efcbc06c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:17.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.374+0000 7efcc1fb5640 1 -- 172.21.15.5:0/2689209962 shutdown_connections 2024-03-24T10:51:17.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.374+0000 7efcc1fb5640 1 -- 172.21.15.5:0/2689209962 wait complete. 2024-03-24T10:51:17.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.376+0000 7efcc1fb5640 1 Processor -- start 2024-03-24T10:51:17.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.376+0000 7efcc1fb5640 1 -- start start 2024-03-24T10:51:17.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.377+0000 7efcc1fb5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcbc073100 0x7efcbc1556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:17.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.377+0000 7efcc1fb5640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcbc0c7350 0x7efcbc1568a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:17.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.377+0000 7efcc1fb5640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efcbc154250 0x7efcbc1f0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:17.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.377+0000 7efcc1fb5640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7efcbc15a4c0 con 0x7efcbc0c7350 2024-03-24T10:51:17.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.377+0000 7efcc1fb5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7efcbc1f08b0 con 0x7efcbc073100 2024-03-24T10:51:17.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.377+0000 7efcc1fb5640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7efcbc1566a0 con 0x7efcbc154250 2024-03-24T10:51:17.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.378+0000 7efcbaffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcbc0c7350 0x7efcbc1568a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:17.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.379+0000 7efcbaffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcbc0c7350 0x7efcbc1568a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36816/0 (socket says 172.21.15.5:36816) 2024-03-24T10:51:17.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.379+0000 7efcbaffd640 1 -- 172.21.15.5:0/1021009332 learned_addr learned my addr 172.21.15.5:0/1021009332 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:17.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.380+0000 7efcbb7fe640 1 --2- 172.21.15.5:0/1021009332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcbc073100 0x7efcbc1556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:17.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.384+0000 7efcbaffd640 1 -- 172.21.15.5:0/1021009332 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efcbc154250 msgr2=0x7efcbc1f0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:17.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.384+0000 7efcbaffd640 1 --2- 172.21.15.5:0/1021009332 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efcbc154250 0x7efcbc1f0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:17.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.384+0000 7efcbaffd640 1 -- 172.21.15.5:0/1021009332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcbc073100 msgr2=0x7efcbc1556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:17.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.384+0000 7efcbaffd640 1 --2- 172.21.15.5:0/1021009332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcbc073100 0x7efcbc1556f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:17.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.384+0000 7efcbaffd640 1 -- 172.21.15.5:0/1021009332 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb4009b30 con 0x7efcbc0c7350 2024-03-24T10:51:17.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.387+0000 7efcbaffd640 1 --2- 172.21.15.5:0/1021009332 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcbc0c7350 0x7efcbc1568a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7efcb000c400 tx=0x7efcb000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:17.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.391+0000 7efca3fff640 1 -- 172.21.15.5:0/1021009332 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb00052d0 con 0x7efcbc0c7350 2024-03-24T10:51:17.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.391+0000 7efca3fff640 1 -- 172.21.15.5:0/1021009332 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcb0005450 con 0x7efcbc0c7350 2024-03-24T10:51:17.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.391+0000 7efca3fff640 1 -- 172.21.15.5:0/1021009332 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb0007260 con 0x7efcbc0c7350 2024-03-24T10:51:17.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.391+0000 7efcc1fb5640 1 -- 172.21.15.5:0/1021009332 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcbc1560e0 con 0x7efcbc0c7350 2024-03-24T10:51:17.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.391+0000 7efcc1fb5640 1 -- 172.21.15.5:0/1021009332 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcbc154c10 con 0x7efcbc0c7350 2024-03-24T10:51:17.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.392+0000 7efcc1fb5640 1 -- 172.21.15.5:0/1021009332 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcbc15c9b0 con 0x7efcbc0c7350 2024-03-24T10:51:17.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.669+0000 7efca3fff640 1 -- 172.21.15.5:0/1021009332 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efcb00073c0 con 0x7efcbc0c7350 2024-03-24T10:51:17.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.670+0000 7efca3fff640 1 --2- 172.21.15.5:0/1021009332 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7efca806b860 0x7efca806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:17.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.670+0000 7efcbb7fe640 1 --2- 172.21.15.5:0/1021009332 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7efca806b860 0x7efca806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:17.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.670+0000 7efcbb7fe640 1 --2- 172.21.15.5:0/1021009332 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7efca806b860 0x7efca806dd20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7efcac009aa0 tx=0x7efcac009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:17.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.691+0000 7efca3fff640 1 -- 172.21.15.5:0/1021009332 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7efcb0007670 con 0x7efcbc0c7350 2024-03-24T10:51:17.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.691+0000 7efca3fff640 1 -- 172.21.15.5:0/1021009332 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efcb0054520 con 0x7efcbc0c7350 2024-03-24T10:51:17.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.924+0000 7efcc1fb5640 1 -- 172.21.15.5:0/1021009332 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efcbc0688e0 con 0x7efcbc0c7350 2024-03-24T10:51:17.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.939+0000 7efca3fff640 1 -- 172.21.15.5:0/1021009332 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7efcb0057be0 con 0x7efcbc0c7350 2024-03-24T10:51:17.939 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:17.939 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:17.940 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:17.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.941+0000 7efcc1fb5640 1 -- 172.21.15.5:0/1021009332 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7efca806b860 msgr2=0x7efca806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:17.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.941+0000 7efcc1fb5640 1 --2- 172.21.15.5:0/1021009332 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7efca806b860 0x7efca806dd20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7efcac009aa0 tx=0x7efcac009190 comp rx=0 tx=0).stop 2024-03-24T10:51:17.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.941+0000 7efcc1fb5640 1 -- 172.21.15.5:0/1021009332 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcbc0c7350 msgr2=0x7efcbc1568a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:17.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.941+0000 7efcc1fb5640 1 --2- 172.21.15.5:0/1021009332 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcbc0c7350 0x7efcbc1568a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7efcb000c400 tx=0x7efcb000c8c0 comp rx=0 tx=0).stop 2024-03-24T10:51:17.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.942+0000 7efcc1fb5640 1 -- 172.21.15.5:0/1021009332 shutdown_connections 2024-03-24T10:51:17.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.942+0000 7efcc1fb5640 1 --2- 172.21.15.5:0/1021009332 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7efca806b860 0x7efca806dd20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:17.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.942+0000 7efcc1fb5640 1 --2- 172.21.15.5:0/1021009332 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7efcbc0c7350 0x7efcbc1568a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:17.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.942+0000 7efcc1fb5640 1 --2- 172.21.15.5:0/1021009332 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7efcbc154250 0x7efcbc1f0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:17.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.942+0000 7efcc1fb5640 1 --2- 172.21.15.5:0/1021009332 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7efcbc073100 0x7efcbc1556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:17.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.942+0000 7efcc1fb5640 1 -- 172.21.15.5:0/1021009332 >> 172.21.15.5:0/1021009332 conn(0x7efcbc06c240 msgr2=0x7efcbc06c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:17.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.942+0000 7efcc1fb5640 1 -- 172.21.15.5:0/1021009332 shutdown_connections 2024-03-24T10:51:17.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:17.942+0000 7efcc1fb5640 1 -- 172.21.15.5:0/1021009332 wait complete. 2024-03-24T10:51:18.557 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:18.557 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:19.558 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:19.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.660+0000 7f9de1023640 1 Processor -- start 2024-03-24T10:51:19.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.660+0000 7f9de1023640 1 -- start start 2024-03-24T10:51:19.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.662+0000 7f9de1023640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9ddc1543a0 0x7f9ddc15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:19.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.662+0000 7f9de1023640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ddc0cb2f0 con 0x7f9ddc06d4f0 2024-03-24T10:51:19.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.662+0000 7f9de1023640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ddc0ca7e0 con 0x7f9ddc06bf50 2024-03-24T10:51:19.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.662+0000 7f9de1023640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f9ddc0c6750 con 0x7f9ddc1543a0 2024-03-24T10:51:19.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.663+0000 7f9dda59c640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f9ddc06d4f0 0x7f9ddc0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:52146/0 (socket says 172.21.15.5:52146) 2024-03-24T10:51:19.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.663+0000 7f9dda59c640 1 -- 172.21.15.5:0/1598874993 learned_addr learned my addr 172.21.15.5:0/1598874993 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:51:19.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.663+0000 7f9ddad9d640 1 --2- 172.21.15.5:0/1598874993 >> v2:172.21.15.57:3300/0 conn(0x7f9ddc1543a0 0x7f9ddc15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:19.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.666+0000 7f9dd9d9b640 1 -- 172.21.15.5:0/1598874993 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3360139229 0 0) 0x7f9ddc0cb2f0 con 0x7f9ddc06d4f0 2024-03-24T10:51:19.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.666+0000 7f9dd9d9b640 1 -- 172.21.15.5:0/1598874993 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dc4003520 con 0x7f9ddc06d4f0 2024-03-24T10:51:19.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.668+0000 7f9dd9d9b640 1 -- 172.21.15.5:0/1598874993 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 344093282 0 0) 0x7f9ddc0ca7e0 con 0x7f9ddc06bf50 2024-03-24T10:51:19.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.668+0000 7f9dd9d9b640 1 -- 172.21.15.5:0/1598874993 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ddc0cb2f0 con 0x7f9ddc06bf50 2024-03-24T10:51:19.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.670+0000 7f9ddad9d640 1 -- 172.21.15.5:0/1598874993 >> v1:172.21.15.57:6789/0 conn(0x7f9ddc06bf50 legacy=0x7f9ddc06c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:19.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.670+0000 7f9ddad9d640 1 -- 172.21.15.5:0/1598874993 >> v1:172.21.15.5:6789/0 conn(0x7f9ddc06d4f0 legacy=0x7f9ddc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:19.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.670+0000 7f9ddad9d640 1 -- 172.21.15.5:0/1598874993 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ddc0cb520 con 0x7f9ddc1543a0 2024-03-24T10:51:19.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.678+0000 7f9ddad9d640 1 --2- 172.21.15.5:0/1598874993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ddc1543a0 0x7f9ddc15ac00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc0097a0 tx=0x7f9dcc02b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfb0c1d01c4a760 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:19.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.683+0000 7f9dd9d9b640 1 -- 172.21.15.5:0/1598874993 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc02f810 con 0x7f9ddc1543a0 2024-03-24T10:51:19.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.683+0000 7f9dd9d9b640 1 -- 172.21.15.5:0/1598874993 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dcc02f970 con 0x7f9ddc1543a0 2024-03-24T10:51:19.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.683+0000 7f9dd9d9b640 1 -- 172.21.15.5:0/1598874993 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc02fd00 con 0x7f9ddc1543a0 2024-03-24T10:51:19.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.684+0000 7f9de1023640 1 -- 172.21.15.5:0/1598874993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ddc1543a0 msgr2=0x7f9ddc15ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:19.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.684+0000 7f9de1023640 1 --2- 172.21.15.5:0/1598874993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ddc1543a0 0x7f9ddc15ac00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc0097a0 tx=0x7f9dcc02b9f0 comp rx=0 tx=0).stop 2024-03-24T10:51:19.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.684+0000 7f9de1023640 1 -- 172.21.15.5:0/1598874993 shutdown_connections 2024-03-24T10:51:19.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.684+0000 7f9de1023640 1 --2- 172.21.15.5:0/1598874993 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ddc1543a0 0x7f9ddc15ac00 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:19.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.684+0000 7f9de1023640 1 -- 172.21.15.5:0/1598874993 >> 172.21.15.5:0/1598874993 conn(0x7f9ddc072ef0 msgr2=0x7f9ddc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:19.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.684+0000 7f9de1023640 1 -- 172.21.15.5:0/1598874993 shutdown_connections 2024-03-24T10:51:19.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.684+0000 7f9de1023640 1 -- 172.21.15.5:0/1598874993 wait complete. 2024-03-24T10:51:19.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.686+0000 7f9de1023640 1 Processor -- start 2024-03-24T10:51:19.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.686+0000 7f9de1023640 1 -- start start 2024-03-24T10:51:19.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.686+0000 7f9de1023640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9ddc06bf50 0x7f9ddc157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:19.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.686+0000 7f9de1023640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9ddc06d4f0 0x7f9ddc0b3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:19.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.686+0000 7f9de1023640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ddc1543a0 0x7f9ddc158b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:19.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.686+0000 7f9de1023640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f9ddc158730 con 0x7f9ddc06bf50 2024-03-24T10:51:19.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.686+0000 7f9de1023640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9ddc1584a0 con 0x7f9ddc1543a0 2024-03-24T10:51:19.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.686+0000 7f9de1023640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f9ddc1efd40 con 0x7f9ddc06d4f0 2024-03-24T10:51:19.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.687+0000 7f9dda59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9ddc06d4f0 0x7f9ddc0b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:19.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.687+0000 7f9ddb59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ddc1543a0 0x7f9ddc158b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:19.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.687+0000 7f9dda59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9ddc06d4f0 0x7f9ddc0b3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:48488/0 (socket says 172.21.15.5:48488) 2024-03-24T10:51:19.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.687+0000 7f9dda59c640 1 -- 172.21.15.5:0/555426771 learned_addr learned my addr 172.21.15.5:0/555426771 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:19.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.689+0000 7f9ddad9d640 1 --2- 172.21.15.5:0/555426771 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9ddc06bf50 0x7f9ddc157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:19.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.691+0000 7f9dda59c640 1 -- 172.21.15.5:0/555426771 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ddc1543a0 msgr2=0x7f9ddc158b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:19.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.691+0000 7f9dda59c640 1 --2- 172.21.15.5:0/555426771 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ddc1543a0 0x7f9ddc158b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:19.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.691+0000 7f9dda59c640 1 -- 172.21.15.5:0/555426771 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9ddc06bf50 msgr2=0x7f9ddc157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:19.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.691+0000 7f9dda59c640 1 --2- 172.21.15.5:0/555426771 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9ddc06bf50 0x7f9ddc157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:19.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.691+0000 7f9dda59c640 1 -- 172.21.15.5:0/555426771 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dcc009400 con 0x7f9ddc06d4f0 2024-03-24T10:51:19.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.693+0000 7f9dda59c640 1 --2- 172.21.15.5:0/555426771 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9ddc06d4f0 0x7f9ddc0b3930 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9dd000c500 tx=0x7f9dd000c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:19.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.696+0000 7f9dc37fe640 1 -- 172.21.15.5:0/555426771 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd0005ce0 con 0x7f9ddc06d4f0 2024-03-24T10:51:19.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.696+0000 7f9de1023640 1 -- 172.21.15.5:0/555426771 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ddc1f0020 con 0x7f9ddc06d4f0 2024-03-24T10:51:19.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.696+0000 7f9de1023640 1 -- 172.21.15.5:0/555426771 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ddc1f05e0 con 0x7f9ddc06d4f0 2024-03-24T10:51:19.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.697+0000 7f9de1023640 1 -- 172.21.15.5:0/555426771 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ddc0681d0 con 0x7f9ddc06d4f0 2024-03-24T10:51:19.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.697+0000 7f9dc37fe640 1 -- 172.21.15.5:0/555426771 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dd0005e40 con 0x7f9ddc06d4f0 2024-03-24T10:51:19.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.697+0000 7f9dc37fe640 1 -- 172.21.15.5:0/555426771 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd00072f0 con 0x7f9ddc06d4f0 2024-03-24T10:51:19.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.968+0000 7f9dc37fe640 1 -- 172.21.15.5:0/555426771 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9dd0007510 con 0x7f9ddc06d4f0 2024-03-24T10:51:19.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.969+0000 7f9dc37fe640 1 --2- 172.21.15.5:0/555426771 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9dc806b810 0x7f9dc806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:19.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.969+0000 7f9ddad9d640 1 --2- 172.21.15.5:0/555426771 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9dc806b810 0x7f9dc806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:19.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.969+0000 7f9ddad9d640 1 --2- 172.21.15.5:0/555426771 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9dc806b810 0x7f9dc806dcd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc036000 tx=0x7f9dcc008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:19.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.972+0000 7f9dc37fe640 1 -- 172.21.15.5:0/555426771 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f9dd0058890 con 0x7f9ddc06d4f0 2024-03-24T10:51:19.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:19.985+0000 7f9dc37fe640 1 -- 172.21.15.5:0/555426771 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9dd0054b30 con 0x7f9ddc06d4f0 2024-03-24T10:51:20.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:20.209+0000 7f9de1023640 1 -- 172.21.15.5:0/555426771 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ddc15e580 con 0x7f9ddc06d4f0 2024-03-24T10:51:20.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:20.295+0000 7f9dc37fe640 1 -- 172.21.15.5:0/555426771 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f9dd0017090 con 0x7f9ddc06d4f0 2024-03-24T10:51:20.296 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:20.296 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:20.296 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:20.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:20.298+0000 7f9de1023640 1 -- 172.21.15.5:0/555426771 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9dc806b810 msgr2=0x7f9dc806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:20.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:20.298+0000 7f9de1023640 1 --2- 172.21.15.5:0/555426771 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9dc806b810 0x7f9dc806dcd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc036000 tx=0x7f9dcc008030 comp rx=0 tx=0).stop 2024-03-24T10:51:20.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:20.298+0000 7f9de1023640 1 -- 172.21.15.5:0/555426771 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9ddc06d4f0 msgr2=0x7f9ddc0b3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:20.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:20.298+0000 7f9de1023640 1 --2- 172.21.15.5:0/555426771 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9ddc06d4f0 0x7f9ddc0b3930 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9dd000c500 tx=0x7f9dd000c9c0 comp rx=0 tx=0).stop 2024-03-24T10:51:20.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:20.298+0000 7f9de1023640 1 -- 172.21.15.5:0/555426771 shutdown_connections 2024-03-24T10:51:20.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:20.298+0000 7f9de1023640 1 --2- 172.21.15.5:0/555426771 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9dc806b810 0x7f9dc806dcd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:20.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:20.298+0000 7f9de1023640 1 --2- 172.21.15.5:0/555426771 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ddc1543a0 0x7f9ddc158b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:20.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:20.298+0000 7f9de1023640 1 --2- 172.21.15.5:0/555426771 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9ddc06d4f0 0x7f9ddc0b3930 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:20.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:20.298+0000 7f9de1023640 1 --2- 172.21.15.5:0/555426771 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9ddc06bf50 0x7f9ddc157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:20.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:20.298+0000 7f9de1023640 1 -- 172.21.15.5:0/555426771 >> 172.21.15.5:0/555426771 conn(0x7f9ddc072ef0 msgr2=0x7f9ddc150e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:20.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:20.298+0000 7f9de1023640 1 -- 172.21.15.5:0/555426771 shutdown_connections 2024-03-24T10:51:20.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:20.298+0000 7f9de1023640 1 -- 172.21.15.5:0/555426771 wait complete. 2024-03-24T10:51:20.912 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:20.912 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:21.913 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:22.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.015+0000 7fdba9088640 1 Processor -- start 2024-03-24T10:51:22.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.015+0000 7fdba9088640 1 -- start start 2024-03-24T10:51:22.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.016+0000 7fdba9088640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fdba415ac00 0x7fdba40cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:22.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.016+0000 7fdba9088640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdba40ccaa0 0x7fdba406bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:22.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.016+0000 7fdba9088640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba40cb2f0 con 0x7fdba41543a0 2024-03-24T10:51:22.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.016+0000 7fdba9088640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fdba40c6750 con 0x7fdba415ac00 2024-03-24T10:51:22.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.016+0000 7fdba9088640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fdba40cceb0 con 0x7fdba40ccaa0 2024-03-24T10:51:22.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.017+0000 7fdba359e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdba40ccaa0 0x7fdba406bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:22.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.017+0000 7fdba359e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdba40ccaa0 0x7fdba406bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:48492/0 (socket says 172.21.15.5:48492) 2024-03-24T10:51:22.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.017+0000 7fdba359e640 1 -- 172.21.15.5:0/915055329 learned_addr learned my addr 172.21.15.5:0/915055329 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:22.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.017+0000 7fdba259c640 1 --2- 172.21.15.5:0/915055329 >> v2:172.21.15.57:3300/0 conn(0x7fdba415ac00 0x7fdba40cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:22.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.020+0000 7fdba359e640 1 -- 172.21.15.5:0/915055329 >> v2:172.21.15.57:3300/0 conn(0x7fdba415ac00 msgr2=0x7fdba40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:22.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.020+0000 7fdba359e640 1 --2- 172.21.15.5:0/915055329 >> v2:172.21.15.57:3300/0 conn(0x7fdba415ac00 0x7fdba40cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:22.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.020+0000 7fdba359e640 1 -- 172.21.15.5:0/915055329 >> v1:172.21.15.57:6789/0 conn(0x7fdba41543a0 legacy=0x7fdba406d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:22.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.020+0000 7fdba359e640 1 -- 172.21.15.5:0/915055329 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba415b070 con 0x7fdba40ccaa0 2024-03-24T10:51:22.022 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.022+0000 7fdba359e640 1 --2- 172.21.15.5:0/915055329 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdba40ccaa0 0x7fdba406bf50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009e50 tx=0x7fdb9c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa4815bc3bf990da server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:22.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.027+0000 7fdba1d9b640 1 -- 172.21.15.5:0/915055329 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c02cb00 con 0x7fdba40ccaa0 2024-03-24T10:51:22.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.028+0000 7fdba1d9b640 1 -- 172.21.15.5:0/915055329 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb9c02cc60 con 0x7fdba40ccaa0 2024-03-24T10:51:22.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.028+0000 7fdba1d9b640 1 -- 172.21.15.5:0/915055329 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c0343f0 con 0x7fdba40ccaa0 2024-03-24T10:51:22.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.028+0000 7fdba9088640 1 -- 172.21.15.5:0/915055329 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdba40ccaa0 msgr2=0x7fdba406bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:22.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.028+0000 7fdba9088640 1 --2- 172.21.15.5:0/915055329 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdba40ccaa0 0x7fdba406bf50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c009e50 tx=0x7fdb9c02bf60 comp rx=0 tx=0).stop 2024-03-24T10:51:22.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.028+0000 7fdba9088640 1 -- 172.21.15.5:0/915055329 shutdown_connections 2024-03-24T10:51:22.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.028+0000 7fdba9088640 1 --2- 172.21.15.5:0/915055329 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdba40ccaa0 0x7fdba406bf50 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:22.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.028+0000 7fdba9088640 1 --2- 172.21.15.5:0/915055329 >> v2:172.21.15.57:3300/0 conn(0x7fdba415ac00 0x7fdba40cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:22.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.028+0000 7fdba9088640 1 -- 172.21.15.5:0/915055329 >> 172.21.15.5:0/915055329 conn(0x7fdba4072ef0 msgr2=0x7fdba4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:22.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.028+0000 7fdba9088640 1 -- 172.21.15.5:0/915055329 shutdown_connections 2024-03-24T10:51:22.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.028+0000 7fdba9088640 1 -- 172.21.15.5:0/915055329 wait complete. 2024-03-24T10:51:22.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.030+0000 7fdba9088640 1 Processor -- start 2024-03-24T10:51:22.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.030+0000 7fdba9088640 1 -- start start 2024-03-24T10:51:22.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.030+0000 7fdba9088640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdba40ccaa0 0x7fdba41447a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:22.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.030+0000 7fdba9088640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdba41543a0 0x7fdba41ef8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:22.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.030+0000 7fdba9088640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdba415ac00 0x7fdba40ce920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:22.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.030+0000 7fdba9088640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fdba415c040 con 0x7fdba40ccaa0 2024-03-24T10:51:22.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.030+0000 7fdba9088640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fdba4159c40 con 0x7fdba415ac00 2024-03-24T10:51:22.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.030+0000 7fdba9088640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fdba4144ce0 con 0x7fdba41543a0 2024-03-24T10:51:22.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.031+0000 7fdba259c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdba41543a0 0x7fdba41ef8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:22.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.031+0000 7fdba2d9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdba40ccaa0 0x7fdba41447a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:22.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.031+0000 7fdba259c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdba41543a0 0x7fdba41ef8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:48508/0 (socket says 172.21.15.5:48508) 2024-03-24T10:51:22.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.031+0000 7fdba259c640 1 -- 172.21.15.5:0/4021072817 learned_addr learned my addr 172.21.15.5:0/4021072817 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:22.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.031+0000 7fdba359e640 1 --2- 172.21.15.5:0/4021072817 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdba415ac00 0x7fdba40ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:22.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.035+0000 7fdba259c640 1 -- 172.21.15.5:0/4021072817 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdba415ac00 msgr2=0x7fdba40ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:22.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.035+0000 7fdba259c640 1 --2- 172.21.15.5:0/4021072817 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdba415ac00 0x7fdba40ce920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:22.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.035+0000 7fdba259c640 1 -- 172.21.15.5:0/4021072817 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdba40ccaa0 msgr2=0x7fdba41447a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:22.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.035+0000 7fdba259c640 1 --2- 172.21.15.5:0/4021072817 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdba40ccaa0 0x7fdba41447a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:22.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.035+0000 7fdba259c640 1 -- 172.21.15.5:0/4021072817 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb9c009b30 con 0x7fdba41543a0 2024-03-24T10:51:22.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.038+0000 7fdba259c640 1 --2- 172.21.15.5:0/4021072817 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdba41543a0 0x7fdba41ef8d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdb9800e870 tx=0x7fdb9800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:22.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.043+0000 7fdb8b7fe640 1 -- 172.21.15.5:0/4021072817 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb980094f0 con 0x7fdba41543a0 2024-03-24T10:51:22.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.043+0000 7fdb8b7fe640 1 -- 172.21.15.5:0/4021072817 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb9800c940 con 0x7fdba41543a0 2024-03-24T10:51:22.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.043+0000 7fdba9088640 1 -- 172.21.15.5:0/4021072817 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdba4158320 con 0x7fdba41543a0 2024-03-24T10:51:22.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.043+0000 7fdb8b7fe640 1 -- 172.21.15.5:0/4021072817 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9800ccf0 con 0x7fdba41543a0 2024-03-24T10:51:22.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.043+0000 7fdba9088640 1 -- 172.21.15.5:0/4021072817 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdba4159e90 con 0x7fdba41543a0 2024-03-24T10:51:22.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.044+0000 7fdba9088640 1 -- 172.21.15.5:0/4021072817 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdba40681d0 con 0x7fdba41543a0 2024-03-24T10:51:22.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.330+0000 7fdb8b7fe640 1 -- 172.21.15.5:0/4021072817 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdb98013470 con 0x7fdba41543a0 2024-03-24T10:51:22.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.330+0000 7fdb8b7fe640 1 --2- 172.21.15.5:0/4021072817 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdb9006b860 0x7fdb9006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:22.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.330+0000 7fdba2d9d640 1 --2- 172.21.15.5:0/4021072817 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdb9006b860 0x7fdb9006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:22.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.330+0000 7fdba2d9d640 1 --2- 172.21.15.5:0/4021072817 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdb9006b860 0x7fdb9006dd20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdb94001780 tx=0x7fdb94009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:22.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.338+0000 7fdb8b7fe640 1 -- 172.21.15.5:0/4021072817 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fdb98059280 con 0x7fdba41543a0 2024-03-24T10:51:22.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.357+0000 7fdb8b7fe640 1 -- 172.21.15.5:0/4021072817 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdb98055520 con 0x7fdba41543a0 2024-03-24T10:51:22.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.606+0000 7fdba9088640 1 -- 172.21.15.5:0/4021072817 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdba415e580 con 0x7fdba41543a0 2024-03-24T10:51:22.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.621+0000 7fdb8b7fe640 1 -- 172.21.15.5:0/4021072817 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fdb98021090 con 0x7fdba41543a0 2024-03-24T10:51:22.622 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:22.622 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:22.622 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:22.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.624+0000 7fdba9088640 1 -- 172.21.15.5:0/4021072817 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdb9006b860 msgr2=0x7fdb9006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:22.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.625+0000 7fdba9088640 1 --2- 172.21.15.5:0/4021072817 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdb9006b860 0x7fdb9006dd20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdb94001780 tx=0x7fdb94009040 comp rx=0 tx=0).stop 2024-03-24T10:51:22.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.625+0000 7fdba9088640 1 -- 172.21.15.5:0/4021072817 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdba41543a0 msgr2=0x7fdba41ef8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:22.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.625+0000 7fdba9088640 1 --2- 172.21.15.5:0/4021072817 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdba41543a0 0x7fdba41ef8d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdb9800e870 tx=0x7fdb9800ed30 comp rx=0 tx=0).stop 2024-03-24T10:51:22.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.625+0000 7fdba9088640 1 -- 172.21.15.5:0/4021072817 shutdown_connections 2024-03-24T10:51:22.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.625+0000 7fdba9088640 1 --2- 172.21.15.5:0/4021072817 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdb9006b860 0x7fdb9006dd20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:22.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.625+0000 7fdba9088640 1 --2- 172.21.15.5:0/4021072817 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdba415ac00 0x7fdba40ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:22.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.625+0000 7fdba9088640 1 --2- 172.21.15.5:0/4021072817 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdba41543a0 0x7fdba41ef8d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:22.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.625+0000 7fdba9088640 1 --2- 172.21.15.5:0/4021072817 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdba40ccaa0 0x7fdba41447a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:22.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.625+0000 7fdba9088640 1 -- 172.21.15.5:0/4021072817 >> 172.21.15.5:0/4021072817 conn(0x7fdba4072ef0 msgr2=0x7fdba4152b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:22.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.625+0000 7fdba9088640 1 -- 172.21.15.5:0/4021072817 shutdown_connections 2024-03-24T10:51:22.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:22.625+0000 7fdba9088640 1 -- 172.21.15.5:0/4021072817 wait complete. 2024-03-24T10:51:23.242 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:23.243 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:24.244 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:24.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.347+0000 7f7cb8855640 1 Processor -- start 2024-03-24T10:51:24.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.347+0000 7f7cb8855640 1 -- start start 2024-03-24T10:51:24.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.348+0000 7f7cb8855640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7cb40cae90 0x7f7cb40cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:24.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.348+0000 7f7cb8855640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7cb40cd960 0x7f7cb40cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:24.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.348+0000 7f7cb8855640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cb40ccba0 con 0x7f7cb40cb270 2024-03-24T10:51:24.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.348+0000 7f7cb8855640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f7cb40cc9b0 con 0x7f7cb40cae90 2024-03-24T10:51:24.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.348+0000 7f7cb8855640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f7cb40cce40 con 0x7f7cb40cd960 2024-03-24T10:51:24.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.348+0000 7f7cb259c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7cb40cae90 0x7f7cb40cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:24.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.349+0000 7f7cb259c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7cb40cae90 0x7f7cb40cf530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43196/0 (socket says 172.21.15.5:43196) 2024-03-24T10:51:24.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.349+0000 7f7cb259c640 1 -- 172.21.15.5:0/573491463 learned_addr learned my addr 172.21.15.5:0/573491463 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:24.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.349+0000 7f7cb2d9d640 1 --2- 172.21.15.5:0/573491463 >> v2:172.21.15.57:3300/0 conn(0x7f7cb40cd960 0x7f7cb40cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:24.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.351+0000 7f7cb259c640 1 -- 172.21.15.5:0/573491463 >> v2:172.21.15.57:3300/0 conn(0x7f7cb40cd960 msgr2=0x7f7cb40cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:24.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.351+0000 7f7cb259c640 1 --2- 172.21.15.5:0/573491463 >> v2:172.21.15.57:3300/0 conn(0x7f7cb40cd960 0x7f7cb40cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:24.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.351+0000 7f7cb259c640 1 -- 172.21.15.5:0/573491463 >> v1:172.21.15.57:6789/0 conn(0x7f7cb40cb270 legacy=0x7f7cb40cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:24.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.351+0000 7f7cb259c640 1 -- 172.21.15.5:0/573491463 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb415a590 con 0x7f7cb40cae90 2024-03-24T10:51:24.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.353+0000 7f7cb259c640 1 --2- 172.21.15.5:0/573491463 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cb40cae90 0x7f7cb40cf530 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7cb40b5cc0 tx=0x7f7ca402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c2b5bddf48bd8c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:24.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.357+0000 7f7cb159a640 1 -- 172.21.15.5:0/573491463 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca402f850 con 0x7f7cb40cae90 2024-03-24T10:51:24.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.357+0000 7f7cb159a640 1 -- 172.21.15.5:0/573491463 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ca402f9b0 con 0x7f7cb40cae90 2024-03-24T10:51:24.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.357+0000 7f7cb159a640 1 -- 172.21.15.5:0/573491463 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca402fd40 con 0x7f7cb40cae90 2024-03-24T10:51:24.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.358+0000 7f7cb8855640 1 -- 172.21.15.5:0/573491463 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cb40cae90 msgr2=0x7f7cb40cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:24.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.358+0000 7f7cb8855640 1 --2- 172.21.15.5:0/573491463 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cb40cae90 0x7f7cb40cf530 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7cb40b5cc0 tx=0x7f7ca402ba30 comp rx=0 tx=0).stop 2024-03-24T10:51:24.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.358+0000 7f7cb8855640 1 -- 172.21.15.5:0/573491463 shutdown_connections 2024-03-24T10:51:24.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.358+0000 7f7cb8855640 1 --2- 172.21.15.5:0/573491463 >> v2:172.21.15.57:3300/0 conn(0x7f7cb40cd960 0x7f7cb40cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:24.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.358+0000 7f7cb8855640 1 --2- 172.21.15.5:0/573491463 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cb40cae90 0x7f7cb40cf530 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:24.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.358+0000 7f7cb8855640 1 -- 172.21.15.5:0/573491463 >> 172.21.15.5:0/573491463 conn(0x7f7cb40cfb80 msgr2=0x7f7cb40cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:24.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.358+0000 7f7cb8855640 1 -- 172.21.15.5:0/573491463 shutdown_connections 2024-03-24T10:51:24.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.358+0000 7f7cb8855640 1 -- 172.21.15.5:0/573491463 wait complete. 2024-03-24T10:51:24.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.360+0000 7f7cb8855640 1 Processor -- start 2024-03-24T10:51:24.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.360+0000 7f7cb8855640 1 -- start start 2024-03-24T10:51:24.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.360+0000 7f7cb8855640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cb40cae90 0x7f7cb40ab160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:24.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.360+0000 7f7cb8855640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7cb40cb270 0x7f7cb40ab6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:24.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.360+0000 7f7cb8855640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cb40cd960 0x7f7cb4155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:24.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.360+0000 7f7cb8855640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb40add60 con 0x7f7cb40cae90 2024-03-24T10:51:24.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.360+0000 7f7cb8855640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7cb40ae350 con 0x7f7cb40cd960 2024-03-24T10:51:24.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.360+0000 7f7cb8855640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7cb4154e90 con 0x7f7cb40cb270 2024-03-24T10:51:24.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.361+0000 7f7cb259c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cb40cae90 0x7f7cb40ab160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:24.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.361+0000 7f7cb1d9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7cb40cb270 0x7f7cb40ab6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:24.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.361+0000 7f7cb259c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cb40cae90 0x7f7cb40ab160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43202/0 (socket says 172.21.15.5:43202) 2024-03-24T10:51:24.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.361+0000 7f7cb259c640 1 -- 172.21.15.5:0/1880697627 learned_addr learned my addr 172.21.15.5:0/1880697627 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:24.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.362+0000 7f7cb2d9d640 1 --2- 172.21.15.5:0/1880697627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cb40cd960 0x7f7cb4155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:24.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.366+0000 7f7cb1d9b640 1 -- 172.21.15.5:0/1880697627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cb40cd960 msgr2=0x7f7cb4155930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:24.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.366+0000 7f7cb1d9b640 1 --2- 172.21.15.5:0/1880697627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cb40cd960 0x7f7cb4155930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:24.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.366+0000 7f7cb1d9b640 1 -- 172.21.15.5:0/1880697627 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cb40cae90 msgr2=0x7f7cb40ab160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:24.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.366+0000 7f7cb1d9b640 1 --2- 172.21.15.5:0/1880697627 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cb40cae90 0x7f7cb40ab160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:24.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.366+0000 7f7cb1d9b640 1 -- 172.21.15.5:0/1880697627 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca40094d0 con 0x7f7cb40cb270 2024-03-24T10:51:24.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.369+0000 7f7cb1d9b640 1 --2- 172.21.15.5:0/1880697627 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7cb40cb270 0x7f7cb40ab6e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800c370 tx=0x7f7ca800c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:24.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.375+0000 7f7ca2ffd640 1 -- 172.21.15.5:0/1880697627 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca80052d0 con 0x7f7cb40cb270 2024-03-24T10:51:24.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.375+0000 7f7ca2ffd640 1 -- 172.21.15.5:0/1880697627 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ca8005450 con 0x7f7cb40cb270 2024-03-24T10:51:24.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.375+0000 7f7cb8855640 1 -- 172.21.15.5:0/1880697627 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cb4156cb0 con 0x7f7cb40cb270 2024-03-24T10:51:24.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.375+0000 7f7ca2ffd640 1 -- 172.21.15.5:0/1880697627 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca8007260 con 0x7f7cb40cb270 2024-03-24T10:51:24.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.375+0000 7f7cb8855640 1 -- 172.21.15.5:0/1880697627 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cb4155130 con 0x7f7cb40cb270 2024-03-24T10:51:24.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.376+0000 7f7cb8855640 1 -- 172.21.15.5:0/1880697627 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cb40732f0 con 0x7f7cb40cb270 2024-03-24T10:51:24.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.664+0000 7f7ca2ffd640 1 -- 172.21.15.5:0/1880697627 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7ca80073c0 con 0x7f7cb40cb270 2024-03-24T10:51:24.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.665+0000 7f7ca2ffd640 1 --2- 172.21.15.5:0/1880697627 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7c9406b890 0x7f7c9406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:24.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.665+0000 7f7cb259c640 1 --2- 172.21.15.5:0/1880697627 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7c9406b890 0x7f7c9406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:24.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.665+0000 7f7cb259c640 1 --2- 172.21.15.5:0/1880697627 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7c9406b890 0x7f7c9406dd50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7ca402ba70 tx=0x7f7ca4008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:24.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.671+0000 7f7ca2ffd640 1 -- 172.21.15.5:0/1880697627 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f7ca80581b0 con 0x7f7cb40cb270 2024-03-24T10:51:24.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.682+0000 7f7ca2ffd640 1 -- 172.21.15.5:0/1880697627 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7ca8054450 con 0x7f7cb40cb270 2024-03-24T10:51:24.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.925+0000 7f7cb8855640 1 -- 172.21.15.5:0/1880697627 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7cb40c8760 con 0x7f7cb40cb270 2024-03-24T10:51:24.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.947+0000 7f7ca2ffd640 1 -- 172.21.15.5:0/1880697627 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f7ca8020020 con 0x7f7cb40cb270 2024-03-24T10:51:24.947 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:24.947 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:24.948 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:24.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.949+0000 7f7cb8855640 1 -- 172.21.15.5:0/1880697627 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7c9406b890 msgr2=0x7f7c9406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:24.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.949+0000 7f7cb8855640 1 --2- 172.21.15.5:0/1880697627 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7c9406b890 0x7f7c9406dd50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7ca402ba70 tx=0x7f7ca4008030 comp rx=0 tx=0).stop 2024-03-24T10:51:24.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.949+0000 7f7cb8855640 1 -- 172.21.15.5:0/1880697627 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7cb40cb270 msgr2=0x7f7cb40ab6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:24.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.949+0000 7f7cb8855640 1 --2- 172.21.15.5:0/1880697627 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7cb40cb270 0x7f7cb40ab6e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7ca800c370 tx=0x7f7ca800c830 comp rx=0 tx=0).stop 2024-03-24T10:51:24.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.949+0000 7f7cb8855640 1 -- 172.21.15.5:0/1880697627 shutdown_connections 2024-03-24T10:51:24.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.949+0000 7f7cb8855640 1 --2- 172.21.15.5:0/1880697627 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7c9406b890 0x7f7c9406dd50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:24.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.949+0000 7f7cb8855640 1 --2- 172.21.15.5:0/1880697627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7cb40cd960 0x7f7cb4155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:24.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.949+0000 7f7cb8855640 1 --2- 172.21.15.5:0/1880697627 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7cb40cb270 0x7f7cb40ab6e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:24.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.950+0000 7f7cb8855640 1 --2- 172.21.15.5:0/1880697627 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7cb40cae90 0x7f7cb40ab160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:24.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.950+0000 7f7cb8855640 1 -- 172.21.15.5:0/1880697627 >> 172.21.15.5:0/1880697627 conn(0x7f7cb40cfb80 msgr2=0x7f7cb4159ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:24.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.950+0000 7f7cb8855640 1 -- 172.21.15.5:0/1880697627 shutdown_connections 2024-03-24T10:51:24.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:24.950+0000 7f7cb8855640 1 -- 172.21.15.5:0/1880697627 wait complete. 2024-03-24T10:51:25.566 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:25.566 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:26.567 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:26.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.670+0000 7fd900193640 1 Processor -- start 2024-03-24T10:51:26.670 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.670+0000 7fd900193640 1 -- start start 2024-03-24T10:51:26.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.671+0000 7fd900193640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd8f8152140 0x7fd8f815bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:26.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.671+0000 7fd900193640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd8f806fc60 0x7fd8f80c4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:26.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.671+0000 7fd900193640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8f815b530 con 0x7fd8f815c1e0 2024-03-24T10:51:26.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.671+0000 7fd900193640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fd8f815b810 con 0x7fd8f806fc60 2024-03-24T10:51:26.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.671+0000 7fd900193640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd8f8159430 con 0x7fd8f8152140 2024-03-24T10:51:26.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.672+0000 7fd8fd72e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd8f806fc60 0x7fd8f80c4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:26.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.672+0000 7fd8fdf2f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd8f8152140 0x7fd8f815bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:26.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.672+0000 7fd8fe730640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fd8f815c1e0 0x7fd8f80c8150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:55566/0 (socket says 172.21.15.5:55566) 2024-03-24T10:51:26.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.672+0000 7fd8fe730640 1 -- 172.21.15.5:0/1196966823 learned_addr learned my addr 172.21.15.5:0/1196966823 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:51:26.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.674+0000 7fd8fd72e640 1 -- 172.21.15.5:0/1196966823 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd8f8152140 msgr2=0x7fd8f815bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:26.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.674+0000 7fd8fd72e640 1 --2- 172.21.15.5:0/1196966823 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd8f8152140 0x7fd8f815bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:26.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.674+0000 7fd8fd72e640 1 -- 172.21.15.5:0/1196966823 >> v1:172.21.15.57:6789/0 conn(0x7fd8f815c1e0 legacy=0x7fd8f80c8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:26.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.674+0000 7fd8fd72e640 1 -- 172.21.15.5:0/1196966823 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f81584c0 con 0x7fd8f806fc60 2024-03-24T10:51:26.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.676+0000 7fd8fd72e640 1 --2- 172.21.15.5:0/1196966823 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8f806fc60 0x7fd8f80c4e40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009d90 tx=0x7fd8ec02bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad440739e55f83af server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:26.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.680+0000 7fd8fcf2d640 1 -- 172.21.15.5:0/1196966823 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec02cce0 con 0x7fd8f806fc60 2024-03-24T10:51:26.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.680+0000 7fd8fcf2d640 1 -- 172.21.15.5:0/1196966823 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8ec02ce40 con 0x7fd8f806fc60 2024-03-24T10:51:26.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.680+0000 7fd8fcf2d640 1 -- 172.21.15.5:0/1196966823 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec034660 con 0x7fd8f806fc60 2024-03-24T10:51:26.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.681+0000 7fd900193640 1 -- 172.21.15.5:0/1196966823 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8f806fc60 msgr2=0x7fd8f80c4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:26.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.681+0000 7fd900193640 1 --2- 172.21.15.5:0/1196966823 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8f806fc60 0x7fd8f80c4e40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009d90 tx=0x7fd8ec02bf30 comp rx=0 tx=0).stop 2024-03-24T10:51:26.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.681+0000 7fd900193640 1 -- 172.21.15.5:0/1196966823 shutdown_connections 2024-03-24T10:51:26.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.681+0000 7fd900193640 1 --2- 172.21.15.5:0/1196966823 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8f806fc60 0x7fd8f80c4e40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:26.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.681+0000 7fd900193640 1 --2- 172.21.15.5:0/1196966823 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd8f8152140 0x7fd8f815bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:26.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.681+0000 7fd900193640 1 -- 172.21.15.5:0/1196966823 >> 172.21.15.5:0/1196966823 conn(0x7fd8f806b130 msgr2=0x7fd8f806b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:26.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.682+0000 7fd900193640 1 -- 172.21.15.5:0/1196966823 shutdown_connections 2024-03-24T10:51:26.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.682+0000 7fd900193640 1 -- 172.21.15.5:0/1196966823 wait complete. 2024-03-24T10:51:26.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.683+0000 7fd900193640 1 Processor -- start 2024-03-24T10:51:26.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.683+0000 7fd900193640 1 -- start start 2024-03-24T10:51:26.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.685+0000 7fd900193640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd8f806fc60 0x7fd8f8072b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:26.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.685+0000 7fd900193640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8f8152140 0x7fd8f8149970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:26.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.685+0000 7fd900193640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8f815c1e0 0x7fd8f8157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:26.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.685+0000 7fd900193640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f814a590 con 0x7fd8f815c1e0 2024-03-24T10:51:26.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.685+0000 7fd900193640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f81f1a70 con 0x7fd8f8152140 2024-03-24T10:51:26.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.685+0000 7fd900193640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd8f81f11d0 con 0x7fd8f806fc60 2024-03-24T10:51:26.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.685+0000 7fd8fe730640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8f815c1e0 0x7fd8f8157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:26.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.685+0000 7fd8fd72e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8f8152140 0x7fd8f8149970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:26.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.685+0000 7fd8fdf2f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd8f806fc60 0x7fd8f8072b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:26.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.686+0000 7fd8fe730640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8f815c1e0 0x7fd8f8157430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43220/0 (socket says 172.21.15.5:43220) 2024-03-24T10:51:26.686 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.686+0000 7fd8fe730640 1 -- 172.21.15.5:0/941606444 learned_addr learned my addr 172.21.15.5:0/941606444 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:26.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.691+0000 7fd8fe730640 1 -- 172.21.15.5:0/941606444 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd8f806fc60 msgr2=0x7fd8f8072b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:26.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.691+0000 7fd8fe730640 1 --2- 172.21.15.5:0/941606444 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd8f806fc60 0x7fd8f8072b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:26.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.691+0000 7fd8fe730640 1 -- 172.21.15.5:0/941606444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8f8152140 msgr2=0x7fd8f8149970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:26.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.691+0000 7fd8fe730640 1 --2- 172.21.15.5:0/941606444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8f8152140 0x7fd8f8149970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:26.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.691+0000 7fd8fe730640 1 -- 172.21.15.5:0/941606444 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8ec009a70 con 0x7fd8f815c1e0 2024-03-24T10:51:26.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.694+0000 7fd8fe730640 1 --2- 172.21.15.5:0/941606444 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8f815c1e0 0x7fd8f8157430 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800c430 tx=0x7fd8e800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:26.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.699+0000 7fd8f67fc640 1 -- 172.21.15.5:0/941606444 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8005ce0 con 0x7fd8f815c1e0 2024-03-24T10:51:26.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.699+0000 7fd8f67fc640 1 -- 172.21.15.5:0/941606444 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e8005e40 con 0x7fd8f815c1e0 2024-03-24T10:51:26.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.699+0000 7fd8f67fc640 1 -- 172.21.15.5:0/941606444 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e80072d0 con 0x7fd8f815c1e0 2024-03-24T10:51:26.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.699+0000 7fd900193640 1 -- 172.21.15.5:0/941606444 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8f81f14b0 con 0x7fd8f815c1e0 2024-03-24T10:51:26.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.699+0000 7fd900193640 1 -- 172.21.15.5:0/941606444 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8f8149170 con 0x7fd8f815c1e0 2024-03-24T10:51:26.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.700+0000 7fd900193640 1 -- 172.21.15.5:0/941606444 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8f8062a00 con 0x7fd8f815c1e0 2024-03-24T10:51:26.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.731+0000 7fd8f67fc640 1 -- 172.21.15.5:0/941606444 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd8e8007430 con 0x7fd8f815c1e0 2024-03-24T10:51:26.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.732+0000 7fd8f67fc640 1 --2- 172.21.15.5:0/941606444 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd8e406b890 0x7fd8e406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:26.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.732+0000 7fd8fdf2f640 1 --2- 172.21.15.5:0/941606444 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd8e406b890 0x7fd8e406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:26.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.732+0000 7fd8fdf2f640 1 --2- 172.21.15.5:0/941606444 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd8e406b890 0x7fd8e406dd50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0009b40 tx=0x7fd8f0009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:26.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:26.736+0000 7fd8f67fc640 1 -- 172.21.15.5:0/941606444 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fd8e80583c0 con 0x7fd8f815c1e0 2024-03-24T10:51:27.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:27.018+0000 7fd8f67fc640 1 -- 172.21.15.5:0/941606444 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd8e8054660 con 0x7fd8f815c1e0 2024-03-24T10:51:27.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:27.245+0000 7fd900193640 1 -- 172.21.15.5:0/941606444 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd8f80681d0 con 0x7fd8f815c1e0 2024-03-24T10:51:27.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:27.260+0000 7fd8f67fc640 1 -- 172.21.15.5:0/941606444 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fd8e8017090 con 0x7fd8f815c1e0 2024-03-24T10:51:27.260 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:27.261 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:27.261 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:27.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:27.263+0000 7fd900193640 1 -- 172.21.15.5:0/941606444 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd8e406b890 msgr2=0x7fd8e406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:27.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:27.263+0000 7fd900193640 1 --2- 172.21.15.5:0/941606444 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd8e406b890 0x7fd8e406dd50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0009b40 tx=0x7fd8f0009190 comp rx=0 tx=0).stop 2024-03-24T10:51:27.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:27.263+0000 7fd900193640 1 -- 172.21.15.5:0/941606444 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8f815c1e0 msgr2=0x7fd8f8157430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:27.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:27.263+0000 7fd900193640 1 --2- 172.21.15.5:0/941606444 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8f815c1e0 0x7fd8f8157430 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800c430 tx=0x7fd8e800c8f0 comp rx=0 tx=0).stop 2024-03-24T10:51:27.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:27.263+0000 7fd900193640 1 -- 172.21.15.5:0/941606444 shutdown_connections 2024-03-24T10:51:27.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:27.263+0000 7fd900193640 1 --2- 172.21.15.5:0/941606444 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd8e406b890 0x7fd8e406dd50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:27.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:27.263+0000 7fd900193640 1 --2- 172.21.15.5:0/941606444 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd8f815c1e0 0x7fd8f8157430 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:27.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:27.263+0000 7fd900193640 1 --2- 172.21.15.5:0/941606444 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd8f8152140 0x7fd8f8149970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:27.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:27.263+0000 7fd900193640 1 --2- 172.21.15.5:0/941606444 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd8f806fc60 0x7fd8f8072b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:27.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:27.263+0000 7fd900193640 1 -- 172.21.15.5:0/941606444 >> 172.21.15.5:0/941606444 conn(0x7fd8f806b130 msgr2=0x7fd8f806b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:27.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:27.263+0000 7fd900193640 1 -- 172.21.15.5:0/941606444 shutdown_connections 2024-03-24T10:51:27.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:27.263+0000 7fd900193640 1 -- 172.21.15.5:0/941606444 wait complete. 2024-03-24T10:51:27.877 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:27.877 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:28.878 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:28.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.982+0000 7fc3fc932640 1 Processor -- start 2024-03-24T10:51:28.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.982+0000 7fc3fc932640 1 -- start start 2024-03-24T10:51:28.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.983+0000 7fc3fc932640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc3f81543a0 0x7fc3f815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:28.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.983+0000 7fc3fc932640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc3f806d4f0 0x7fc3f80cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:28.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.983+0000 7fc3fc932640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3f80cb2f0 con 0x7fc3f80ccaa0 2024-03-24T10:51:28.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.983+0000 7fc3fc932640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fc3f80c6750 con 0x7fc3f81543a0 2024-03-24T10:51:28.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.983+0000 7fc3fc932640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc3f80cc380 con 0x7fc3f806d4f0 2024-03-24T10:51:28.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.983+0000 7fc3f659c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc3f81543a0 0x7fc3f815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:28.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.983+0000 7fc3f5d9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc3f806d4f0 0x7fc3f80cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:28.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.983+0000 7fc3f6d9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fc3f80ccaa0 0x7fc3f806bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:55580/0 (socket says 172.21.15.5:55580) 2024-03-24T10:51:28.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.983+0000 7fc3f6d9d640 1 -- 172.21.15.5:0/264250897 learned_addr learned my addr 172.21.15.5:0/264250897 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:51:28.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.986+0000 7fc3f659c640 1 -- 172.21.15.5:0/264250897 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc3f806d4f0 msgr2=0x7fc3f80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:28.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.986+0000 7fc3f659c640 1 --2- 172.21.15.5:0/264250897 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc3f806d4f0 0x7fc3f80cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:28.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.986+0000 7fc3f659c640 1 -- 172.21.15.5:0/264250897 >> v1:172.21.15.57:6789/0 conn(0x7fc3f80ccaa0 legacy=0x7fc3f806bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:28.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.986+0000 7fc3f659c640 1 -- 172.21.15.5:0/264250897 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f80ca7e0 con 0x7fc3f81543a0 2024-03-24T10:51:28.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.988+0000 7fc3f659c640 1 --2- 172.21.15.5:0/264250897 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3f81543a0 0x7fc3f815ac00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80097a0 tx=0x7fc3e802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b3c94aedf98e52d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:28.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.992+0000 7fc3f559a640 1 -- 172.21.15.5:0/264250897 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8030610 con 0x7fc3f81543a0 2024-03-24T10:51:28.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.992+0000 7fc3f559a640 1 -- 172.21.15.5:0/264250897 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3e8030770 con 0x7fc3f81543a0 2024-03-24T10:51:28.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.992+0000 7fc3f559a640 1 -- 172.21.15.5:0/264250897 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8030b00 con 0x7fc3f81543a0 2024-03-24T10:51:28.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.992+0000 7fc3fc932640 1 -- 172.21.15.5:0/264250897 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3f81543a0 msgr2=0x7fc3f815ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:28.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.992+0000 7fc3fc932640 1 --2- 172.21.15.5:0/264250897 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3f81543a0 0x7fc3f815ac00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80097a0 tx=0x7fc3e802b960 comp rx=0 tx=0).stop 2024-03-24T10:51:28.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.993+0000 7fc3fc932640 1 -- 172.21.15.5:0/264250897 shutdown_connections 2024-03-24T10:51:28.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.993+0000 7fc3fc932640 1 --2- 172.21.15.5:0/264250897 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc3f806d4f0 0x7fc3f80cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:28.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.993+0000 7fc3fc932640 1 --2- 172.21.15.5:0/264250897 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3f81543a0 0x7fc3f815ac00 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:28.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.993+0000 7fc3fc932640 1 -- 172.21.15.5:0/264250897 >> 172.21.15.5:0/264250897 conn(0x7fc3f8072ef0 msgr2=0x7fc3f8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:28.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.993+0000 7fc3fc932640 1 -- 172.21.15.5:0/264250897 shutdown_connections 2024-03-24T10:51:28.993 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.993+0000 7fc3fc932640 1 -- 172.21.15.5:0/264250897 wait complete. 2024-03-24T10:51:28.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.994+0000 7fc3fc932640 1 Processor -- start 2024-03-24T10:51:28.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.994+0000 7fc3fc932640 1 -- start start 2024-03-24T10:51:28.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.995+0000 7fc3fc932640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc3f806d4f0 0x7fc3f80ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:28.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.995+0000 7fc3fc932640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3f80ccaa0 0x7fc3f8156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:28.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.995+0000 7fc3fc932640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3f81543a0 0x7fc3f81571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:28.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.995+0000 7fc3fc932640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f8159900 con 0x7fc3f81543a0 2024-03-24T10:51:28.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.995+0000 7fc3fc932640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f8159b30 con 0x7fc3f80ccaa0 2024-03-24T10:51:28.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.995+0000 7fc3fc932640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc3f8159e10 con 0x7fc3f806d4f0 2024-03-24T10:51:28.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.996+0000 7fc3f6d9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3f81543a0 0x7fc3f81571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:28.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.996+0000 7fc3f659c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc3f806d4f0 0x7fc3f80ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:28.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.996+0000 7fc3f5d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3f80ccaa0 0x7fc3f8156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:28.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.996+0000 7fc3f6d9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3f81543a0 0x7fc3f81571d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43252/0 (socket says 172.21.15.5:43252) 2024-03-24T10:51:28.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:28.996+0000 7fc3f6d9d640 1 -- 172.21.15.5:0/3033119269 learned_addr learned my addr 172.21.15.5:0/3033119269 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:29.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.001+0000 7fc3f5d9b640 1 -- 172.21.15.5:0/3033119269 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc3f806d4f0 msgr2=0x7fc3f80ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:29.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.001+0000 7fc3f5d9b640 1 --2- 172.21.15.5:0/3033119269 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc3f806d4f0 0x7fc3f80ceb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:29.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.001+0000 7fc3f5d9b640 1 -- 172.21.15.5:0/3033119269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3f81543a0 msgr2=0x7fc3f81571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:29.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.001+0000 7fc3f5d9b640 1 --2- 172.21.15.5:0/3033119269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3f81543a0 0x7fc3f81571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:29.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.001+0000 7fc3f5d9b640 1 -- 172.21.15.5:0/3033119269 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e8009400 con 0x7fc3f80ccaa0 2024-03-24T10:51:29.005 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.005+0000 7fc3f5d9b640 1 --2- 172.21.15.5:0/3033119269 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3f80ccaa0 0x7fc3f8156c90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec00e880 tx=0x7fc3ec00ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:29.009 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.009+0000 7fc3e6ffd640 1 -- 172.21.15.5:0/3033119269 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec00c940 con 0x7fc3f80ccaa0 2024-03-24T10:51:29.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.009+0000 7fc3e6ffd640 1 -- 172.21.15.5:0/3033119269 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3ec00caa0 con 0x7fc3f80ccaa0 2024-03-24T10:51:29.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.009+0000 7fc3fc932640 1 -- 172.21.15.5:0/3033119269 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3f8154a60 con 0x7fc3f80ccaa0 2024-03-24T10:51:29.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.009+0000 7fc3e6ffd640 1 -- 172.21.15.5:0/3033119269 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec00ce50 con 0x7fc3f80ccaa0 2024-03-24T10:51:29.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.010+0000 7fc3fc932640 1 -- 172.21.15.5:0/3033119269 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3f8155020 con 0x7fc3f80ccaa0 2024-03-24T10:51:29.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.010+0000 7fc3fc932640 1 -- 172.21.15.5:0/3033119269 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3f80681d0 con 0x7fc3f80ccaa0 2024-03-24T10:51:29.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.333+0000 7fc3e6ffd640 1 -- 172.21.15.5:0/3033119269 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc3ec013470 con 0x7fc3f80ccaa0 2024-03-24T10:51:29.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.333+0000 7fc3e6ffd640 1 --2- 172.21.15.5:0/3033119269 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc3d806b860 0x7fc3d806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:29.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.333+0000 7fc3f659c640 1 --2- 172.21.15.5:0/3033119269 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc3d806b860 0x7fc3d806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:29.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.334+0000 7fc3f659c640 1 --2- 172.21.15.5:0/3033119269 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc3d806b860 0x7fc3d806dd20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8036000 tx=0x7fc3e803b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:29.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.340+0000 7fc3e6ffd640 1 -- 172.21.15.5:0/3033119269 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fc3ec059280 con 0x7fc3f80ccaa0 2024-03-24T10:51:29.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.359+0000 7fc3e6ffd640 1 -- 172.21.15.5:0/3033119269 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc3ec055520 con 0x7fc3f80ccaa0 2024-03-24T10:51:29.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.581+0000 7fc3fc932640 1 -- 172.21.15.5:0/3033119269 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3f80caec0 con 0x7fc3f80ccaa0 2024-03-24T10:51:29.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.601+0000 7fc3e6ffd640 1 -- 172.21.15.5:0/3033119269 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fc3ec017090 con 0x7fc3f80ccaa0 2024-03-24T10:51:29.602 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:29.602 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:29.602 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:29.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.604+0000 7fc3fc932640 1 -- 172.21.15.5:0/3033119269 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc3d806b860 msgr2=0x7fc3d806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:29.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.604+0000 7fc3fc932640 1 --2- 172.21.15.5:0/3033119269 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc3d806b860 0x7fc3d806dd20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8036000 tx=0x7fc3e803b040 comp rx=0 tx=0).stop 2024-03-24T10:51:29.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.604+0000 7fc3fc932640 1 -- 172.21.15.5:0/3033119269 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3f80ccaa0 msgr2=0x7fc3f8156c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:29.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.604+0000 7fc3fc932640 1 --2- 172.21.15.5:0/3033119269 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3f80ccaa0 0x7fc3f8156c90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec00e880 tx=0x7fc3ec00ed40 comp rx=0 tx=0).stop 2024-03-24T10:51:29.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.604+0000 7fc3fc932640 1 -- 172.21.15.5:0/3033119269 shutdown_connections 2024-03-24T10:51:29.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.604+0000 7fc3fc932640 1 --2- 172.21.15.5:0/3033119269 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc3d806b860 0x7fc3d806dd20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:29.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.604+0000 7fc3fc932640 1 --2- 172.21.15.5:0/3033119269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc3f81543a0 0x7fc3f81571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:29.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.604+0000 7fc3fc932640 1 --2- 172.21.15.5:0/3033119269 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc3f80ccaa0 0x7fc3f8156c90 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:29.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.604+0000 7fc3fc932640 1 --2- 172.21.15.5:0/3033119269 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc3f806d4f0 0x7fc3f80ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:29.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.604+0000 7fc3fc932640 1 -- 172.21.15.5:0/3033119269 >> 172.21.15.5:0/3033119269 conn(0x7fc3f8072ef0 msgr2=0x7fc3f806be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:29.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.604+0000 7fc3fc932640 1 -- 172.21.15.5:0/3033119269 shutdown_connections 2024-03-24T10:51:29.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:29.604+0000 7fc3fc932640 1 -- 172.21.15.5:0/3033119269 wait complete. 2024-03-24T10:51:30.219 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:30.219 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:31.220 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:31.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.322+0000 7ff5ef92e640 1 Processor -- start 2024-03-24T10:51:31.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.323+0000 7ff5ef92e640 1 -- start start 2024-03-24T10:51:31.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.323+0000 7ff5ef92e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff5e80cae90 0x7ff5e80cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:31.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.323+0000 7ff5ef92e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5e80ccba0 con 0x7ff5e80cd960 2024-03-24T10:51:31.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.323+0000 7ff5ef92e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5e80cce40 con 0x7ff5e80cb270 2024-03-24T10:51:31.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.323+0000 7ff5ef92e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7ff5e80cc9b0 con 0x7ff5e80cae90 2024-03-24T10:51:31.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.324+0000 7ff5ecec9640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff5e80cb270 0x7ff5e80cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:55582/0 (socket says 172.21.15.5:55582) 2024-03-24T10:51:31.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.324+0000 7ff5ecec9640 1 -- 172.21.15.5:0/764581069 learned_addr learned my addr 172.21.15.5:0/764581069 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:51:31.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.325+0000 7ff5ed6ca640 1 --2- 172.21.15.5:0/764581069 >> v2:172.21.15.5:3300/0 conn(0x7ff5e80cae90 0x7ff5e80cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:31.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.327+0000 7ff5dffff640 1 -- 172.21.15.5:0/764581069 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1643343542 0 0) 0x7ff5e80cce40 con 0x7ff5e80cb270 2024-03-24T10:51:31.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.327+0000 7ff5dffff640 1 -- 172.21.15.5:0/764581069 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5d4003520 con 0x7ff5e80cb270 2024-03-24T10:51:31.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.330+0000 7ff5ed6ca640 1 -- 172.21.15.5:0/764581069 >> v1:172.21.15.57:6789/0 conn(0x7ff5e80cb270 legacy=0x7ff5e80cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:31.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.330+0000 7ff5ed6ca640 1 -- 172.21.15.5:0/764581069 >> v1:172.21.15.5:6789/0 conn(0x7ff5e80cd960 legacy=0x7ff5e80cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:31.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.330+0000 7ff5ed6ca640 1 -- 172.21.15.5:0/764581069 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e815a590 con 0x7ff5e80cae90 2024-03-24T10:51:31.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.332+0000 7ff5ed6ca640 1 --2- 172.21.15.5:0/764581069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5e80cae90 0x7ff5e80cf530 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40097a0 tx=0x7ff5e402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37204a40755533fd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:31.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.337+0000 7ff5dffff640 1 -- 172.21.15.5:0/764581069 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e402f810 con 0x7ff5e80cae90 2024-03-24T10:51:31.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.337+0000 7ff5dffff640 1 -- 172.21.15.5:0/764581069 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5e402f970 con 0x7ff5e80cae90 2024-03-24T10:51:31.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.337+0000 7ff5dffff640 1 -- 172.21.15.5:0/764581069 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e402fd00 con 0x7ff5e80cae90 2024-03-24T10:51:31.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.338+0000 7ff5ef92e640 1 -- 172.21.15.5:0/764581069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5e80cae90 msgr2=0x7ff5e80cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:31.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.338+0000 7ff5ef92e640 1 --2- 172.21.15.5:0/764581069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5e80cae90 0x7ff5e80cf530 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40097a0 tx=0x7ff5e402b9f0 comp rx=0 tx=0).stop 2024-03-24T10:51:31.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.338+0000 7ff5ef92e640 1 -- 172.21.15.5:0/764581069 shutdown_connections 2024-03-24T10:51:31.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.338+0000 7ff5ef92e640 1 --2- 172.21.15.5:0/764581069 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5e80cae90 0x7ff5e80cf530 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:31.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.338+0000 7ff5ef92e640 1 -- 172.21.15.5:0/764581069 >> 172.21.15.5:0/764581069 conn(0x7ff5e80cfb80 msgr2=0x7ff5e80cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:31.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.338+0000 7ff5ef92e640 1 -- 172.21.15.5:0/764581069 shutdown_connections 2024-03-24T10:51:31.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.338+0000 7ff5ef92e640 1 -- 172.21.15.5:0/764581069 wait complete. 2024-03-24T10:51:31.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.339+0000 7ff5ef92e640 1 Processor -- start 2024-03-24T10:51:31.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.339+0000 7ff5ef92e640 1 -- start start 2024-03-24T10:51:31.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.340+0000 7ff5ef92e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5e80cae90 0x7ff5e80b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:31.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.340+0000 7ff5ef92e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff5e80cb270 0x7ff5e80b1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:31.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.340+0000 7ff5ef92e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5e80cd960 0x7ff5e80b4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:31.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.340+0000 7ff5ef92e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e81f3280 con 0x7ff5e80cd960 2024-03-24T10:51:31.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.340+0000 7ff5ef92e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e80ab9a0 con 0x7ff5e80cae90 2024-03-24T10:51:31.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.340+0000 7ff5ef92e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff5e80b2d70 con 0x7ff5e80cb270 2024-03-24T10:51:31.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.340+0000 7ff5edecb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5e80cd960 0x7ff5e80b4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:31.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.341+0000 7ff5ed6ca640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5e80cae90 0x7ff5e80b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:31.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.341+0000 7ff5edecb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5e80cd960 0x7ff5e80b4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43278/0 (socket says 172.21.15.5:43278) 2024-03-24T10:51:31.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.341+0000 7ff5edecb640 1 -- 172.21.15.5:0/1496217743 learned_addr learned my addr 172.21.15.5:0/1496217743 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:31.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.341+0000 7ff5ecec9640 1 --2- 172.21.15.5:0/1496217743 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff5e80cb270 0x7ff5e80b1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:31.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.344+0000 7ff5edecb640 1 -- 172.21.15.5:0/1496217743 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff5e80cb270 msgr2=0x7ff5e80b1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:31.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.344+0000 7ff5edecb640 1 --2- 172.21.15.5:0/1496217743 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff5e80cb270 0x7ff5e80b1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:31.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.344+0000 7ff5edecb640 1 -- 172.21.15.5:0/1496217743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5e80cae90 msgr2=0x7ff5e80b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:31.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.344+0000 7ff5edecb640 1 --2- 172.21.15.5:0/1496217743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5e80cae90 0x7ff5e80b43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:31.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.344+0000 7ff5edecb640 1 -- 172.21.15.5:0/1496217743 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e4009400 con 0x7ff5e80cd960 2024-03-24T10:51:31.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.347+0000 7ff5edecb640 1 --2- 172.21.15.5:0/1496217743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5e80cd960 0x7ff5e80b4d20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff5e000c6b0 tx=0x7ff5e000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:31.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.351+0000 7ff5ddffb640 1 -- 172.21.15.5:0/1496217743 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0003b70 con 0x7ff5e80cd960 2024-03-24T10:51:31.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.351+0000 7ff5ddffb640 1 -- 172.21.15.5:0/1496217743 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5e0003cf0 con 0x7ff5e80cd960 2024-03-24T10:51:31.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.351+0000 7ff5ef92e640 1 -- 172.21.15.5:0/1496217743 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5e8152b00 con 0x7ff5e80cd960 2024-03-24T10:51:31.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.351+0000 7ff5ddffb640 1 -- 172.21.15.5:0/1496217743 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0003ea0 con 0x7ff5e80cd960 2024-03-24T10:51:31.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.352+0000 7ff5ef92e640 1 -- 172.21.15.5:0/1496217743 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5e81f3460 con 0x7ff5e80cd960 2024-03-24T10:51:31.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.353+0000 7ff5ef92e640 1 -- 172.21.15.5:0/1496217743 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5e80b4a40 con 0x7ff5e80cd960 2024-03-24T10:51:31.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.621+0000 7ff5ddffb640 1 -- 172.21.15.5:0/1496217743 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff5e0005310 con 0x7ff5e80cd960 2024-03-24T10:51:31.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.622+0000 7ff5ddffb640 1 --2- 172.21.15.5:0/1496217743 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff5cc06b890 0x7ff5cc06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:31.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.622+0000 7ff5ed6ca640 1 --2- 172.21.15.5:0/1496217743 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff5cc06b890 0x7ff5cc06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:31.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.622+0000 7ff5ed6ca640 1 --2- 172.21.15.5:0/1496217743 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff5cc06b890 0x7ff5cc06dd50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4036000 tx=0x7ff5e4008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:31.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.637+0000 7ff5ddffb640 1 -- 172.21.15.5:0/1496217743 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7ff5e0058f20 con 0x7ff5e80cd960 2024-03-24T10:51:31.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.637+0000 7ff5ddffb640 1 -- 172.21.15.5:0/1496217743 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff5e00551c0 con 0x7ff5e80cd960 2024-03-24T10:51:31.875 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.875+0000 7ff5ef92e640 1 -- 172.21.15.5:0/1496217743 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5e80b4a40 con 0x7ff5e80cd960 2024-03-24T10:51:31.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.893+0000 7ff5ddffb640 1 -- 172.21.15.5:0/1496217743 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7ff5e80b4a40 con 0x7ff5e80cd960 2024-03-24T10:51:31.894 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:31.894 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:31.894 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:31.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.896+0000 7ff5ef92e640 1 -- 172.21.15.5:0/1496217743 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff5cc06b890 msgr2=0x7ff5cc06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:31.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.896+0000 7ff5ef92e640 1 --2- 172.21.15.5:0/1496217743 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff5cc06b890 0x7ff5cc06dd50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4036000 tx=0x7ff5e4008030 comp rx=0 tx=0).stop 2024-03-24T10:51:31.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.896+0000 7ff5ef92e640 1 -- 172.21.15.5:0/1496217743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5e80cd960 msgr2=0x7ff5e80b4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:31.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.896+0000 7ff5ef92e640 1 --2- 172.21.15.5:0/1496217743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5e80cd960 0x7ff5e80b4d20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff5e000c6b0 tx=0x7ff5e000cb70 comp rx=0 tx=0).stop 2024-03-24T10:51:31.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.896+0000 7ff5ef92e640 1 -- 172.21.15.5:0/1496217743 shutdown_connections 2024-03-24T10:51:31.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.896+0000 7ff5ef92e640 1 --2- 172.21.15.5:0/1496217743 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff5cc06b890 0x7ff5cc06dd50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:31.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.896+0000 7ff5ef92e640 1 --2- 172.21.15.5:0/1496217743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5e80cd960 0x7ff5e80b4d20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:31.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.896+0000 7ff5ef92e640 1 --2- 172.21.15.5:0/1496217743 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff5e80cb270 0x7ff5e80b1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:31.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.896+0000 7ff5ef92e640 1 --2- 172.21.15.5:0/1496217743 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5e80cae90 0x7ff5e80b43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:31.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.896+0000 7ff5ef92e640 1 -- 172.21.15.5:0/1496217743 >> 172.21.15.5:0/1496217743 conn(0x7ff5e80cfb80 msgr2=0x7ff5e80c7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:31.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.896+0000 7ff5ef92e640 1 -- 172.21.15.5:0/1496217743 shutdown_connections 2024-03-24T10:51:31.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:31.896+0000 7ff5ef92e640 1 -- 172.21.15.5:0/1496217743 wait complete. 2024-03-24T10:51:31.966 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:51:31.968 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:51:32.513 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:32.513 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:33.514 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:33.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.617+0000 7fb699af4640 1 Processor -- start 2024-03-24T10:51:33.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.617+0000 7fb699af4640 1 -- start start 2024-03-24T10:51:33.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.619+0000 7fb699af4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6941543a0 0x7fb69415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:33.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.619+0000 7fb699af4640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb69406d4f0 0x7fb6940cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:33.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.619+0000 7fb699af4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6940cb2f0 con 0x7fb6940ccaa0 2024-03-24T10:51:33.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.619+0000 7fb699af4640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fb6940c6750 con 0x7fb69406d4f0 2024-03-24T10:51:33.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.619+0000 7fb699af4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fb6940cc380 con 0x7fb6941543a0 2024-03-24T10:51:33.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.619+0000 7fb6937fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6941543a0 0x7fb69415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:33.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.619+0000 7fb692ffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb69406d4f0 0x7fb6940cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:33.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.620+0000 7fb693fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb6940ccaa0 0x7fb69406bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:53562/0 (socket says 172.21.15.5:53562) 2024-03-24T10:51:33.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.620+0000 7fb693fff640 1 -- 172.21.15.5:0/2511691932 learned_addr learned my addr 172.21.15.5:0/2511691932 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:51:33.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.622+0000 7fb6937fe640 1 -- 172.21.15.5:0/2511691932 >> v2:172.21.15.5:3300/0 conn(0x7fb69406d4f0 msgr2=0x7fb6940cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:33.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.622+0000 7fb6937fe640 1 --2- 172.21.15.5:0/2511691932 >> v2:172.21.15.5:3300/0 conn(0x7fb69406d4f0 0x7fb6940cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:33.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.622+0000 7fb6937fe640 1 -- 172.21.15.5:0/2511691932 >> v1:172.21.15.57:6789/0 conn(0x7fb6940ccaa0 legacy=0x7fb69406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:33.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.622+0000 7fb6937fe640 1 -- 172.21.15.5:0/2511691932 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6940ca7e0 con 0x7fb6941543a0 2024-03-24T10:51:33.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.625+0000 7fb6937fe640 1 --2- 172.21.15.5:0/2511691932 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6941543a0 0x7fb69415ac00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb684009d90 tx=0x7fb68402bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad13d5b1ae185888 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:33.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.629+0000 7fb6927fc640 1 -- 172.21.15.5:0/2511691932 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68402cce0 con 0x7fb6941543a0 2024-03-24T10:51:33.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.629+0000 7fb6927fc640 1 -- 172.21.15.5:0/2511691932 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb68402ce40 con 0x7fb6941543a0 2024-03-24T10:51:33.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.629+0000 7fb6927fc640 1 -- 172.21.15.5:0/2511691932 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb684034660 con 0x7fb6941543a0 2024-03-24T10:51:33.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.630+0000 7fb699af4640 1 -- 172.21.15.5:0/2511691932 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6941543a0 msgr2=0x7fb69415ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:33.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.630+0000 7fb699af4640 1 --2- 172.21.15.5:0/2511691932 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6941543a0 0x7fb69415ac00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb684009d90 tx=0x7fb68402bf30 comp rx=0 tx=0).stop 2024-03-24T10:51:33.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.630+0000 7fb699af4640 1 -- 172.21.15.5:0/2511691932 shutdown_connections 2024-03-24T10:51:33.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.630+0000 7fb699af4640 1 --2- 172.21.15.5:0/2511691932 >> v2:172.21.15.5:3300/0 conn(0x7fb69406d4f0 0x7fb6940cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:33.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.630+0000 7fb699af4640 1 --2- 172.21.15.5:0/2511691932 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6941543a0 0x7fb69415ac00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:33.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.630+0000 7fb699af4640 1 -- 172.21.15.5:0/2511691932 >> 172.21.15.5:0/2511691932 conn(0x7fb694072ef0 msgr2=0x7fb694061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:33.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.630+0000 7fb699af4640 1 -- 172.21.15.5:0/2511691932 shutdown_connections 2024-03-24T10:51:33.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.630+0000 7fb699af4640 1 -- 172.21.15.5:0/2511691932 wait complete. 2024-03-24T10:51:33.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.632+0000 7fb699af4640 1 Processor -- start 2024-03-24T10:51:33.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.632+0000 7fb699af4640 1 -- start start 2024-03-24T10:51:33.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.633+0000 7fb699af4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb69406d4f0 0x7fb6940b3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:33.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.633+0000 7fb699af4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6940ccaa0 0x7fb6941effe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:33.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.633+0000 7fb699af4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb6941543a0 0x7fb6941f0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:33.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.633+0000 7fb699af4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fb69415bdd0 con 0x7fb6941543a0 2024-03-24T10:51:33.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.633+0000 7fb699af4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb6940b3c20 con 0x7fb69406d4f0 2024-03-24T10:51:33.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.633+0000 7fb699af4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fb69415b350 con 0x7fb6940ccaa0 2024-03-24T10:51:33.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.634+0000 7fb692ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6940ccaa0 0x7fb6941effe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:33.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.634+0000 7fb693fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb6941543a0 0x7fb6941f0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:33.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.634+0000 7fb6937fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb69406d4f0 0x7fb6940b3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:33.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.634+0000 7fb692ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6940ccaa0 0x7fb6941effe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:34354/0 (socket says 172.21.15.5:34354) 2024-03-24T10:51:33.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.634+0000 7fb692ffd640 1 -- 172.21.15.5:0/953139869 learned_addr learned my addr 172.21.15.5:0/953139869 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:33.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.639+0000 7fb693fff640 1 -- 172.21.15.5:0/953139869 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6940ccaa0 msgr2=0x7fb6941effe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:33.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.639+0000 7fb693fff640 1 --2- 172.21.15.5:0/953139869 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6940ccaa0 0x7fb6941effe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:33.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.639+0000 7fb693fff640 1 -- 172.21.15.5:0/953139869 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb69406d4f0 msgr2=0x7fb6940b3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:33.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.639+0000 7fb693fff640 1 --2- 172.21.15.5:0/953139869 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb69406d4f0 0x7fb6940b3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:33.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.639+0000 7fb693fff640 1 -- 172.21.15.5:0/953139869 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb684009a70 con 0x7fb6941543a0 2024-03-24T10:51:33.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.639+0000 7fb692ffd640 1 --2- 172.21.15.5:0/953139869 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6940ccaa0 0x7fb6941effe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:51:33.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.642+0000 7fb693fff640 1 --2- 172.21.15.5:0/953139869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb6941543a0 0x7fb6941f0520 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb68c00c3c0 tx=0x7fb68c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:33.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.647+0000 7fb67bfff640 1 -- 172.21.15.5:0/953139869 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68c005ce0 con 0x7fb6941543a0 2024-03-24T10:51:33.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.647+0000 7fb67bfff640 1 -- 172.21.15.5:0/953139869 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb68c005e40 con 0x7fb6941543a0 2024-03-24T10:51:33.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.647+0000 7fb67bfff640 1 -- 172.21.15.5:0/953139869 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68c0072d0 con 0x7fb6941543a0 2024-03-24T10:51:33.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.647+0000 7fb699af4640 1 -- 172.21.15.5:0/953139869 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb69415b630 con 0x7fb6941543a0 2024-03-24T10:51:33.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.647+0000 7fb699af4640 1 -- 172.21.15.5:0/953139869 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6940b3f00 con 0x7fb6941543a0 2024-03-24T10:51:33.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.648+0000 7fb699af4640 1 -- 172.21.15.5:0/953139869 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6941efd00 con 0x7fb6941543a0 2024-03-24T10:51:33.913 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.913+0000 7fb67bfff640 1 -- 172.21.15.5:0/953139869 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb68c007430 con 0x7fb6941543a0 2024-03-24T10:51:33.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.913+0000 7fb67bfff640 1 --2- 172.21.15.5:0/953139869 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb68006b810 0x7fb68006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:33.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.914+0000 7fb6937fe640 1 --2- 172.21.15.5:0/953139869 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb68006b810 0x7fb68006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:33.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.914+0000 7fb6937fe640 1 --2- 172.21.15.5:0/953139869 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb68006b810 0x7fb68006dcd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb684036000 tx=0x7fb684030000 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:33.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.929+0000 7fb67bfff640 1 -- 172.21.15.5:0/953139869 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fb68c0580e0 con 0x7fb6941543a0 2024-03-24T10:51:33.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:33.929+0000 7fb67bfff640 1 -- 172.21.15.5:0/953139869 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb68c054380 con 0x7fb6941543a0 2024-03-24T10:51:34.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:34.157+0000 7fb699af4640 1 -- 172.21.15.5:0/953139869 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6941efd00 con 0x7fb6941543a0 2024-03-24T10:51:34.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:34.173+0000 7fb67bfff640 1 -- 172.21.15.5:0/953139869 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fb6941efd00 con 0x7fb6941543a0 2024-03-24T10:51:34.173 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:34.174 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:34.174 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:34.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:34.175+0000 7fb699af4640 1 -- 172.21.15.5:0/953139869 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb68006b810 msgr2=0x7fb68006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:34.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:34.175+0000 7fb699af4640 1 --2- 172.21.15.5:0/953139869 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb68006b810 0x7fb68006dcd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb684036000 tx=0x7fb684030000 comp rx=0 tx=0).stop 2024-03-24T10:51:34.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:34.175+0000 7fb699af4640 1 -- 172.21.15.5:0/953139869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb6941543a0 msgr2=0x7fb6941f0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:34.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:34.175+0000 7fb699af4640 1 --2- 172.21.15.5:0/953139869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb6941543a0 0x7fb6941f0520 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb68c00c3c0 tx=0x7fb68c00c880 comp rx=0 tx=0).stop 2024-03-24T10:51:34.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:34.175+0000 7fb699af4640 1 -- 172.21.15.5:0/953139869 shutdown_connections 2024-03-24T10:51:34.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:34.175+0000 7fb699af4640 1 --2- 172.21.15.5:0/953139869 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb68006b810 0x7fb68006dcd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:34.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:34.175+0000 7fb699af4640 1 --2- 172.21.15.5:0/953139869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb6941543a0 0x7fb6941f0520 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:34.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:34.175+0000 7fb699af4640 1 --2- 172.21.15.5:0/953139869 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6940ccaa0 0x7fb6941effe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:34.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:34.175+0000 7fb699af4640 1 --2- 172.21.15.5:0/953139869 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb69406d4f0 0x7fb6940b3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:34.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:34.175+0000 7fb699af4640 1 -- 172.21.15.5:0/953139869 >> 172.21.15.5:0/953139869 conn(0x7fb694072ef0 msgr2=0x7fb69406be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:34.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:34.176+0000 7fb699af4640 1 -- 172.21.15.5:0/953139869 shutdown_connections 2024-03-24T10:51:34.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:34.176+0000 7fb699af4640 1 -- 172.21.15.5:0/953139869 wait complete. 2024-03-24T10:51:34.790 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:34.790 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:35.791 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:35.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.892+0000 7f2a7b055640 1 Processor -- start 2024-03-24T10:51:35.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.892+0000 7f2a7b055640 1 -- start start 2024-03-24T10:51:35.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.894+0000 7f2a7b055640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2a740cae90 0x7f2a740cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:35.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.894+0000 7f2a7b055640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2a740cd960 0x7f2a740cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:35.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.894+0000 7f2a7b055640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a740ccba0 con 0x7f2a740cb270 2024-03-24T10:51:35.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.894+0000 7f2a7b055640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f2a740cc9b0 con 0x7f2a740cd960 2024-03-24T10:51:35.895 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.894+0000 7f2a7b055640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f2a740cce40 con 0x7f2a740cae90 2024-03-24T10:51:35.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.894+0000 7f2a78df1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2a740cae90 0x7f2a740cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:35.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.895+0000 7f2a795f2640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2a740cd960 0x7f2a740cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:35.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.895+0000 7f2a78df1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2a740cae90 0x7f2a740cf530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:34362/0 (socket says 172.21.15.5:34362) 2024-03-24T10:51:35.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.895+0000 7f2a78df1640 1 -- 172.21.15.5:0/1271198389 learned_addr learned my addr 172.21.15.5:0/1271198389 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:35.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.897+0000 7f2a78df1640 1 -- 172.21.15.5:0/1271198389 >> v2:172.21.15.5:3300/0 conn(0x7f2a740cd960 msgr2=0x7f2a740cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:35.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.897+0000 7f2a78df1640 1 --2- 172.21.15.5:0/1271198389 >> v2:172.21.15.5:3300/0 conn(0x7f2a740cd960 0x7f2a740cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:35.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.897+0000 7f2a78df1640 1 -- 172.21.15.5:0/1271198389 >> v1:172.21.15.5:6789/0 conn(0x7f2a740cb270 legacy=0x7f2a740cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:35.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.897+0000 7f2a78df1640 1 -- 172.21.15.5:0/1271198389 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a7415a590 con 0x7f2a740cae90 2024-03-24T10:51:35.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.899+0000 7f2a78df1640 1 --2- 172.21.15.5:0/1271198389 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2a740cae90 0x7f2a740cf530 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2a68009870 tx=0x7f2a6802ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e43663414c86b003 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:35.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.904+0000 7f2a737fe640 1 -- 172.21.15.5:0/1271198389 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6802f840 con 0x7f2a740cae90 2024-03-24T10:51:35.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.904+0000 7f2a737fe640 1 -- 172.21.15.5:0/1271198389 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a6802f9a0 con 0x7f2a740cae90 2024-03-24T10:51:35.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.904+0000 7f2a737fe640 1 -- 172.21.15.5:0/1271198389 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a6802fd30 con 0x7f2a740cae90 2024-03-24T10:51:35.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.905+0000 7f2a7b055640 1 -- 172.21.15.5:0/1271198389 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2a740cae90 msgr2=0x7f2a740cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:35.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.905+0000 7f2a7b055640 1 --2- 172.21.15.5:0/1271198389 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2a740cae90 0x7f2a740cf530 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2a68009870 tx=0x7f2a6802ba20 comp rx=0 tx=0).stop 2024-03-24T10:51:35.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.905+0000 7f2a7b055640 1 -- 172.21.15.5:0/1271198389 shutdown_connections 2024-03-24T10:51:35.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.905+0000 7f2a7b055640 1 --2- 172.21.15.5:0/1271198389 >> v2:172.21.15.5:3300/0 conn(0x7f2a740cd960 0x7f2a740cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:35.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.905+0000 7f2a7b055640 1 --2- 172.21.15.5:0/1271198389 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2a740cae90 0x7f2a740cf530 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:35.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.905+0000 7f2a7b055640 1 -- 172.21.15.5:0/1271198389 >> 172.21.15.5:0/1271198389 conn(0x7f2a740cfb80 msgr2=0x7f2a740cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:35.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.905+0000 7f2a7b055640 1 -- 172.21.15.5:0/1271198389 shutdown_connections 2024-03-24T10:51:35.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.905+0000 7f2a7b055640 1 -- 172.21.15.5:0/1271198389 wait complete. 2024-03-24T10:51:35.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.907+0000 7f2a7b055640 1 Processor -- start 2024-03-24T10:51:35.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.907+0000 7f2a7b055640 1 -- start start 2024-03-24T10:51:35.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.908+0000 7f2a7b055640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2a740cae90 0x7f2a740c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:35.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.908+0000 7f2a7b055640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a740cb270 0x7f2a7414f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:35.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.908+0000 7f2a7b055640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a740cd960 0x7f2a740c37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:35.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.908+0000 7f2a7b055640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f2a740c54e0 con 0x7f2a740cb270 2024-03-24T10:51:35.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.908+0000 7f2a7b055640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2a741f0e10 con 0x7f2a740cd960 2024-03-24T10:51:35.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.908+0000 7f2a7b055640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f2a741f0200 con 0x7f2a740cae90 2024-03-24T10:51:35.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.909+0000 7f2a73fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a740cb270 0x7f2a7414f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:35.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.909+0000 7f2a78df1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2a740cae90 0x7f2a740c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:35.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.909+0000 7f2a795f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a740cd960 0x7f2a740c37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:35.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.909+0000 7f2a78df1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2a740cae90 0x7f2a740c4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:34366/0 (socket says 172.21.15.5:34366) 2024-03-24T10:51:35.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.909+0000 7f2a73fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a740cb270 0x7f2a7414f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35828/0 (socket says 172.21.15.5:35828) 2024-03-24T10:51:35.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.909+0000 7f2a78df1640 1 -- 172.21.15.5:0/58704089 learned_addr learned my addr 172.21.15.5:0/58704089 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:35.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.913+0000 7f2a73fff640 1 -- 172.21.15.5:0/58704089 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2a740cae90 msgr2=0x7f2a740c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:35.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.913+0000 7f2a73fff640 1 --2- 172.21.15.5:0/58704089 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2a740cae90 0x7f2a740c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:35.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.913+0000 7f2a73fff640 1 -- 172.21.15.5:0/58704089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a740cd960 msgr2=0x7f2a740c37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:35.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.913+0000 7f2a73fff640 1 --2- 172.21.15.5:0/58704089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a740cd960 0x7f2a740c37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:35.914 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.913+0000 7f2a73fff640 1 -- 172.21.15.5:0/58704089 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a680094d0 con 0x7f2a740cb270 2024-03-24T10:51:35.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.915+0000 7f2a73fff640 1 --2- 172.21.15.5:0/58704089 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a740cb270 0x7f2a7414f770 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2a6000c340 tx=0x7f2a6000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:35.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.919+0000 7f2a717fa640 1 -- 172.21.15.5:0/58704089 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a600052d0 con 0x7f2a740cb270 2024-03-24T10:51:35.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.919+0000 7f2a717fa640 1 -- 172.21.15.5:0/58704089 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a60005450 con 0x7f2a740cb270 2024-03-24T10:51:35.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.919+0000 7f2a717fa640 1 -- 172.21.15.5:0/58704089 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a60007260 con 0x7f2a740cb270 2024-03-24T10:51:35.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.919+0000 7f2a7b055640 1 -- 172.21.15.5:0/58704089 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a741f04e0 con 0x7f2a740cb270 2024-03-24T10:51:35.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.919+0000 7f2a7b055640 1 -- 172.21.15.5:0/58704089 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a741f0aa0 con 0x7f2a740cb270 2024-03-24T10:51:35.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:35.920+0000 7f2a7b055640 1 -- 172.21.15.5:0/58704089 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a740c34e0 con 0x7f2a740cb270 2024-03-24T10:51:36.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:36.191+0000 7f2a717fa640 1 -- 172.21.15.5:0/58704089 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2a600073c0 con 0x7f2a740cb270 2024-03-24T10:51:36.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:36.191+0000 7f2a717fa640 1 --2- 172.21.15.5:0/58704089 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2a5806b840 0x7f2a5806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:36.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:36.191+0000 7f2a78df1640 1 --2- 172.21.15.5:0/58704089 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2a5806b840 0x7f2a5806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:36.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:36.192+0000 7f2a78df1640 1 --2- 172.21.15.5:0/58704089 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2a5806b840 0x7f2a5806dd00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2a68009870 tx=0x7f2a68008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:36.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:36.205+0000 7f2a717fa640 1 -- 172.21.15.5:0/58704089 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f2a60058390 con 0x7f2a740cb270 2024-03-24T10:51:36.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:36.205+0000 7f2a717fa640 1 -- 172.21.15.5:0/58704089 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2a60054630 con 0x7f2a740cb270 2024-03-24T10:51:36.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:36.435+0000 7f2a7b055640 1 -- 172.21.15.5:0/58704089 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a740c34e0 con 0x7f2a740cb270 2024-03-24T10:51:36.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:36.455+0000 7f2a717fa640 1 -- 172.21.15.5:0/58704089 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f2a740c34e0 con 0x7f2a740cb270 2024-03-24T10:51:36.456 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:36.456 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:36.457 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:36.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:36.457+0000 7f2a7b055640 1 -- 172.21.15.5:0/58704089 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2a5806b840 msgr2=0x7f2a5806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:36.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:36.457+0000 7f2a7b055640 1 --2- 172.21.15.5:0/58704089 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2a5806b840 0x7f2a5806dd00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2a68009870 tx=0x7f2a68008030 comp rx=0 tx=0).stop 2024-03-24T10:51:36.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:36.457+0000 7f2a7b055640 1 -- 172.21.15.5:0/58704089 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a740cb270 msgr2=0x7f2a7414f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:36.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:36.457+0000 7f2a7b055640 1 --2- 172.21.15.5:0/58704089 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a740cb270 0x7f2a7414f770 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2a6000c340 tx=0x7f2a6000c800 comp rx=0 tx=0).stop 2024-03-24T10:51:36.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:36.457+0000 7f2a7b055640 1 -- 172.21.15.5:0/58704089 shutdown_connections 2024-03-24T10:51:36.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:36.457+0000 7f2a7b055640 1 --2- 172.21.15.5:0/58704089 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2a5806b840 0x7f2a5806dd00 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:36.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:36.457+0000 7f2a7b055640 1 --2- 172.21.15.5:0/58704089 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a740cb270 0x7f2a7414f770 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:36.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:36.457+0000 7f2a7b055640 1 --2- 172.21.15.5:0/58704089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a740cd960 0x7f2a740c37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:36.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:36.457+0000 7f2a7b055640 1 --2- 172.21.15.5:0/58704089 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2a740cae90 0x7f2a740c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:36.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:36.457+0000 7f2a7b055640 1 -- 172.21.15.5:0/58704089 >> 172.21.15.5:0/58704089 conn(0x7f2a740cfb80 msgr2=0x7f2a74159e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:36.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:36.458+0000 7f2a7b055640 1 -- 172.21.15.5:0/58704089 shutdown_connections 2024-03-24T10:51:36.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:36.458+0000 7f2a7b055640 1 -- 172.21.15.5:0/58704089 wait complete. 2024-03-24T10:51:37.073 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:37.073 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:38.074 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:38.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.175+0000 7fd2edec2640 1 Processor -- start 2024-03-24T10:51:38.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.175+0000 7fd2edec2640 1 -- start start 2024-03-24T10:51:38.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.176+0000 7fd2edec2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2e80cae90 0x7fd2e80cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:38.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.176+0000 7fd2edec2640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd2e80cb270 0x7fd2e80cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:38.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.176+0000 7fd2edec2640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2e80ccba0 con 0x7fd2e80cd960 2024-03-24T10:51:38.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.176+0000 7fd2edec2640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fd2e80cce40 con 0x7fd2e80cb270 2024-03-24T10:51:38.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.176+0000 7fd2edec2640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd2e815a590 con 0x7fd2e80cae90 2024-03-24T10:51:38.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.177+0000 7fd2e77fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2e80cae90 0x7fd2e80cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:38.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.177+0000 7fd2e7fff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fd2e80cd960 0x7fd2e80cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:52932/0 (socket says 172.21.15.5:52932) 2024-03-24T10:51:38.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.177+0000 7fd2e7fff640 1 -- 172.21.15.5:0/3339012053 learned_addr learned my addr 172.21.15.5:0/3339012053 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:51:38.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.177+0000 7fd2e6ffd640 1 --2- 172.21.15.5:0/3339012053 >> v2:172.21.15.5:3300/0 conn(0x7fd2e80cb270 0x7fd2e80cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:38.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.180+0000 7fd2e77fe640 1 -- 172.21.15.5:0/3339012053 >> v2:172.21.15.5:3300/0 conn(0x7fd2e80cb270 msgr2=0x7fd2e80cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:38.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.180+0000 7fd2e77fe640 1 --2- 172.21.15.5:0/3339012053 >> v2:172.21.15.5:3300/0 conn(0x7fd2e80cb270 0x7fd2e80cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:38.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.180+0000 7fd2e77fe640 1 -- 172.21.15.5:0/3339012053 >> v1:172.21.15.5:6789/0 conn(0x7fd2e80cd960 legacy=0x7fd2e80cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:38.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.180+0000 7fd2e77fe640 1 -- 172.21.15.5:0/3339012053 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2e815a870 con 0x7fd2e80cae90 2024-03-24T10:51:38.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.182+0000 7fd2e77fe640 1 --2- 172.21.15.5:0/3339012053 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2e80cae90 0x7fd2e80cf530 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80097a0 tx=0x7fd2d802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5756cd68aa600d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:38.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.186+0000 7fd2e67fc640 1 -- 172.21.15.5:0/3339012053 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8030610 con 0x7fd2e80cae90 2024-03-24T10:51:38.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.186+0000 7fd2e67fc640 1 -- 172.21.15.5:0/3339012053 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2d8030770 con 0x7fd2e80cae90 2024-03-24T10:51:38.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.186+0000 7fd2e67fc640 1 -- 172.21.15.5:0/3339012053 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d8030b00 con 0x7fd2e80cae90 2024-03-24T10:51:38.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.187+0000 7fd2edec2640 1 -- 172.21.15.5:0/3339012053 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2e80cae90 msgr2=0x7fd2e80cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:38.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.187+0000 7fd2edec2640 1 --2- 172.21.15.5:0/3339012053 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2e80cae90 0x7fd2e80cf530 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd2d80097a0 tx=0x7fd2d802b960 comp rx=0 tx=0).stop 2024-03-24T10:51:38.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.187+0000 7fd2edec2640 1 -- 172.21.15.5:0/3339012053 shutdown_connections 2024-03-24T10:51:38.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.187+0000 7fd2edec2640 1 --2- 172.21.15.5:0/3339012053 >> v2:172.21.15.5:3300/0 conn(0x7fd2e80cb270 0x7fd2e80cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:38.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.187+0000 7fd2edec2640 1 --2- 172.21.15.5:0/3339012053 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2e80cae90 0x7fd2e80cf530 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:38.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.187+0000 7fd2edec2640 1 -- 172.21.15.5:0/3339012053 >> 172.21.15.5:0/3339012053 conn(0x7fd2e80cfb80 msgr2=0x7fd2e80cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:38.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.187+0000 7fd2edec2640 1 -- 172.21.15.5:0/3339012053 shutdown_connections 2024-03-24T10:51:38.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.187+0000 7fd2edec2640 1 -- 172.21.15.5:0/3339012053 wait complete. 2024-03-24T10:51:38.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.189+0000 7fd2edec2640 1 Processor -- start 2024-03-24T10:51:38.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.189+0000 7fd2edec2640 1 -- start start 2024-03-24T10:51:38.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.190+0000 7fd2edec2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2e80cae90 0x7fd2e81f1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:38.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.190+0000 7fd2edec2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2e80cb270 0x7fd2e8153dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:38.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.190+0000 7fd2edec2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2e80cd960 0x7fd2e8154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:38.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.190+0000 7fd2edec2640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e80ac3f0 con 0x7fd2e80cb270 2024-03-24T10:51:38.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.190+0000 7fd2edec2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd2e8154850 con 0x7fd2e80cae90 2024-03-24T10:51:38.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.190+0000 7fd2edec2640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd2e80c3fe0 con 0x7fd2e80cd960 2024-03-24T10:51:38.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.191+0000 7fd2e6ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2e80cb270 0x7fd2e8153dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:38.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.191+0000 7fd2e77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2e80cae90 0x7fd2e81f1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:38.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.191+0000 7fd2e6ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2e80cb270 0x7fd2e8153dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35848/0 (socket says 172.21.15.5:35848) 2024-03-24T10:51:38.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.191+0000 7fd2e6ffd640 1 -- 172.21.15.5:0/1832551728 learned_addr learned my addr 172.21.15.5:0/1832551728 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:38.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.194+0000 7fd2e7fff640 1 --2- 172.21.15.5:0/1832551728 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2e80cd960 0x7fd2e8154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:38.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.195+0000 7fd2e6ffd640 1 -- 172.21.15.5:0/1832551728 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2e80cd960 msgr2=0x7fd2e8154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:38.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.195+0000 7fd2e6ffd640 1 --2- 172.21.15.5:0/1832551728 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2e80cd960 0x7fd2e8154310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:38.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.195+0000 7fd2e6ffd640 1 -- 172.21.15.5:0/1832551728 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2e80cae90 msgr2=0x7fd2e81f1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:38.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.195+0000 7fd2e6ffd640 1 --2- 172.21.15.5:0/1832551728 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2e80cae90 0x7fd2e81f1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:38.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.195+0000 7fd2e6ffd640 1 -- 172.21.15.5:0/1832551728 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d8009400 con 0x7fd2e80cb270 2024-03-24T10:51:38.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.197+0000 7fd2e6ffd640 1 --2- 172.21.15.5:0/1832551728 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2e80cb270 0x7fd2e8153dd0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc00e9b0 tx=0x7fd2dc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:38.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.201+0000 7fd2cffff640 1 -- 172.21.15.5:0/1832551728 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc009670 con 0x7fd2e80cb270 2024-03-24T10:51:38.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.201+0000 7fd2cffff640 1 -- 172.21.15.5:0/1832551728 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2dc00c9a0 con 0x7fd2e80cb270 2024-03-24T10:51:38.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.201+0000 7fd2edec2640 1 -- 172.21.15.5:0/1832551728 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2e80c42c0 con 0x7fd2e80cb270 2024-03-24T10:51:38.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.201+0000 7fd2cffff640 1 -- 172.21.15.5:0/1832551728 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc00cd50 con 0x7fd2e80cb270 2024-03-24T10:51:38.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.201+0000 7fd2edec2640 1 -- 172.21.15.5:0/1832551728 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2e80c4880 con 0x7fd2e80cb270 2024-03-24T10:51:38.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.202+0000 7fd2edec2640 1 -- 172.21.15.5:0/1832551728 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2e806d5a0 con 0x7fd2e80cb270 2024-03-24T10:51:38.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.469+0000 7fd2cffff640 1 -- 172.21.15.5:0/1832551728 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd2dc013470 con 0x7fd2e80cb270 2024-03-24T10:51:38.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.469+0000 7fd2cffff640 1 --2- 172.21.15.5:0/1832551728 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd2d406b810 0x7fd2d406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:38.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.469+0000 7fd2e77fe640 1 --2- 172.21.15.5:0/1832551728 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd2d406b810 0x7fd2d406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:38.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.470+0000 7fd2e77fe640 1 --2- 172.21.15.5:0/1832551728 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd2d406b810 0x7fd2d406dcd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8036000 tx=0x7fd2d8008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:38.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.484+0000 7fd2cffff640 1 -- 172.21.15.5:0/1832551728 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fd2dc059460 con 0x7fd2e80cb270 2024-03-24T10:51:38.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.484+0000 7fd2cffff640 1 -- 172.21.15.5:0/1832551728 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd2dc055700 con 0x7fd2e80cb270 2024-03-24T10:51:38.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.753+0000 7fd2edec2640 1 -- 172.21.15.5:0/1832551728 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2e80681d0 con 0x7fd2e80cb270 2024-03-24T10:51:38.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.766+0000 7fd2cffff640 1 -- 172.21.15.5:0/1832551728 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fd2dc017020 con 0x7fd2e80cb270 2024-03-24T10:51:38.767 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:38.767 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:38.767 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:38.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.768+0000 7fd2edec2640 1 -- 172.21.15.5:0/1832551728 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd2d406b810 msgr2=0x7fd2d406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:38.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.768+0000 7fd2edec2640 1 --2- 172.21.15.5:0/1832551728 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd2d406b810 0x7fd2d406dcd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8036000 tx=0x7fd2d8008030 comp rx=0 tx=0).stop 2024-03-24T10:51:38.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.768+0000 7fd2edec2640 1 -- 172.21.15.5:0/1832551728 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2e80cb270 msgr2=0x7fd2e8153dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:38.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.768+0000 7fd2edec2640 1 --2- 172.21.15.5:0/1832551728 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2e80cb270 0x7fd2e8153dd0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc00e9b0 tx=0x7fd2dc00ee70 comp rx=0 tx=0).stop 2024-03-24T10:51:38.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.768+0000 7fd2edec2640 1 -- 172.21.15.5:0/1832551728 shutdown_connections 2024-03-24T10:51:38.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.768+0000 7fd2edec2640 1 --2- 172.21.15.5:0/1832551728 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd2d406b810 0x7fd2d406dcd0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:38.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.768+0000 7fd2edec2640 1 --2- 172.21.15.5:0/1832551728 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2e80cb270 0x7fd2e8153dd0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:38.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.768+0000 7fd2edec2640 1 --2- 172.21.15.5:0/1832551728 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2e80cd960 0x7fd2e8154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:38.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.768+0000 7fd2edec2640 1 --2- 172.21.15.5:0/1832551728 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2e80cae90 0x7fd2e81f1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:38.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.768+0000 7fd2edec2640 1 -- 172.21.15.5:0/1832551728 >> 172.21.15.5:0/1832551728 conn(0x7fd2e80cfb80 msgr2=0x7fd2e80b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:38.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.768+0000 7fd2edec2640 1 -- 172.21.15.5:0/1832551728 shutdown_connections 2024-03-24T10:51:38.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:38.768+0000 7fd2edec2640 1 -- 172.21.15.5:0/1832551728 wait complete. 2024-03-24T10:51:39.384 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:39.384 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:40.385 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:40.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.489+0000 7f4a6193e640 1 Processor -- start 2024-03-24T10:51:40.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.489+0000 7f4a6193e640 1 -- start start 2024-03-24T10:51:40.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.490+0000 7f4a6193e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a5c1543a0 0x7f4a5c15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:40.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.490+0000 7f4a6193e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4a5c06d4f0 0x7f4a5c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:40.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.490+0000 7f4a6193e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a5c0cb2f0 con 0x7f4a5c0ccaa0 2024-03-24T10:51:40.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.490+0000 7f4a6193e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f4a5c0c6750 con 0x7f4a5c06d4f0 2024-03-24T10:51:40.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.490+0000 7f4a6193e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4a5c0cc380 con 0x7f4a5c1543a0 2024-03-24T10:51:40.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.491+0000 7f4a5affd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a5c1543a0 0x7f4a5c15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:40.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.492+0000 7f4a5affd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a5c1543a0 0x7f4a5c15ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:34400/0 (socket says 172.21.15.5:34400) 2024-03-24T10:51:40.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.492+0000 7f4a5affd640 1 -- 172.21.15.5:0/617680341 learned_addr learned my addr 172.21.15.5:0/617680341 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:40.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.494+0000 7f4a5a7fc640 1 --2- 172.21.15.5:0/617680341 >> v2:172.21.15.5:3300/0 conn(0x7f4a5c06d4f0 0x7f4a5c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:40.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.496+0000 7f4a5affd640 1 -- 172.21.15.5:0/617680341 >> v2:172.21.15.5:3300/0 conn(0x7f4a5c06d4f0 msgr2=0x7f4a5c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:40.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.496+0000 7f4a5affd640 1 --2- 172.21.15.5:0/617680341 >> v2:172.21.15.5:3300/0 conn(0x7f4a5c06d4f0 0x7f4a5c0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:40.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.496+0000 7f4a5affd640 1 -- 172.21.15.5:0/617680341 >> v1:172.21.15.5:6789/0 conn(0x7f4a5c0ccaa0 legacy=0x7f4a5c06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:40.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.496+0000 7f4a5affd640 1 -- 172.21.15.5:0/617680341 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a5c0ca7e0 con 0x7f4a5c1543a0 2024-03-24T10:51:40.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.498+0000 7f4a5affd640 1 --2- 172.21.15.5:0/617680341 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a5c1543a0 0x7f4a5c15ac00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c009870 tx=0x7f4a4c02ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c140ed6616432834 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:40.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.503+0000 7f4a59ffb640 1 -- 172.21.15.5:0/617680341 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c02f840 con 0x7f4a5c1543a0 2024-03-24T10:51:40.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.503+0000 7f4a59ffb640 1 -- 172.21.15.5:0/617680341 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a4c02f9a0 con 0x7f4a5c1543a0 2024-03-24T10:51:40.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.504+0000 7f4a59ffb640 1 -- 172.21.15.5:0/617680341 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c02fd30 con 0x7f4a5c1543a0 2024-03-24T10:51:40.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.504+0000 7f4a6193e640 1 -- 172.21.15.5:0/617680341 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a5c1543a0 msgr2=0x7f4a5c15ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:40.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.504+0000 7f4a6193e640 1 --2- 172.21.15.5:0/617680341 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a5c1543a0 0x7f4a5c15ac00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c009870 tx=0x7f4a4c02ba20 comp rx=0 tx=0).stop 2024-03-24T10:51:40.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.504+0000 7f4a6193e640 1 -- 172.21.15.5:0/617680341 shutdown_connections 2024-03-24T10:51:40.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.504+0000 7f4a6193e640 1 --2- 172.21.15.5:0/617680341 >> v2:172.21.15.5:3300/0 conn(0x7f4a5c06d4f0 0x7f4a5c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:40.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.504+0000 7f4a6193e640 1 --2- 172.21.15.5:0/617680341 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a5c1543a0 0x7f4a5c15ac00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:40.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.504+0000 7f4a6193e640 1 -- 172.21.15.5:0/617680341 >> 172.21.15.5:0/617680341 conn(0x7f4a5c072ef0 msgr2=0x7f4a5c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:40.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.504+0000 7f4a6193e640 1 -- 172.21.15.5:0/617680341 shutdown_connections 2024-03-24T10:51:40.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.504+0000 7f4a6193e640 1 -- 172.21.15.5:0/617680341 wait complete. 2024-03-24T10:51:40.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.505+0000 7f4a6193e640 1 Processor -- start 2024-03-24T10:51:40.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.506+0000 7f4a6193e640 1 -- start start 2024-03-24T10:51:40.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.506+0000 7f4a6193e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a5c06d4f0 0x7f4a5c0b3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:40.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.506+0000 7f4a6193e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a5c0ccaa0 0x7f4a5c1effe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:40.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.506+0000 7f4a6193e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a5c1543a0 0x7f4a5c1f0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:40.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.506+0000 7f4a6193e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5c15bdd0 con 0x7f4a5c0ccaa0 2024-03-24T10:51:40.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.506+0000 7f4a6193e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5c0b3c20 con 0x7f4a5c1543a0 2024-03-24T10:51:40.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.506+0000 7f4a6193e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4a5c15b350 con 0x7f4a5c06d4f0 2024-03-24T10:51:40.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.507+0000 7f4a5a7fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a5c0ccaa0 0x7f4a5c1effe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:40.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.507+0000 7f4a5b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a5c1543a0 0x7f4a5c1f0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:40.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.507+0000 7f4a5affd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a5c06d4f0 0x7f4a5c0b3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:40.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.507+0000 7f4a5a7fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a5c0ccaa0 0x7f4a5c1effe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35866/0 (socket says 172.21.15.5:35866) 2024-03-24T10:51:40.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.507+0000 7f4a5a7fc640 1 -- 172.21.15.5:0/3261576211 learned_addr learned my addr 172.21.15.5:0/3261576211 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:40.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.511+0000 7f4a5a7fc640 1 -- 172.21.15.5:0/3261576211 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a5c06d4f0 msgr2=0x7f4a5c0b3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:40.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.511+0000 7f4a5a7fc640 1 --2- 172.21.15.5:0/3261576211 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a5c06d4f0 0x7f4a5c0b3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:40.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.511+0000 7f4a5a7fc640 1 -- 172.21.15.5:0/3261576211 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a5c1543a0 msgr2=0x7f4a5c1f0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:40.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.511+0000 7f4a5a7fc640 1 --2- 172.21.15.5:0/3261576211 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a5c1543a0 0x7f4a5c1f0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:40.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.511+0000 7f4a5a7fc640 1 -- 172.21.15.5:0/3261576211 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a4c0094d0 con 0x7f4a5c0ccaa0 2024-03-24T10:51:40.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.514+0000 7f4a5a7fc640 1 --2- 172.21.15.5:0/3261576211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a5c0ccaa0 0x7f4a5c1effe0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000d5f0 tx=0x7f4a5000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:40.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.518+0000 7f4a43fff640 1 -- 172.21.15.5:0/3261576211 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a500125f0 con 0x7f4a5c0ccaa0 2024-03-24T10:51:40.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.518+0000 7f4a43fff640 1 -- 172.21.15.5:0/3261576211 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a50012750 con 0x7f4a5c0ccaa0 2024-03-24T10:51:40.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.518+0000 7f4a6193e640 1 -- 172.21.15.5:0/3261576211 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a5c15b630 con 0x7f4a5c0ccaa0 2024-03-24T10:51:40.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.518+0000 7f4a43fff640 1 -- 172.21.15.5:0/3261576211 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a50012b00 con 0x7f4a5c0ccaa0 2024-03-24T10:51:40.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.518+0000 7f4a6193e640 1 -- 172.21.15.5:0/3261576211 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a5c0b3f00 con 0x7f4a5c0ccaa0 2024-03-24T10:51:40.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.519+0000 7f4a6193e640 1 -- 172.21.15.5:0/3261576211 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a5c0681d0 con 0x7f4a5c0ccaa0 2024-03-24T10:51:40.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.538+0000 7f4a43fff640 1 -- 172.21.15.5:0/3261576211 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4a50012c60 con 0x7f4a5c0ccaa0 2024-03-24T10:51:40.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.538+0000 7f4a43fff640 1 --2- 172.21.15.5:0/3261576211 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4a4806b810 0x7f4a4806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:40.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.538+0000 7f4a5affd640 1 --2- 172.21.15.5:0/3261576211 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4a4806b810 0x7f4a4806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:40.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.539+0000 7f4a5affd640 1 --2- 172.21.15.5:0/3261576211 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4a4806b810 0x7f4a4806dcd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c036000 tx=0x7f4a4c008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:40.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.543+0000 7f4a43fff640 1 -- 172.21.15.5:0/3261576211 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f4a50059870 con 0x7f4a5c0ccaa0 2024-03-24T10:51:40.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:40.804+0000 7f4a43fff640 1 -- 172.21.15.5:0/3261576211 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4a50055b10 con 0x7f4a5c0ccaa0 2024-03-24T10:51:41.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:41.048+0000 7f4a6193e640 1 -- 172.21.15.5:0/3261576211 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a5c152ef0 con 0x7f4a5c0ccaa0 2024-03-24T10:51:41.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:41.069+0000 7f4a43fff640 1 -- 172.21.15.5:0/3261576211 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f4a500591d0 con 0x7f4a5c0ccaa0 2024-03-24T10:51:41.071 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:41.071 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:41.071 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:41.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:41.071+0000 7f4a6193e640 1 -- 172.21.15.5:0/3261576211 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4a4806b810 msgr2=0x7f4a4806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:41.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:41.072+0000 7f4a6193e640 1 --2- 172.21.15.5:0/3261576211 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4a4806b810 0x7f4a4806dcd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c036000 tx=0x7f4a4c008030 comp rx=0 tx=0).stop 2024-03-24T10:51:41.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:41.072+0000 7f4a6193e640 1 -- 172.21.15.5:0/3261576211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a5c0ccaa0 msgr2=0x7f4a5c1effe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:41.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:41.072+0000 7f4a6193e640 1 --2- 172.21.15.5:0/3261576211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a5c0ccaa0 0x7f4a5c1effe0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000d5f0 tx=0x7f4a5000dab0 comp rx=0 tx=0).stop 2024-03-24T10:51:41.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:41.072+0000 7f4a6193e640 1 -- 172.21.15.5:0/3261576211 shutdown_connections 2024-03-24T10:51:41.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:41.072+0000 7f4a6193e640 1 --2- 172.21.15.5:0/3261576211 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4a4806b810 0x7f4a4806dcd0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:41.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:41.072+0000 7f4a6193e640 1 --2- 172.21.15.5:0/3261576211 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4a5c0ccaa0 0x7f4a5c1effe0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:41.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:41.072+0000 7f4a6193e640 1 --2- 172.21.15.5:0/3261576211 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a5c1543a0 0x7f4a5c1f0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:41.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:41.072+0000 7f4a6193e640 1 --2- 172.21.15.5:0/3261576211 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4a5c06d4f0 0x7f4a5c0b3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:41.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:41.072+0000 7f4a6193e640 1 -- 172.21.15.5:0/3261576211 >> 172.21.15.5:0/3261576211 conn(0x7f4a5c072ef0 msgr2=0x7f4a5c06be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:41.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:41.072+0000 7f4a6193e640 1 -- 172.21.15.5:0/3261576211 shutdown_connections 2024-03-24T10:51:41.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:41.072+0000 7f4a6193e640 1 -- 172.21.15.5:0/3261576211 wait complete. 2024-03-24T10:51:41.688 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:41.688 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:42.689 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:42.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.792+0000 7f9a060cd640 1 Processor -- start 2024-03-24T10:51:42.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.792+0000 7f9a060cd640 1 -- start start 2024-03-24T10:51:42.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.793+0000 7f9a060cd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9a000cae90 0x7f9a000cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:42.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.793+0000 7f9a060cd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9a000cd960 0x7f9a000cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:42.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.793+0000 7f9a060cd640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a000ccba0 con 0x7f9a000cb270 2024-03-24T10:51:42.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.793+0000 7f9a060cd640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f9a000cc9b0 con 0x7f9a000cd960 2024-03-24T10:51:42.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.793+0000 7f9a060cd640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f9a000cce40 con 0x7f9a000cae90 2024-03-24T10:51:42.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.793+0000 7f99fffff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9a000cd960 0x7f9a000cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:42.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.794+0000 7f99fffff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9a000cd960 0x7f9a000cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36338/0 (socket says 172.21.15.5:36338) 2024-03-24T10:51:42.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.794+0000 7f99fffff640 1 -- 172.21.15.5:0/297845306 learned_addr learned my addr 172.21.15.5:0/297845306 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:42.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.794+0000 7f99ff7fe640 1 --2- 172.21.15.5:0/297845306 >> v2:172.21.15.57:3300/0 conn(0x7f9a000cae90 0x7f9a000cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:42.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.796+0000 7f99fffff640 1 -- 172.21.15.5:0/297845306 >> v2:172.21.15.57:3300/0 conn(0x7f9a000cae90 msgr2=0x7f9a000cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:42.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.796+0000 7f99fffff640 1 --2- 172.21.15.5:0/297845306 >> v2:172.21.15.57:3300/0 conn(0x7f9a000cae90 0x7f9a000cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:42.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.796+0000 7f99fffff640 1 -- 172.21.15.5:0/297845306 >> v1:172.21.15.57:6789/0 conn(0x7f9a000cb270 legacy=0x7f9a000cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:42.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.796+0000 7f99fffff640 1 -- 172.21.15.5:0/297845306 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a0015a590 con 0x7f9a000cd960 2024-03-24T10:51:42.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.798+0000 7f99fffff640 1 --2- 172.21.15.5:0/297845306 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a000cd960 0x7f9a000cb6f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9a000b26a0 tx=0x7f99f802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a7ce884d34526e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:42.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.803+0000 7f99feffd640 1 -- 172.21.15.5:0/297845306 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f802cbb0 con 0x7f9a000cd960 2024-03-24T10:51:42.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.803+0000 7f99feffd640 1 -- 172.21.15.5:0/297845306 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f802cd10 con 0x7f9a000cd960 2024-03-24T10:51:42.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.803+0000 7f99feffd640 1 -- 172.21.15.5:0/297845306 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f8034610 con 0x7f9a000cd960 2024-03-24T10:51:42.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.803+0000 7f9a060cd640 1 -- 172.21.15.5:0/297845306 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a000cd960 msgr2=0x7f9a000cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:42.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.803+0000 7f9a060cd640 1 --2- 172.21.15.5:0/297845306 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a000cd960 0x7f9a000cb6f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9a000b26a0 tx=0x7f99f802c010 comp rx=0 tx=0).stop 2024-03-24T10:51:42.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.803+0000 7f9a060cd640 1 -- 172.21.15.5:0/297845306 shutdown_connections 2024-03-24T10:51:42.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.803+0000 7f9a060cd640 1 --2- 172.21.15.5:0/297845306 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a000cd960 0x7f9a000cb6f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:42.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.803+0000 7f9a060cd640 1 --2- 172.21.15.5:0/297845306 >> v2:172.21.15.57:3300/0 conn(0x7f9a000cae90 0x7f9a000cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:42.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.803+0000 7f9a060cd640 1 -- 172.21.15.5:0/297845306 >> 172.21.15.5:0/297845306 conn(0x7f9a000cfb80 msgr2=0x7f9a000cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:42.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.803+0000 7f9a060cd640 1 -- 172.21.15.5:0/297845306 shutdown_connections 2024-03-24T10:51:42.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.803+0000 7f9a060cd640 1 -- 172.21.15.5:0/297845306 wait complete. 2024-03-24T10:51:42.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.805+0000 7f9a060cd640 1 Processor -- start 2024-03-24T10:51:42.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.806+0000 7f9a060cd640 1 -- start start 2024-03-24T10:51:42.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.807+0000 7f9a060cd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a000cae90 0x7f9a00153720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:42.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.807+0000 7f9a060cd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a000cb270 0x7f9a001566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:42.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.807+0000 7f9a060cd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9a000cd960 0x7f9a00156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:42.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.807+0000 7f9a060cd640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f9a001f0020 con 0x7f9a000cb270 2024-03-24T10:51:42.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.807+0000 7f9a060cd640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9a00156250 con 0x7f9a000cae90 2024-03-24T10:51:42.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.807+0000 7f9a060cd640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f9a001f0510 con 0x7f9a000cd960 2024-03-24T10:51:42.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.807+0000 7f99f6ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a000cb270 0x7f9a001566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:42.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.808+0000 7f99f6ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a000cb270 0x7f9a001566e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36344/0 (socket says 172.21.15.5:36344) 2024-03-24T10:51:42.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.808+0000 7f99f6ffd640 1 -- 172.21.15.5:0/3540777369 learned_addr learned my addr 172.21.15.5:0/3540777369 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:42.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.809+0000 7f99fffff640 1 --2- 172.21.15.5:0/3540777369 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9a000cd960 0x7f9a00156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:42.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.809+0000 7f99ff7fe640 1 --2- 172.21.15.5:0/3540777369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a000cae90 0x7f9a00153720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:42.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.811+0000 7f99f6ffd640 1 -- 172.21.15.5:0/3540777369 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9a000cd960 msgr2=0x7f9a00156c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:42.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.811+0000 7f99f6ffd640 1 --2- 172.21.15.5:0/3540777369 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9a000cd960 0x7f9a00156c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:42.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.811+0000 7f99f6ffd640 1 -- 172.21.15.5:0/3540777369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a000cae90 msgr2=0x7f9a00153720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:42.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.811+0000 7f99f6ffd640 1 --2- 172.21.15.5:0/3540777369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a000cae90 0x7f9a00153720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:42.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.811+0000 7f99f6ffd640 1 -- 172.21.15.5:0/3540777369 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99f8009b30 con 0x7f9a000cb270 2024-03-24T10:51:42.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.813+0000 7f99f6ffd640 1 --2- 172.21.15.5:0/3540777369 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a000cb270 0x7f9a001566e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f99e800c430 tx=0x7f99e800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:42.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.818+0000 7f99fcff9640 1 -- 172.21.15.5:0/3540777369 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e80052d0 con 0x7f9a000cb270 2024-03-24T10:51:42.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.818+0000 7f99fcff9640 1 -- 172.21.15.5:0/3540777369 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99e8005450 con 0x7f9a000cb270 2024-03-24T10:51:42.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.818+0000 7f99fcff9640 1 -- 172.21.15.5:0/3540777369 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e8007260 con 0x7f9a000cb270 2024-03-24T10:51:42.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.818+0000 7f9a060cd640 1 -- 172.21.15.5:0/3540777369 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a00157620 con 0x7f9a000cb270 2024-03-24T10:51:42.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.818+0000 7f9a060cd640 1 -- 172.21.15.5:0/3540777369 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a00155ee0 con 0x7f9a000cb270 2024-03-24T10:51:42.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.820+0000 7f9a060cd640 1 -- 172.21.15.5:0/3540777369 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a001560a0 con 0x7f9a000cb270 2024-03-24T10:51:42.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.844+0000 7f99fcff9640 1 -- 172.21.15.5:0/3540777369 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f99e80073c0 con 0x7f9a000cb270 2024-03-24T10:51:42.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.845+0000 7f99fcff9640 1 --2- 172.21.15.5:0/3540777369 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f99e006b810 0x7f99e006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:42.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.845+0000 7f99ff7fe640 1 --2- 172.21.15.5:0/3540777369 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f99e006b810 0x7f99e006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:42.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.845+0000 7f99ff7fe640 1 --2- 172.21.15.5:0/3540777369 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f99e006b810 0x7f99e006dcd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f99f0009b40 tx=0x7f99f0009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:42.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:42.848+0000 7f99fcff9640 1 -- 172.21.15.5:0/3540777369 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f99e8058310 con 0x7f9a000cb270 2024-03-24T10:51:43.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:43.138+0000 7f99fcff9640 1 -- 172.21.15.5:0/3540777369 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f99e80545b0 con 0x7f9a000cb270 2024-03-24T10:51:43.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:43.362+0000 7f9a060cd640 1 -- 172.21.15.5:0/3540777369 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a000681d0 con 0x7f9a000cb270 2024-03-24T10:51:43.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:43.375+0000 7f99fcff9640 1 -- 172.21.15.5:0/3540777369 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f99e8020020 con 0x7f9a000cb270 2024-03-24T10:51:43.375 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:43.375 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:43.376 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:43.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:43.378+0000 7f9a060cd640 1 -- 172.21.15.5:0/3540777369 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f99e006b810 msgr2=0x7f99e006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:43.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:43.378+0000 7f9a060cd640 1 --2- 172.21.15.5:0/3540777369 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f99e006b810 0x7f99e006dcd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f99f0009b40 tx=0x7f99f0009190 comp rx=0 tx=0).stop 2024-03-24T10:51:43.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:43.378+0000 7f9a060cd640 1 -- 172.21.15.5:0/3540777369 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a000cb270 msgr2=0x7f9a001566e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:43.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:43.378+0000 7f9a060cd640 1 --2- 172.21.15.5:0/3540777369 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a000cb270 0x7f9a001566e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f99e800c430 tx=0x7f99e800c8f0 comp rx=0 tx=0).stop 2024-03-24T10:51:43.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:43.378+0000 7f9a060cd640 1 -- 172.21.15.5:0/3540777369 shutdown_connections 2024-03-24T10:51:43.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:43.378+0000 7f9a060cd640 1 --2- 172.21.15.5:0/3540777369 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f99e006b810 0x7f99e006dcd0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:43.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:43.378+0000 7f9a060cd640 1 --2- 172.21.15.5:0/3540777369 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a000cb270 0x7f9a001566e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:43.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:43.378+0000 7f9a060cd640 1 --2- 172.21.15.5:0/3540777369 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9a000cd960 0x7f9a00156c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:43.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:43.378+0000 7f9a060cd640 1 --2- 172.21.15.5:0/3540777369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a000cae90 0x7f9a00153720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:43.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:43.378+0000 7f9a060cd640 1 -- 172.21.15.5:0/3540777369 >> 172.21.15.5:0/3540777369 conn(0x7f9a000cfb80 msgr2=0x7f9a00159a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:43.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:43.378+0000 7f9a060cd640 1 -- 172.21.15.5:0/3540777369 shutdown_connections 2024-03-24T10:51:43.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:43.378+0000 7f9a060cd640 1 -- 172.21.15.5:0/3540777369 wait complete. 2024-03-24T10:51:43.993 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:43.994 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:44.995 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:45.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.097+0000 7f18dbb9a640 1 Processor -- start 2024-03-24T10:51:45.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.097+0000 7f18dbb9a640 1 -- start start 2024-03-24T10:51:45.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.098+0000 7f18dbb9a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f18d40c59c0 0x7f18d40c2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:45.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.098+0000 7f18dbb9a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f18d4157140 0x7f18d40ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:45.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.098+0000 7f18dbb9a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18d40cb470 con 0x7f18d40c55e0 2024-03-24T10:51:45.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.098+0000 7f18dbb9a640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f18d41565e0 con 0x7f18d4157140 2024-03-24T10:51:45.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.098+0000 7f18dbb9a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f18d406eca0 con 0x7f18d40c59c0 2024-03-24T10:51:45.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.099+0000 7f18d9936640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f18d40c55e0 0x7f18d4156a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:33864/0 (socket says 172.21.15.5:33864) 2024-03-24T10:51:45.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.099+0000 7f18d9936640 1 -- 172.21.15.5:0/3204727611 learned_addr learned my addr 172.21.15.5:0/3204727611 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:51:45.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.099+0000 7f18da137640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f18d4157140 0x7f18d40ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:45.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.099+0000 7f18d9135640 1 --2- 172.21.15.5:0/3204727611 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f18d40c59c0 0x7f18d40c2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:45.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.102+0000 7f18da137640 1 -- 172.21.15.5:0/3204727611 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f18d40c59c0 msgr2=0x7f18d40c2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:45.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.102+0000 7f18da137640 1 --2- 172.21.15.5:0/3204727611 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f18d40c59c0 0x7f18d40c2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:45.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.102+0000 7f18da137640 1 -- 172.21.15.5:0/3204727611 >> v1:172.21.15.57:6789/0 conn(0x7f18d40c55e0 legacy=0x7f18d4156a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:45.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.102+0000 7f18da137640 1 -- 172.21.15.5:0/3204727611 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d406ef40 con 0x7f18d4157140 2024-03-24T10:51:45.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.105+0000 7f18da137640 1 --2- 172.21.15.5:0/3204727611 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d4157140 0x7f18d40ca840 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00aea0 tx=0x7f18cc02d030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9cfc8ab89c39674 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:45.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.109+0000 7f18d8934640 1 -- 172.21.15.5:0/3204727611 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18cc02dbd0 con 0x7f18d4157140 2024-03-24T10:51:45.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.109+0000 7f18d8934640 1 -- 172.21.15.5:0/3204727611 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18cc02dd30 con 0x7f18d4157140 2024-03-24T10:51:45.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.109+0000 7f18d8934640 1 -- 172.21.15.5:0/3204727611 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18cc02dea0 con 0x7f18d4157140 2024-03-24T10:51:45.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.110+0000 7f18dbb9a640 1 -- 172.21.15.5:0/3204727611 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d4157140 msgr2=0x7f18d40ca840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:45.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.110+0000 7f18dbb9a640 1 --2- 172.21.15.5:0/3204727611 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d4157140 0x7f18d40ca840 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f18cc00aea0 tx=0x7f18cc02d030 comp rx=0 tx=0).stop 2024-03-24T10:51:45.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.110+0000 7f18dbb9a640 1 -- 172.21.15.5:0/3204727611 shutdown_connections 2024-03-24T10:51:45.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.110+0000 7f18dbb9a640 1 --2- 172.21.15.5:0/3204727611 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f18d40c59c0 0x7f18d40c2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:45.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.110+0000 7f18dbb9a640 1 --2- 172.21.15.5:0/3204727611 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d4157140 0x7f18d40ca840 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:45.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.110+0000 7f18dbb9a640 1 -- 172.21.15.5:0/3204727611 >> 172.21.15.5:0/3204727611 conn(0x7f18d406b130 msgr2=0x7f18d406b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:45.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.110+0000 7f18dbb9a640 1 -- 172.21.15.5:0/3204727611 shutdown_connections 2024-03-24T10:51:45.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.110+0000 7f18dbb9a640 1 -- 172.21.15.5:0/3204727611 wait complete. 2024-03-24T10:51:45.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.112+0000 7f18dbb9a640 1 Processor -- start 2024-03-24T10:51:45.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.112+0000 7f18dbb9a640 1 -- start start 2024-03-24T10:51:45.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.112+0000 7f18dbb9a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f18d40c55e0 0x7f18d41ee050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:45.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.112+0000 7f18dbb9a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d40c59c0 0x7f18d40c47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:45.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.112+0000 7f18dbb9a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18d4157140 0x7f18d414d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:45.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.112+0000 7f18dbb9a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f18d4151ca0 con 0x7f18d40c59c0 2024-03-24T10:51:45.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.112+0000 7f18dbb9a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f18d414a7f0 con 0x7f18d4157140 2024-03-24T10:51:45.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.112+0000 7f18dbb9a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f18d4150eb0 con 0x7f18d40c55e0 2024-03-24T10:51:45.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.113+0000 7f18d9135640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d40c59c0 0x7f18d40c47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:45.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.113+0000 7f18d9936640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f18d40c55e0 0x7f18d41ee050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:45.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.113+0000 7f18da137640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18d4157140 0x7f18d414d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:45.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.113+0000 7f18d9135640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d40c59c0 0x7f18d40c47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36364/0 (socket says 172.21.15.5:36364) 2024-03-24T10:51:45.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.113+0000 7f18d9135640 1 -- 172.21.15.5:0/4224720871 learned_addr learned my addr 172.21.15.5:0/4224720871 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:45.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.117+0000 7f18d9135640 1 -- 172.21.15.5:0/4224720871 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f18d40c55e0 msgr2=0x7f18d41ee050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:45.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.117+0000 7f18d9135640 1 --2- 172.21.15.5:0/4224720871 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f18d40c55e0 0x7f18d41ee050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:45.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.117+0000 7f18d9135640 1 -- 172.21.15.5:0/4224720871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18d4157140 msgr2=0x7f18d414d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:45.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.117+0000 7f18d9135640 1 --2- 172.21.15.5:0/4224720871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18d4157140 0x7f18d414d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:45.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.117+0000 7f18d9135640 1 -- 172.21.15.5:0/4224720871 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18cc00ab80 con 0x7f18d40c59c0 2024-03-24T10:51:45.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.119+0000 7f18d9135640 1 --2- 172.21.15.5:0/4224720871 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d40c59c0 0x7f18d40c47a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f18c400e9b0 tx=0x7f18c400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:45.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.127+0000 7f18ca7fc640 1 -- 172.21.15.5:0/4224720871 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4009670 con 0x7f18d40c59c0 2024-03-24T10:51:45.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.127+0000 7f18ca7fc640 1 -- 172.21.15.5:0/4224720871 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18c400c9a0 con 0x7f18d40c59c0 2024-03-24T10:51:45.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.127+0000 7f18ca7fc640 1 -- 172.21.15.5:0/4224720871 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c400cd50 con 0x7f18d40c59c0 2024-03-24T10:51:45.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.127+0000 7f18dbb9a640 1 -- 172.21.15.5:0/4224720871 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18d41ee590 con 0x7f18d40c59c0 2024-03-24T10:51:45.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.127+0000 7f18dbb9a640 1 -- 172.21.15.5:0/4224720871 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18d4151110 con 0x7f18d40c59c0 2024-03-24T10:51:45.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.129+0000 7f18dbb9a640 1 -- 172.21.15.5:0/4224720871 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18d4062a00 con 0x7f18d40c59c0 2024-03-24T10:51:45.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.397+0000 7f18ca7fc640 1 -- 172.21.15.5:0/4224720871 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f18c4013470 con 0x7f18d40c59c0 2024-03-24T10:51:45.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.397+0000 7f18ca7fc640 1 --2- 172.21.15.5:0/4224720871 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f18b806b810 0x7f18b806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:45.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.397+0000 7f18d9936640 1 --2- 172.21.15.5:0/4224720871 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f18b806b810 0x7f18b806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:45.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.398+0000 7f18d9936640 1 --2- 172.21.15.5:0/4224720871 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f18b806b810 0x7f18b806dcd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f18d0005880 tx=0x7f18d0009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:45.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.411+0000 7f18ca7fc640 1 -- 172.21.15.5:0/4224720871 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f18c4059420 con 0x7f18d40c59c0 2024-03-24T10:51:45.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.411+0000 7f18ca7fc640 1 -- 172.21.15.5:0/4224720871 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f18c40556c0 con 0x7f18d40c59c0 2024-03-24T10:51:45.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.638+0000 7f18dbb9a640 1 -- 172.21.15.5:0/4224720871 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18d40681d0 con 0x7f18d40c59c0 2024-03-24T10:51:45.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.697+0000 7f18ca7fc640 1 -- 172.21.15.5:0/4224720871 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f18c4021020 con 0x7f18d40c59c0 2024-03-24T10:51:45.698 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:45.698 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:45.698 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:45.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.700+0000 7f18dbb9a640 1 -- 172.21.15.5:0/4224720871 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f18b806b810 msgr2=0x7f18b806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:45.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.700+0000 7f18dbb9a640 1 --2- 172.21.15.5:0/4224720871 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f18b806b810 0x7f18b806dcd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f18d0005880 tx=0x7f18d0009040 comp rx=0 tx=0).stop 2024-03-24T10:51:45.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.700+0000 7f18dbb9a640 1 -- 172.21.15.5:0/4224720871 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d40c59c0 msgr2=0x7f18d40c47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:45.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.700+0000 7f18dbb9a640 1 --2- 172.21.15.5:0/4224720871 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d40c59c0 0x7f18d40c47a0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f18c400e9b0 tx=0x7f18c400ee70 comp rx=0 tx=0).stop 2024-03-24T10:51:45.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.700+0000 7f18dbb9a640 1 -- 172.21.15.5:0/4224720871 shutdown_connections 2024-03-24T10:51:45.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.700+0000 7f18dbb9a640 1 --2- 172.21.15.5:0/4224720871 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f18b806b810 0x7f18b806dcd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:45.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.700+0000 7f18dbb9a640 1 --2- 172.21.15.5:0/4224720871 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f18d40c59c0 0x7f18d40c47a0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:45.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.700+0000 7f18dbb9a640 1 --2- 172.21.15.5:0/4224720871 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f18d4157140 0x7f18d414d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:45.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.700+0000 7f18dbb9a640 1 --2- 172.21.15.5:0/4224720871 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f18d40c55e0 0x7f18d41ee050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:45.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.700+0000 7f18dbb9a640 1 -- 172.21.15.5:0/4224720871 >> 172.21.15.5:0/4224720871 conn(0x7f18d406b130 msgr2=0x7f18d40cc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:45.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.700+0000 7f18dbb9a640 1 -- 172.21.15.5:0/4224720871 shutdown_connections 2024-03-24T10:51:45.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:45.700+0000 7f18dbb9a640 1 -- 172.21.15.5:0/4224720871 wait complete. 2024-03-24T10:51:46.315 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:46.315 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:47.317 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:47.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.419+0000 7f9b9cd43640 1 Processor -- start 2024-03-24T10:51:47.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.419+0000 7f9b9cd43640 1 -- start start 2024-03-24T10:51:47.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.420+0000 7f9b9cd43640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9b980c58c0 0x7f9b980caab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:47.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.420+0000 7f9b9cd43640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9b980c9610 0x7f9b980c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:47.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.420+0000 7f9b9cd43640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b980ca000 con 0x7f9b980c2ea0 2024-03-24T10:51:47.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.420+0000 7f9b9cd43640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f9b980cb210 con 0x7f9b980c58c0 2024-03-24T10:51:47.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.420+0000 7f9b9cd43640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f9b98156db0 con 0x7f9b980c9610 2024-03-24T10:51:47.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.420+0000 7f9b8ffff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9b980c58c0 0x7f9b980caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:47.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.421+0000 7f9b8ffff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9b980c58c0 0x7f9b980caab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36380/0 (socket says 172.21.15.5:36380) 2024-03-24T10:51:47.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.421+0000 7f9b8ffff640 1 -- 172.21.15.5:0/596153330 learned_addr learned my addr 172.21.15.5:0/596153330 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:47.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.421+0000 7f9b96d9d640 1 --2- 172.21.15.5:0/596153330 >> v2:172.21.15.57:3300/0 conn(0x7f9b980c9610 0x7f9b980c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:47.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.423+0000 7f9b8ffff640 1 -- 172.21.15.5:0/596153330 >> v2:172.21.15.57:3300/0 conn(0x7f9b980c9610 msgr2=0x7f9b980c9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:47.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.423+0000 7f9b8ffff640 1 --2- 172.21.15.5:0/596153330 >> v2:172.21.15.57:3300/0 conn(0x7f9b980c9610 0x7f9b980c9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:47.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.423+0000 7f9b8ffff640 1 -- 172.21.15.5:0/596153330 >> v1:172.21.15.57:6789/0 conn(0x7f9b980c2ea0 legacy=0x7f9b980c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:47.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.423+0000 7f9b8ffff640 1 -- 172.21.15.5:0/596153330 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b98156fc0 con 0x7f9b980c58c0 2024-03-24T10:51:47.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.425+0000 7f9b8ffff640 1 --2- 172.21.15.5:0/596153330 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b980c58c0 0x7f9b980caab0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9b980c69a0 tx=0x7f9b8002e200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=946d7d87eee5451b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:47.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.430+0000 7f9b95d9b640 1 -- 172.21.15.5:0/596153330 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80036440 con 0x7f9b980c58c0 2024-03-24T10:51:47.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.430+0000 7f9b95d9b640 1 -- 172.21.15.5:0/596153330 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b800365a0 con 0x7f9b980c58c0 2024-03-24T10:51:47.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.430+0000 7f9b95d9b640 1 -- 172.21.15.5:0/596153330 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b80036930 con 0x7f9b980c58c0 2024-03-24T10:51:47.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.431+0000 7f9b9cd43640 1 -- 172.21.15.5:0/596153330 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b980c58c0 msgr2=0x7f9b980caab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:47.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.431+0000 7f9b9cd43640 1 --2- 172.21.15.5:0/596153330 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b980c58c0 0x7f9b980caab0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9b980c69a0 tx=0x7f9b8002e200 comp rx=0 tx=0).stop 2024-03-24T10:51:47.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.431+0000 7f9b9cd43640 1 -- 172.21.15.5:0/596153330 shutdown_connections 2024-03-24T10:51:47.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.431+0000 7f9b9cd43640 1 --2- 172.21.15.5:0/596153330 >> v2:172.21.15.57:3300/0 conn(0x7f9b980c9610 0x7f9b980c9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:47.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.431+0000 7f9b9cd43640 1 --2- 172.21.15.5:0/596153330 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b980c58c0 0x7f9b980caab0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:47.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.431+0000 7f9b9cd43640 1 -- 172.21.15.5:0/596153330 >> 172.21.15.5:0/596153330 conn(0x7f9b9806b130 msgr2=0x7f9b9806b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:47.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.431+0000 7f9b9cd43640 1 -- 172.21.15.5:0/596153330 shutdown_connections 2024-03-24T10:51:47.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.431+0000 7f9b9cd43640 1 -- 172.21.15.5:0/596153330 wait complete. 2024-03-24T10:51:47.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.433+0000 7f9b9cd43640 1 Processor -- start 2024-03-24T10:51:47.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.433+0000 7f9b9cd43640 1 -- start start 2024-03-24T10:51:47.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.433+0000 7f9b9cd43640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b980c2ea0 0x7f9b98155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:47.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.433+0000 7f9b9cd43640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b980c58c0 0x7f9b981ebb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:47.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.433+0000 7f9b9cd43640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9b980c9610 0x7f9b981ec090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:47.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.433+0000 7f9b9cd43640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f9b980c8810 con 0x7f9b980c2ea0 2024-03-24T10:51:47.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.433+0000 7f9b9cd43640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9b981efcc0 con 0x7f9b980c58c0 2024-03-24T10:51:47.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.433+0000 7f9b9cd43640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f9b981ef0f0 con 0x7f9b980c9610 2024-03-24T10:51:47.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.434+0000 7f9b9659c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b980c2ea0 0x7f9b98155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:47.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.434+0000 7f9b8ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b980c58c0 0x7f9b981ebb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:47.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.434+0000 7f9b96d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9b980c9610 0x7f9b981ec090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:47.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.434+0000 7f9b9659c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b980c2ea0 0x7f9b98155cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36394/0 (socket says 172.21.15.5:36394) 2024-03-24T10:51:47.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.434+0000 7f9b9659c640 1 -- 172.21.15.5:0/2607615011 learned_addr learned my addr 172.21.15.5:0/2607615011 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:47.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.438+0000 7f9b9659c640 1 -- 172.21.15.5:0/2607615011 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9b980c9610 msgr2=0x7f9b981ec090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:47.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.438+0000 7f9b9659c640 1 --2- 172.21.15.5:0/2607615011 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9b980c9610 0x7f9b981ec090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:47.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.438+0000 7f9b9659c640 1 -- 172.21.15.5:0/2607615011 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b980c58c0 msgr2=0x7f9b981ebb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:47.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.438+0000 7f9b9659c640 1 --2- 172.21.15.5:0/2607615011 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b980c58c0 0x7f9b981ebb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:47.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.438+0000 7f9b9659c640 1 -- 172.21.15.5:0/2607615011 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b8000b770 con 0x7f9b980c2ea0 2024-03-24T10:51:47.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.440+0000 7f9b9659c640 1 --2- 172.21.15.5:0/2607615011 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b980c2ea0 0x7f9b98155cc0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800c3c0 tx=0x7f9b8800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:47.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.443+0000 7f9b8effd640 1 -- 172.21.15.5:0/2607615011 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b880052d0 con 0x7f9b980c2ea0 2024-03-24T10:51:47.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.443+0000 7f9b9cd43640 1 -- 172.21.15.5:0/2607615011 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b981ef390 con 0x7f9b980c2ea0 2024-03-24T10:51:47.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.443+0000 7f9b9cd43640 1 -- 172.21.15.5:0/2607615011 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b981ef950 con 0x7f9b980c2ea0 2024-03-24T10:51:47.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.444+0000 7f9b9cd43640 1 -- 172.21.15.5:0/2607615011 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b98062a00 con 0x7f9b980c2ea0 2024-03-24T10:51:47.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.446+0000 7f9b8effd640 1 -- 172.21.15.5:0/2607615011 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b88005430 con 0x7f9b980c2ea0 2024-03-24T10:51:47.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.446+0000 7f9b8effd640 1 -- 172.21.15.5:0/2607615011 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b88007250 con 0x7f9b980c2ea0 2024-03-24T10:51:47.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.755+0000 7f9b8effd640 1 -- 172.21.15.5:0/2607615011 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9b880073b0 con 0x7f9b980c2ea0 2024-03-24T10:51:47.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.756+0000 7f9b8effd640 1 --2- 172.21.15.5:0/2607615011 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9b7806b840 0x7f9b7806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:47.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.756+0000 7f9b8ffff640 1 --2- 172.21.15.5:0/2607615011 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9b7806b840 0x7f9b7806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:47.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.756+0000 7f9b8ffff640 1 --2- 172.21.15.5:0/2607615011 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9b7806b840 0x7f9b7806dd00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9b80038000 tx=0x7f9b80032000 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:47.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.774+0000 7f9b8effd640 1 -- 172.21.15.5:0/2607615011 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f9b88007660 con 0x7f9b980c2ea0 2024-03-24T10:51:47.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:47.774+0000 7f9b8effd640 1 -- 172.21.15.5:0/2607615011 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9b88054460 con 0x7f9b980c2ea0 2024-03-24T10:51:48.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:48.032+0000 7f9b9cd43640 1 -- 172.21.15.5:0/2607615011 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b980681d0 con 0x7f9b980c2ea0 2024-03-24T10:51:48.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:48.060+0000 7f9b8effd640 1 -- 172.21.15.5:0/2607615011 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f9b88020020 con 0x7f9b980c2ea0 2024-03-24T10:51:48.060 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:48.060 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:48.060 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:48.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:48.062+0000 7f9b9cd43640 1 -- 172.21.15.5:0/2607615011 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9b7806b840 msgr2=0x7f9b7806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:48.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:48.062+0000 7f9b9cd43640 1 --2- 172.21.15.5:0/2607615011 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9b7806b840 0x7f9b7806dd00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9b80038000 tx=0x7f9b80032000 comp rx=0 tx=0).stop 2024-03-24T10:51:48.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:48.062+0000 7f9b9cd43640 1 -- 172.21.15.5:0/2607615011 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b980c2ea0 msgr2=0x7f9b98155cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:48.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:48.062+0000 7f9b9cd43640 1 --2- 172.21.15.5:0/2607615011 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b980c2ea0 0x7f9b98155cc0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800c3c0 tx=0x7f9b8800c880 comp rx=0 tx=0).stop 2024-03-24T10:51:48.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:48.062+0000 7f9b9cd43640 1 -- 172.21.15.5:0/2607615011 shutdown_connections 2024-03-24T10:51:48.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:48.062+0000 7f9b9cd43640 1 --2- 172.21.15.5:0/2607615011 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9b7806b840 0x7f9b7806dd00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:48.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:48.062+0000 7f9b9cd43640 1 --2- 172.21.15.5:0/2607615011 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9b980c9610 0x7f9b981ec090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:48.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:48.062+0000 7f9b9cd43640 1 --2- 172.21.15.5:0/2607615011 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9b980c58c0 0x7f9b981ebb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:48.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:48.062+0000 7f9b9cd43640 1 --2- 172.21.15.5:0/2607615011 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9b980c2ea0 0x7f9b98155cc0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:48.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:48.062+0000 7f9b9cd43640 1 -- 172.21.15.5:0/2607615011 >> 172.21.15.5:0/2607615011 conn(0x7f9b9806b130 msgr2=0x7f9b98150c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:48.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:48.062+0000 7f9b9cd43640 1 -- 172.21.15.5:0/2607615011 shutdown_connections 2024-03-24T10:51:48.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:48.062+0000 7f9b9cd43640 1 -- 172.21.15.5:0/2607615011 wait complete. 2024-03-24T10:51:48.677 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:48.677 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:49.678 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:49.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.781+0000 7fdd4ff19640 1 Processor -- start 2024-03-24T10:51:49.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.781+0000 7fdd4ff19640 1 -- start start 2024-03-24T10:51:49.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.782+0000 7fdd4ff19640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdd4815ac00 0x7fdd480cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:49.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.782+0000 7fdd4ff19640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd480ccaa0 0x7fdd4806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:49.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.782+0000 7fdd4ff19640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd480cb2f0 con 0x7fdd481543a0 2024-03-24T10:51:49.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.782+0000 7fdd4ff19640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fdd480c6750 con 0x7fdd4815ac00 2024-03-24T10:51:49.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.782+0000 7fdd4ff19640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fdd480cceb0 con 0x7fdd480ccaa0 2024-03-24T10:51:49.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.782+0000 7fdd4e4b6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd480ccaa0 0x7fdd4806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:49.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.783+0000 7fdd4d4b4640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdd4815ac00 0x7fdd480cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:49.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.783+0000 7fdd4dcb5640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fdd481543a0 0x7fdd4806d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:45974/0 (socket says 172.21.15.5:45974) 2024-03-24T10:51:49.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.783+0000 7fdd4dcb5640 1 -- 172.21.15.5:0/1997375486 learned_addr learned my addr 172.21.15.5:0/1997375486 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:51:49.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.786+0000 7fdd4e4b6640 1 -- 172.21.15.5:0/1997375486 >> v2:172.21.15.5:3300/0 conn(0x7fdd4815ac00 msgr2=0x7fdd480cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:49.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.786+0000 7fdd4e4b6640 1 --2- 172.21.15.5:0/1997375486 >> v2:172.21.15.5:3300/0 conn(0x7fdd4815ac00 0x7fdd480cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:49.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.786+0000 7fdd4e4b6640 1 -- 172.21.15.5:0/1997375486 >> v1:172.21.15.5:6789/0 conn(0x7fdd481543a0 legacy=0x7fdd4806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:49.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.786+0000 7fdd4e4b6640 1 -- 172.21.15.5:0/1997375486 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd4815b070 con 0x7fdd480ccaa0 2024-03-24T10:51:49.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.789+0000 7fdd4e4b6640 1 --2- 172.21.15.5:0/1997375486 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd480ccaa0 0x7fdd4806bf50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdd40009d90 tx=0x7fdd4002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16df6b47f07b33ed server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:49.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.794+0000 7fdd4ccb3640 1 -- 172.21.15.5:0/1997375486 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4002cc50 con 0x7fdd480ccaa0 2024-03-24T10:51:49.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.795+0000 7fdd4ccb3640 1 -- 172.21.15.5:0/1997375486 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd4002cdb0 con 0x7fdd480ccaa0 2024-03-24T10:51:49.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.795+0000 7fdd4ccb3640 1 -- 172.21.15.5:0/1997375486 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd40034660 con 0x7fdd480ccaa0 2024-03-24T10:51:49.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.795+0000 7fdd4ff19640 1 -- 172.21.15.5:0/1997375486 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd480ccaa0 msgr2=0x7fdd4806bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:49.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.795+0000 7fdd4ff19640 1 --2- 172.21.15.5:0/1997375486 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd480ccaa0 0x7fdd4806bf50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdd40009d90 tx=0x7fdd4002bea0 comp rx=0 tx=0).stop 2024-03-24T10:51:49.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.796+0000 7fdd4ff19640 1 -- 172.21.15.5:0/1997375486 shutdown_connections 2024-03-24T10:51:49.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.796+0000 7fdd4ff19640 1 --2- 172.21.15.5:0/1997375486 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd480ccaa0 0x7fdd4806bf50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:49.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.796+0000 7fdd4ff19640 1 --2- 172.21.15.5:0/1997375486 >> v2:172.21.15.5:3300/0 conn(0x7fdd4815ac00 0x7fdd480cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:49.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.796+0000 7fdd4ff19640 1 -- 172.21.15.5:0/1997375486 >> 172.21.15.5:0/1997375486 conn(0x7fdd48072ef0 msgr2=0x7fdd48061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:49.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.796+0000 7fdd4ff19640 1 -- 172.21.15.5:0/1997375486 shutdown_connections 2024-03-24T10:51:49.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.796+0000 7fdd4ff19640 1 -- 172.21.15.5:0/1997375486 wait complete. 2024-03-24T10:51:49.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.798+0000 7fdd4ff19640 1 Processor -- start 2024-03-24T10:51:49.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.798+0000 7fdd4ff19640 1 -- start start 2024-03-24T10:51:49.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.799+0000 7fdd4ff19640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd480ccaa0 0x7fdd481f4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:49.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.799+0000 7fdd4ff19640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd481543a0 0x7fdd481f0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:49.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.799+0000 7fdd4ff19640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4815ac00 0x7fdd4815e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:49.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.799+0000 7fdd4ff19640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fdd481f06b0 con 0x7fdd481543a0 2024-03-24T10:51:49.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.799+0000 7fdd4ff19640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fdd481f3dd0 con 0x7fdd4815ac00 2024-03-24T10:51:49.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.799+0000 7fdd4ff19640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fdd481f1100 con 0x7fdd480ccaa0 2024-03-24T10:51:49.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.799+0000 7fdd4e4b6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4815ac00 0x7fdd4815e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:49.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.800+0000 7fdd4dcb5640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd480ccaa0 0x7fdd481f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:49.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.800+0000 7fdd4d4b4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd481543a0 0x7fdd481f0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:49.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.800+0000 7fdd4e4b6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4815ac00 0x7fdd4815e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:55530/0 (socket says 172.21.15.5:55530) 2024-03-24T10:51:49.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.800+0000 7fdd4e4b6640 1 -- 172.21.15.5:0/187771381 learned_addr learned my addr 172.21.15.5:0/187771381 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:49.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.805+0000 7fdd4e4b6640 1 -- 172.21.15.5:0/187771381 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd480ccaa0 msgr2=0x7fdd481f4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:49.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.805+0000 7fdd4e4b6640 1 --2- 172.21.15.5:0/187771381 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd480ccaa0 0x7fdd481f4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:49.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.805+0000 7fdd4e4b6640 1 -- 172.21.15.5:0/187771381 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd481543a0 msgr2=0x7fdd481f0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:49.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.805+0000 7fdd4e4b6640 1 --2- 172.21.15.5:0/187771381 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd481543a0 0x7fdd481f0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:49.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.805+0000 7fdd4e4b6640 1 -- 172.21.15.5:0/187771381 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd40009a70 con 0x7fdd4815ac00 2024-03-24T10:51:49.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.805+0000 7fdd4dcb5640 1 --2- 172.21.15.5:0/187771381 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd480ccaa0 0x7fdd481f4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:51:49.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.808+0000 7fdd4e4b6640 1 --2- 172.21.15.5:0/187771381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4815ac00 0x7fdd4815e050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdd4002c4b0 tx=0x7fdd40004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:49.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.814+0000 7fdd3e7fc640 1 -- 172.21.15.5:0/187771381 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4003e600 con 0x7fdd4815ac00 2024-03-24T10:51:49.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.814+0000 7fdd3e7fc640 1 -- 172.21.15.5:0/187771381 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd4003e760 con 0x7fdd4815ac00 2024-03-24T10:51:49.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.814+0000 7fdd3e7fc640 1 -- 172.21.15.5:0/187771381 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4003eb10 con 0x7fdd4815ac00 2024-03-24T10:51:49.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.814+0000 7fdd4ff19640 1 -- 172.21.15.5:0/187771381 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd4815e590 con 0x7fdd4815ac00 2024-03-24T10:51:49.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.814+0000 7fdd4ff19640 1 -- 172.21.15.5:0/187771381 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd48147f80 con 0x7fdd4815ac00 2024-03-24T10:51:49.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:49.815+0000 7fdd4ff19640 1 -- 172.21.15.5:0/187771381 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd481f3f80 con 0x7fdd4815ac00 2024-03-24T10:51:50.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:50.091+0000 7fdd3e7fc640 1 -- 172.21.15.5:0/187771381 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdd4003ec70 con 0x7fdd4815ac00 2024-03-24T10:51:50.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:50.091+0000 7fdd3e7fc640 1 --2- 172.21.15.5:0/187771381 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdd2c06b860 0x7fdd2c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:50.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:50.092+0000 7fdd4dcb5640 1 --2- 172.21.15.5:0/187771381 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdd2c06b860 0x7fdd2c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:50.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:50.092+0000 7fdd4dcb5640 1 --2- 172.21.15.5:0/187771381 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdd2c06b860 0x7fdd2c06dd20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdd44007fd0 tx=0x7fdd44007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:50.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:50.094+0000 7fdd3e7fc640 1 -- 172.21.15.5:0/187771381 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fdd4007b9b0 con 0x7fdd4815ac00 2024-03-24T10:51:50.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:50.105+0000 7fdd3e7fc640 1 -- 172.21.15.5:0/187771381 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdd40077c50 con 0x7fdd4815ac00 2024-03-24T10:51:50.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:50.358+0000 7fdd4ff19640 1 -- 172.21.15.5:0/187771381 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd481f3f80 con 0x7fdd4815ac00 2024-03-24T10:51:50.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:50.374+0000 7fdd3e7fc640 1 -- 172.21.15.5:0/187771381 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fdd481f3f80 con 0x7fdd4815ac00 2024-03-24T10:51:50.374 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:50.374 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:50.375 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:50.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:50.376+0000 7fdd4ff19640 1 -- 172.21.15.5:0/187771381 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdd2c06b860 msgr2=0x7fdd2c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:50.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:50.376+0000 7fdd4ff19640 1 --2- 172.21.15.5:0/187771381 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdd2c06b860 0x7fdd2c06dd20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdd44007fd0 tx=0x7fdd44007e80 comp rx=0 tx=0).stop 2024-03-24T10:51:50.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:50.376+0000 7fdd4ff19640 1 -- 172.21.15.5:0/187771381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4815ac00 msgr2=0x7fdd4815e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:50.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:50.376+0000 7fdd4ff19640 1 --2- 172.21.15.5:0/187771381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4815ac00 0x7fdd4815e050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fdd4002c4b0 tx=0x7fdd40004a40 comp rx=0 tx=0).stop 2024-03-24T10:51:50.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:50.376+0000 7fdd4ff19640 1 -- 172.21.15.5:0/187771381 shutdown_connections 2024-03-24T10:51:50.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:50.376+0000 7fdd4ff19640 1 --2- 172.21.15.5:0/187771381 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdd2c06b860 0x7fdd2c06dd20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:50.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:50.376+0000 7fdd4ff19640 1 --2- 172.21.15.5:0/187771381 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd481543a0 0x7fdd481f0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:50.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:50.376+0000 7fdd4ff19640 1 --2- 172.21.15.5:0/187771381 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd4815ac00 0x7fdd4815e050 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:50.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:50.376+0000 7fdd4ff19640 1 --2- 172.21.15.5:0/187771381 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd480ccaa0 0x7fdd481f4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:50.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:50.376+0000 7fdd4ff19640 1 -- 172.21.15.5:0/187771381 >> 172.21.15.5:0/187771381 conn(0x7fdd48072ef0 msgr2=0x7fdd481511e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:50.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:50.377+0000 7fdd4ff19640 1 -- 172.21.15.5:0/187771381 shutdown_connections 2024-03-24T10:51:50.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:50.377+0000 7fdd4ff19640 1 -- 172.21.15.5:0/187771381 wait complete. 2024-03-24T10:51:50.991 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:50.992 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:51.992 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:52.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.094+0000 7ff0b1a1c640 1 Processor -- start 2024-03-24T10:51:52.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.094+0000 7ff0b1a1c640 1 -- start start 2024-03-24T10:51:52.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.095+0000 7ff0b1a1c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff0ac1543a0 0x7ff0ac15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:52.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.095+0000 7ff0b1a1c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff0ac06d4f0 0x7ff0ac0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:52.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.095+0000 7ff0b1a1c640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0ac0cb2f0 con 0x7ff0ac0ccaa0 2024-03-24T10:51:52.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.095+0000 7ff0b1a1c640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7ff0ac0c6750 con 0x7ff0ac1543a0 2024-03-24T10:51:52.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.095+0000 7ff0b1a1c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7ff0ac0cc380 con 0x7ff0ac06d4f0 2024-03-24T10:51:52.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.096+0000 7ff0a3fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff0ac06d4f0 0x7ff0ac0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:52.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.096+0000 7ff0ab7fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7ff0ac0ccaa0 0x7ff0ac06bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:45976/0 (socket says 172.21.15.5:45976) 2024-03-24T10:51:52.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.096+0000 7ff0ab7fe640 1 -- 172.21.15.5:0/2443993238 learned_addr learned my addr 172.21.15.5:0/2443993238 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:51:52.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.096+0000 7ff0aaffd640 1 --2- 172.21.15.5:0/2443993238 >> v2:172.21.15.5:3300/0 conn(0x7ff0ac1543a0 0x7ff0ac15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:52.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.098+0000 7ff0a3fff640 1 -- 172.21.15.5:0/2443993238 >> v2:172.21.15.5:3300/0 conn(0x7ff0ac1543a0 msgr2=0x7ff0ac15ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:52.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.098+0000 7ff0a3fff640 1 --2- 172.21.15.5:0/2443993238 >> v2:172.21.15.5:3300/0 conn(0x7ff0ac1543a0 0x7ff0ac15ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:52.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.098+0000 7ff0a3fff640 1 -- 172.21.15.5:0/2443993238 >> v1:172.21.15.5:6789/0 conn(0x7ff0ac0ccaa0 legacy=0x7ff0ac06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:52.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.098+0000 7ff0a3fff640 1 -- 172.21.15.5:0/2443993238 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0ac0ca7e0 con 0x7ff0ac06d4f0 2024-03-24T10:51:52.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.100+0000 7ff0a3fff640 1 --2- 172.21.15.5:0/2443993238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0ac06d4f0 0x7ff0ac0cc560 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac0c6990 tx=0x7ff09402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8731279e3b7da3e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:52.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.105+0000 7ff0aa7fc640 1 -- 172.21.15.5:0/2443993238 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff094030610 con 0x7ff0ac06d4f0 2024-03-24T10:51:52.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.105+0000 7ff0aa7fc640 1 -- 172.21.15.5:0/2443993238 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff094030770 con 0x7ff0ac06d4f0 2024-03-24T10:51:52.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.105+0000 7ff0aa7fc640 1 -- 172.21.15.5:0/2443993238 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff094030b00 con 0x7ff0ac06d4f0 2024-03-24T10:51:52.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.106+0000 7ff0b1a1c640 1 -- 172.21.15.5:0/2443993238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0ac06d4f0 msgr2=0x7ff0ac0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:52.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.106+0000 7ff0b1a1c640 1 --2- 172.21.15.5:0/2443993238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0ac06d4f0 0x7ff0ac0cc560 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac0c6990 tx=0x7ff09402b960 comp rx=0 tx=0).stop 2024-03-24T10:51:52.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.106+0000 7ff0b1a1c640 1 -- 172.21.15.5:0/2443993238 shutdown_connections 2024-03-24T10:51:52.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.106+0000 7ff0b1a1c640 1 --2- 172.21.15.5:0/2443993238 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0ac06d4f0 0x7ff0ac0cc560 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:52.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.106+0000 7ff0b1a1c640 1 --2- 172.21.15.5:0/2443993238 >> v2:172.21.15.5:3300/0 conn(0x7ff0ac1543a0 0x7ff0ac15ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:52.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.106+0000 7ff0b1a1c640 1 -- 172.21.15.5:0/2443993238 >> 172.21.15.5:0/2443993238 conn(0x7ff0ac072ef0 msgr2=0x7ff0ac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:52.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.106+0000 7ff0b1a1c640 1 -- 172.21.15.5:0/2443993238 shutdown_connections 2024-03-24T10:51:52.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.106+0000 7ff0b1a1c640 1 -- 172.21.15.5:0/2443993238 wait complete. 2024-03-24T10:51:52.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.107+0000 7ff0b1a1c640 1 Processor -- start 2024-03-24T10:51:52.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.107+0000 7ff0b1a1c640 1 -- start start 2024-03-24T10:51:52.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.108+0000 7ff0b1a1c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0ac06d4f0 0x7ff0ac14e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:52.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.108+0000 7ff0b1a1c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff0ac0ccaa0 0x7ff0ac150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:52.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.108+0000 7ff0b1a1c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0ac1543a0 0x7ff0ac1506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:52.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.108+0000 7ff0b1a1c640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff0ac14fd40 con 0x7ff0ac1543a0 2024-03-24T10:51:52.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.108+0000 7ff0b1a1c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff0ac1f06e0 con 0x7ff0ac06d4f0 2024-03-24T10:51:52.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.108+0000 7ff0b1a1c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff0ac159e50 con 0x7ff0ac0ccaa0 2024-03-24T10:51:52.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.109+0000 7ff0ab7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0ac1543a0 0x7ff0ac1506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:52.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.109+0000 7ff0ab7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0ac1543a0 0x7ff0ac1506a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36440/0 (socket says 172.21.15.5:36440) 2024-03-24T10:51:52.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.109+0000 7ff0ab7fe640 1 -- 172.21.15.5:0/720716615 learned_addr learned my addr 172.21.15.5:0/720716615 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:52.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.109+0000 7ff0a3fff640 1 --2- 172.21.15.5:0/720716615 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff0ac0ccaa0 0x7ff0ac150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:52.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.109+0000 7ff0aaffd640 1 --2- 172.21.15.5:0/720716615 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0ac06d4f0 0x7ff0ac14e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:52.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.112+0000 7ff0ab7fe640 1 -- 172.21.15.5:0/720716615 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff0ac0ccaa0 msgr2=0x7ff0ac150160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:52.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.112+0000 7ff0ab7fe640 1 --2- 172.21.15.5:0/720716615 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff0ac0ccaa0 0x7ff0ac150160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:52.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.112+0000 7ff0ab7fe640 1 -- 172.21.15.5:0/720716615 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0ac06d4f0 msgr2=0x7ff0ac14e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:52.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.112+0000 7ff0ab7fe640 1 --2- 172.21.15.5:0/720716615 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0ac06d4f0 0x7ff0ac14e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:52.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.112+0000 7ff0ab7fe640 1 -- 172.21.15.5:0/720716615 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff094009400 con 0x7ff0ac1543a0 2024-03-24T10:51:52.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.114+0000 7ff0ab7fe640 1 --2- 172.21.15.5:0/720716615 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0ac1543a0 0x7ff0ac1506a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff0a400cbc0 tx=0x7ff0a400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:52.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.119+0000 7ff0a8ff9640 1 -- 172.21.15.5:0/720716615 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a400ad40 con 0x7ff0ac1543a0 2024-03-24T10:51:52.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.119+0000 7ff0a8ff9640 1 -- 172.21.15.5:0/720716615 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0a4012400 con 0x7ff0ac1543a0 2024-03-24T10:51:52.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.119+0000 7ff0a8ff9640 1 -- 172.21.15.5:0/720716615 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a400aea0 con 0x7ff0ac1543a0 2024-03-24T10:51:52.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.119+0000 7ff0b1a1c640 1 -- 172.21.15.5:0/720716615 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0ac1f0950 con 0x7ff0ac1543a0 2024-03-24T10:51:52.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.119+0000 7ff0b1a1c640 1 -- 172.21.15.5:0/720716615 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0ac14f600 con 0x7ff0ac1543a0 2024-03-24T10:51:52.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.120+0000 7ff0b1a1c640 1 -- 172.21.15.5:0/720716615 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0ac14fe80 con 0x7ff0ac1543a0 2024-03-24T10:51:52.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.144+0000 7ff0a8ff9640 1 -- 172.21.15.5:0/720716615 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff0a4007070 con 0x7ff0ac1543a0 2024-03-24T10:51:52.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.144+0000 7ff0a8ff9640 1 --2- 172.21.15.5:0/720716615 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff08c06b860 0x7ff08c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:52.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.145+0000 7ff0aaffd640 1 --2- 172.21.15.5:0/720716615 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff08c06b860 0x7ff08c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:52.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.145+0000 7ff0aaffd640 1 --2- 172.21.15.5:0/720716615 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff08c06b860 0x7ff08c06dd20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff09c009aa0 tx=0x7ff09c009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:52.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.147+0000 7ff0a8ff9640 1 -- 172.21.15.5:0/720716615 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7ff0a4058c40 con 0x7ff0ac1543a0 2024-03-24T10:51:52.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.440+0000 7ff0a8ff9640 1 -- 172.21.15.5:0/720716615 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff0a4054ee0 con 0x7ff0ac1543a0 2024-03-24T10:51:52.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.700+0000 7ff0b1a1c640 1 -- 172.21.15.5:0/720716615 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0ac14fe80 con 0x7ff0ac1543a0 2024-03-24T10:51:52.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.721+0000 7ff0a8ff9640 1 -- 172.21.15.5:0/720716615 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7ff0ac14fe80 con 0x7ff0ac1543a0 2024-03-24T10:51:52.721 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:52.721 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:52.722 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.723+0000 7ff0b1a1c640 1 -- 172.21.15.5:0/720716615 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff08c06b860 msgr2=0x7ff08c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.723+0000 7ff0b1a1c640 1 --2- 172.21.15.5:0/720716615 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff08c06b860 0x7ff08c06dd20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff09c009aa0 tx=0x7ff09c009190 comp rx=0 tx=0).stop 2024-03-24T10:51:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.723+0000 7ff0b1a1c640 1 -- 172.21.15.5:0/720716615 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0ac1543a0 msgr2=0x7ff0ac1506a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.723+0000 7ff0b1a1c640 1 --2- 172.21.15.5:0/720716615 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0ac1543a0 0x7ff0ac1506a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff0a400cbc0 tx=0x7ff0a400a430 comp rx=0 tx=0).stop 2024-03-24T10:51:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.723+0000 7ff0b1a1c640 1 -- 172.21.15.5:0/720716615 shutdown_connections 2024-03-24T10:51:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.723+0000 7ff0b1a1c640 1 --2- 172.21.15.5:0/720716615 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff08c06b860 0x7ff08c06dd20 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.723+0000 7ff0b1a1c640 1 --2- 172.21.15.5:0/720716615 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0ac1543a0 0x7ff0ac1506a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.723+0000 7ff0b1a1c640 1 --2- 172.21.15.5:0/720716615 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff0ac0ccaa0 0x7ff0ac150160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.723+0000 7ff0b1a1c640 1 --2- 172.21.15.5:0/720716615 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0ac06d4f0 0x7ff0ac14e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.723+0000 7ff0b1a1c640 1 -- 172.21.15.5:0/720716615 >> 172.21.15.5:0/720716615 conn(0x7ff0ac072ef0 msgr2=0x7ff0ac157f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.723+0000 7ff0b1a1c640 1 -- 172.21.15.5:0/720716615 shutdown_connections 2024-03-24T10:51:52.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:52.723+0000 7ff0b1a1c640 1 -- 172.21.15.5:0/720716615 wait complete. 2024-03-24T10:51:53.339 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:53.339 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:54.340 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:54.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.443+0000 7f227fd1e640 1 Processor -- start 2024-03-24T10:51:54.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.443+0000 7f227fd1e640 1 -- start start 2024-03-24T10:51:54.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.444+0000 7f227fd1e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f22780c4300 0x7f22780cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:54.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.444+0000 7f227fd1e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22780c4720 con 0x7f22780cdfb0 2024-03-24T10:51:54.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.444+0000 7f227fd1e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22780c9050 con 0x7f2278154220 2024-03-24T10:51:54.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.444+0000 7f227fd1e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f22780c8bf0 con 0x7f22780c4300 2024-03-24T10:51:54.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.444+0000 7f227e2bb640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f22780cdfb0 0x7f227806c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:50792/0 (socket says 172.21.15.5:50792) 2024-03-24T10:51:54.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.444+0000 7f227e2bb640 1 -- 172.21.15.5:0/2872900385 learned_addr learned my addr 172.21.15.5:0/2872900385 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:51:54.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.445+0000 7f227d2b9640 1 --2- 172.21.15.5:0/2872900385 >> v2:172.21.15.57:3300/0 conn(0x7f22780c4300 0x7f22780cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:54.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.447+0000 7f227cab8640 1 -- 172.21.15.5:0/2872900385 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 449910451 0 0) 0x7f22780c4720 con 0x7f22780cdfb0 2024-03-24T10:51:54.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.447+0000 7f227cab8640 1 -- 172.21.15.5:0/2872900385 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2264003520 con 0x7f22780cdfb0 2024-03-24T10:51:54.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.451+0000 7f227d2b9640 1 -- 172.21.15.5:0/2872900385 >> v1:172.21.15.57:6789/0 conn(0x7f2278154220 legacy=0x7f22780c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:54.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.451+0000 7f227d2b9640 1 -- 172.21.15.5:0/2872900385 >> v1:172.21.15.5:6789/0 conn(0x7f22780cdfb0 legacy=0x7f227806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:54.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.451+0000 7f227d2b9640 1 -- 172.21.15.5:0/2872900385 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f227815a5f0 con 0x7f22780c4300 2024-03-24T10:51:54.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.455+0000 7f227d2b9640 1 --2- 172.21.15.5:0/2872900385 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22780c4300 0x7f22780cda70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f22680097a0 tx=0x7f226802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a23e3aa5be0257f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:54.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.460+0000 7f227cab8640 1 -- 172.21.15.5:0/2872900385 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226802f810 con 0x7f22780c4300 2024-03-24T10:51:54.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.460+0000 7f227cab8640 1 -- 172.21.15.5:0/2872900385 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f226802f970 con 0x7f22780c4300 2024-03-24T10:51:54.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.460+0000 7f227cab8640 1 -- 172.21.15.5:0/2872900385 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f226802fce0 con 0x7f22780c4300 2024-03-24T10:51:54.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.461+0000 7f227fd1e640 1 -- 172.21.15.5:0/2872900385 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22780c4300 msgr2=0x7f22780cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:54.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.461+0000 7f227fd1e640 1 --2- 172.21.15.5:0/2872900385 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22780c4300 0x7f22780cda70 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f22680097a0 tx=0x7f226802b9f0 comp rx=0 tx=0).stop 2024-03-24T10:51:54.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.461+0000 7f227fd1e640 1 -- 172.21.15.5:0/2872900385 shutdown_connections 2024-03-24T10:51:54.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.461+0000 7f227fd1e640 1 --2- 172.21.15.5:0/2872900385 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f22780c4300 0x7f22780cda70 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:54.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.461+0000 7f227fd1e640 1 -- 172.21.15.5:0/2872900385 >> 172.21.15.5:0/2872900385 conn(0x7f227806f830 msgr2=0x7f227806fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:54.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.461+0000 7f227fd1e640 1 -- 172.21.15.5:0/2872900385 shutdown_connections 2024-03-24T10:51:54.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.461+0000 7f227fd1e640 1 -- 172.21.15.5:0/2872900385 wait complete. 2024-03-24T10:51:54.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.462+0000 7f227fd1e640 1 Processor -- start 2024-03-24T10:51:54.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.463+0000 7f227fd1e640 1 -- start start 2024-03-24T10:51:54.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.463+0000 7f227fd1e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f22780c4300 0x7f22780adb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:54.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.463+0000 7f227fd1e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f22780cdfb0 0x7f22781e47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:54.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.463+0000 7f227fd1e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2278154220 0x7f22781e4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:54.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.463+0000 7f227fd1e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f22780aede0 con 0x7f22780c4300 2024-03-24T10:51:54.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.463+0000 7f227fd1e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f22781f0f50 con 0x7f2278154220 2024-03-24T10:51:54.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.463+0000 7f227fd1e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f22780c7cd0 con 0x7f22780cdfb0 2024-03-24T10:51:54.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.464+0000 7f227daba640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f22780c4300 0x7f22780adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:54.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.464+0000 7f227e2bb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2278154220 0x7f22781e4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:54.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.464+0000 7f227d2b9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f22780cdfb0 0x7f22781e47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:54.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.464+0000 7f227daba640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f22780c4300 0x7f22780adb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38772/0 (socket says 172.21.15.5:38772) 2024-03-24T10:51:54.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.464+0000 7f227daba640 1 -- 172.21.15.5:0/1520590589 learned_addr learned my addr 172.21.15.5:0/1520590589 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:54.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.468+0000 7f227daba640 1 -- 172.21.15.5:0/1520590589 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f22780cdfb0 msgr2=0x7f22781e47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:54.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.468+0000 7f227daba640 1 --2- 172.21.15.5:0/1520590589 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f22780cdfb0 0x7f22781e47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:54.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.468+0000 7f227daba640 1 -- 172.21.15.5:0/1520590589 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2278154220 msgr2=0x7f22781e4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:54.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.468+0000 7f227daba640 1 --2- 172.21.15.5:0/1520590589 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2278154220 0x7f22781e4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:54.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.468+0000 7f227daba640 1 -- 172.21.15.5:0/1520590589 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2268009400 con 0x7f22780c4300 2024-03-24T10:51:54.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.470+0000 7f227daba640 1 --2- 172.21.15.5:0/1520590589 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f22780c4300 0x7f22780adb20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f227400c450 tx=0x7f227400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:54.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.475+0000 7f226e7fc640 1 -- 172.21.15.5:0/1520590589 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22740052d0 con 0x7f22780c4300 2024-03-24T10:51:54.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.475+0000 7f226e7fc640 1 -- 172.21.15.5:0/1520590589 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2274005430 con 0x7f22780c4300 2024-03-24T10:51:54.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.475+0000 7f227fd1e640 1 -- 172.21.15.5:0/1520590589 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22781f0340 con 0x7f22780c4300 2024-03-24T10:51:54.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.475+0000 7f226e7fc640 1 -- 172.21.15.5:0/1520590589 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2274012870 con 0x7f22780c4300 2024-03-24T10:51:54.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.475+0000 7f227fd1e640 1 -- 172.21.15.5:0/1520590589 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22781f0900 con 0x7f22780c4300 2024-03-24T10:51:54.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.476+0000 7f227fd1e640 1 -- 172.21.15.5:0/1520590589 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2278153dd0 con 0x7f22780c4300 2024-03-24T10:51:54.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.495+0000 7f226e7fc640 1 -- 172.21.15.5:0/1520590589 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f22740129d0 con 0x7f22780c4300 2024-03-24T10:51:54.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.496+0000 7f226e7fc640 1 --2- 172.21.15.5:0/1520590589 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f225c06b860 0x7f225c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:54.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.496+0000 7f227d2b9640 1 --2- 172.21.15.5:0/1520590589 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f225c06b860 0x7f225c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:54.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.497+0000 7f227d2b9640 1 --2- 172.21.15.5:0/1520590589 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f225c06b860 0x7f225c06dd20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2268036000 tx=0x7f2268008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:54.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.758+0000 7f226e7fc640 1 -- 172.21.15.5:0/1520590589 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f2274058ed0 con 0x7f22780c4300 2024-03-24T10:51:54.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.758+0000 7f226e7fc640 1 -- 172.21.15.5:0/1520590589 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2274055170 con 0x7f22780c4300 2024-03-24T10:51:54.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:54.999+0000 7f227fd1e640 1 -- 172.21.15.5:0/1520590589 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22780c2e80 con 0x7f22780c4300 2024-03-24T10:51:55.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:55.023+0000 7f226e7fc640 1 -- 172.21.15.5:0/1520590589 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f2274058830 con 0x7f22780c4300 2024-03-24T10:51:55.023 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:55.024 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:55.024 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:55.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:55.026+0000 7f227fd1e640 1 -- 172.21.15.5:0/1520590589 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f225c06b860 msgr2=0x7f225c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:55.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:55.026+0000 7f227fd1e640 1 --2- 172.21.15.5:0/1520590589 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f225c06b860 0x7f225c06dd20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2268036000 tx=0x7f2268008030 comp rx=0 tx=0).stop 2024-03-24T10:51:55.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:55.026+0000 7f227fd1e640 1 -- 172.21.15.5:0/1520590589 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f22780c4300 msgr2=0x7f22780adb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:55.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:55.026+0000 7f227fd1e640 1 --2- 172.21.15.5:0/1520590589 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f22780c4300 0x7f22780adb20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f227400c450 tx=0x7f227400c910 comp rx=0 tx=0).stop 2024-03-24T10:51:55.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:55.026+0000 7f227fd1e640 1 -- 172.21.15.5:0/1520590589 shutdown_connections 2024-03-24T10:51:55.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:55.026+0000 7f227fd1e640 1 --2- 172.21.15.5:0/1520590589 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f225c06b860 0x7f225c06dd20 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:55.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:55.026+0000 7f227fd1e640 1 --2- 172.21.15.5:0/1520590589 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2278154220 0x7f22781e4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:55.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:55.026+0000 7f227fd1e640 1 --2- 172.21.15.5:0/1520590589 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f22780cdfb0 0x7f22781e47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:55.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:55.026+0000 7f227fd1e640 1 --2- 172.21.15.5:0/1520590589 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f22780c4300 0x7f22780adb20 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:55.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:55.026+0000 7f227fd1e640 1 -- 172.21.15.5:0/1520590589 >> 172.21.15.5:0/1520590589 conn(0x7f227806f830 msgr2=0x7f2278151c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:55.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:55.026+0000 7f227fd1e640 1 -- 172.21.15.5:0/1520590589 shutdown_connections 2024-03-24T10:51:55.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:55.026+0000 7f227fd1e640 1 -- 172.21.15.5:0/1520590589 wait complete. 2024-03-24T10:51:55.644 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:55.644 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:56.645 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:56.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.748+0000 7f09ced93640 1 Processor -- start 2024-03-24T10:51:56.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.748+0000 7f09ced93640 1 -- start start 2024-03-24T10:51:56.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.749+0000 7f09ced93640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f09c8154220 0x7f09c80c4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:56.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.749+0000 7f09ced93640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f09c80c84e0 0x7f09c80cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:56.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.749+0000 7f09ced93640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09c80c9050 con 0x7f09c80c8900 2024-03-24T10:51:56.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.749+0000 7f09ced93640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f09c815a5f0 con 0x7f09c80c84e0 2024-03-24T10:51:56.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.749+0000 7f09ced93640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f09c815a8d0 con 0x7f09c8154220 2024-03-24T10:51:56.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.749+0000 7f09c7fff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f09c80c84e0 0x7f09c80cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:56.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.749+0000 7f09ccb2f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f09c8154220 0x7f09c80c4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:56.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.749+0000 7f09cd330640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f09c80c8900 0x7f09c806c910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:46216/0 (socket says 172.21.15.5:46216) 2024-03-24T10:51:56.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.749+0000 7f09cd330640 1 -- 172.21.15.5:0/297309258 learned_addr learned my addr 172.21.15.5:0/297309258 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:51:56.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.752+0000 7f09c7fff640 1 -- 172.21.15.5:0/297309258 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f09c8154220 msgr2=0x7f09c80c4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:56.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.752+0000 7f09c7fff640 1 --2- 172.21.15.5:0/297309258 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f09c8154220 0x7f09c80c4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:56.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.752+0000 7f09c7fff640 1 -- 172.21.15.5:0/297309258 >> v1:172.21.15.57:6789/0 conn(0x7f09c80c8900 legacy=0x7f09c806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:56.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.752+0000 7f09c7fff640 1 -- 172.21.15.5:0/297309258 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c815abb0 con 0x7f09c80c84e0 2024-03-24T10:51:56.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.754+0000 7f09c7fff640 1 --2- 172.21.15.5:0/297309258 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09c80c84e0 0x7f09c80cda70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f09b40097a0 tx=0x7f09b402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=894755b52f7cbdad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:56.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.758+0000 7f09c77fe640 1 -- 172.21.15.5:0/297309258 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4030610 con 0x7f09c80c84e0 2024-03-24T10:51:56.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.758+0000 7f09c77fe640 1 -- 172.21.15.5:0/297309258 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09b4030770 con 0x7f09c80c84e0 2024-03-24T10:51:56.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.758+0000 7f09c77fe640 1 -- 172.21.15.5:0/297309258 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4030b00 con 0x7f09c80c84e0 2024-03-24T10:51:56.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.759+0000 7f09ced93640 1 -- 172.21.15.5:0/297309258 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09c80c84e0 msgr2=0x7f09c80cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:56.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.759+0000 7f09ced93640 1 --2- 172.21.15.5:0/297309258 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09c80c84e0 0x7f09c80cda70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f09b40097a0 tx=0x7f09b402b960 comp rx=0 tx=0).stop 2024-03-24T10:51:56.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.759+0000 7f09ced93640 1 -- 172.21.15.5:0/297309258 shutdown_connections 2024-03-24T10:51:56.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.759+0000 7f09ced93640 1 --2- 172.21.15.5:0/297309258 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09c80c84e0 0x7f09c80cda70 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:56.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.759+0000 7f09ced93640 1 --2- 172.21.15.5:0/297309258 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f09c8154220 0x7f09c80c4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:56.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.759+0000 7f09ced93640 1 -- 172.21.15.5:0/297309258 >> 172.21.15.5:0/297309258 conn(0x7f09c806f830 msgr2=0x7f09c806fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:56.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.759+0000 7f09ced93640 1 -- 172.21.15.5:0/297309258 shutdown_connections 2024-03-24T10:51:56.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.759+0000 7f09ced93640 1 -- 172.21.15.5:0/297309258 wait complete. 2024-03-24T10:51:56.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.760+0000 7f09ced93640 1 Processor -- start 2024-03-24T10:51:56.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.760+0000 7f09ced93640 1 -- start start 2024-03-24T10:51:56.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.761+0000 7f09ced93640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09c80c84e0 0x7f09c80ca890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:56.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.761+0000 7f09ced93640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09c80c8900 0x7f09c81e4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:56.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.761+0000 7f09ced93640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f09c8154220 0x7f09c81f39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:56.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.761+0000 7f09ced93640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f09c80c9e80 con 0x7f09c80c8900 2024-03-24T10:51:56.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.761+0000 7f09ced93640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f09c80cadd0 con 0x7f09c80c84e0 2024-03-24T10:51:56.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.761+0000 7f09ced93640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f09c80ca400 con 0x7f09c8154220 2024-03-24T10:51:56.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.762+0000 7f09c7fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09c80c8900 0x7f09c81e4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:56.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.762+0000 7f09cd330640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f09c8154220 0x7f09c81f39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:56.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.762+0000 7f09c7fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09c80c8900 0x7f09c81e4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38782/0 (socket says 172.21.15.5:38782) 2024-03-24T10:51:56.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.762+0000 7f09c7fff640 1 -- 172.21.15.5:0/2677518563 learned_addr learned my addr 172.21.15.5:0/2677518563 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:56.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.762+0000 7f09ccb2f640 1 --2- 172.21.15.5:0/2677518563 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09c80c84e0 0x7f09c80ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:56.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.766+0000 7f09c7fff640 1 -- 172.21.15.5:0/2677518563 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f09c8154220 msgr2=0x7f09c81f39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:56.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.766+0000 7f09c7fff640 1 --2- 172.21.15.5:0/2677518563 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f09c8154220 0x7f09c81f39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:56.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.766+0000 7f09c7fff640 1 -- 172.21.15.5:0/2677518563 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09c80c84e0 msgr2=0x7f09c80ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:56.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.766+0000 7f09c7fff640 1 --2- 172.21.15.5:0/2677518563 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09c80c84e0 0x7f09c80ca890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:56.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.766+0000 7f09c7fff640 1 -- 172.21.15.5:0/2677518563 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09b4009400 con 0x7f09c80c8900 2024-03-24T10:51:56.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.768+0000 7f09c7fff640 1 --2- 172.21.15.5:0/2677518563 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09c80c8900 0x7f09c81e4ab0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f09b4009890 tx=0x7f09b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:56.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.773+0000 7f09c57fa640 1 -- 172.21.15.5:0/2677518563 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b403e070 con 0x7f09c80c8900 2024-03-24T10:51:56.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.773+0000 7f09c57fa640 1 -- 172.21.15.5:0/2677518563 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09b4030e10 con 0x7f09c80c8900 2024-03-24T10:51:56.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.773+0000 7f09ced93640 1 -- 172.21.15.5:0/2677518563 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c8150420 con 0x7f09c80c8900 2024-03-24T10:51:56.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.773+0000 7f09c57fa640 1 -- 172.21.15.5:0/2677518563 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4039c60 con 0x7f09c80c8900 2024-03-24T10:51:56.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.773+0000 7f09ced93640 1 -- 172.21.15.5:0/2677518563 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c80ca090 con 0x7f09c80c8900 2024-03-24T10:51:56.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:56.774+0000 7f09ced93640 1 -- 172.21.15.5:0/2677518563 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09c814e290 con 0x7f09c80c8900 2024-03-24T10:51:57.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:57.070+0000 7f09c57fa640 1 -- 172.21.15.5:0/2677518563 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f09b40423f0 con 0x7f09c80c8900 2024-03-24T10:51:57.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:57.071+0000 7f09c57fa640 1 --2- 172.21.15.5:0/2677518563 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f09ac06b860 0x7f09ac06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:57.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:57.071+0000 7f09ccb2f640 1 --2- 172.21.15.5:0/2677518563 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f09ac06b860 0x7f09ac06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:57.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:57.071+0000 7f09ccb2f640 1 --2- 172.21.15.5:0/2677518563 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f09ac06b860 0x7f09ac06dd20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f09bc009b40 tx=0x7f09bc009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:57.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:57.083+0000 7f09c57fa640 1 -- 172.21.15.5:0/2677518563 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f09b407ad30 con 0x7f09c80c8900 2024-03-24T10:51:57.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:57.083+0000 7f09c57fa640 1 -- 172.21.15.5:0/2677518563 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f09b4076fd0 con 0x7f09c80c8900 2024-03-24T10:51:57.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:57.302+0000 7f09ced93640 1 -- 172.21.15.5:0/2677518563 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09c80681d0 con 0x7f09c80c8900 2024-03-24T10:51:57.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:57.316+0000 7f09c57fa640 1 -- 172.21.15.5:0/2677518563 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f09b4048090 con 0x7f09c80c8900 2024-03-24T10:51:57.316 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:57.316 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:57.316 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:57.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:57.318+0000 7f09ced93640 1 -- 172.21.15.5:0/2677518563 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f09ac06b860 msgr2=0x7f09ac06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:57.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:57.318+0000 7f09ced93640 1 --2- 172.21.15.5:0/2677518563 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f09ac06b860 0x7f09ac06dd20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f09bc009b40 tx=0x7f09bc009190 comp rx=0 tx=0).stop 2024-03-24T10:51:57.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:57.318+0000 7f09ced93640 1 -- 172.21.15.5:0/2677518563 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09c80c8900 msgr2=0x7f09c81e4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:57.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:57.318+0000 7f09ced93640 1 --2- 172.21.15.5:0/2677518563 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09c80c8900 0x7f09c81e4ab0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f09b4009890 tx=0x7f09b40046c0 comp rx=0 tx=0).stop 2024-03-24T10:51:57.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:57.318+0000 7f09ced93640 1 -- 172.21.15.5:0/2677518563 shutdown_connections 2024-03-24T10:51:57.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:57.318+0000 7f09ced93640 1 --2- 172.21.15.5:0/2677518563 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f09ac06b860 0x7f09ac06dd20 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:57.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:57.318+0000 7f09ced93640 1 --2- 172.21.15.5:0/2677518563 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09c80c8900 0x7f09c81e4ab0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:57.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:57.318+0000 7f09ced93640 1 --2- 172.21.15.5:0/2677518563 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f09c8154220 0x7f09c81f39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:57.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:57.318+0000 7f09ced93640 1 --2- 172.21.15.5:0/2677518563 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09c80c84e0 0x7f09c80ca890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:57.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:57.318+0000 7f09ced93640 1 -- 172.21.15.5:0/2677518563 >> 172.21.15.5:0/2677518563 conn(0x7f09c806f830 msgr2=0x7f09c81592d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:57.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:57.318+0000 7f09ced93640 1 -- 172.21.15.5:0/2677518563 shutdown_connections 2024-03-24T10:51:57.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:57.318+0000 7f09ced93640 1 -- 172.21.15.5:0/2677518563 wait complete. 2024-03-24T10:51:57.932 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:51:57.933 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:51:58.934 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:51:59.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.035+0000 7fdd267d0640 1 Processor -- start 2024-03-24T10:51:59.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.035+0000 7fdd267d0640 1 -- start start 2024-03-24T10:51:59.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.036+0000 7fdd267d0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdd201548e0 0x7fdd2006b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:59.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.036+0000 7fdd267d0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fdd200732e0 0x7fdd2015e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:59.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.036+0000 7fdd267d0640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd200c4d70 con 0x7fdd20154500 2024-03-24T10:51:59.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.036+0000 7fdd267d0640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fdd200ca160 con 0x7fdd201548e0 2024-03-24T10:51:59.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.036+0000 7fdd267d0640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fdd200c9550 con 0x7fdd200732e0 2024-03-24T10:51:59.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.037+0000 7fdd24d6d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fdd200732e0 0x7fdd2015e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:59.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.037+0000 7fdd1f7fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fdd201548e0 0x7fdd2006b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:59.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.037+0000 7fdd1ffff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fdd20154500 0x7fdd200cc8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:50806/0 (socket says 172.21.15.5:50806) 2024-03-24T10:51:59.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.037+0000 7fdd1ffff640 1 -- 172.21.15.5:0/1505886898 learned_addr learned my addr 172.21.15.5:0/1505886898 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:51:59.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.041+0000 7fdd24d6d640 1 -- 172.21.15.5:0/1505886898 >> v2:172.21.15.5:3300/0 conn(0x7fdd201548e0 msgr2=0x7fdd2006b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:51:59.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.041+0000 7fdd24d6d640 1 --2- 172.21.15.5:0/1505886898 >> v2:172.21.15.5:3300/0 conn(0x7fdd201548e0 0x7fdd2006b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:59.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.041+0000 7fdd24d6d640 1 -- 172.21.15.5:0/1505886898 >> v1:172.21.15.5:6789/0 conn(0x7fdd20154500 legacy=0x7fdd200cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:59.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.041+0000 7fdd24d6d640 1 -- 172.21.15.5:0/1505886898 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd200c9830 con 0x7fdd200732e0 2024-03-24T10:51:59.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.043+0000 7fdd24d6d640 1 --2- 172.21.15.5:0/1505886898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd200732e0 0x7fdd2015e2d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdd2006fb50 tx=0x7fdd1802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=871f63999f0cad04 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:59.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.047+0000 7fdd1effd640 1 -- 172.21.15.5:0/1505886898 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1802cc80 con 0x7fdd200732e0 2024-03-24T10:51:59.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.050+0000 7fdd1effd640 1 -- 172.21.15.5:0/1505886898 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd1802cde0 con 0x7fdd200732e0 2024-03-24T10:51:59.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.050+0000 7fdd1effd640 1 -- 172.21.15.5:0/1505886898 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd18034660 con 0x7fdd200732e0 2024-03-24T10:51:59.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.050+0000 7fdd267d0640 1 -- 172.21.15.5:0/1505886898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd200732e0 msgr2=0x7fdd2015e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:59.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.050+0000 7fdd267d0640 1 --2- 172.21.15.5:0/1505886898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd200732e0 0x7fdd2015e2d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdd2006fb50 tx=0x7fdd1802bed0 comp rx=0 tx=0).stop 2024-03-24T10:51:59.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.050+0000 7fdd267d0640 1 -- 172.21.15.5:0/1505886898 shutdown_connections 2024-03-24T10:51:59.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.050+0000 7fdd267d0640 1 --2- 172.21.15.5:0/1505886898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd200732e0 0x7fdd2015e2d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:59.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.050+0000 7fdd267d0640 1 --2- 172.21.15.5:0/1505886898 >> v2:172.21.15.5:3300/0 conn(0x7fdd201548e0 0x7fdd2006b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:59.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.050+0000 7fdd267d0640 1 -- 172.21.15.5:0/1505886898 >> 172.21.15.5:0/1505886898 conn(0x7fdd2006c240 msgr2=0x7fdd2006c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:59.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.050+0000 7fdd267d0640 1 -- 172.21.15.5:0/1505886898 shutdown_connections 2024-03-24T10:51:59.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.050+0000 7fdd267d0640 1 -- 172.21.15.5:0/1505886898 wait complete. 2024-03-24T10:51:59.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.052+0000 7fdd267d0640 1 Processor -- start 2024-03-24T10:51:59.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.052+0000 7fdd267d0640 1 -- start start 2024-03-24T10:51:59.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.052+0000 7fdd267d0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd200732e0 0x7fdd2015adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:59.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.052+0000 7fdd267d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd20154500 0x7fdd201f0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:59.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.052+0000 7fdd267d0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd201548e0 0x7fdd2014f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:59.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.052+0000 7fdd267d0640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2006b8d0 con 0x7fdd201548e0 2024-03-24T10:51:59.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.052+0000 7fdd267d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fdd201f06f0 con 0x7fdd20154500 2024-03-24T10:51:59.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.052+0000 7fdd267d0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fdd20155e60 con 0x7fdd200732e0 2024-03-24T10:51:59.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.053+0000 7fdd24d6d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd201548e0 0x7fdd2014f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:59.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.053+0000 7fdd1f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd20154500 0x7fdd201f0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:59.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.053+0000 7fdd1ffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd200732e0 0x7fdd2015adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:59.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.053+0000 7fdd24d6d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd201548e0 0x7fdd2014f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38808/0 (socket says 172.21.15.5:38808) 2024-03-24T10:51:59.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.053+0000 7fdd24d6d640 1 -- 172.21.15.5:0/94273961 learned_addr learned my addr 172.21.15.5:0/94273961 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:51:59.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.057+0000 7fdd24d6d640 1 -- 172.21.15.5:0/94273961 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd200732e0 msgr2=0x7fdd2015adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:59.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.057+0000 7fdd24d6d640 1 --2- 172.21.15.5:0/94273961 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd200732e0 0x7fdd2015adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:59.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.057+0000 7fdd24d6d640 1 -- 172.21.15.5:0/94273961 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd20154500 msgr2=0x7fdd201f0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:59.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.057+0000 7fdd24d6d640 1 --2- 172.21.15.5:0/94273961 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd20154500 0x7fdd201f0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:59.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.057+0000 7fdd24d6d640 1 -- 172.21.15.5:0/94273961 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd18009a70 con 0x7fdd201548e0 2024-03-24T10:51:59.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.061+0000 7fdd24d6d640 1 --2- 172.21.15.5:0/94273961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd201548e0 0x7fdd2014f140 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdd18009e80 tx=0x7fdd18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:59.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.066+0000 7fdd1cff9640 1 -- 172.21.15.5:0/94273961 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd18040650 con 0x7fdd201548e0 2024-03-24T10:51:59.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.066+0000 7fdd1cff9640 1 -- 172.21.15.5:0/94273961 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd180407d0 con 0x7fdd201548e0 2024-03-24T10:51:59.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.066+0000 7fdd1cff9640 1 -- 172.21.15.5:0/94273961 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd18040b80 con 0x7fdd201548e0 2024-03-24T10:51:59.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.066+0000 7fdd267d0640 1 -- 172.21.15.5:0/94273961 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd20156140 con 0x7fdd201548e0 2024-03-24T10:51:59.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.066+0000 7fdd267d0640 1 -- 172.21.15.5:0/94273961 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd20156700 con 0x7fdd201548e0 2024-03-24T10:51:59.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.067+0000 7fdd267d0640 1 -- 172.21.15.5:0/94273961 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd200c91e0 con 0x7fdd201548e0 2024-03-24T10:51:59.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.099+0000 7fdd1cff9640 1 -- 172.21.15.5:0/94273961 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdd18040ce0 con 0x7fdd201548e0 2024-03-24T10:51:59.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.099+0000 7fdd1cff9640 1 --2- 172.21.15.5:0/94273961 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdd0c06b810 0x7fdd0c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:51:59.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.099+0000 7fdd1ffff640 1 --2- 172.21.15.5:0/94273961 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdd0c06b810 0x7fdd0c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:51:59.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.100+0000 7fdd1ffff640 1 --2- 172.21.15.5:0/94273961 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdd0c06b810 0x7fdd0c06dcd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdd10007fd0 tx=0x7fdd10007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:51:59.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.369+0000 7fdd1cff9640 1 -- 172.21.15.5:0/94273961 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fdd1807bc10 con 0x7fdd201548e0 2024-03-24T10:51:59.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.369+0000 7fdd1cff9640 1 -- 172.21.15.5:0/94273961 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdd18077eb0 con 0x7fdd201548e0 2024-03-24T10:51:59.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.592+0000 7fdd267d0640 1 -- 172.21.15.5:0/94273961 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd200681d0 con 0x7fdd201548e0 2024-03-24T10:51:59.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.610+0000 7fdd1cff9640 1 -- 172.21.15.5:0/94273961 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fdd18039090 con 0x7fdd201548e0 2024-03-24T10:51:59.610 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:51:59.610 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:51:59.610 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:51:59.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.612+0000 7fdd267d0640 1 -- 172.21.15.5:0/94273961 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdd0c06b810 msgr2=0x7fdd0c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:59.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.612+0000 7fdd267d0640 1 --2- 172.21.15.5:0/94273961 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdd0c06b810 0x7fdd0c06dcd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdd10007fd0 tx=0x7fdd10007e80 comp rx=0 tx=0).stop 2024-03-24T10:51:59.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.612+0000 7fdd267d0640 1 -- 172.21.15.5:0/94273961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd201548e0 msgr2=0x7fdd2014f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:51:59.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.612+0000 7fdd267d0640 1 --2- 172.21.15.5:0/94273961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd201548e0 0x7fdd2014f140 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdd18009e80 tx=0x7fdd18004a40 comp rx=0 tx=0).stop 2024-03-24T10:51:59.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.612+0000 7fdd267d0640 1 -- 172.21.15.5:0/94273961 shutdown_connections 2024-03-24T10:51:59.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.612+0000 7fdd267d0640 1 --2- 172.21.15.5:0/94273961 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdd0c06b810 0x7fdd0c06dcd0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:59.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.612+0000 7fdd267d0640 1 --2- 172.21.15.5:0/94273961 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd201548e0 0x7fdd2014f140 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:59.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.612+0000 7fdd267d0640 1 --2- 172.21.15.5:0/94273961 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd20154500 0x7fdd201f0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:59.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.612+0000 7fdd267d0640 1 --2- 172.21.15.5:0/94273961 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd200732e0 0x7fdd2015adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:51:59.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.612+0000 7fdd267d0640 1 -- 172.21.15.5:0/94273961 >> 172.21.15.5:0/94273961 conn(0x7fdd2006c240 msgr2=0x7fdd200c52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:51:59.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.613+0000 7fdd267d0640 1 -- 172.21.15.5:0/94273961 shutdown_connections 2024-03-24T10:51:59.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:51:59.613+0000 7fdd267d0640 1 -- 172.21.15.5:0/94273961 wait complete. 2024-03-24T10:52:00.228 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:00.228 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:01.229 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:01.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.332+0000 7f9241956640 1 Processor -- start 2024-03-24T10:52:01.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.332+0000 7f9241956640 1 -- start start 2024-03-24T10:52:01.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.333+0000 7f9241956640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f923c0cae90 0x7f923c0cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:01.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.333+0000 7f9241956640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f923c0ccba0 con 0x7f923c0cb270 2024-03-24T10:52:01.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.333+0000 7f9241956640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f923c0cce40 con 0x7f923c0cd960 2024-03-24T10:52:01.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.333+0000 7f9241956640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f923c0cc9b0 con 0x7f923c0cae90 2024-03-24T10:52:01.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.334+0000 7f923b7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f923c0cd960 0x7f923c0cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:46232/0 (socket says 172.21.15.5:46232) 2024-03-24T10:52:01.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.334+0000 7f923b7fe640 1 -- 172.21.15.5:0/3729794528 learned_addr learned my addr 172.21.15.5:0/3729794528 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:52:01.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.334+0000 7f923affd640 1 --2- 172.21.15.5:0/3729794528 >> v2:172.21.15.5:3300/0 conn(0x7f923c0cae90 0x7f923c0cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:01.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.337+0000 7f923a7fc640 1 -- 172.21.15.5:0/3729794528 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 757000534 0 0) 0x7f923c0cce40 con 0x7f923c0cd960 2024-03-24T10:52:01.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.337+0000 7f923a7fc640 1 -- 172.21.15.5:0/3729794528 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9228003520 con 0x7f923c0cd960 2024-03-24T10:52:01.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.339+0000 7f923affd640 1 -- 172.21.15.5:0/3729794528 >> v1:172.21.15.57:6789/0 conn(0x7f923c0cd960 legacy=0x7f923c0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:01.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.339+0000 7f923affd640 1 -- 172.21.15.5:0/3729794528 >> v1:172.21.15.5:6789/0 conn(0x7f923c0cb270 legacy=0x7f923c0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:01.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.339+0000 7f923affd640 1 -- 172.21.15.5:0/3729794528 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f923c15a590 con 0x7f923c0cae90 2024-03-24T10:52:01.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.343+0000 7f923affd640 1 --2- 172.21.15.5:0/3729794528 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f923c0cae90 0x7f923c0cf530 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f922c0097a0 tx=0x7f922c02b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f5379e7e584af80 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:01.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.348+0000 7f923a7fc640 1 -- 172.21.15.5:0/3729794528 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c02f810 con 0x7f923c0cae90 2024-03-24T10:52:01.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.348+0000 7f923a7fc640 1 -- 172.21.15.5:0/3729794528 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f922c02f970 con 0x7f923c0cae90 2024-03-24T10:52:01.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.348+0000 7f923a7fc640 1 -- 172.21.15.5:0/3729794528 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c02fd00 con 0x7f923c0cae90 2024-03-24T10:52:01.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.349+0000 7f9241956640 1 -- 172.21.15.5:0/3729794528 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f923c0cae90 msgr2=0x7f923c0cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:01.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.349+0000 7f9241956640 1 --2- 172.21.15.5:0/3729794528 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f923c0cae90 0x7f923c0cf530 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f922c0097a0 tx=0x7f922c02b9f0 comp rx=0 tx=0).stop 2024-03-24T10:52:01.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.349+0000 7f9241956640 1 -- 172.21.15.5:0/3729794528 shutdown_connections 2024-03-24T10:52:01.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.349+0000 7f9241956640 1 --2- 172.21.15.5:0/3729794528 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f923c0cae90 0x7f923c0cf530 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:01.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.349+0000 7f9241956640 1 -- 172.21.15.5:0/3729794528 >> 172.21.15.5:0/3729794528 conn(0x7f923c0cfb80 msgr2=0x7f923c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:01.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.349+0000 7f9241956640 1 -- 172.21.15.5:0/3729794528 shutdown_connections 2024-03-24T10:52:01.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.349+0000 7f9241956640 1 -- 172.21.15.5:0/3729794528 wait complete. 2024-03-24T10:52:01.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.350+0000 7f9241956640 1 Processor -- start 2024-03-24T10:52:01.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.351+0000 7f9241956640 1 -- start start 2024-03-24T10:52:01.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.351+0000 7f9241956640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f923c0cae90 0x7f923c153110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:01.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.351+0000 7f9241956640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f923c0cb270 0x7f923c153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:01.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.351+0000 7f9241956640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f923c0cd960 0x7f923c14c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:01.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.351+0000 7f9241956640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f923c0c3340 con 0x7f923c0cae90 2024-03-24T10:52:01.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.351+0000 7f9241956640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f923c154e80 con 0x7f923c0cd960 2024-03-24T10:52:01.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.351+0000 7f9241956640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f923c1f3430 con 0x7f923c0cb270 2024-03-24T10:52:01.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.352+0000 7f923affd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f923c0cae90 0x7f923c153110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:01.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.352+0000 7f923b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f923c0cd960 0x7f923c14c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:01.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.352+0000 7f9233fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f923c0cb270 0x7f923c153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:01.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.352+0000 7f923affd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f923c0cae90 0x7f923c153110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38828/0 (socket says 172.21.15.5:38828) 2024-03-24T10:52:01.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.352+0000 7f923affd640 1 -- 172.21.15.5:0/3578817350 learned_addr learned my addr 172.21.15.5:0/3578817350 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:01.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.356+0000 7f923affd640 1 -- 172.21.15.5:0/3578817350 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f923c0cb270 msgr2=0x7f923c153650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:01.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.356+0000 7f923affd640 1 --2- 172.21.15.5:0/3578817350 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f923c0cb270 0x7f923c153650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:01.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.356+0000 7f923affd640 1 -- 172.21.15.5:0/3578817350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f923c0cd960 msgr2=0x7f923c14c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:01.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.356+0000 7f923affd640 1 --2- 172.21.15.5:0/3578817350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f923c0cd960 0x7f923c14c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:01.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.356+0000 7f923affd640 1 -- 172.21.15.5:0/3578817350 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f922c009400 con 0x7f923c0cae90 2024-03-24T10:52:01.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.359+0000 7f923affd640 1 --2- 172.21.15.5:0/3578817350 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f923c0cae90 0x7f923c153110 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f922c02c000 tx=0x7f922c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:01.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.364+0000 7f9238ff9640 1 -- 172.21.15.5:0/3578817350 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c03c040 con 0x7f923c0cae90 2024-03-24T10:52:01.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.364+0000 7f9238ff9640 1 -- 172.21.15.5:0/3578817350 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f922c0378f0 con 0x7f923c0cae90 2024-03-24T10:52:01.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.364+0000 7f9241956640 1 -- 172.21.15.5:0/3578817350 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f923c1f28b0 con 0x7f923c0cae90 2024-03-24T10:52:01.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.364+0000 7f9238ff9640 1 -- 172.21.15.5:0/3578817350 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922c037ca0 con 0x7f923c0cae90 2024-03-24T10:52:01.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.364+0000 7f9241956640 1 -- 172.21.15.5:0/3578817350 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f923c1f2e70 con 0x7f923c0cae90 2024-03-24T10:52:01.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.365+0000 7f9241956640 1 -- 172.21.15.5:0/3578817350 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f923c0ca1d0 con 0x7f923c0cae90 2024-03-24T10:52:01.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.402+0000 7f9238ff9640 1 -- 172.21.15.5:0/3578817350 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f922c04b3b0 con 0x7f923c0cae90 2024-03-24T10:52:01.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.402+0000 7f9238ff9640 1 --2- 172.21.15.5:0/3578817350 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f921c06b860 0x7f921c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:01.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.403+0000 7f9233fff640 1 --2- 172.21.15.5:0/3578817350 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f921c06b860 0x7f921c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:01.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.403+0000 7f9233fff640 1 --2- 172.21.15.5:0/3578817350 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f921c06b860 0x7f921c06dd20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f923c0c97b0 tx=0x7f9224009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:01.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.664+0000 7f9238ff9640 1 -- 172.21.15.5:0/3578817350 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f922c07af10 con 0x7f923c0cae90 2024-03-24T10:52:01.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.664+0000 7f9238ff9640 1 -- 172.21.15.5:0/3578817350 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f922c077210 con 0x7f923c0cae90 2024-03-24T10:52:01.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.966+0000 7f9241956640 1 -- 172.21.15.5:0/3578817350 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f923c0681d0 con 0x7f923c0cae90 2024-03-24T10:52:01.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.981+0000 7f9238ff9640 1 -- 172.21.15.5:0/3578817350 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f922c047020 con 0x7f923c0cae90 2024-03-24T10:52:01.982 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:01.982 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:01.982 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:01.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.984+0000 7f9241956640 1 -- 172.21.15.5:0/3578817350 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f921c06b860 msgr2=0x7f921c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:01.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.984+0000 7f9241956640 1 --2- 172.21.15.5:0/3578817350 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f921c06b860 0x7f921c06dd20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f923c0c97b0 tx=0x7f9224009040 comp rx=0 tx=0).stop 2024-03-24T10:52:01.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.984+0000 7f9241956640 1 -- 172.21.15.5:0/3578817350 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f923c0cae90 msgr2=0x7f923c153110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:01.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.984+0000 7f9241956640 1 --2- 172.21.15.5:0/3578817350 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f923c0cae90 0x7f923c153110 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f922c02c000 tx=0x7f922c004730 comp rx=0 tx=0).stop 2024-03-24T10:52:01.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.984+0000 7f9241956640 1 -- 172.21.15.5:0/3578817350 shutdown_connections 2024-03-24T10:52:01.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.984+0000 7f9241956640 1 --2- 172.21.15.5:0/3578817350 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f921c06b860 0x7f921c06dd20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:01.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.984+0000 7f9241956640 1 --2- 172.21.15.5:0/3578817350 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f923c0cd960 0x7f923c14c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:01.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.984+0000 7f9241956640 1 --2- 172.21.15.5:0/3578817350 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f923c0cb270 0x7f923c153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:01.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.984+0000 7f9241956640 1 --2- 172.21.15.5:0/3578817350 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f923c0cae90 0x7f923c153110 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:01.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.984+0000 7f9241956640 1 -- 172.21.15.5:0/3578817350 >> 172.21.15.5:0/3578817350 conn(0x7f923c0cfb80 msgr2=0x7f923c0c8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:01.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.984+0000 7f9241956640 1 -- 172.21.15.5:0/3578817350 shutdown_connections 2024-03-24T10:52:01.985 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:01.984+0000 7f9241956640 1 -- 172.21.15.5:0/3578817350 wait complete. 2024-03-24T10:52:02.061 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:52:02.063 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:52:02.599 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:02.599 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:03.601 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:03.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.705+0000 7f42f3b4f640 1 Processor -- start 2024-03-24T10:52:03.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.705+0000 7f42f3b4f640 1 -- start start 2024-03-24T10:52:03.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.706+0000 7f42f3b4f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f42ec152140 0x7f42ec15bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:03.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.706+0000 7f42f3b4f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f42ec15c1e0 0x7f42ec1628c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:03.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.706+0000 7f42f3b4f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42ec15b530 con 0x7f42ec06fc60 2024-03-24T10:52:03.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.706+0000 7f42f3b4f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f42ec15b810 con 0x7f42ec15c1e0 2024-03-24T10:52:03.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.706+0000 7f42f3b4f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f42ec159430 con 0x7f42ec152140 2024-03-24T10:52:03.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.707+0000 7f42f18eb640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f42ec152140 0x7f42ec15bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:03.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.707+0000 7f42f10ea640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f42ec06fc60 0x7f42ec0c4e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:58802/0 (socket says 172.21.15.5:58802) 2024-03-24T10:52:03.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.707+0000 7f42f10ea640 1 -- 172.21.15.5:0/3263201694 learned_addr learned my addr 172.21.15.5:0/3263201694 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:52:03.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.707+0000 7f42f20ec640 1 --2- 172.21.15.5:0/3263201694 >> v2:172.21.15.57:3300/0 conn(0x7f42ec15c1e0 0x7f42ec1628c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:03.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.711+0000 7f42f18eb640 1 -- 172.21.15.5:0/3263201694 >> v2:172.21.15.57:3300/0 conn(0x7f42ec15c1e0 msgr2=0x7f42ec1628c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:03.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.711+0000 7f42f18eb640 1 --2- 172.21.15.5:0/3263201694 >> v2:172.21.15.57:3300/0 conn(0x7f42ec15c1e0 0x7f42ec1628c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:03.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.711+0000 7f42f18eb640 1 -- 172.21.15.5:0/3263201694 >> v1:172.21.15.57:6789/0 conn(0x7f42ec06fc60 legacy=0x7f42ec0c4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:03.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.711+0000 7f42f18eb640 1 -- 172.21.15.5:0/3263201694 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42ec1584c0 con 0x7f42ec152140 2024-03-24T10:52:03.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.716+0000 7f42f18eb640 1 --2- 172.21.15.5:0/3263201694 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f42ec152140 0x7f42ec15bca0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f42e80097a0 tx=0x7f42e802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4d13893b4708b63 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:03.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.721+0000 7f42f08e9640 1 -- 172.21.15.5:0/3263201694 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e8030610 con 0x7f42ec152140 2024-03-24T10:52:03.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.721+0000 7f42f08e9640 1 -- 172.21.15.5:0/3263201694 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42e8030770 con 0x7f42ec152140 2024-03-24T10:52:03.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.721+0000 7f42f08e9640 1 -- 172.21.15.5:0/3263201694 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e8030b00 con 0x7f42ec152140 2024-03-24T10:52:03.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.721+0000 7f42f3b4f640 1 -- 172.21.15.5:0/3263201694 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f42ec152140 msgr2=0x7f42ec15bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:03.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.721+0000 7f42f3b4f640 1 --2- 172.21.15.5:0/3263201694 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f42ec152140 0x7f42ec15bca0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f42e80097a0 tx=0x7f42e802b960 comp rx=0 tx=0).stop 2024-03-24T10:52:03.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.721+0000 7f42f3b4f640 1 -- 172.21.15.5:0/3263201694 shutdown_connections 2024-03-24T10:52:03.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.721+0000 7f42f3b4f640 1 --2- 172.21.15.5:0/3263201694 >> v2:172.21.15.57:3300/0 conn(0x7f42ec15c1e0 0x7f42ec1628c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:03.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.721+0000 7f42f3b4f640 1 --2- 172.21.15.5:0/3263201694 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f42ec152140 0x7f42ec15bca0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:03.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.721+0000 7f42f3b4f640 1 -- 172.21.15.5:0/3263201694 >> 172.21.15.5:0/3263201694 conn(0x7f42ec06b130 msgr2=0x7f42ec06b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:03.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.722+0000 7f42f3b4f640 1 -- 172.21.15.5:0/3263201694 shutdown_connections 2024-03-24T10:52:03.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.722+0000 7f42f3b4f640 1 -- 172.21.15.5:0/3263201694 wait complete. 2024-03-24T10:52:03.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.723+0000 7f42f3b4f640 1 Processor -- start 2024-03-24T10:52:03.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.723+0000 7f42f3b4f640 1 -- start start 2024-03-24T10:52:03.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.723+0000 7f42f3b4f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42ec06fc60 0x7f42ec156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:03.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.723+0000 7f42f3b4f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f42ec152140 0x7f42ec1f0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:03.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.723+0000 7f42f3b4f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42ec15c1e0 0x7f42ec1f0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:03.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.723+0000 7f42f3b4f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f42ec1efc70 con 0x7f42ec06fc60 2024-03-24T10:52:03.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.723+0000 7f42f3b4f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f42ec14fb30 con 0x7f42ec15c1e0 2024-03-24T10:52:03.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.723+0000 7f42f3b4f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f42ec14fdd0 con 0x7f42ec152140 2024-03-24T10:52:03.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.724+0000 7f42f20ec640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42ec15c1e0 0x7f42ec1f0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:03.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.724+0000 7f42f10ea640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f42ec152140 0x7f42ec1f0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:03.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.724+0000 7f42f18eb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42ec06fc60 0x7f42ec156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:03.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.724+0000 7f42f20ec640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42ec15c1e0 0x7f42ec1f0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:50624/0 (socket says 172.21.15.5:50624) 2024-03-24T10:52:03.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.724+0000 7f42f20ec640 1 -- 172.21.15.5:0/2395372538 learned_addr learned my addr 172.21.15.5:0/2395372538 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:03.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.728+0000 7f42f20ec640 1 -- 172.21.15.5:0/2395372538 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f42ec152140 msgr2=0x7f42ec1f0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:03.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.728+0000 7f42f20ec640 1 --2- 172.21.15.5:0/2395372538 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f42ec152140 0x7f42ec1f0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:03.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.728+0000 7f42f20ec640 1 -- 172.21.15.5:0/2395372538 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42ec06fc60 msgr2=0x7f42ec156f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:03.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.728+0000 7f42f20ec640 1 --2- 172.21.15.5:0/2395372538 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42ec06fc60 0x7f42ec156f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:03.728 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.728+0000 7f42f20ec640 1 -- 172.21.15.5:0/2395372538 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e8009400 con 0x7f42ec15c1e0 2024-03-24T10:52:03.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.731+0000 7f42f20ec640 1 --2- 172.21.15.5:0/2395372538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42ec15c1e0 0x7f42ec1f0640 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f42e400efc0 tx=0x7f42e400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:03.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.735+0000 7f42e27fc640 1 -- 172.21.15.5:0/2395372538 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e4009ca0 con 0x7f42ec15c1e0 2024-03-24T10:52:03.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.735+0000 7f42e27fc640 1 -- 172.21.15.5:0/2395372538 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42e400f040 con 0x7f42ec15c1e0 2024-03-24T10:52:03.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.735+0000 7f42e27fc640 1 -- 172.21.15.5:0/2395372538 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e40146c0 con 0x7f42ec15c1e0 2024-03-24T10:52:03.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.735+0000 7f42f3b4f640 1 -- 172.21.15.5:0/2395372538 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42ec14b430 con 0x7f42ec15c1e0 2024-03-24T10:52:03.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.735+0000 7f42f3b4f640 1 -- 172.21.15.5:0/2395372538 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42ec14b9f0 con 0x7f42ec15c1e0 2024-03-24T10:52:03.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.737+0000 7f42f3b4f640 1 -- 172.21.15.5:0/2395372538 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42ec062a00 con 0x7f42ec15c1e0 2024-03-24T10:52:03.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.769+0000 7f42e27fc640 1 -- 172.21.15.5:0/2395372538 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f42e4014820 con 0x7f42ec15c1e0 2024-03-24T10:52:03.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.769+0000 7f42e27fc640 1 --2- 172.21.15.5:0/2395372538 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f42d006b890 0x7f42d006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:03.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.769+0000 7f42f18eb640 1 --2- 172.21.15.5:0/2395372538 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f42d006b890 0x7f42d006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:03.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:03.770+0000 7f42f18eb640 1 --2- 172.21.15.5:0/2395372538 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f42d006b890 0x7f42d006dd50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f42e8036000 tx=0x7f42e803b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:04.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:04.025+0000 7f42e27fc640 1 -- 172.21.15.5:0/2395372538 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f42e4059d90 con 0x7f42ec15c1e0 2024-03-24T10:52:04.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:04.045+0000 7f42e27fc640 1 -- 172.21.15.5:0/2395372538 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f42e4056030 con 0x7f42ec15c1e0 2024-03-24T10:52:04.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:04.293+0000 7f42f3b4f640 1 -- 172.21.15.5:0/2395372538 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42ec1efe20 con 0x7f42ec15c1e0 2024-03-24T10:52:04.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:04.316+0000 7f42e27fc640 1 -- 172.21.15.5:0/2395372538 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f42ec1efe20 con 0x7f42ec15c1e0 2024-03-24T10:52:04.316 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:04.316 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:04.316 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:04.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:04.318+0000 7f42f3b4f640 1 -- 172.21.15.5:0/2395372538 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f42d006b890 msgr2=0x7f42d006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:04.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:04.318+0000 7f42f3b4f640 1 --2- 172.21.15.5:0/2395372538 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f42d006b890 0x7f42d006dd50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f42e8036000 tx=0x7f42e803b040 comp rx=0 tx=0).stop 2024-03-24T10:52:04.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:04.318+0000 7f42f3b4f640 1 -- 172.21.15.5:0/2395372538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42ec15c1e0 msgr2=0x7f42ec1f0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:04.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:04.318+0000 7f42f3b4f640 1 --2- 172.21.15.5:0/2395372538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42ec15c1e0 0x7f42ec1f0640 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f42e400efc0 tx=0x7f42e400c490 comp rx=0 tx=0).stop 2024-03-24T10:52:04.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:04.318+0000 7f42f3b4f640 1 -- 172.21.15.5:0/2395372538 shutdown_connections 2024-03-24T10:52:04.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:04.318+0000 7f42f3b4f640 1 --2- 172.21.15.5:0/2395372538 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f42d006b890 0x7f42d006dd50 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:04.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:04.318+0000 7f42f3b4f640 1 --2- 172.21.15.5:0/2395372538 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f42ec15c1e0 0x7f42ec1f0640 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:04.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:04.318+0000 7f42f3b4f640 1 --2- 172.21.15.5:0/2395372538 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f42ec152140 0x7f42ec1f0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:04.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:04.318+0000 7f42f3b4f640 1 --2- 172.21.15.5:0/2395372538 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f42ec06fc60 0x7f42ec156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:04.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:04.318+0000 7f42f3b4f640 1 -- 172.21.15.5:0/2395372538 >> 172.21.15.5:0/2395372538 conn(0x7f42ec06b130 msgr2=0x7f42ec0c5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:04.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:04.318+0000 7f42f3b4f640 1 -- 172.21.15.5:0/2395372538 shutdown_connections 2024-03-24T10:52:04.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:04.318+0000 7f42f3b4f640 1 -- 172.21.15.5:0/2395372538 wait complete. 2024-03-24T10:52:04.932 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:04.933 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:05.933 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:06.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.036+0000 7f4e54d1e640 1 Processor -- start 2024-03-24T10:52:06.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.037+0000 7f4e54d1e640 1 -- start start 2024-03-24T10:52:06.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.037+0000 7f4e54d1e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e501543a0 0x7f4e5015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:06.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.037+0000 7f4e54d1e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e500cb2f0 con 0x7f4e5006d4f0 2024-03-24T10:52:06.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.037+0000 7f4e54d1e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e500ca7e0 con 0x7f4e5006bf50 2024-03-24T10:52:06.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.037+0000 7f4e54d1e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4e500c6750 con 0x7f4e501543a0 2024-03-24T10:52:06.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.038+0000 7f4e47fff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f4e5006d4f0 0x7f4e500cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:58086/0 (socket says 172.21.15.5:58086) 2024-03-24T10:52:06.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.038+0000 7f4e47fff640 1 -- 172.21.15.5:0/4193921041 learned_addr learned my addr 172.21.15.5:0/4193921041 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:52:06.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.038+0000 7f4e4e59c640 1 --2- 172.21.15.5:0/4193921041 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e501543a0 0x7f4e5015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:06.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.041+0000 7f4e4dd9b640 1 -- 172.21.15.5:0/4193921041 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 679529477 0 0) 0x7f4e500cb2f0 con 0x7f4e5006d4f0 2024-03-24T10:52:06.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.041+0000 7f4e4dd9b640 1 -- 172.21.15.5:0/4193921041 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e3c003520 con 0x7f4e5006d4f0 2024-03-24T10:52:06.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.041+0000 7f4e4dd9b640 1 -- 172.21.15.5:0/4193921041 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1786022826 0 0) 0x7f4e500ca7e0 con 0x7f4e5006bf50 2024-03-24T10:52:06.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.041+0000 7f4e4dd9b640 1 -- 172.21.15.5:0/4193921041 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e500cb2f0 con 0x7f4e5006bf50 2024-03-24T10:52:06.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.042+0000 7f4e4e59c640 1 -- 172.21.15.5:0/4193921041 >> v1:172.21.15.57:6789/0 conn(0x7f4e5006bf50 legacy=0x7f4e5006c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:06.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.042+0000 7f4e4e59c640 1 -- 172.21.15.5:0/4193921041 >> v1:172.21.15.5:6789/0 conn(0x7f4e5006d4f0 legacy=0x7f4e500cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:06.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.042+0000 7f4e4e59c640 1 -- 172.21.15.5:0/4193921041 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e500cb520 con 0x7f4e501543a0 2024-03-24T10:52:06.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.045+0000 7f4e4e59c640 1 --2- 172.21.15.5:0/4193921041 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e501543a0 0x7f4e5015ac00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4e400097d0 tx=0x7f4e4002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7aeb03cacae1c002 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:06.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.050+0000 7f4e4dd9b640 1 -- 172.21.15.5:0/4193921041 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e40030880 con 0x7f4e501543a0 2024-03-24T10:52:06.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.050+0000 7f4e4dd9b640 1 -- 172.21.15.5:0/4193921041 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e400309e0 con 0x7f4e501543a0 2024-03-24T10:52:06.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.050+0000 7f4e4dd9b640 1 -- 172.21.15.5:0/4193921041 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e40030d70 con 0x7f4e501543a0 2024-03-24T10:52:06.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.051+0000 7f4e54d1e640 1 -- 172.21.15.5:0/4193921041 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e501543a0 msgr2=0x7f4e5015ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:06.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.051+0000 7f4e54d1e640 1 --2- 172.21.15.5:0/4193921041 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e501543a0 0x7f4e5015ac00 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4e400097d0 tx=0x7f4e4002b9c0 comp rx=0 tx=0).stop 2024-03-24T10:52:06.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.051+0000 7f4e54d1e640 1 -- 172.21.15.5:0/4193921041 shutdown_connections 2024-03-24T10:52:06.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.051+0000 7f4e54d1e640 1 --2- 172.21.15.5:0/4193921041 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e501543a0 0x7f4e5015ac00 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:06.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.051+0000 7f4e54d1e640 1 -- 172.21.15.5:0/4193921041 >> 172.21.15.5:0/4193921041 conn(0x7f4e50072ef0 msgr2=0x7f4e50061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:06.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.051+0000 7f4e54d1e640 1 -- 172.21.15.5:0/4193921041 shutdown_connections 2024-03-24T10:52:06.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.051+0000 7f4e54d1e640 1 -- 172.21.15.5:0/4193921041 wait complete. 2024-03-24T10:52:06.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.052+0000 7f4e54d1e640 1 Processor -- start 2024-03-24T10:52:06.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.052+0000 7f4e54d1e640 1 -- start start 2024-03-24T10:52:06.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.053+0000 7f4e54d1e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e5006bf50 0x7f4e50157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:06.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.053+0000 7f4e54d1e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e5006d4f0 0x7f4e5015b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:06.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.053+0000 7f4e54d1e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e501543a0 0x7f4e5015b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:06.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.053+0000 7f4e54d1e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5015ce40 con 0x7f4e5006d4f0 2024-03-24T10:52:06.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.053+0000 7f4e54d1e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4e50157f60 con 0x7f4e501543a0 2024-03-24T10:52:06.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.053+0000 7f4e54d1e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4e5015dee0 con 0x7f4e5006bf50 2024-03-24T10:52:06.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.053+0000 7f4e47fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e5006d4f0 0x7f4e5015b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:06.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.054+0000 7f4e4ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e501543a0 0x7f4e5015b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:06.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.054+0000 7f4e4e59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e5006bf50 0x7f4e50157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:06.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.054+0000 7f4e47fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e5006d4f0 0x7f4e5015b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50720/0 (socket says 172.21.15.5:50720) 2024-03-24T10:52:06.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.054+0000 7f4e47fff640 1 -- 172.21.15.5:0/2033302640 learned_addr learned my addr 172.21.15.5:0/2033302640 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:06.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.057+0000 7f4e47fff640 1 -- 172.21.15.5:0/2033302640 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e5006bf50 msgr2=0x7f4e50157380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:06.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.057+0000 7f4e47fff640 1 --2- 172.21.15.5:0/2033302640 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e5006bf50 0x7f4e50157380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:06.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.057+0000 7f4e47fff640 1 -- 172.21.15.5:0/2033302640 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e501543a0 msgr2=0x7f4e5015b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:06.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.057+0000 7f4e47fff640 1 --2- 172.21.15.5:0/2033302640 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e501543a0 0x7f4e5015b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:06.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.057+0000 7f4e47fff640 1 -- 172.21.15.5:0/2033302640 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e40009400 con 0x7f4e5006d4f0 2024-03-24T10:52:06.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.060+0000 7f4e47fff640 1 --2- 172.21.15.5:0/2033302640 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e5006d4f0 0x7f4e5015b2b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4e3800c500 tx=0x7f4e3800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:06.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.064+0000 7f4e46ffd640 1 -- 172.21.15.5:0/2033302640 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e38005ce0 con 0x7f4e5006d4f0 2024-03-24T10:52:06.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.064+0000 7f4e46ffd640 1 -- 172.21.15.5:0/2033302640 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e38005e60 con 0x7f4e5006d4f0 2024-03-24T10:52:06.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.064+0000 7f4e54d1e640 1 -- 172.21.15.5:0/2033302640 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e5015e1a0 con 0x7f4e5006d4f0 2024-03-24T10:52:06.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.064+0000 7f4e46ffd640 1 -- 172.21.15.5:0/2033302640 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e38007330 con 0x7f4e5006d4f0 2024-03-24T10:52:06.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.065+0000 7f4e54d1e640 1 -- 172.21.15.5:0/2033302640 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e501584e0 con 0x7f4e5006d4f0 2024-03-24T10:52:06.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.066+0000 7f4e54d1e640 1 -- 172.21.15.5:0/2033302640 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e5015e5a0 con 0x7f4e5006d4f0 2024-03-24T10:52:06.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.096+0000 7f4e46ffd640 1 -- 172.21.15.5:0/2033302640 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4e38007550 con 0x7f4e5006d4f0 2024-03-24T10:52:06.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.096+0000 7f4e46ffd640 1 --2- 172.21.15.5:0/2033302640 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e3006b860 0x7f4e3006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:06.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.097+0000 7f4e4e59c640 1 --2- 172.21.15.5:0/2033302640 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e3006b860 0x7f4e3006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:06.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.097+0000 7f4e4e59c640 1 --2- 172.21.15.5:0/2033302640 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e3006b860 0x7f4e3006dd20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4e40036000 tx=0x7f4e40008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:06.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.100+0000 7f4e46ffd640 1 -- 172.21.15.5:0/2033302640 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f4e38058530 con 0x7f4e5006d4f0 2024-03-24T10:52:06.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.356+0000 7f4e46ffd640 1 -- 172.21.15.5:0/2033302640 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4e380b5e10 con 0x7f4e5006d4f0 2024-03-24T10:52:06.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.582+0000 7f4e54d1e640 1 -- 172.21.15.5:0/2033302640 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e500681d0 con 0x7f4e5006d4f0 2024-03-24T10:52:06.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.596+0000 7f4e46ffd640 1 -- 172.21.15.5:0/2033302640 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f4e38058300 con 0x7f4e5006d4f0 2024-03-24T10:52:06.596 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:06.596 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:06.596 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:06.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.598+0000 7f4e54d1e640 1 -- 172.21.15.5:0/2033302640 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e3006b860 msgr2=0x7f4e3006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:06.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.598+0000 7f4e54d1e640 1 --2- 172.21.15.5:0/2033302640 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e3006b860 0x7f4e3006dd20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4e40036000 tx=0x7f4e40008030 comp rx=0 tx=0).stop 2024-03-24T10:52:06.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.598+0000 7f4e54d1e640 1 -- 172.21.15.5:0/2033302640 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e5006d4f0 msgr2=0x7f4e5015b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:06.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.598+0000 7f4e54d1e640 1 --2- 172.21.15.5:0/2033302640 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e5006d4f0 0x7f4e5015b2b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4e3800c500 tx=0x7f4e3800c9c0 comp rx=0 tx=0).stop 2024-03-24T10:52:06.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.599+0000 7f4e54d1e640 1 -- 172.21.15.5:0/2033302640 shutdown_connections 2024-03-24T10:52:06.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.599+0000 7f4e54d1e640 1 --2- 172.21.15.5:0/2033302640 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e3006b860 0x7f4e3006dd20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:06.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.599+0000 7f4e54d1e640 1 --2- 172.21.15.5:0/2033302640 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e5006d4f0 0x7f4e5015b2b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:06.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.599+0000 7f4e54d1e640 1 --2- 172.21.15.5:0/2033302640 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e501543a0 0x7f4e5015b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:06.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.599+0000 7f4e54d1e640 1 --2- 172.21.15.5:0/2033302640 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e5006bf50 0x7f4e50157380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:06.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.599+0000 7f4e54d1e640 1 -- 172.21.15.5:0/2033302640 >> 172.21.15.5:0/2033302640 conn(0x7f4e50072ef0 msgr2=0x7f4e501533d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:06.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.599+0000 7f4e54d1e640 1 -- 172.21.15.5:0/2033302640 shutdown_connections 2024-03-24T10:52:06.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:06.599+0000 7f4e54d1e640 1 -- 172.21.15.5:0/2033302640 wait complete. 2024-03-24T10:52:07.214 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:07.214 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:08.215 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:08.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.318+0000 7f56fbc88640 1 Processor -- start 2024-03-24T10:52:08.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.318+0000 7f56fbc88640 1 -- start start 2024-03-24T10:52:08.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.318+0000 7f56fbc88640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56f41548e0 0x7f56f406b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:08.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.319+0000 7f56fbc88640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f40c4d70 con 0x7f56f40732e0 2024-03-24T10:52:08.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.319+0000 7f56fbc88640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f40ca160 con 0x7f56f4154500 2024-03-24T10:52:08.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.319+0000 7f56fbc88640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f56f40c9550 con 0x7f56f41548e0 2024-03-24T10:52:08.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.319+0000 7f56fa225640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f56f40732e0 0x7f56f40c52b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:58102/0 (socket says 172.21.15.5:58102) 2024-03-24T10:52:08.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.319+0000 7f56fa225640 1 -- 172.21.15.5:0/2550016904 learned_addr learned my addr 172.21.15.5:0/2550016904 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:52:08.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.319+0000 7f56f9a24640 1 --1- 172.21.15.5:0/2550016904 >> v1:172.21.15.57:6789/0 conn(0x7f56f4154500 0x7f56f40cc8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:58822/0 (socket says 172.21.15.5:58822) 2024-03-24T10:52:08.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.319+0000 7f56f9223640 1 --2- 172.21.15.5:0/2550016904 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56f41548e0 0x7f56f406b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:08.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.322+0000 7f56f8a22640 1 -- 172.21.15.5:0/2550016904 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2871493728 0 0) 0x7f56f40c4d70 con 0x7f56f40732e0 2024-03-24T10:52:08.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.322+0000 7f56f8a22640 1 -- 172.21.15.5:0/2550016904 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56e0003520 con 0x7f56f40732e0 2024-03-24T10:52:08.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.322+0000 7f56f8a22640 1 -- 172.21.15.5:0/2550016904 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4013141694 0 0) 0x7f56f40ca160 con 0x7f56f4154500 2024-03-24T10:52:08.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.322+0000 7f56f8a22640 1 -- 172.21.15.5:0/2550016904 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56f40c4d70 con 0x7f56f4154500 2024-03-24T10:52:08.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.324+0000 7f56f9223640 1 -- 172.21.15.5:0/2550016904 >> v1:172.21.15.57:6789/0 conn(0x7f56f4154500 legacy=0x7f56f40cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:08.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.324+0000 7f56f9223640 1 -- 172.21.15.5:0/2550016904 >> v1:172.21.15.5:6789/0 conn(0x7f56f40732e0 legacy=0x7f56f40c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:08.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.324+0000 7f56f9223640 1 -- 172.21.15.5:0/2550016904 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f40c9830 con 0x7f56f41548e0 2024-03-24T10:52:08.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.327+0000 7f56f9223640 1 --2- 172.21.15.5:0/2550016904 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56f41548e0 0x7f56f406b8c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f56e40097d0 tx=0x7f56e402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c20e7c9a81f66bb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:08.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.332+0000 7f56f8a22640 1 -- 172.21.15.5:0/2550016904 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e4031890 con 0x7f56f41548e0 2024-03-24T10:52:08.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.332+0000 7f56f8a22640 1 -- 172.21.15.5:0/2550016904 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56e40319f0 con 0x7f56f41548e0 2024-03-24T10:52:08.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.332+0000 7f56f8a22640 1 -- 172.21.15.5:0/2550016904 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56e4031d80 con 0x7f56f41548e0 2024-03-24T10:52:08.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.333+0000 7f56fbc88640 1 -- 172.21.15.5:0/2550016904 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56f41548e0 msgr2=0x7f56f406b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:08.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.333+0000 7f56fbc88640 1 --2- 172.21.15.5:0/2550016904 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56f41548e0 0x7f56f406b8c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f56e40097d0 tx=0x7f56e402b930 comp rx=0 tx=0).stop 2024-03-24T10:52:08.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.333+0000 7f56fbc88640 1 -- 172.21.15.5:0/2550016904 shutdown_connections 2024-03-24T10:52:08.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.333+0000 7f56fbc88640 1 --2- 172.21.15.5:0/2550016904 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56f41548e0 0x7f56f406b8c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:08.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.333+0000 7f56fbc88640 1 -- 172.21.15.5:0/2550016904 >> 172.21.15.5:0/2550016904 conn(0x7f56f406c240 msgr2=0x7f56f406c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:08.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.333+0000 7f56fbc88640 1 -- 172.21.15.5:0/2550016904 shutdown_connections 2024-03-24T10:52:08.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.333+0000 7f56fbc88640 1 -- 172.21.15.5:0/2550016904 wait complete. 2024-03-24T10:52:08.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.334+0000 7f56fbc88640 1 Processor -- start 2024-03-24T10:52:08.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.334+0000 7f56fbc88640 1 -- start start 2024-03-24T10:52:08.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.335+0000 7f56fbc88640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56f40732e0 0x7f56f4154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:08.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.335+0000 7f56fbc88640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56f4154500 0x7f56f41f0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:08.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.335+0000 7f56fbc88640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56f41548e0 0x7f56f41f1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:08.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.335+0000 7f56fbc88640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f56f4155350 con 0x7f56f41548e0 2024-03-24T10:52:08.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.335+0000 7f56fbc88640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f56f4156820 con 0x7f56f40732e0 2024-03-24T10:52:08.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.335+0000 7f56fbc88640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f56f41f11e0 con 0x7f56f4154500 2024-03-24T10:52:08.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.335+0000 7f56f9a24640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56f40732e0 0x7f56f4154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:08.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.335+0000 7f56fa225640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56f41548e0 0x7f56f41f1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:08.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.335+0000 7f56f9223640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56f4154500 0x7f56f41f0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:08.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.335+0000 7f56f9a24640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56f40732e0 0x7f56f4154cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:50642/0 (socket says 172.21.15.5:50642) 2024-03-24T10:52:08.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.335+0000 7f56f9a24640 1 -- 172.21.15.5:0/2325478149 learned_addr learned my addr 172.21.15.5:0/2325478149 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:08.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.339+0000 7f56fa225640 1 -- 172.21.15.5:0/2325478149 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56f4154500 msgr2=0x7f56f41f0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:08.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.339+0000 7f56fa225640 1 --2- 172.21.15.5:0/2325478149 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56f4154500 0x7f56f41f0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:08.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.339+0000 7f56fa225640 1 -- 172.21.15.5:0/2325478149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56f40732e0 msgr2=0x7f56f4154cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:08.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.339+0000 7f56fa225640 1 --2- 172.21.15.5:0/2325478149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56f40732e0 0x7f56f4154cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:08.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.339+0000 7f56fa225640 1 -- 172.21.15.5:0/2325478149 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56e4009400 con 0x7f56f41548e0 2024-03-24T10:52:08.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.339+0000 7f56f9a24640 1 --2- 172.21.15.5:0/2325478149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56f40732e0 0x7f56f4154cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:52:08.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.342+0000 7f56fa225640 1 --2- 172.21.15.5:0/2325478149 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56f41548e0 0x7f56f41f1690 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f56f4073750 tx=0x7f56ec00cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:08.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.346+0000 7f56ea7fc640 1 -- 172.21.15.5:0/2325478149 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec00ab70 con 0x7f56f41548e0 2024-03-24T10:52:08.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.346+0000 7f56ea7fc640 1 -- 172.21.15.5:0/2325478149 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56ec00acd0 con 0x7f56f41548e0 2024-03-24T10:52:08.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.346+0000 7f56fbc88640 1 -- 172.21.15.5:0/2325478149 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56f4156a50 con 0x7f56f41548e0 2024-03-24T10:52:08.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.346+0000 7f56ea7fc640 1 -- 172.21.15.5:0/2325478149 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec012600 con 0x7f56f41548e0 2024-03-24T10:52:08.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.346+0000 7f56fbc88640 1 -- 172.21.15.5:0/2325478149 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56f4157010 con 0x7f56f41548e0 2024-03-24T10:52:08.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.347+0000 7f56fbc88640 1 -- 172.21.15.5:0/2325478149 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f40681d0 con 0x7f56f41548e0 2024-03-24T10:52:08.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.615+0000 7f56ea7fc640 1 -- 172.21.15.5:0/2325478149 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f56ec007070 con 0x7f56f41548e0 2024-03-24T10:52:08.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.616+0000 7f56ea7fc640 1 --2- 172.21.15.5:0/2325478149 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f56d806b890 0x7f56d806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:08.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.616+0000 7f56f9a24640 1 --2- 172.21.15.5:0/2325478149 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f56d806b890 0x7f56d806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:08.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.616+0000 7f56f9a24640 1 --2- 172.21.15.5:0/2325478149 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f56d806b890 0x7f56d806dd50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f56f00011e0 tx=0x7f56f0007df0 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:08.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.631+0000 7f56ea7fc640 1 -- 172.21.15.5:0/2325478149 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f56ec058cd0 con 0x7f56f41548e0 2024-03-24T10:52:08.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.632+0000 7f56ea7fc640 1 -- 172.21.15.5:0/2325478149 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f56ec054f70 con 0x7f56f41548e0 2024-03-24T10:52:08.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.858+0000 7f56fbc88640 1 -- 172.21.15.5:0/2325478149 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56f4072eb0 con 0x7f56f41548e0 2024-03-24T10:52:08.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.882+0000 7f56ea7fc640 1 -- 172.21.15.5:0/2325478149 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f56ec058630 con 0x7f56f41548e0 2024-03-24T10:52:08.882 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:08.883 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:08.883 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:08.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.884+0000 7f56fbc88640 1 -- 172.21.15.5:0/2325478149 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f56d806b890 msgr2=0x7f56d806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.884+0000 7f56fbc88640 1 --2- 172.21.15.5:0/2325478149 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f56d806b890 0x7f56d806dd50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f56f00011e0 tx=0x7f56f0007df0 comp rx=0 tx=0).stop 2024-03-24T10:52:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.884+0000 7f56fbc88640 1 -- 172.21.15.5:0/2325478149 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56f41548e0 msgr2=0x7f56f41f1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.884+0000 7f56fbc88640 1 --2- 172.21.15.5:0/2325478149 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56f41548e0 0x7f56f41f1690 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f56f4073750 tx=0x7f56ec00cf00 comp rx=0 tx=0).stop 2024-03-24T10:52:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.884+0000 7f56fbc88640 1 -- 172.21.15.5:0/2325478149 shutdown_connections 2024-03-24T10:52:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.884+0000 7f56fbc88640 1 --2- 172.21.15.5:0/2325478149 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f56d806b890 0x7f56d806dd50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.884+0000 7f56fbc88640 1 --2- 172.21.15.5:0/2325478149 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f56f41548e0 0x7f56f41f1690 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.884+0000 7f56fbc88640 1 --2- 172.21.15.5:0/2325478149 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56f4154500 0x7f56f41f0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.884+0000 7f56fbc88640 1 --2- 172.21.15.5:0/2325478149 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56f40732e0 0x7f56f4154cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.884+0000 7f56fbc88640 1 -- 172.21.15.5:0/2325478149 >> 172.21.15.5:0/2325478149 conn(0x7f56f406c240 msgr2=0x7f56f40ca8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.884+0000 7f56fbc88640 1 -- 172.21.15.5:0/2325478149 shutdown_connections 2024-03-24T10:52:08.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:08.884+0000 7f56fbc88640 1 -- 172.21.15.5:0/2325478149 wait complete. 2024-03-24T10:52:09.499 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:09.499 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:10.501 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:10.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.604+0000 7fcb153d0640 1 Processor -- start 2024-03-24T10:52:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.604+0000 7fcb153d0640 1 -- start start 2024-03-24T10:52:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.605+0000 7fcb153d0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcb101543a0 0x7fcb1015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.605+0000 7fcb153d0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcb1006d4f0 0x7fcb100cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:10.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.605+0000 7fcb153d0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcb100ccaa0 0x7fcb1006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:10.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.605+0000 7fcb153d0640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fcb100c6750 con 0x7fcb100ccaa0 2024-03-24T10:52:10.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.605+0000 7fcb153d0640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fcb100cc380 con 0x7fcb1006d4f0 2024-03-24T10:52:10.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.605+0000 7fcb153d0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fcb100cb2f0 con 0x7fcb101543a0 2024-03-24T10:52:10.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.606+0000 7fcb0f7fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcb100ccaa0 0x7fcb1006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:10.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.606+0000 7fcb0e7fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcb1006d4f0 0x7fcb100cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:10.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.606+0000 7fcb0effd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcb101543a0 0x7fcb1015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:10.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.606+0000 7fcb0f7fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcb100ccaa0 0x7fcb1006bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50742/0 (socket says 172.21.15.5:50742) 2024-03-24T10:52:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.606+0000 7fcb0e7fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcb1006d4f0 0x7fcb100cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:50654/0 (socket says 172.21.15.5:50654) 2024-03-24T10:52:10.607 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.606+0000 7fcb0f7fe640 1 -- 172.21.15.5:0/1263091374 learned_addr learned my addr 172.21.15.5:0/1263091374 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.608+0000 7fcb0e7fc640 1 -- 172.21.15.5:0/1263091374 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcb101543a0 msgr2=0x7fcb1015ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.608+0000 7fcb0e7fc640 1 --2- 172.21.15.5:0/1263091374 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcb101543a0 0x7fcb1015ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.608+0000 7fcb0e7fc640 1 -- 172.21.15.5:0/1263091374 >> v2:172.21.15.5:3300/0 conn(0x7fcb100ccaa0 msgr2=0x7fcb1006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.608+0000 7fcb0e7fc640 1 --2- 172.21.15.5:0/1263091374 >> v2:172.21.15.5:3300/0 conn(0x7fcb100ccaa0 0x7fcb1006bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:10.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.608+0000 7fcb0e7fc640 1 -- 172.21.15.5:0/1263091374 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb100ca7e0 con 0x7fcb1006d4f0 2024-03-24T10:52:10.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.611+0000 7fcb0e7fc640 1 --2- 172.21.15.5:0/1263091374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb1006d4f0 0x7fcb100cc560 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcb10062a50 tx=0x7fcb0402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2d037ce1cc13fbe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:10.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.616+0000 7fcb0dffb640 1 -- 172.21.15.5:0/1263091374 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb04031830 con 0x7fcb1006d4f0 2024-03-24T10:52:10.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.616+0000 7fcb0dffb640 1 -- 172.21.15.5:0/1263091374 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb04031990 con 0x7fcb1006d4f0 2024-03-24T10:52:10.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.616+0000 7fcb0dffb640 1 -- 172.21.15.5:0/1263091374 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb04031d20 con 0x7fcb1006d4f0 2024-03-24T10:52:10.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.616+0000 7fcb153d0640 1 -- 172.21.15.5:0/1263091374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb1006d4f0 msgr2=0x7fcb100cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:10.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.616+0000 7fcb153d0640 1 --2- 172.21.15.5:0/1263091374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb1006d4f0 0x7fcb100cc560 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcb10062a50 tx=0x7fcb0402b8d0 comp rx=0 tx=0).stop 2024-03-24T10:52:10.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.616+0000 7fcb153d0640 1 -- 172.21.15.5:0/1263091374 shutdown_connections 2024-03-24T10:52:10.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.616+0000 7fcb153d0640 1 --2- 172.21.15.5:0/1263091374 >> v2:172.21.15.5:3300/0 conn(0x7fcb100ccaa0 0x7fcb1006bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:10.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.616+0000 7fcb153d0640 1 --2- 172.21.15.5:0/1263091374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb1006d4f0 0x7fcb100cc560 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:10.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.616+0000 7fcb153d0640 1 --2- 172.21.15.5:0/1263091374 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcb101543a0 0x7fcb1015ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:10.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.616+0000 7fcb153d0640 1 -- 172.21.15.5:0/1263091374 >> 172.21.15.5:0/1263091374 conn(0x7fcb10072ef0 msgr2=0x7fcb10061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:10.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.616+0000 7fcb153d0640 1 -- 172.21.15.5:0/1263091374 shutdown_connections 2024-03-24T10:52:10.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.616+0000 7fcb153d0640 1 -- 172.21.15.5:0/1263091374 wait complete. 2024-03-24T10:52:10.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.618+0000 7fcb153d0640 1 Processor -- start 2024-03-24T10:52:10.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.618+0000 7fcb153d0640 1 -- start start 2024-03-24T10:52:10.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.618+0000 7fcb153d0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb1006d4f0 0x7fcb101f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:10.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.618+0000 7fcb153d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb100ccaa0 0x7fcb100ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:10.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.618+0000 7fcb153d0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcb101543a0 0x7fcb100cf400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:10.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.618+0000 7fcb153d0640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fcb10156a00 con 0x7fcb1006d4f0 2024-03-24T10:52:10.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.618+0000 7fcb153d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fcb10157a70 con 0x7fcb100ccaa0 2024-03-24T10:52:10.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.618+0000 7fcb153d0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fcb10152d40 con 0x7fcb101543a0 2024-03-24T10:52:10.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.619+0000 7fcb0effd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb1006d4f0 0x7fcb101f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:10.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.619+0000 7fcb0e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb100ccaa0 0x7fcb100ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:10.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.619+0000 7fcb0f7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcb101543a0 0x7fcb100cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:10.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.619+0000 7fcb0effd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb1006d4f0 0x7fcb101f2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50744/0 (socket says 172.21.15.5:50744) 2024-03-24T10:52:10.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.619+0000 7fcb0effd640 1 -- 172.21.15.5:0/138902287 learned_addr learned my addr 172.21.15.5:0/138902287 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:10.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.623+0000 7fcb0effd640 1 -- 172.21.15.5:0/138902287 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcb101543a0 msgr2=0x7fcb100cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:10.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.623+0000 7fcb0effd640 1 --2- 172.21.15.5:0/138902287 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcb101543a0 0x7fcb100cf400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:10.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.623+0000 7fcb0effd640 1 -- 172.21.15.5:0/138902287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb100ccaa0 msgr2=0x7fcb100ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:10.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.623+0000 7fcb0effd640 1 --2- 172.21.15.5:0/138902287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb100ccaa0 0x7fcb100ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:10.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.623+0000 7fcb0effd640 1 -- 172.21.15.5:0/138902287 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb04009400 con 0x7fcb1006d4f0 2024-03-24T10:52:10.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.625+0000 7fcb0effd640 1 --2- 172.21.15.5:0/138902287 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb1006d4f0 0x7fcb101f2a10 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fcb0000e870 tx=0x7fcb0000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:10.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.629+0000 7fcaf77fe640 1 -- 172.21.15.5:0/138902287 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb000094f0 con 0x7fcb1006d4f0 2024-03-24T10:52:10.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.630+0000 7fcaf77fe640 1 -- 172.21.15.5:0/138902287 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb0000c940 con 0x7fcb1006d4f0 2024-03-24T10:52:10.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.630+0000 7fcaf77fe640 1 -- 172.21.15.5:0/138902287 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0000ccf0 con 0x7fcb1006d4f0 2024-03-24T10:52:10.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.630+0000 7fcb153d0640 1 -- 172.21.15.5:0/138902287 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb10154b90 con 0x7fcb1006d4f0 2024-03-24T10:52:10.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.630+0000 7fcb153d0640 1 -- 172.21.15.5:0/138902287 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb10154e30 con 0x7fcb1006d4f0 2024-03-24T10:52:10.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.631+0000 7fcb153d0640 1 -- 172.21.15.5:0/138902287 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb10157bb0 con 0x7fcb1006d4f0 2024-03-24T10:52:10.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.651+0000 7fcaf77fe640 1 -- 172.21.15.5:0/138902287 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcb00013470 con 0x7fcb1006d4f0 2024-03-24T10:52:10.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.651+0000 7fcaf77fe640 1 --2- 172.21.15.5:0/138902287 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcafc06b840 0x7fcafc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:10.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.651+0000 7fcb0e7fc640 1 --2- 172.21.15.5:0/138902287 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcafc06b840 0x7fcafc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:10.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.652+0000 7fcb0e7fc640 1 --2- 172.21.15.5:0/138902287 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcafc06b840 0x7fcafc06dd00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcb04036000 tx=0x7fcb0403c040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:10.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.962+0000 7fcaf77fe640 1 -- 172.21.15.5:0/138902287 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fcb00059020 con 0x7fcb1006d4f0 2024-03-24T10:52:10.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:10.962+0000 7fcaf77fe640 1 -- 172.21.15.5:0/138902287 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcb000552c0 con 0x7fcb1006d4f0 2024-03-24T10:52:11.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:11.187+0000 7fcb153d0640 1 -- 172.21.15.5:0/138902287 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb10157bb0 con 0x7fcb1006d4f0 2024-03-24T10:52:11.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:11.201+0000 7fcaf77fe640 1 -- 172.21.15.5:0/138902287 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fcb10157bb0 con 0x7fcb1006d4f0 2024-03-24T10:52:11.202 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:11.202 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:11.202 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:11.203+0000 7fcb153d0640 1 -- 172.21.15.5:0/138902287 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcafc06b840 msgr2=0x7fcafc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:11.203+0000 7fcb153d0640 1 --2- 172.21.15.5:0/138902287 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcafc06b840 0x7fcafc06dd00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcb04036000 tx=0x7fcb0403c040 comp rx=0 tx=0).stop 2024-03-24T10:52:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:11.203+0000 7fcb153d0640 1 -- 172.21.15.5:0/138902287 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb1006d4f0 msgr2=0x7fcb101f2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:11.203+0000 7fcb153d0640 1 --2- 172.21.15.5:0/138902287 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb1006d4f0 0x7fcb101f2a10 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fcb0000e870 tx=0x7fcb0000ed30 comp rx=0 tx=0).stop 2024-03-24T10:52:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:11.203+0000 7fcb153d0640 1 -- 172.21.15.5:0/138902287 shutdown_connections 2024-03-24T10:52:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:11.203+0000 7fcb153d0640 1 --2- 172.21.15.5:0/138902287 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcafc06b840 0x7fcafc06dd00 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:11.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:11.203+0000 7fcb153d0640 1 --2- 172.21.15.5:0/138902287 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcb101543a0 0x7fcb100cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:11.203+0000 7fcb153d0640 1 --2- 172.21.15.5:0/138902287 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcb100ccaa0 0x7fcb100ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:11.203+0000 7fcb153d0640 1 --2- 172.21.15.5:0/138902287 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcb1006d4f0 0x7fcb101f2a10 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:11.203+0000 7fcb153d0640 1 -- 172.21.15.5:0/138902287 >> 172.21.15.5:0/138902287 conn(0x7fcb10072ef0 msgr2=0x7fcb10151bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:11.204+0000 7fcb153d0640 1 -- 172.21.15.5:0/138902287 shutdown_connections 2024-03-24T10:52:11.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:11.204+0000 7fcb153d0640 1 -- 172.21.15.5:0/138902287 wait complete. 2024-03-24T10:52:11.818 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:11.821 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:12.822 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:12.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.924+0000 7f4cafcd4640 1 Processor -- start 2024-03-24T10:52:12.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.925+0000 7f4cafcd4640 1 -- start start 2024-03-24T10:52:12.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.925+0000 7f4cafcd4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4ca80cb270 0x7f4ca80cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:12.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.925+0000 7f4cafcd4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ca80ccba0 con 0x7f4ca80cd960 2024-03-24T10:52:12.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.925+0000 7f4cafcd4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ca80cce40 con 0x7f4ca80cae90 2024-03-24T10:52:12.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.925+0000 7f4cafcd4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f4ca80cc9b0 con 0x7f4ca80cb270 2024-03-24T10:52:12.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.926+0000 7f4cae271640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f4ca80cd960 0x7f4ca80cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60552/0 (socket says 172.21.15.5:60552) 2024-03-24T10:52:12.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.926+0000 7f4cae271640 1 -- 172.21.15.5:0/1793147596 learned_addr learned my addr 172.21.15.5:0/1793147596 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:52:12.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.926+0000 7f4cad26f640 1 --2- 172.21.15.5:0/1793147596 >> v2:172.21.15.57:3300/0 conn(0x7f4ca80cb270 0x7f4ca80cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:12.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.929+0000 7f4caca6e640 1 -- 172.21.15.5:0/1793147596 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1712230954 0 0) 0x7f4ca80ccba0 con 0x7f4ca80cd960 2024-03-24T10:52:12.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.929+0000 7f4caca6e640 1 -- 172.21.15.5:0/1793147596 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c94003520 con 0x7f4ca80cd960 2024-03-24T10:52:12.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.930+0000 7f4cad26f640 1 -- 172.21.15.5:0/1793147596 >> v1:172.21.15.57:6789/0 conn(0x7f4ca80cae90 legacy=0x7f4ca80cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:12.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.930+0000 7f4cad26f640 1 -- 172.21.15.5:0/1793147596 >> v1:172.21.15.5:6789/0 conn(0x7f4ca80cd960 legacy=0x7f4ca80cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:12.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.930+0000 7f4cad26f640 1 -- 172.21.15.5:0/1793147596 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca815a590 con 0x7f4ca80cb270 2024-03-24T10:52:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.933+0000 7f4cad26f640 1 --2- 172.21.15.5:0/1793147596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ca80cb270 0x7f4ca80cf530 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4c980097a0 tx=0x7f4c9802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8f17bfb88435ab0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.936+0000 7f4caca6e640 1 -- 172.21.15.5:0/1793147596 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9802f810 con 0x7f4ca80cb270 2024-03-24T10:52:12.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.939+0000 7f4caca6e640 1 -- 172.21.15.5:0/1793147596 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c9802f970 con 0x7f4ca80cb270 2024-03-24T10:52:12.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.939+0000 7f4caca6e640 1 -- 172.21.15.5:0/1793147596 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9802fce0 con 0x7f4ca80cb270 2024-03-24T10:52:12.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.940+0000 7f4cafcd4640 1 -- 172.21.15.5:0/1793147596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ca80cb270 msgr2=0x7f4ca80cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:12.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.940+0000 7f4cafcd4640 1 --2- 172.21.15.5:0/1793147596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ca80cb270 0x7f4ca80cf530 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4c980097a0 tx=0x7f4c9802b9f0 comp rx=0 tx=0).stop 2024-03-24T10:52:12.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.940+0000 7f4cafcd4640 1 -- 172.21.15.5:0/1793147596 shutdown_connections 2024-03-24T10:52:12.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.940+0000 7f4cafcd4640 1 --2- 172.21.15.5:0/1793147596 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ca80cb270 0x7f4ca80cf530 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:12.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.940+0000 7f4cafcd4640 1 -- 172.21.15.5:0/1793147596 >> 172.21.15.5:0/1793147596 conn(0x7f4ca80cfb80 msgr2=0x7f4ca80cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:12.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.940+0000 7f4cafcd4640 1 -- 172.21.15.5:0/1793147596 shutdown_connections 2024-03-24T10:52:12.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.940+0000 7f4cafcd4640 1 -- 172.21.15.5:0/1793147596 wait complete. 2024-03-24T10:52:12.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.942+0000 7f4cafcd4640 1 Processor -- start 2024-03-24T10:52:12.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.942+0000 7f4cafcd4640 1 -- start start 2024-03-24T10:52:12.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.943+0000 7f4cafcd4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ca80cae90 0x7f4ca81f00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:12.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.943+0000 7f4cafcd4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ca80cb270 0x7f4ca81f0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:12.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.943+0000 7f4cafcd4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ca80cd960 0x7f4ca8159920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:12.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.943+0000 7f4cafcd4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca80ab620 con 0x7f4ca80cae90 2024-03-24T10:52:12.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.943+0000 7f4cafcd4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca80c3650 con 0x7f4ca80cd960 2024-03-24T10:52:12.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.943+0000 7f4cafcd4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4ca80c38a0 con 0x7f4ca80cb270 2024-03-24T10:52:12.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.943+0000 7f4cae271640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ca80cd960 0x7f4ca8159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:12.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.944+0000 7f4cada70640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ca80cae90 0x7f4ca81f00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:12.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.944+0000 7f4cae271640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ca80cd960 0x7f4ca8159920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:39216/0 (socket says 172.21.15.5:39216) 2024-03-24T10:52:12.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.944+0000 7f4cae271640 1 -- 172.21.15.5:0/3714270588 learned_addr learned my addr 172.21.15.5:0/3714270588 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:12.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.944+0000 7f4cad26f640 1 --2- 172.21.15.5:0/3714270588 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ca80cb270 0x7f4ca81f0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:12.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.949+0000 7f4cada70640 1 -- 172.21.15.5:0/3714270588 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ca80cb270 msgr2=0x7f4ca81f0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:12.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.949+0000 7f4cada70640 1 --2- 172.21.15.5:0/3714270588 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ca80cb270 0x7f4ca81f0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:12.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.949+0000 7f4cada70640 1 -- 172.21.15.5:0/3714270588 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ca80cd960 msgr2=0x7f4ca8159920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:12.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.949+0000 7f4cada70640 1 --2- 172.21.15.5:0/3714270588 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ca80cd960 0x7f4ca8159920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:12.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.949+0000 7f4cada70640 1 -- 172.21.15.5:0/3714270588 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c98009400 con 0x7f4ca80cae90 2024-03-24T10:52:12.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.952+0000 7f4cada70640 1 --2- 172.21.15.5:0/3714270588 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ca80cae90 0x7f4ca81f00e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400c3c0 tx=0x7f4ca400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:12.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.956+0000 7f4c9e7fc640 1 -- 172.21.15.5:0/3714270588 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca40052d0 con 0x7f4ca80cae90 2024-03-24T10:52:12.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.956+0000 7f4c9e7fc640 1 -- 172.21.15.5:0/3714270588 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ca4005430 con 0x7f4ca80cae90 2024-03-24T10:52:12.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.956+0000 7f4cafcd4640 1 -- 172.21.15.5:0/3714270588 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca80c3b80 con 0x7f4ca80cae90 2024-03-24T10:52:12.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.956+0000 7f4c9e7fc640 1 -- 172.21.15.5:0/3714270588 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4003830 con 0x7f4ca80cae90 2024-03-24T10:52:12.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.956+0000 7f4cafcd4640 1 -- 172.21.15.5:0/3714270588 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca8156120 con 0x7f4ca80cae90 2024-03-24T10:52:12.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:12.958+0000 7f4cafcd4640 1 -- 172.21.15.5:0/3714270588 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ca8156a50 con 0x7f4ca80cae90 2024-03-24T10:52:13.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:13.240+0000 7f4c9e7fc640 1 -- 172.21.15.5:0/3714270588 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4ca4003990 con 0x7f4ca80cae90 2024-03-24T10:52:13.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:13.240+0000 7f4c9e7fc640 1 --2- 172.21.15.5:0/3714270588 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4c8c06b860 0x7f4c8c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:13.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:13.240+0000 7f4cad26f640 1 --2- 172.21.15.5:0/3714270588 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4c8c06b860 0x7f4c8c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:13.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:13.241+0000 7f4cad26f640 1 --2- 172.21.15.5:0/3714270588 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4c8c06b860 0x7f4c8c06dd20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4c98036000 tx=0x7f4c98008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:13.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:13.264+0000 7f4c9e7fc640 1 -- 172.21.15.5:0/3714270588 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f4ca4058300 con 0x7f4ca80cae90 2024-03-24T10:52:13.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:13.264+0000 7f4c9e7fc640 1 -- 172.21.15.5:0/3714270588 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4ca40545a0 con 0x7f4ca80cae90 2024-03-24T10:52:13.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:13.484+0000 7f4cafcd4640 1 -- 172.21.15.5:0/3714270588 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ca80681d0 con 0x7f4ca80cae90 2024-03-24T10:52:13.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:13.497+0000 7f4c9e7fc640 1 -- 172.21.15.5:0/3714270588 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f4ca4017020 con 0x7f4ca80cae90 2024-03-24T10:52:13.498 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:13.498 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:13.498 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:13.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:13.500+0000 7f4cafcd4640 1 -- 172.21.15.5:0/3714270588 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4c8c06b860 msgr2=0x7f4c8c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:13.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:13.500+0000 7f4cafcd4640 1 --2- 172.21.15.5:0/3714270588 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4c8c06b860 0x7f4c8c06dd20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4c98036000 tx=0x7f4c98008030 comp rx=0 tx=0).stop 2024-03-24T10:52:13.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:13.500+0000 7f4cafcd4640 1 -- 172.21.15.5:0/3714270588 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ca80cae90 msgr2=0x7f4ca81f00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:13.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:13.500+0000 7f4cafcd4640 1 --2- 172.21.15.5:0/3714270588 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ca80cae90 0x7f4ca81f00e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4ca400c3c0 tx=0x7f4ca400c880 comp rx=0 tx=0).stop 2024-03-24T10:52:13.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:13.500+0000 7f4cafcd4640 1 -- 172.21.15.5:0/3714270588 shutdown_connections 2024-03-24T10:52:13.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:13.500+0000 7f4cafcd4640 1 --2- 172.21.15.5:0/3714270588 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4c8c06b860 0x7f4c8c06dd20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:13.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:13.500+0000 7f4cafcd4640 1 --2- 172.21.15.5:0/3714270588 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4ca80cd960 0x7f4ca8159920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:13.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:13.500+0000 7f4cafcd4640 1 --2- 172.21.15.5:0/3714270588 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4ca80cb270 0x7f4ca81f0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:13.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:13.500+0000 7f4cafcd4640 1 --2- 172.21.15.5:0/3714270588 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4ca80cae90 0x7f4ca81f00e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:13.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:13.500+0000 7f4cafcd4640 1 -- 172.21.15.5:0/3714270588 >> 172.21.15.5:0/3714270588 conn(0x7f4ca80cfb80 msgr2=0x7f4ca80c6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:13.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:13.500+0000 7f4cafcd4640 1 -- 172.21.15.5:0/3714270588 shutdown_connections 2024-03-24T10:52:13.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:13.500+0000 7f4cafcd4640 1 -- 172.21.15.5:0/3714270588 wait complete. 2024-03-24T10:52:14.119 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:14.120 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:15.121 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:15.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.223+0000 7f6833313640 1 Processor -- start 2024-03-24T10:52:15.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.224+0000 7f6833313640 1 -- start start 2024-03-24T10:52:15.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.224+0000 7f6833313640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f682c0cb270 0x7f682c0cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:15.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.224+0000 7f6833313640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f682c0cd960 0x7f682c0cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:15.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.224+0000 7f6833313640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f682c0ccba0 con 0x7f682c0cae90 2024-03-24T10:52:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.224+0000 7f6833313640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f682c0cc9b0 con 0x7f682c0cd960 2024-03-24T10:52:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.224+0000 7f6833313640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f682c0cce40 con 0x7f682c0cb270 2024-03-24T10:52:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.225+0000 7f68318b0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f682c0cd960 0x7f682c0cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.225+0000 7f68310af640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f682c0cae90 0x7f682c0cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:45154/0 (socket says 172.21.15.5:45154) 2024-03-24T10:52:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.225+0000 7f68310af640 1 -- 172.21.15.5:0/1655995019 learned_addr learned my addr 172.21.15.5:0/1655995019 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:52:15.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.226+0000 7f68308ae640 1 --2- 172.21.15.5:0/1655995019 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f682c0cb270 0x7f682c0cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:15.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.228+0000 7f68318b0640 1 -- 172.21.15.5:0/1655995019 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f682c0cb270 msgr2=0x7f682c0cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:15.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.228+0000 7f68318b0640 1 --2- 172.21.15.5:0/1655995019 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f682c0cb270 0x7f682c0cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:15.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.228+0000 7f68318b0640 1 -- 172.21.15.5:0/1655995019 >> v1:172.21.15.57:6789/0 conn(0x7f682c0cae90 legacy=0x7f682c0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:15.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.228+0000 7f68318b0640 1 -- 172.21.15.5:0/1655995019 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f682c15a590 con 0x7f682c0cd960 2024-03-24T10:52:15.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.230+0000 7f68318b0640 1 --2- 172.21.15.5:0/1655995019 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f682c0cd960 0x7f682c0cb6f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6824009d90 tx=0x7f682402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5b842deb17de6fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:15.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.234+0000 7f6823fff640 1 -- 172.21.15.5:0/1655995019 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f682402cce0 con 0x7f682c0cd960 2024-03-24T10:52:15.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.234+0000 7f6823fff640 1 -- 172.21.15.5:0/1655995019 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f682402ce40 con 0x7f682c0cd960 2024-03-24T10:52:15.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.234+0000 7f6823fff640 1 -- 172.21.15.5:0/1655995019 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6824034660 con 0x7f682c0cd960 2024-03-24T10:52:15.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.235+0000 7f6833313640 1 -- 172.21.15.5:0/1655995019 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f682c0cd960 msgr2=0x7f682c0cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:15.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.235+0000 7f6833313640 1 --2- 172.21.15.5:0/1655995019 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f682c0cd960 0x7f682c0cb6f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6824009d90 tx=0x7f682402bf30 comp rx=0 tx=0).stop 2024-03-24T10:52:15.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.235+0000 7f6833313640 1 -- 172.21.15.5:0/1655995019 shutdown_connections 2024-03-24T10:52:15.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.235+0000 7f6833313640 1 --2- 172.21.15.5:0/1655995019 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f682c0cb270 0x7f682c0cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:15.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.235+0000 7f6833313640 1 --2- 172.21.15.5:0/1655995019 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f682c0cd960 0x7f682c0cb6f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:15.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.235+0000 7f6833313640 1 -- 172.21.15.5:0/1655995019 >> 172.21.15.5:0/1655995019 conn(0x7f682c0cfb80 msgr2=0x7f682c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:15.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.235+0000 7f6833313640 1 -- 172.21.15.5:0/1655995019 shutdown_connections 2024-03-24T10:52:15.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.235+0000 7f6833313640 1 -- 172.21.15.5:0/1655995019 wait complete. 2024-03-24T10:52:15.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.237+0000 7f6833313640 1 Processor -- start 2024-03-24T10:52:15.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.237+0000 7f6833313640 1 -- start start 2024-03-24T10:52:15.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.238+0000 7f6833313640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f682c0cae90 0x7f682c0c5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:15.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.238+0000 7f6833313640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f682c0cb270 0x7f682c14de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:15.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.238+0000 7f6833313640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f682c0cd960 0x7f682c14e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:15.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.238+0000 7f6833313640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f682c1f0800 con 0x7f682c0cb270 2024-03-24T10:52:15.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.238+0000 7f6833313640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f682c0c32b0 con 0x7f682c0cd960 2024-03-24T10:52:15.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.238+0000 7f6833313640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f682c0c6530 con 0x7f682c0cae90 2024-03-24T10:52:15.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.238+0000 7f68308ae640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f682c0cb270 0x7f682c14de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:15.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.238+0000 7f68310af640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f682c0cae90 0x7f682c0c5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:15.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.238+0000 7f68318b0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f682c0cd960 0x7f682c14e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:15.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.238+0000 7f68308ae640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f682c0cb270 0x7f682c14de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54588/0 (socket says 172.21.15.5:54588) 2024-03-24T10:52:15.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.238+0000 7f68308ae640 1 -- 172.21.15.5:0/1279101222 learned_addr learned my addr 172.21.15.5:0/1279101222 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:15.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.242+0000 7f68308ae640 1 -- 172.21.15.5:0/1279101222 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f682c0cae90 msgr2=0x7f682c0c5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:15.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.242+0000 7f68308ae640 1 --2- 172.21.15.5:0/1279101222 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f682c0cae90 0x7f682c0c5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:15.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.242+0000 7f68308ae640 1 -- 172.21.15.5:0/1279101222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f682c0cd960 msgr2=0x7f682c14e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:15.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.242+0000 7f68308ae640 1 --2- 172.21.15.5:0/1279101222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f682c0cd960 0x7f682c14e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:15.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.242+0000 7f68308ae640 1 -- 172.21.15.5:0/1279101222 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6824009a70 con 0x7f682c0cb270 2024-03-24T10:52:15.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.244+0000 7f68308ae640 1 --2- 172.21.15.5:0/1279101222 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f682c0cb270 0x7f682c14de70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f681c00e9b0 tx=0x7f681c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:15.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.248+0000 7f6821ffb640 1 -- 172.21.15.5:0/1279101222 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681c009670 con 0x7f682c0cb270 2024-03-24T10:52:15.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.248+0000 7f6821ffb640 1 -- 172.21.15.5:0/1279101222 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f681c00c9a0 con 0x7f682c0cb270 2024-03-24T10:52:15.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.248+0000 7f6821ffb640 1 -- 172.21.15.5:0/1279101222 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681c00cd50 con 0x7f682c0cb270 2024-03-24T10:52:15.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.248+0000 7f6833313640 1 -- 172.21.15.5:0/1279101222 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f682c15c670 con 0x7f682c0cb270 2024-03-24T10:52:15.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.248+0000 7f6833313640 1 -- 172.21.15.5:0/1279101222 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f682c1f0a50 con 0x7f682c0cb270 2024-03-24T10:52:15.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.250+0000 7f6833313640 1 -- 172.21.15.5:0/1279101222 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f682c1f0c10 con 0x7f682c0cb270 2024-03-24T10:52:15.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.274+0000 7f6821ffb640 1 -- 172.21.15.5:0/1279101222 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f681c013470 con 0x7f682c0cb270 2024-03-24T10:52:15.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.275+0000 7f6821ffb640 1 --2- 172.21.15.5:0/1279101222 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f681006b860 0x7f681006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:15.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.275+0000 7f68310af640 1 --2- 172.21.15.5:0/1279101222 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f681006b860 0x7f681006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:15.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.275+0000 7f68310af640 1 --2- 172.21.15.5:0/1279101222 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f681006b860 0x7f681006dd20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6828007fd0 tx=0x7f6828007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:15.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.396+0000 7f6821ffb640 1 -- 172.21.15.5:0/1279101222 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f681c059370 con 0x7f682c0cb270 2024-03-24T10:52:15.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.560+0000 7f6821ffb640 1 -- 172.21.15.5:0/1279101222 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f681c055610 con 0x7f682c0cb270 2024-03-24T10:52:15.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.788+0000 7f6833313640 1 -- 172.21.15.5:0/1279101222 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f682c0732f0 con 0x7f682c0cb270 2024-03-24T10:52:15.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.811+0000 7f6821ffb640 1 -- 172.21.15.5:0/1279101222 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f681c017090 con 0x7f682c0cb270 2024-03-24T10:52:15.811 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:15.811 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:15.811 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:15.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.813+0000 7f6833313640 1 -- 172.21.15.5:0/1279101222 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f681006b860 msgr2=0x7f681006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:15.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.813+0000 7f6833313640 1 --2- 172.21.15.5:0/1279101222 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f681006b860 0x7f681006dd20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6828007fd0 tx=0x7f6828007e80 comp rx=0 tx=0).stop 2024-03-24T10:52:15.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.813+0000 7f6833313640 1 -- 172.21.15.5:0/1279101222 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f682c0cb270 msgr2=0x7f682c14de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:15.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.813+0000 7f6833313640 1 --2- 172.21.15.5:0/1279101222 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f682c0cb270 0x7f682c14de70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f681c00e9b0 tx=0x7f681c00ee70 comp rx=0 tx=0).stop 2024-03-24T10:52:15.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.813+0000 7f6833313640 1 -- 172.21.15.5:0/1279101222 shutdown_connections 2024-03-24T10:52:15.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.813+0000 7f6833313640 1 --2- 172.21.15.5:0/1279101222 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f681006b860 0x7f681006dd20 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:15.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.813+0000 7f6833313640 1 --2- 172.21.15.5:0/1279101222 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f682c0cb270 0x7f682c14de70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:15.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.813+0000 7f6833313640 1 --2- 172.21.15.5:0/1279101222 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f682c0cd960 0x7f682c14e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:15.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.813+0000 7f6833313640 1 --2- 172.21.15.5:0/1279101222 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f682c0cae90 0x7f682c0c5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:15.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.813+0000 7f6833313640 1 -- 172.21.15.5:0/1279101222 >> 172.21.15.5:0/1279101222 conn(0x7f682c0cfb80 msgr2=0x7f682c159da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:15.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.813+0000 7f6833313640 1 -- 172.21.15.5:0/1279101222 shutdown_connections 2024-03-24T10:52:15.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:15.813+0000 7f6833313640 1 -- 172.21.15.5:0/1279101222 wait complete. 2024-03-24T10:52:16.427 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:16.428 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:17.428 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:17.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.531+0000 7f7fc5381640 1 Processor -- start 2024-03-24T10:52:17.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.531+0000 7f7fc5381640 1 -- start start 2024-03-24T10:52:17.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.532+0000 7f7fc5381640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7fc015ac00 0x7f7fc00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:17.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.532+0000 7f7fc5381640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7fc00ccaa0 0x7f7fc006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:17.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.532+0000 7f7fc5381640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fc00cb2f0 con 0x7f7fc01543a0 2024-03-24T10:52:17.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.532+0000 7f7fc5381640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f7fc00c6750 con 0x7f7fc00ccaa0 2024-03-24T10:52:17.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.532+0000 7f7fc5381640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7fc00cc380 con 0x7f7fc015ac00 2024-03-24T10:52:17.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.532+0000 7f7fbe7fc640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7fc015ac00 0x7f7fc00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:17.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.533+0000 7f7fbf7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7fc00ccaa0 0x7f7fc006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:17.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.533+0000 7f7fbe7fc640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7fc015ac00 0x7f7fc00cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:40426/0 (socket says 172.21.15.5:40426) 2024-03-24T10:52:17.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.533+0000 7f7fbe7fc640 1 -- 172.21.15.5:0/3807399003 learned_addr learned my addr 172.21.15.5:0/3807399003 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:17.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.536+0000 7f7fbe7fc640 1 -- 172.21.15.5:0/3807399003 >> v2:172.21.15.57:3300/0 conn(0x7f7fc00ccaa0 msgr2=0x7f7fc006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:17.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.536+0000 7f7fbe7fc640 1 --2- 172.21.15.5:0/3807399003 >> v2:172.21.15.57:3300/0 conn(0x7f7fc00ccaa0 0x7f7fc006bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:17.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.536+0000 7f7fbe7fc640 1 -- 172.21.15.5:0/3807399003 >> v1:172.21.15.57:6789/0 conn(0x7f7fc01543a0 legacy=0x7f7fc006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:17.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.536+0000 7f7fbe7fc640 1 -- 172.21.15.5:0/3807399003 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fc015b070 con 0x7f7fc015ac00 2024-03-24T10:52:17.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.539+0000 7f7fbe7fc640 1 --2- 172.21.15.5:0/3807399003 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7fc015ac00 0x7f7fc00cc560 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009870 tx=0x7f7fb402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1b4b3605608fede server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:17.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.542+0000 7f7fbdffb640 1 -- 172.21.15.5:0/3807399003 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4030840 con 0x7f7fc015ac00 2024-03-24T10:52:17.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.545+0000 7f7fbdffb640 1 -- 172.21.15.5:0/3807399003 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fb40309a0 con 0x7f7fc015ac00 2024-03-24T10:52:17.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.545+0000 7f7fbdffb640 1 -- 172.21.15.5:0/3807399003 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4030d30 con 0x7f7fc015ac00 2024-03-24T10:52:17.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.545+0000 7f7fc5381640 1 -- 172.21.15.5:0/3807399003 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7fc015ac00 msgr2=0x7f7fc00cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.545+0000 7f7fc5381640 1 --2- 172.21.15.5:0/3807399003 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7fc015ac00 0x7f7fc00cc560 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009870 tx=0x7f7fb402b980 comp rx=0 tx=0).stop 2024-03-24T10:52:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.545+0000 7f7fc5381640 1 -- 172.21.15.5:0/3807399003 shutdown_connections 2024-03-24T10:52:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.545+0000 7f7fc5381640 1 --2- 172.21.15.5:0/3807399003 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7fc015ac00 0x7f7fc00cc560 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.545+0000 7f7fc5381640 1 --2- 172.21.15.5:0/3807399003 >> v2:172.21.15.57:3300/0 conn(0x7f7fc00ccaa0 0x7f7fc006bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.545+0000 7f7fc5381640 1 -- 172.21.15.5:0/3807399003 >> 172.21.15.5:0/3807399003 conn(0x7f7fc0072ef0 msgr2=0x7f7fc0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.545+0000 7f7fc5381640 1 -- 172.21.15.5:0/3807399003 shutdown_connections 2024-03-24T10:52:17.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.545+0000 7f7fc5381640 1 -- 172.21.15.5:0/3807399003 wait complete. 2024-03-24T10:52:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.547+0000 7f7fc5381640 1 Processor -- start 2024-03-24T10:52:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.547+0000 7f7fc5381640 1 -- start start 2024-03-24T10:52:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.547+0000 7f7fc5381640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fc00ccaa0 0x7f7fc0157360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:17.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.547+0000 7f7fc5381640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fc01543a0 0x7f7fc014d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.547+0000 7f7fc5381640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7fc015ac00 0x7f7fc014a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.547+0000 7f7fc5381640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc014db60 con 0x7f7fc01543a0 2024-03-24T10:52:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.547+0000 7f7fc5381640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7fc014cdc0 con 0x7f7fc00ccaa0 2024-03-24T10:52:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.547+0000 7f7fc5381640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7fc006d790 con 0x7f7fc015ac00 2024-03-24T10:52:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.548+0000 7f7fbe7fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fc01543a0 0x7f7fc014d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.548+0000 7f7fbf7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7fc015ac00 0x7f7fc014a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.548+0000 7f7fbeffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fc00ccaa0 0x7f7fc0157360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.548+0000 7f7fbe7fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fc01543a0 0x7f7fc014d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54602/0 (socket says 172.21.15.5:54602) 2024-03-24T10:52:17.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.548+0000 7f7fbe7fc640 1 -- 172.21.15.5:0/2260835241 learned_addr learned my addr 172.21.15.5:0/2260835241 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:17.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.552+0000 7f7fbe7fc640 1 -- 172.21.15.5:0/2260835241 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7fc015ac00 msgr2=0x7f7fc014a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:17.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.552+0000 7f7fbe7fc640 1 --2- 172.21.15.5:0/2260835241 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7fc015ac00 0x7f7fc014a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:17.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.552+0000 7f7fbe7fc640 1 -- 172.21.15.5:0/2260835241 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fc00ccaa0 msgr2=0x7f7fc0157360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:17.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.552+0000 7f7fbe7fc640 1 --2- 172.21.15.5:0/2260835241 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fc00ccaa0 0x7f7fc0157360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:17.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.552+0000 7f7fbe7fc640 1 -- 172.21.15.5:0/2260835241 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fb40094d0 con 0x7f7fc01543a0 2024-03-24T10:52:17.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.555+0000 7f7fbe7fc640 1 --2- 172.21.15.5:0/2260835241 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fc01543a0 0x7f7fc014d250 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4000c00 tx=0x7f7fb4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:17.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.560+0000 7f7fa77fe640 1 -- 172.21.15.5:0/2260835241 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4039a90 con 0x7f7fc01543a0 2024-03-24T10:52:17.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.560+0000 7f7fa77fe640 1 -- 172.21.15.5:0/2260835241 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fb4039bf0 con 0x7f7fc01543a0 2024-03-24T10:52:17.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.560+0000 7f7fc5381640 1 -- 172.21.15.5:0/2260835241 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fc014b6e0 con 0x7f7fc01543a0 2024-03-24T10:52:17.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.560+0000 7f7fa77fe640 1 -- 172.21.15.5:0/2260835241 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fb4043410 con 0x7f7fc01543a0 2024-03-24T10:52:17.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.560+0000 7f7fc5381640 1 -- 172.21.15.5:0/2260835241 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fc014b400 con 0x7f7fc01543a0 2024-03-24T10:52:17.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.561+0000 7f7fc5381640 1 -- 172.21.15.5:0/2260835241 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fc00681d0 con 0x7f7fc01543a0 2024-03-24T10:52:17.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.856+0000 7f7fa77fe640 1 -- 172.21.15.5:0/2260835241 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7fb404c3b0 con 0x7f7fc01543a0 2024-03-24T10:52:17.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.857+0000 7f7fa77fe640 1 --2- 172.21.15.5:0/2260835241 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7fac06b860 0x7f7fac06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:17.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.857+0000 7f7fbeffd640 1 --2- 172.21.15.5:0/2260835241 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7fac06b860 0x7f7fac06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:17.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.858+0000 7f7fbeffd640 1 --2- 172.21.15.5:0/2260835241 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7fac06b860 0x7f7fac06dd20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0001780 tx=0x7f7fb0009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:17.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.871+0000 7f7fa77fe640 1 -- 172.21.15.5:0/2260835241 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f7fb407b850 con 0x7f7fc01543a0 2024-03-24T10:52:17.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:17.872+0000 7f7fa77fe640 1 -- 172.21.15.5:0/2260835241 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7fb4077af0 con 0x7f7fc01543a0 2024-03-24T10:52:18.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:18.099+0000 7f7fc5381640 1 -- 172.21.15.5:0/2260835241 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7fc0061bf0 con 0x7f7fc01543a0 2024-03-24T10:52:18.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:18.163+0000 7f7fa77fe640 1 -- 172.21.15.5:0/2260835241 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f7fb4034090 con 0x7f7fc01543a0 2024-03-24T10:52:18.163 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:18.164 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:18.164 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:18.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:18.165+0000 7f7fc5381640 1 -- 172.21.15.5:0/2260835241 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7fac06b860 msgr2=0x7f7fac06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:18.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:18.165+0000 7f7fc5381640 1 --2- 172.21.15.5:0/2260835241 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7fac06b860 0x7f7fac06dd20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0001780 tx=0x7f7fb0009040 comp rx=0 tx=0).stop 2024-03-24T10:52:18.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:18.165+0000 7f7fc5381640 1 -- 172.21.15.5:0/2260835241 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fc01543a0 msgr2=0x7f7fc014d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:18.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:18.165+0000 7f7fc5381640 1 --2- 172.21.15.5:0/2260835241 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fc01543a0 0x7f7fc014d250 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4000c00 tx=0x7f7fb4004730 comp rx=0 tx=0).stop 2024-03-24T10:52:18.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:18.165+0000 7f7fc5381640 1 -- 172.21.15.5:0/2260835241 shutdown_connections 2024-03-24T10:52:18.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:18.165+0000 7f7fc5381640 1 --2- 172.21.15.5:0/2260835241 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7fac06b860 0x7f7fac06dd20 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:18.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:18.165+0000 7f7fc5381640 1 --2- 172.21.15.5:0/2260835241 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7fc01543a0 0x7f7fc014d250 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:18.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:18.165+0000 7f7fc5381640 1 --2- 172.21.15.5:0/2260835241 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7fc015ac00 0x7f7fc014a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:18.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:18.165+0000 7f7fc5381640 1 --2- 172.21.15.5:0/2260835241 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7fc00ccaa0 0x7f7fc0157360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:18.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:18.165+0000 7f7fc5381640 1 -- 172.21.15.5:0/2260835241 >> 172.21.15.5:0/2260835241 conn(0x7f7fc0072ef0 msgr2=0x7f7fc0152380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:18.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:18.165+0000 7f7fc5381640 1 -- 172.21.15.5:0/2260835241 shutdown_connections 2024-03-24T10:52:18.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:18.165+0000 7f7fc5381640 1 -- 172.21.15.5:0/2260835241 wait complete. 2024-03-24T10:52:18.780 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:18.780 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:19.781 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:19.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.887+0000 7f7f3bb0f640 1 Processor -- start 2024-03-24T10:52:19.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.887+0000 7f7f3bb0f640 1 -- start start 2024-03-24T10:52:19.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.888+0000 7f7f3bb0f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f3415aac0 0x7f7f3415aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:19.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.888+0000 7f7f3bb0f640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f7f3406c230 0x7f7f3406c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:19.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.888+0000 7f7f3bb0f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f3406cd50 con 0x7f7f340cebc0 2024-03-24T10:52:19.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.888+0000 7f7f3bb0f640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f7f340ce790 con 0x7f7f3406c230 2024-03-24T10:52:19.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.888+0000 7f7f3bb0f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7f3415b3e0 con 0x7f7f3415aac0 2024-03-24T10:52:19.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.888+0000 7f7f3a0ac640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f7f340cebc0 0x7f7f340ca5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:45166/0 (socket says 172.21.15.5:45166) 2024-03-24T10:52:19.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.888+0000 7f7f3a0ac640 1 -- 172.21.15.5:0/2069408305 learned_addr learned my addr 172.21.15.5:0/2069408305 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:52:19.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.888+0000 7f7f390aa640 1 --2- 172.21.15.5:0/2069408305 >> v2:172.21.15.5:3300/0 conn(0x7f7f3406c230 0x7f7f3406c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:19.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.889+0000 7f7f398ab640 1 --2- 172.21.15.5:0/2069408305 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f3415aac0 0x7f7f3415aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:19.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.891+0000 7f7f390aa640 1 -- 172.21.15.5:0/2069408305 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f3415aac0 msgr2=0x7f7f3415aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:19.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.891+0000 7f7f390aa640 1 --2- 172.21.15.5:0/2069408305 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f3415aac0 0x7f7f3415aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:19.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.891+0000 7f7f390aa640 1 -- 172.21.15.5:0/2069408305 >> v1:172.21.15.57:6789/0 conn(0x7f7f340cebc0 legacy=0x7f7f340ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:19.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.891+0000 7f7f390aa640 1 -- 172.21.15.5:0/2069408305 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f340cdb10 con 0x7f7f3406c230 2024-03-24T10:52:19.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.893+0000 7f7f390aa640 1 --2- 172.21.15.5:0/2069408305 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f3406c230 0x7f7f3406c670 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7f240097a0 tx=0x7f7f2402b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c6bc1937b362698 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:19.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.898+0000 7f7f388a9640 1 -- 172.21.15.5:0/2069408305 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f24030800 con 0x7f7f3406c230 2024-03-24T10:52:19.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.898+0000 7f7f388a9640 1 -- 172.21.15.5:0/2069408305 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f24030960 con 0x7f7f3406c230 2024-03-24T10:52:19.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.898+0000 7f7f388a9640 1 -- 172.21.15.5:0/2069408305 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f24030cf0 con 0x7f7f3406c230 2024-03-24T10:52:19.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.899+0000 7f7f3bb0f640 1 -- 172.21.15.5:0/2069408305 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f3406c230 msgr2=0x7f7f3406c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:19.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.899+0000 7f7f3bb0f640 1 --2- 172.21.15.5:0/2069408305 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f3406c230 0x7f7f3406c670 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f7f240097a0 tx=0x7f7f2402b940 comp rx=0 tx=0).stop 2024-03-24T10:52:19.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.899+0000 7f7f3bb0f640 1 -- 172.21.15.5:0/2069408305 shutdown_connections 2024-03-24T10:52:19.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.899+0000 7f7f3bb0f640 1 --2- 172.21.15.5:0/2069408305 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f3406c230 0x7f7f3406c670 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:19.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.899+0000 7f7f3bb0f640 1 --2- 172.21.15.5:0/2069408305 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f3415aac0 0x7f7f3415aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:19.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.899+0000 7f7f3bb0f640 1 -- 172.21.15.5:0/2069408305 >> 172.21.15.5:0/2069408305 conn(0x7f7f340cfb90 msgr2=0x7f7f340cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:19.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.899+0000 7f7f3bb0f640 1 -- 172.21.15.5:0/2069408305 shutdown_connections 2024-03-24T10:52:19.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.899+0000 7f7f3bb0f640 1 -- 172.21.15.5:0/2069408305 wait complete. 2024-03-24T10:52:19.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.900+0000 7f7f3bb0f640 1 Processor -- start 2024-03-24T10:52:19.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.900+0000 7f7f3bb0f640 1 -- start start 2024-03-24T10:52:19.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.901+0000 7f7f3bb0f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f3406c230 0x7f7f340b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:19.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.901+0000 7f7f3bb0f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f340cebc0 0x7f7f341e4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:19.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.901+0000 7f7f3bb0f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f3415aac0 0x7f7f34155590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:19.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.901+0000 7f7f3bb0f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f7f34157320 con 0x7f7f340cebc0 2024-03-24T10:52:19.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.901+0000 7f7f3bb0f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7f341e47c0 con 0x7f7f3406c230 2024-03-24T10:52:19.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.901+0000 7f7f3bb0f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7f341e6630 con 0x7f7f3415aac0 2024-03-24T10:52:19.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.902+0000 7f7f390aa640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f340cebc0 0x7f7f341e4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:19.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.902+0000 7f7f3a0ac640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f3415aac0 0x7f7f34155590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:19.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.902+0000 7f7f398ab640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f3406c230 0x7f7f340b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:19.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.902+0000 7f7f390aa640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f340cebc0 0x7f7f341e4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54616/0 (socket says 172.21.15.5:54616) 2024-03-24T10:52:19.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.902+0000 7f7f390aa640 1 -- 172.21.15.5:0/3165665952 learned_addr learned my addr 172.21.15.5:0/3165665952 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:19.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.905+0000 7f7f390aa640 1 -- 172.21.15.5:0/3165665952 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f3415aac0 msgr2=0x7f7f34155590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:19.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.905+0000 7f7f390aa640 1 --2- 172.21.15.5:0/3165665952 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f3415aac0 0x7f7f34155590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:19.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.905+0000 7f7f390aa640 1 -- 172.21.15.5:0/3165665952 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f3406c230 msgr2=0x7f7f340b07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:19.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.905+0000 7f7f390aa640 1 --2- 172.21.15.5:0/3165665952 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f3406c230 0x7f7f340b07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:19.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.905+0000 7f7f390aa640 1 -- 172.21.15.5:0/3165665952 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f24009400 con 0x7f7f340cebc0 2024-03-24T10:52:19.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.908+0000 7f7f390aa640 1 --2- 172.21.15.5:0/3165665952 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f340cebc0 0x7f7f341e4c50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7f240098c0 tx=0x7f7f24004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:19.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.911+0000 7f7f2a7fc640 1 -- 172.21.15.5:0/3165665952 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2403e040 con 0x7f7f340cebc0 2024-03-24T10:52:19.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.911+0000 7f7f2a7fc640 1 -- 172.21.15.5:0/3165665952 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f24039cd0 con 0x7f7f340cebc0 2024-03-24T10:52:19.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.911+0000 7f7f2a7fc640 1 -- 172.21.15.5:0/3165665952 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f24044630 con 0x7f7f340cebc0 2024-03-24T10:52:19.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.911+0000 7f7f3bb0f640 1 -- 172.21.15.5:0/3165665952 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f341e4490 con 0x7f7f340cebc0 2024-03-24T10:52:19.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.912+0000 7f7f3bb0f640 1 -- 172.21.15.5:0/3165665952 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f34156d60 con 0x7f7f340cebc0 2024-03-24T10:52:19.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.913+0000 7f7f3bb0f640 1 -- 172.21.15.5:0/3165665952 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f341e42f0 con 0x7f7f340cebc0 2024-03-24T10:52:19.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.938+0000 7f7f2a7fc640 1 -- 172.21.15.5:0/3165665952 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7f2404d3b0 con 0x7f7f340cebc0 2024-03-24T10:52:19.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.939+0000 7f7f2a7fc640 1 --2- 172.21.15.5:0/3165665952 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7f1806b860 0x7f7f1806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:19.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.939+0000 7f7f398ab640 1 --2- 172.21.15.5:0/3165665952 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7f1806b860 0x7f7f1806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:19.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.939+0000 7f7f398ab640 1 --2- 172.21.15.5:0/3165665952 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7f1806b860 0x7f7f1806dd20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7f30009aa0 tx=0x7f7f30009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:19.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:19.942+0000 7f7f2a7fc640 1 -- 172.21.15.5:0/3165665952 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f7f24039e40 con 0x7f7f340cebc0 2024-03-24T10:52:20.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:20.202+0000 7f7f2a7fc640 1 -- 172.21.15.5:0/3165665952 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7f24077540 con 0x7f7f340cebc0 2024-03-24T10:52:20.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:20.429+0000 7f7f3bb0f640 1 -- 172.21.15.5:0/3165665952 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f341e42f0 con 0x7f7f340cebc0 2024-03-24T10:52:20.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:20.443+0000 7f7f2a7fc640 1 -- 172.21.15.5:0/3165665952 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f7f341e42f0 con 0x7f7f340cebc0 2024-03-24T10:52:20.443 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:20.443 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:20.443 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:20.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:20.445+0000 7f7f3bb0f640 1 -- 172.21.15.5:0/3165665952 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7f1806b860 msgr2=0x7f7f1806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:20.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:20.445+0000 7f7f3bb0f640 1 --2- 172.21.15.5:0/3165665952 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7f1806b860 0x7f7f1806dd20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7f30009aa0 tx=0x7f7f30009190 comp rx=0 tx=0).stop 2024-03-24T10:52:20.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:20.445+0000 7f7f3bb0f640 1 -- 172.21.15.5:0/3165665952 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f340cebc0 msgr2=0x7f7f341e4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:20.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:20.445+0000 7f7f3bb0f640 1 --2- 172.21.15.5:0/3165665952 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f340cebc0 0x7f7f341e4c50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7f240098c0 tx=0x7f7f24004730 comp rx=0 tx=0).stop 2024-03-24T10:52:20.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:20.445+0000 7f7f3bb0f640 1 -- 172.21.15.5:0/3165665952 shutdown_connections 2024-03-24T10:52:20.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:20.446+0000 7f7f3bb0f640 1 --2- 172.21.15.5:0/3165665952 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7f1806b860 0x7f7f1806dd20 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:20.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:20.446+0000 7f7f3bb0f640 1 --2- 172.21.15.5:0/3165665952 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f340cebc0 0x7f7f341e4c50 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:20.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:20.446+0000 7f7f3bb0f640 1 --2- 172.21.15.5:0/3165665952 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f3415aac0 0x7f7f34155590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:20.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:20.446+0000 7f7f3bb0f640 1 --2- 172.21.15.5:0/3165665952 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f3406c230 0x7f7f340b07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:20.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:20.446+0000 7f7f3bb0f640 1 -- 172.21.15.5:0/3165665952 >> 172.21.15.5:0/3165665952 conn(0x7f7f340cfb90 msgr2=0x7f7f340c8810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:20.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:20.446+0000 7f7f3bb0f640 1 -- 172.21.15.5:0/3165665952 shutdown_connections 2024-03-24T10:52:20.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:20.446+0000 7f7f3bb0f640 1 -- 172.21.15.5:0/3165665952 wait complete. 2024-03-24T10:52:21.061 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:21.061 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:22.062 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:22.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.165+0000 7fd2bdf32640 1 Processor -- start 2024-03-24T10:52:22.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.165+0000 7fd2bdf32640 1 -- start start 2024-03-24T10:52:22.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.165+0000 7fd2bdf32640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2b81543a0 0x7fd2b815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:22.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.165+0000 7fd2bdf32640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2b80cb2f0 con 0x7fd2b806d4f0 2024-03-24T10:52:22.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.165+0000 7fd2bdf32640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2b80ca7e0 con 0x7fd2b806bf50 2024-03-24T10:52:22.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.165+0000 7fd2bdf32640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd2b80c6750 con 0x7fd2b81543a0 2024-03-24T10:52:22.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.166+0000 7fd2b77fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2b81543a0 0x7fd2b815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:22.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.166+0000 7fd2b7fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fd2b806bf50 0x7fd2b806c380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:45174/0 (socket says 172.21.15.5:45174) 2024-03-24T10:52:22.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.166+0000 7fd2b7fff640 1 -- 172.21.15.5:0/2702700217 learned_addr learned my addr 172.21.15.5:0/2702700217 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:52:22.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.169+0000 7fd2b6ffd640 1 -- 172.21.15.5:0/2702700217 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3200317737 0 0) 0x7fd2b80cb2f0 con 0x7fd2b806d4f0 2024-03-24T10:52:22.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.169+0000 7fd2b6ffd640 1 -- 172.21.15.5:0/2702700217 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2a4003520 con 0x7fd2b806d4f0 2024-03-24T10:52:22.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.170+0000 7fd2b6ffd640 1 -- 172.21.15.5:0/2702700217 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2316596146 0 0) 0x7fd2b80ca7e0 con 0x7fd2b806bf50 2024-03-24T10:52:22.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.170+0000 7fd2b6ffd640 1 -- 172.21.15.5:0/2702700217 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2b80cb2f0 con 0x7fd2b806bf50 2024-03-24T10:52:22.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.170+0000 7fd2b77fe640 1 -- 172.21.15.5:0/2702700217 >> v1:172.21.15.57:6789/0 conn(0x7fd2b806bf50 legacy=0x7fd2b806c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:22.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.170+0000 7fd2b77fe640 1 -- 172.21.15.5:0/2702700217 >> v1:172.21.15.5:6789/0 conn(0x7fd2b806d4f0 legacy=0x7fd2b80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:22.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.170+0000 7fd2b77fe640 1 -- 172.21.15.5:0/2702700217 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b80cb520 con 0x7fd2b81543a0 2024-03-24T10:52:22.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.173+0000 7fd2b77fe640 1 --2- 172.21.15.5:0/2702700217 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2b81543a0 0x7fd2b815ac00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd2a80097d0 tx=0x7fd2a802b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ece4d20d11a523e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:22.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.179+0000 7fd2b6ffd640 1 -- 172.21.15.5:0/2702700217 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8030880 con 0x7fd2b81543a0 2024-03-24T10:52:22.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.179+0000 7fd2b6ffd640 1 -- 172.21.15.5:0/2702700217 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2a80309e0 con 0x7fd2b81543a0 2024-03-24T10:52:22.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.179+0000 7fd2b6ffd640 1 -- 172.21.15.5:0/2702700217 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a8030d70 con 0x7fd2b81543a0 2024-03-24T10:52:22.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.179+0000 7fd2bdf32640 1 -- 172.21.15.5:0/2702700217 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2b81543a0 msgr2=0x7fd2b815ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:22.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.179+0000 7fd2bdf32640 1 --2- 172.21.15.5:0/2702700217 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2b81543a0 0x7fd2b815ac00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd2a80097d0 tx=0x7fd2a802b9c0 comp rx=0 tx=0).stop 2024-03-24T10:52:22.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.179+0000 7fd2bdf32640 1 -- 172.21.15.5:0/2702700217 shutdown_connections 2024-03-24T10:52:22.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.179+0000 7fd2bdf32640 1 --2- 172.21.15.5:0/2702700217 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2b81543a0 0x7fd2b815ac00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:22.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.179+0000 7fd2bdf32640 1 -- 172.21.15.5:0/2702700217 >> 172.21.15.5:0/2702700217 conn(0x7fd2b8072ef0 msgr2=0x7fd2b8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:22.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.179+0000 7fd2bdf32640 1 -- 172.21.15.5:0/2702700217 shutdown_connections 2024-03-24T10:52:22.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.179+0000 7fd2bdf32640 1 -- 172.21.15.5:0/2702700217 wait complete. 2024-03-24T10:52:22.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.181+0000 7fd2bdf32640 1 Processor -- start 2024-03-24T10:52:22.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.181+0000 7fd2bdf32640 1 -- start start 2024-03-24T10:52:22.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.181+0000 7fd2bdf32640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2b806bf50 0x7fd2b80c6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:22.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.181+0000 7fd2bdf32640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2b806d4f0 0x7fd2b81f2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:22.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.181+0000 7fd2bdf32640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2b81543a0 0x7fd2b81f1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:22.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.181+0000 7fd2bdf32640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b80c7850 con 0x7fd2b81543a0 2024-03-24T10:52:22.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.181+0000 7fd2bdf32640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b80c7410 con 0x7fd2b806bf50 2024-03-24T10:52:22.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.181+0000 7fd2bdf32640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd2b81f3900 con 0x7fd2b806d4f0 2024-03-24T10:52:22.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.182+0000 7fd2b7fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2b81543a0 0x7fd2b81f1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:22.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.182+0000 7fd2aeffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2b806d4f0 0x7fd2b81f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:22.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.182+0000 7fd2b77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2b806bf50 0x7fd2b80c6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:22.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.182+0000 7fd2b7fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2b81543a0 0x7fd2b81f1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54622/0 (socket says 172.21.15.5:54622) 2024-03-24T10:52:22.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.182+0000 7fd2b7fff640 1 -- 172.21.15.5:0/4286496022 learned_addr learned my addr 172.21.15.5:0/4286496022 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:22.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.186+0000 7fd2b7fff640 1 -- 172.21.15.5:0/4286496022 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2b806d4f0 msgr2=0x7fd2b81f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:22.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.186+0000 7fd2b7fff640 1 --2- 172.21.15.5:0/4286496022 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2b806d4f0 0x7fd2b81f2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:22.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.186+0000 7fd2b7fff640 1 -- 172.21.15.5:0/4286496022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2b806bf50 msgr2=0x7fd2b80c6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:22.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.186+0000 7fd2b7fff640 1 --2- 172.21.15.5:0/4286496022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2b806bf50 0x7fd2b80c6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:22.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.186+0000 7fd2b7fff640 1 -- 172.21.15.5:0/4286496022 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a8009400 con 0x7fd2b81543a0 2024-03-24T10:52:22.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.188+0000 7fd2b7fff640 1 --2- 172.21.15.5:0/4286496022 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2b81543a0 0x7fd2b81f1ef0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd2b000caa0 tx=0x7fd2b000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:22.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.191+0000 7fd2b4ff9640 1 -- 172.21.15.5:0/4286496022 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b000ac00 con 0x7fd2b81543a0 2024-03-24T10:52:22.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.192+0000 7fd2b4ff9640 1 -- 172.21.15.5:0/4286496022 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2b000ad60 con 0x7fd2b81543a0 2024-03-24T10:52:22.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.192+0000 7fd2bdf32640 1 -- 172.21.15.5:0/4286496022 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2b80c7b10 con 0x7fd2b81543a0 2024-03-24T10:52:22.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.192+0000 7fd2b4ff9640 1 -- 172.21.15.5:0/4286496022 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b0012620 con 0x7fd2b81543a0 2024-03-24T10:52:22.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.192+0000 7fd2bdf32640 1 -- 172.21.15.5:0/4286496022 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2b81f19d0 con 0x7fd2b81543a0 2024-03-24T10:52:22.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.193+0000 7fd2bdf32640 1 -- 172.21.15.5:0/4286496022 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2b81f1bc0 con 0x7fd2b81543a0 2024-03-24T10:52:22.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.212+0000 7fd2b4ff9640 1 -- 172.21.15.5:0/4286496022 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd2b0007070 con 0x7fd2b81543a0 2024-03-24T10:52:22.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.212+0000 7fd2b4ff9640 1 --2- 172.21.15.5:0/4286496022 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd29806b890 0x7fd29806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:22.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.212+0000 7fd2b77fe640 1 --2- 172.21.15.5:0/4286496022 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd29806b890 0x7fd29806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:22.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.213+0000 7fd2b77fe640 1 --2- 172.21.15.5:0/4286496022 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd29806b890 0x7fd29806dd50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8036000 tx=0x7fd2a8008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:22.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.476+0000 7fd2b4ff9640 1 -- 172.21.15.5:0/4286496022 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fd2b0058d70 con 0x7fd2b81543a0 2024-03-24T10:52:22.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.476+0000 7fd2b4ff9640 1 -- 172.21.15.5:0/4286496022 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd2b0055010 con 0x7fd2b81543a0 2024-03-24T10:52:22.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.723+0000 7fd2bdf32640 1 -- 172.21.15.5:0/4286496022 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2b80681d0 con 0x7fd2b81543a0 2024-03-24T10:52:22.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.736+0000 7fd2b4ff9640 1 -- 172.21.15.5:0/4286496022 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fd2b0017020 con 0x7fd2b81543a0 2024-03-24T10:52:22.736 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:22.736 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:22.736 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:22.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.738+0000 7fd2bdf32640 1 -- 172.21.15.5:0/4286496022 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd29806b890 msgr2=0x7fd29806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:22.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.738+0000 7fd2bdf32640 1 --2- 172.21.15.5:0/4286496022 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd29806b890 0x7fd29806dd50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8036000 tx=0x7fd2a8008030 comp rx=0 tx=0).stop 2024-03-24T10:52:22.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.738+0000 7fd2bdf32640 1 -- 172.21.15.5:0/4286496022 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2b81543a0 msgr2=0x7fd2b81f1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:22.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.738+0000 7fd2bdf32640 1 --2- 172.21.15.5:0/4286496022 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2b81543a0 0x7fd2b81f1ef0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd2b000caa0 tx=0x7fd2b000cf60 comp rx=0 tx=0).stop 2024-03-24T10:52:22.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.738+0000 7fd2bdf32640 1 -- 172.21.15.5:0/4286496022 shutdown_connections 2024-03-24T10:52:22.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.738+0000 7fd2bdf32640 1 --2- 172.21.15.5:0/4286496022 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd29806b890 0x7fd29806dd50 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:22.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.738+0000 7fd2bdf32640 1 --2- 172.21.15.5:0/4286496022 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd2b81543a0 0x7fd2b81f1ef0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:22.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.738+0000 7fd2bdf32640 1 --2- 172.21.15.5:0/4286496022 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd2b806d4f0 0x7fd2b81f2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:22.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.738+0000 7fd2bdf32640 1 --2- 172.21.15.5:0/4286496022 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2b806bf50 0x7fd2b80c6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:22.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.738+0000 7fd2bdf32640 1 -- 172.21.15.5:0/4286496022 >> 172.21.15.5:0/4286496022 conn(0x7fd2b8072ef0 msgr2=0x7fd2b8152b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:22.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.738+0000 7fd2bdf32640 1 -- 172.21.15.5:0/4286496022 shutdown_connections 2024-03-24T10:52:22.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:22.738+0000 7fd2bdf32640 1 -- 172.21.15.5:0/4286496022 wait complete. 2024-03-24T10:52:23.356 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:23.357 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:24.357 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:24.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.459+0000 7f5a50634640 1 Processor -- start 2024-03-24T10:52:24.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.459+0000 7f5a50634640 1 -- start start 2024-03-24T10:52:24.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.460+0000 7f5a50634640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a4815aac0 0x7f5a4815aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:24.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.460+0000 7f5a50634640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f5a4806c230 0x7f5a4806c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:24.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.460+0000 7f5a50634640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a4806cd50 con 0x7f5a480cebc0 2024-03-24T10:52:24.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.460+0000 7f5a50634640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f5a480ce790 con 0x7f5a4806c230 2024-03-24T10:52:24.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.460+0000 7f5a50634640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f5a4815b3e0 con 0x7f5a4815aac0 2024-03-24T10:52:24.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.461+0000 7f5a4e3d0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a4815aac0 0x7f5a4815aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:24.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.461+0000 7f5a4ebd1640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f5a480cebc0 0x7f5a480ca5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:35786/0 (socket says 172.21.15.5:35786) 2024-03-24T10:52:24.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.461+0000 7f5a4ebd1640 1 -- 172.21.15.5:0/1065973316 learned_addr learned my addr 172.21.15.5:0/1065973316 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:52:24.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.461+0000 7f5a4dbcf640 1 --2- 172.21.15.5:0/1065973316 >> v2:172.21.15.5:3300/0 conn(0x7f5a4806c230 0x7f5a4806c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:24.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.465+0000 7f5a4e3d0640 1 -- 172.21.15.5:0/1065973316 >> v2:172.21.15.5:3300/0 conn(0x7f5a4806c230 msgr2=0x7f5a4806c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:24.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.465+0000 7f5a4e3d0640 1 --2- 172.21.15.5:0/1065973316 >> v2:172.21.15.5:3300/0 conn(0x7f5a4806c230 0x7f5a4806c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:24.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.465+0000 7f5a4e3d0640 1 -- 172.21.15.5:0/1065973316 >> v1:172.21.15.5:6789/0 conn(0x7f5a480cebc0 legacy=0x7f5a480ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:24.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.465+0000 7f5a4e3d0640 1 -- 172.21.15.5:0/1065973316 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a480cdb10 con 0x7f5a4815aac0 2024-03-24T10:52:24.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.467+0000 7f5a4e3d0640 1 --2- 172.21.15.5:0/1065973316 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a4815aac0 0x7f5a4815aea0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5a440097a0 tx=0x7f5a4402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4990726e2e77aa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:24.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.472+0000 7f5a4d3ce640 1 -- 172.21.15.5:0/1065973316 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a44030610 con 0x7f5a4815aac0 2024-03-24T10:52:24.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.472+0000 7f5a4d3ce640 1 -- 172.21.15.5:0/1065973316 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a44030770 con 0x7f5a4815aac0 2024-03-24T10:52:24.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.472+0000 7f5a4d3ce640 1 -- 172.21.15.5:0/1065973316 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a44030b00 con 0x7f5a4815aac0 2024-03-24T10:52:24.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.472+0000 7f5a50634640 1 -- 172.21.15.5:0/1065973316 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a4815aac0 msgr2=0x7f5a4815aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:24.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.472+0000 7f5a50634640 1 --2- 172.21.15.5:0/1065973316 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a4815aac0 0x7f5a4815aea0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5a440097a0 tx=0x7f5a4402b960 comp rx=0 tx=0).stop 2024-03-24T10:52:24.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.473+0000 7f5a50634640 1 -- 172.21.15.5:0/1065973316 shutdown_connections 2024-03-24T10:52:24.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.473+0000 7f5a50634640 1 --2- 172.21.15.5:0/1065973316 >> v2:172.21.15.5:3300/0 conn(0x7f5a4806c230 0x7f5a4806c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:24.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.473+0000 7f5a50634640 1 --2- 172.21.15.5:0/1065973316 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a4815aac0 0x7f5a4815aea0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:24.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.473+0000 7f5a50634640 1 -- 172.21.15.5:0/1065973316 >> 172.21.15.5:0/1065973316 conn(0x7f5a480cfb90 msgr2=0x7f5a480cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:24.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.473+0000 7f5a50634640 1 -- 172.21.15.5:0/1065973316 shutdown_connections 2024-03-24T10:52:24.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.473+0000 7f5a50634640 1 -- 172.21.15.5:0/1065973316 wait complete. 2024-03-24T10:52:24.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.474+0000 7f5a50634640 1 Processor -- start 2024-03-24T10:52:24.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.474+0000 7f5a50634640 1 -- start start 2024-03-24T10:52:24.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.474+0000 7f5a50634640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a4806c230 0x7f5a4815cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:24.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.474+0000 7f5a50634640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a480cebc0 0x7f5a48159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:24.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.475+0000 7f5a50634640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a4815aac0 0x7f5a4815d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:24.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.475+0000 7f5a50634640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4815d130 con 0x7f5a4815aac0 2024-03-24T10:52:24.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.475+0000 7f5a50634640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4814f700 con 0x7f5a480cebc0 2024-03-24T10:52:24.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.475+0000 7f5a50634640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f5a4814e680 con 0x7f5a4806c230 2024-03-24T10:52:24.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.475+0000 7f5a4ebd1640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a4815aac0 0x7f5a4815d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:24.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.475+0000 7f5a4e3d0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a4806c230 0x7f5a4815cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:24.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.475+0000 7f5a4dbcf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a480cebc0 0x7f5a48159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:24.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.475+0000 7f5a4ebd1640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a4815aac0 0x7f5a4815d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:53730/0 (socket says 172.21.15.5:53730) 2024-03-24T10:52:24.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.475+0000 7f5a4ebd1640 1 -- 172.21.15.5:0/711750698 learned_addr learned my addr 172.21.15.5:0/711750698 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:24.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.480+0000 7f5a4ebd1640 1 -- 172.21.15.5:0/711750698 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a4806c230 msgr2=0x7f5a4815cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:24.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.480+0000 7f5a4ebd1640 1 --2- 172.21.15.5:0/711750698 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a4806c230 0x7f5a4815cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:24.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.480+0000 7f5a4ebd1640 1 -- 172.21.15.5:0/711750698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a480cebc0 msgr2=0x7f5a48159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:24.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.480+0000 7f5a4ebd1640 1 --2- 172.21.15.5:0/711750698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a480cebc0 0x7f5a48159ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:24.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.480+0000 7f5a4ebd1640 1 -- 172.21.15.5:0/711750698 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a44009400 con 0x7f5a4815aac0 2024-03-24T10:52:24.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.484+0000 7f5a4ebd1640 1 --2- 172.21.15.5:0/711750698 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a4815aac0 0x7f5a4815d3d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5a4000cbc0 tx=0x7f5a4000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:24.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.489+0000 7f5a3effd640 1 -- 172.21.15.5:0/711750698 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4000ad40 con 0x7f5a4815aac0 2024-03-24T10:52:24.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.489+0000 7f5a3effd640 1 -- 172.21.15.5:0/711750698 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a40012400 con 0x7f5a4815aac0 2024-03-24T10:52:24.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.489+0000 7f5a3effd640 1 -- 172.21.15.5:0/711750698 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4000aea0 con 0x7f5a4815aac0 2024-03-24T10:52:24.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.489+0000 7f5a50634640 1 -- 172.21.15.5:0/711750698 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a481f40c0 con 0x7f5a4815aac0 2024-03-24T10:52:24.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.489+0000 7f5a50634640 1 -- 172.21.15.5:0/711750698 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a4814ec20 con 0x7f5a4815aac0 2024-03-24T10:52:24.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.490+0000 7f5a50634640 1 -- 172.21.15.5:0/711750698 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a480ca1e0 con 0x7f5a4815aac0 2024-03-24T10:52:24.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.766+0000 7f5a3effd640 1 -- 172.21.15.5:0/711750698 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5a40007070 con 0x7f5a4815aac0 2024-03-24T10:52:24.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.767+0000 7f5a3effd640 1 --2- 172.21.15.5:0/711750698 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5a2c06b890 0x7f5a2c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:24.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.767+0000 7f5a4e3d0640 1 --2- 172.21.15.5:0/711750698 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5a2c06b890 0x7f5a2c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:24.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.767+0000 7f5a4e3d0640 1 --2- 172.21.15.5:0/711750698 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5a2c06b890 0x7f5a2c06dd50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5a44036000 tx=0x7f5a4403b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:24.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.791+0000 7f5a3effd640 1 -- 172.21.15.5:0/711750698 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f5a40058db0 con 0x7f5a4815aac0 2024-03-24T10:52:24.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:24.791+0000 7f5a3effd640 1 -- 172.21.15.5:0/711750698 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5a40055050 con 0x7f5a4815aac0 2024-03-24T10:52:25.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:25.026+0000 7f5a50634640 1 -- 172.21.15.5:0/711750698 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a480681d0 con 0x7f5a4815aac0 2024-03-24T10:52:25.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:25.040+0000 7f5a3effd640 1 -- 172.21.15.5:0/711750698 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f5a40058710 con 0x7f5a4815aac0 2024-03-24T10:52:25.041 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:25.041 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:25.042 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:25.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:25.042+0000 7f5a50634640 1 -- 172.21.15.5:0/711750698 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5a2c06b890 msgr2=0x7f5a2c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:25.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:25.042+0000 7f5a50634640 1 --2- 172.21.15.5:0/711750698 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5a2c06b890 0x7f5a2c06dd50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5a44036000 tx=0x7f5a4403b040 comp rx=0 tx=0).stop 2024-03-24T10:52:25.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:25.042+0000 7f5a50634640 1 -- 172.21.15.5:0/711750698 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a4815aac0 msgr2=0x7f5a4815d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:25.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:25.042+0000 7f5a50634640 1 --2- 172.21.15.5:0/711750698 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a4815aac0 0x7f5a4815d3d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5a4000cbc0 tx=0x7f5a4000a430 comp rx=0 tx=0).stop 2024-03-24T10:52:25.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:25.042+0000 7f5a50634640 1 -- 172.21.15.5:0/711750698 shutdown_connections 2024-03-24T10:52:25.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:25.042+0000 7f5a50634640 1 --2- 172.21.15.5:0/711750698 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5a2c06b890 0x7f5a2c06dd50 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:25.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:25.042+0000 7f5a50634640 1 --2- 172.21.15.5:0/711750698 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5a4815aac0 0x7f5a4815d3d0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:25.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:25.042+0000 7f5a50634640 1 --2- 172.21.15.5:0/711750698 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a480cebc0 0x7f5a48159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:25.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:25.042+0000 7f5a50634640 1 --2- 172.21.15.5:0/711750698 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5a4806c230 0x7f5a4815cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:25.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:25.042+0000 7f5a50634640 1 -- 172.21.15.5:0/711750698 >> 172.21.15.5:0/711750698 conn(0x7f5a480cfb90 msgr2=0x7f5a480cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:25.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:25.042+0000 7f5a50634640 1 -- 172.21.15.5:0/711750698 shutdown_connections 2024-03-24T10:52:25.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:25.042+0000 7f5a50634640 1 -- 172.21.15.5:0/711750698 wait complete. 2024-03-24T10:52:25.658 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:25.658 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:26.659 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:26.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.761+0000 7f1b4b215640 1 Processor -- start 2024-03-24T10:52:26.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.761+0000 7f1b4b215640 1 -- start start 2024-03-24T10:52:26.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.761+0000 7f1b4b215640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1b4415ac00 0x7f1b440cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:26.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.761+0000 7f1b4b215640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b440ccaa0 0x7f1b4406bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:26.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.761+0000 7f1b4b215640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b440cb2f0 con 0x7f1b441543a0 2024-03-24T10:52:26.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.762+0000 7f1b4b215640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f1b440c6750 con 0x7f1b4415ac00 2024-03-24T10:52:26.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.762+0000 7f1b4b215640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1b440cceb0 con 0x7f1b440ccaa0 2024-03-24T10:52:26.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.762+0000 7f1b497b2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b440ccaa0 0x7f1b4406bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:26.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.763+0000 7f1b48fb1640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f1b441543a0 0x7f1b4406d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:35796/0 (socket says 172.21.15.5:35796) 2024-03-24T10:52:26.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.763+0000 7f1b48fb1640 1 -- 172.21.15.5:0/3623673947 learned_addr learned my addr 172.21.15.5:0/3623673947 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:52:26.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.763+0000 7f1b497b2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b440ccaa0 0x7f1b4406bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:36046/0 (socket says 172.21.15.5:36046) 2024-03-24T10:52:26.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.763+0000 7f1b43fff640 1 --2- 172.21.15.5:0/3623673947 >> v2:172.21.15.5:3300/0 conn(0x7f1b4415ac00 0x7f1b440cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:26.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.765+0000 7f1b497b2640 1 -- 172.21.15.5:0/3623673947 >> v2:172.21.15.5:3300/0 conn(0x7f1b4415ac00 msgr2=0x7f1b440cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:26.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.765+0000 7f1b497b2640 1 --2- 172.21.15.5:0/3623673947 >> v2:172.21.15.5:3300/0 conn(0x7f1b4415ac00 0x7f1b440cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:26.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.765+0000 7f1b497b2640 1 -- 172.21.15.5:0/3623673947 >> v1:172.21.15.5:6789/0 conn(0x7f1b441543a0 legacy=0x7f1b4406d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:26.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.765+0000 7f1b497b2640 1 -- 172.21.15.5:0/3623673947 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b4415b070 con 0x7f1b440ccaa0 2024-03-24T10:52:26.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.767+0000 7f1b497b2640 1 --2- 172.21.15.5:0/3623673947 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b440ccaa0 0x7f1b4406bf50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009d90 tx=0x7f1b3c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1a7953fdbd44c0b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:26.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.772+0000 7f1b437fe640 1 -- 172.21.15.5:0/3623673947 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c02cc50 con 0x7f1b440ccaa0 2024-03-24T10:52:26.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.773+0000 7f1b437fe640 1 -- 172.21.15.5:0/3623673947 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b3c02cdb0 con 0x7f1b440ccaa0 2024-03-24T10:52:26.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.773+0000 7f1b437fe640 1 -- 172.21.15.5:0/3623673947 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c034660 con 0x7f1b440ccaa0 2024-03-24T10:52:26.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.773+0000 7f1b4b215640 1 -- 172.21.15.5:0/3623673947 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b440ccaa0 msgr2=0x7f1b4406bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:26.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.773+0000 7f1b4b215640 1 --2- 172.21.15.5:0/3623673947 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b440ccaa0 0x7f1b4406bf50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009d90 tx=0x7f1b3c02bea0 comp rx=0 tx=0).stop 2024-03-24T10:52:26.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.773+0000 7f1b4b215640 1 -- 172.21.15.5:0/3623673947 shutdown_connections 2024-03-24T10:52:26.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.773+0000 7f1b4b215640 1 --2- 172.21.15.5:0/3623673947 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b440ccaa0 0x7f1b4406bf50 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:26.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.773+0000 7f1b4b215640 1 --2- 172.21.15.5:0/3623673947 >> v2:172.21.15.5:3300/0 conn(0x7f1b4415ac00 0x7f1b440cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:26.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.773+0000 7f1b4b215640 1 -- 172.21.15.5:0/3623673947 >> 172.21.15.5:0/3623673947 conn(0x7f1b44072ef0 msgr2=0x7f1b44061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:26.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.773+0000 7f1b4b215640 1 -- 172.21.15.5:0/3623673947 shutdown_connections 2024-03-24T10:52:26.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.773+0000 7f1b4b215640 1 -- 172.21.15.5:0/3623673947 wait complete. 2024-03-24T10:52:26.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.774+0000 7f1b4b215640 1 Processor -- start 2024-03-24T10:52:26.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.775+0000 7f1b4b215640 1 -- start start 2024-03-24T10:52:26.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.775+0000 7f1b4b215640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b440ccaa0 0x7f1b440c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:26.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.775+0000 7f1b4b215640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b441543a0 0x7f1b44158970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:26.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.775+0000 7f1b4b215640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b4415ac00 0x7f1b44158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:26.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.775+0000 7f1b4b215640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f1b44157f60 con 0x7f1b4415ac00 2024-03-24T10:52:26.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.775+0000 7f1b4b215640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1b440ce940 con 0x7f1b440ccaa0 2024-03-24T10:52:26.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.775+0000 7f1b4b215640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1b440cec20 con 0x7f1b441543a0 2024-03-24T10:52:26.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.776+0000 7f1b43fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b441543a0 0x7f1b44158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:26.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.776+0000 7f1b497b2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b4415ac00 0x7f1b44158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:26.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.776+0000 7f1b48fb1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b440ccaa0 0x7f1b440c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:26.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.776+0000 7f1b43fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b441543a0 0x7f1b44158970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:36052/0 (socket says 172.21.15.5:36052) 2024-03-24T10:52:26.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.776+0000 7f1b43fff640 1 -- 172.21.15.5:0/1498876880 learned_addr learned my addr 172.21.15.5:0/1498876880 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:26.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.781+0000 7f1b43fff640 1 -- 172.21.15.5:0/1498876880 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b440ccaa0 msgr2=0x7f1b440c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:26.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.781+0000 7f1b43fff640 1 --2- 172.21.15.5:0/1498876880 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b440ccaa0 0x7f1b440c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:26.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.781+0000 7f1b43fff640 1 -- 172.21.15.5:0/1498876880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b4415ac00 msgr2=0x7f1b44158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:26.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.781+0000 7f1b43fff640 1 --2- 172.21.15.5:0/1498876880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b4415ac00 0x7f1b44158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:26.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.781+0000 7f1b43fff640 1 -- 172.21.15.5:0/1498876880 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3c009a70 con 0x7f1b441543a0 2024-03-24T10:52:26.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.785+0000 7f1b43fff640 1 --2- 172.21.15.5:0/1498876880 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b441543a0 0x7f1b44158970 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1b3000e870 tx=0x7f1b3000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:26.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.790+0000 7f1b417fa640 1 -- 172.21.15.5:0/1498876880 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b300094f0 con 0x7f1b441543a0 2024-03-24T10:52:26.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.790+0000 7f1b417fa640 1 -- 172.21.15.5:0/1498876880 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b3000c940 con 0x7f1b441543a0 2024-03-24T10:52:26.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.790+0000 7f1b417fa640 1 -- 172.21.15.5:0/1498876880 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3000ccf0 con 0x7f1b441543a0 2024-03-24T10:52:26.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.790+0000 7f1b4b215640 1 -- 172.21.15.5:0/1498876880 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b440cef00 con 0x7f1b441543a0 2024-03-24T10:52:26.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.790+0000 7f1b4b215640 1 -- 172.21.15.5:0/1498876880 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b440b4dd0 con 0x7f1b441543a0 2024-03-24T10:52:26.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.791+0000 7f1b4b215640 1 -- 172.21.15.5:0/1498876880 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b4406d060 con 0x7f1b441543a0 2024-03-24T10:52:26.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.817+0000 7f1b417fa640 1 -- 172.21.15.5:0/1498876880 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1b30013470 con 0x7f1b441543a0 2024-03-24T10:52:26.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.817+0000 7f1b417fa640 1 --2- 172.21.15.5:0/1498876880 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1b2806b810 0x7f1b2806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:26.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.817+0000 7f1b48fb1640 1 --2- 172.21.15.5:0/1498876880 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1b2806b810 0x7f1b2806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:26.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:26.818+0000 7f1b48fb1640 1 --2- 172.21.15.5:0/1498876880 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1b2806b810 0x7f1b2806dcd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1b38007fd0 tx=0x7f1b38007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:27.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:27.081+0000 7f1b417fa640 1 -- 172.21.15.5:0/1498876880 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f1b30059210 con 0x7f1b441543a0 2024-03-24T10:52:27.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:27.100+0000 7f1b417fa640 1 -- 172.21.15.5:0/1498876880 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1b300554b0 con 0x7f1b441543a0 2024-03-24T10:52:27.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:27.321+0000 7f1b4b215640 1 -- 172.21.15.5:0/1498876880 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b440681d0 con 0x7f1b441543a0 2024-03-24T10:52:27.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:27.342+0000 7f1b417fa640 1 -- 172.21.15.5:0/1498876880 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f1b30021090 con 0x7f1b441543a0 2024-03-24T10:52:27.344 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:27.344 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:27.345 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:27.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:27.344+0000 7f1b4b215640 1 -- 172.21.15.5:0/1498876880 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1b2806b810 msgr2=0x7f1b2806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:27.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:27.344+0000 7f1b4b215640 1 --2- 172.21.15.5:0/1498876880 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1b2806b810 0x7f1b2806dcd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1b38007fd0 tx=0x7f1b38007e80 comp rx=0 tx=0).stop 2024-03-24T10:52:27.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:27.344+0000 7f1b4b215640 1 -- 172.21.15.5:0/1498876880 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b441543a0 msgr2=0x7f1b44158970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:27.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:27.344+0000 7f1b4b215640 1 --2- 172.21.15.5:0/1498876880 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b441543a0 0x7f1b44158970 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1b3000e870 tx=0x7f1b3000ed30 comp rx=0 tx=0).stop 2024-03-24T10:52:27.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:27.344+0000 7f1b4b215640 1 -- 172.21.15.5:0/1498876880 shutdown_connections 2024-03-24T10:52:27.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:27.344+0000 7f1b4b215640 1 --2- 172.21.15.5:0/1498876880 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1b2806b810 0x7f1b2806dcd0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:27.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:27.344+0000 7f1b4b215640 1 --2- 172.21.15.5:0/1498876880 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b4415ac00 0x7f1b44158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:27.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:27.344+0000 7f1b4b215640 1 --2- 172.21.15.5:0/1498876880 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b441543a0 0x7f1b44158970 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:27.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:27.344+0000 7f1b4b215640 1 --2- 172.21.15.5:0/1498876880 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b440ccaa0 0x7f1b440c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:27.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:27.344+0000 7f1b4b215640 1 -- 172.21.15.5:0/1498876880 >> 172.21.15.5:0/1498876880 conn(0x7f1b44072ef0 msgr2=0x7f1b4406c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:27.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:27.345+0000 7f1b4b215640 1 -- 172.21.15.5:0/1498876880 shutdown_connections 2024-03-24T10:52:27.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:27.345+0000 7f1b4b215640 1 -- 172.21.15.5:0/1498876880 wait complete. 2024-03-24T10:52:27.963 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:27.963 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:28.963 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:29.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.065+0000 7f613d823640 1 Processor -- start 2024-03-24T10:52:29.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.065+0000 7f613d823640 1 -- start start 2024-03-24T10:52:29.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.066+0000 7f613d823640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61380c92f0 0x7f61380cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:29.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.066+0000 7f613d823640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f613806c750 0x7f6138164c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:29.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.066+0000 7f613d823640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61380ce350 con 0x7f61380c96d0 2024-03-24T10:52:29.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.066+0000 7f613d823640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f613815e320 con 0x7f613806c750 2024-03-24T10:52:29.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.066+0000 7f613d823640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f613815e7b0 con 0x7f61380c92f0 2024-03-24T10:52:29.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.067+0000 7f6136ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61380c92f0 0x7f61380cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:29.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.067+0000 7f61377fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f613806c750 0x7f6138164c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:29.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.067+0000 7f61367fc640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f61380c96d0 0x7f613815ad50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:35800/0 (socket says 172.21.15.5:35800) 2024-03-24T10:52:29.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.067+0000 7f61367fc640 1 -- 172.21.15.5:0/2173568885 learned_addr learned my addr 172.21.15.5:0/2173568885 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:52:29.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.069+0000 7f6136ffd640 1 -- 172.21.15.5:0/2173568885 >> v2:172.21.15.5:3300/0 conn(0x7f613806c750 msgr2=0x7f6138164c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:29.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.069+0000 7f6136ffd640 1 --2- 172.21.15.5:0/2173568885 >> v2:172.21.15.5:3300/0 conn(0x7f613806c750 0x7f6138164c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:29.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.069+0000 7f6136ffd640 1 -- 172.21.15.5:0/2173568885 >> v1:172.21.15.5:6789/0 conn(0x7f61380c96d0 legacy=0x7f613815ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:29.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.069+0000 7f6136ffd640 1 -- 172.21.15.5:0/2173568885 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61380ce070 con 0x7f61380c92f0 2024-03-24T10:52:29.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.072+0000 7f6136ffd640 1 --2- 172.21.15.5:0/2173568885 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61380c92f0 0x7f61380cd9c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f61280097a0 tx=0x7f612802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6cf6f227dd896c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:29.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.077+0000 7f6135ffb640 1 -- 172.21.15.5:0/2173568885 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128031830 con 0x7f61380c92f0 2024-03-24T10:52:29.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.077+0000 7f6135ffb640 1 -- 172.21.15.5:0/2173568885 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6128031990 con 0x7f61380c92f0 2024-03-24T10:52:29.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.077+0000 7f6135ffb640 1 -- 172.21.15.5:0/2173568885 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128031d20 con 0x7f61380c92f0 2024-03-24T10:52:29.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.078+0000 7f613d823640 1 -- 172.21.15.5:0/2173568885 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61380c92f0 msgr2=0x7f61380cd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:29.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.078+0000 7f613d823640 1 --2- 172.21.15.5:0/2173568885 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61380c92f0 0x7f61380cd9c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f61280097a0 tx=0x7f612802b8d0 comp rx=0 tx=0).stop 2024-03-24T10:52:29.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.078+0000 7f613d823640 1 -- 172.21.15.5:0/2173568885 shutdown_connections 2024-03-24T10:52:29.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.078+0000 7f613d823640 1 --2- 172.21.15.5:0/2173568885 >> v2:172.21.15.5:3300/0 conn(0x7f613806c750 0x7f6138164c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:29.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.078+0000 7f613d823640 1 --2- 172.21.15.5:0/2173568885 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61380c92f0 0x7f61380cd9c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:29.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.078+0000 7f613d823640 1 -- 172.21.15.5:0/2173568885 >> 172.21.15.5:0/2173568885 conn(0x7f613806f820 msgr2=0x7f6138061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:29.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.078+0000 7f613d823640 1 -- 172.21.15.5:0/2173568885 shutdown_connections 2024-03-24T10:52:29.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.078+0000 7f613d823640 1 -- 172.21.15.5:0/2173568885 wait complete. 2024-03-24T10:52:29.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.080+0000 7f613d823640 1 Processor -- start 2024-03-24T10:52:29.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.080+0000 7f613d823640 1 -- start start 2024-03-24T10:52:29.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.081+0000 7f613d823640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f613806c750 0x7f6138152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:29.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.081+0000 7f613d823640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61380c92f0 0x7f61380af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:29.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.081+0000 7f613d823640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61380c96d0 0x7f61380add00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:29.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.081+0000 7f613d823640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f61381e3ff0 con 0x7f613806c750 2024-03-24T10:52:29.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.081+0000 7f613d823640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f61380cc8e0 con 0x7f61380c92f0 2024-03-24T10:52:29.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.081+0000 7f613d823640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f61380cb9f0 con 0x7f61380c96d0 2024-03-24T10:52:29.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.082+0000 7f61367fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61380c92f0 0x7f61380af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:29.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.082+0000 7f6136ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f613806c750 0x7f6138152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:29.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.082+0000 7f61377fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61380c96d0 0x7f61380add00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:29.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.082+0000 7f61367fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61380c92f0 0x7f61380af8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:60566/0 (socket says 172.21.15.5:60566) 2024-03-24T10:52:29.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.082+0000 7f61367fc640 1 -- 172.21.15.5:0/2999407931 learned_addr learned my addr 172.21.15.5:0/2999407931 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:29.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.086+0000 7f61367fc640 1 -- 172.21.15.5:0/2999407931 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61380c96d0 msgr2=0x7f61380add00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:29.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.086+0000 7f61367fc640 1 --2- 172.21.15.5:0/2999407931 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61380c96d0 0x7f61380add00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:29.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.086+0000 7f61367fc640 1 -- 172.21.15.5:0/2999407931 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f613806c750 msgr2=0x7f6138152bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:29.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.086+0000 7f61367fc640 1 --2- 172.21.15.5:0/2999407931 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f613806c750 0x7f6138152bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:29.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.086+0000 7f61367fc640 1 -- 172.21.15.5:0/2999407931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6128009400 con 0x7f61380c92f0 2024-03-24T10:52:29.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.088+0000 7f61367fc640 1 --2- 172.21.15.5:0/2999407931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61380c92f0 0x7f61380af8b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f612c00c400 tx=0x7f612c00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:29.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.093+0000 7f611ffff640 1 -- 172.21.15.5:0/2999407931 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612c005ce0 con 0x7f61380c92f0 2024-03-24T10:52:29.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.093+0000 7f611ffff640 1 -- 172.21.15.5:0/2999407931 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f612c005e40 con 0x7f61380c92f0 2024-03-24T10:52:29.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.093+0000 7f613d823640 1 -- 172.21.15.5:0/2999407931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61380cbcd0 con 0x7f61380c92f0 2024-03-24T10:52:29.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.093+0000 7f611ffff640 1 -- 172.21.15.5:0/2999407931 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612c0072d0 con 0x7f61380c92f0 2024-03-24T10:52:29.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.093+0000 7f613d823640 1 -- 172.21.15.5:0/2999407931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61380cc290 con 0x7f61380c92f0 2024-03-24T10:52:29.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.095+0000 7f613d823640 1 -- 172.21.15.5:0/2999407931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61380681d0 con 0x7f61380c92f0 2024-03-24T10:52:29.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.119+0000 7f611ffff640 1 -- 172.21.15.5:0/2999407931 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f612c007430 con 0x7f61380c92f0 2024-03-24T10:52:29.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.120+0000 7f611ffff640 1 --2- 172.21.15.5:0/2999407931 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f612406b890 0x7f612406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:29.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.120+0000 7f6136ffd640 1 --2- 172.21.15.5:0/2999407931 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f612406b890 0x7f612406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:29.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.120+0000 7f6136ffd640 1 --2- 172.21.15.5:0/2999407931 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f612406b890 0x7f612406dd50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6128036000 tx=0x7f612803c040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:29.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.405+0000 7f611ffff640 1 -- 172.21.15.5:0/2999407931 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f612c058430 con 0x7f61380c92f0 2024-03-24T10:52:29.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.416+0000 7f611ffff640 1 -- 172.21.15.5:0/2999407931 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f612c0546d0 con 0x7f61380c92f0 2024-03-24T10:52:29.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.678+0000 7f613d823640 1 -- 172.21.15.5:0/2999407931 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61380c8590 con 0x7f61380c92f0 2024-03-24T10:52:29.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.695+0000 7f611ffff640 1 -- 172.21.15.5:0/2999407931 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f612c017020 con 0x7f61380c92f0 2024-03-24T10:52:29.696 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:29.696 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:29.696 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:29.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.697+0000 7f613d823640 1 -- 172.21.15.5:0/2999407931 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f612406b890 msgr2=0x7f612406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:29.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.697+0000 7f613d823640 1 --2- 172.21.15.5:0/2999407931 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f612406b890 0x7f612406dd50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6128036000 tx=0x7f612803c040 comp rx=0 tx=0).stop 2024-03-24T10:52:29.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.697+0000 7f613d823640 1 -- 172.21.15.5:0/2999407931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61380c92f0 msgr2=0x7f61380af8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:29.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.697+0000 7f613d823640 1 --2- 172.21.15.5:0/2999407931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61380c92f0 0x7f61380af8b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f612c00c400 tx=0x7f612c00c8c0 comp rx=0 tx=0).stop 2024-03-24T10:52:29.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.697+0000 7f613d823640 1 -- 172.21.15.5:0/2999407931 shutdown_connections 2024-03-24T10:52:29.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.697+0000 7f613d823640 1 --2- 172.21.15.5:0/2999407931 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f612406b890 0x7f612406dd50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:29.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.697+0000 7f613d823640 1 --2- 172.21.15.5:0/2999407931 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61380c96d0 0x7f61380add00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:29.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.697+0000 7f613d823640 1 --2- 172.21.15.5:0/2999407931 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61380c92f0 0x7f61380af8b0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:29.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.697+0000 7f613d823640 1 --2- 172.21.15.5:0/2999407931 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f613806c750 0x7f6138152bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:29.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.697+0000 7f613d823640 1 -- 172.21.15.5:0/2999407931 >> 172.21.15.5:0/2999407931 conn(0x7f613806f820 msgr2=0x7f61380c3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:29.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.697+0000 7f613d823640 1 -- 172.21.15.5:0/2999407931 shutdown_connections 2024-03-24T10:52:29.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:29.697+0000 7f613d823640 1 -- 172.21.15.5:0/2999407931 wait complete. 2024-03-24T10:52:30.313 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:30.313 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:31.314 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:31.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.415+0000 7f13de1bc640 1 Processor -- start 2024-03-24T10:52:31.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.415+0000 7f13de1bc640 1 -- start start 2024-03-24T10:52:31.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.416+0000 7f13de1bc640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f13d80c4300 0x7f13d80cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:31.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.416+0000 7f13de1bc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f13d80cdfb0 0x7f13d806c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:31.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.416+0000 7f13de1bc640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13d80c4720 con 0x7f13d8154220 2024-03-24T10:52:31.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.416+0000 7f13de1bc640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f13d80c8bf0 con 0x7f13d80c4300 2024-03-24T10:52:31.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.416+0000 7f13de1bc640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f13d80c9050 con 0x7f13d80cdfb0 2024-03-24T10:52:31.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.417+0000 7f13d7fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f13d80cdfb0 0x7f13d806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:31.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.417+0000 7f13d77fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f13d8154220 0x7f13d80c84e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:35816/0 (socket says 172.21.15.5:35816) 2024-03-24T10:52:31.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.417+0000 7f13d77fe640 1 -- 172.21.15.5:0/177947555 learned_addr learned my addr 172.21.15.5:0/177947555 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:52:31.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.417+0000 7f13d6ffd640 1 --2- 172.21.15.5:0/177947555 >> v2:172.21.15.5:3300/0 conn(0x7f13d80c4300 0x7f13d80cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:31.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.421+0000 7f13d7fff640 1 -- 172.21.15.5:0/177947555 >> v2:172.21.15.5:3300/0 conn(0x7f13d80c4300 msgr2=0x7f13d80cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:31.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.421+0000 7f13d7fff640 1 --2- 172.21.15.5:0/177947555 >> v2:172.21.15.5:3300/0 conn(0x7f13d80c4300 0x7f13d80cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:31.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.421+0000 7f13d7fff640 1 -- 172.21.15.5:0/177947555 >> v1:172.21.15.5:6789/0 conn(0x7f13d8154220 legacy=0x7f13d80c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:31.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.421+0000 7f13d7fff640 1 -- 172.21.15.5:0/177947555 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d815a5f0 con 0x7f13d80cdfb0 2024-03-24T10:52:31.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.424+0000 7f13d7fff640 1 --2- 172.21.15.5:0/177947555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13d80cdfb0 0x7f13d806c910 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f13d80c30c0 tx=0x7f13d002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e01b2642ac76f65 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:31.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.428+0000 7f13d67fc640 1 -- 172.21.15.5:0/177947555 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d002cd00 con 0x7f13d80cdfb0 2024-03-24T10:52:31.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.429+0000 7f13d67fc640 1 -- 172.21.15.5:0/177947555 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d002ce60 con 0x7f13d80cdfb0 2024-03-24T10:52:31.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.429+0000 7f13d67fc640 1 -- 172.21.15.5:0/177947555 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0034660 con 0x7f13d80cdfb0 2024-03-24T10:52:31.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.429+0000 7f13de1bc640 1 -- 172.21.15.5:0/177947555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13d80cdfb0 msgr2=0x7f13d806c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:31.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.429+0000 7f13de1bc640 1 --2- 172.21.15.5:0/177947555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13d80cdfb0 0x7f13d806c910 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f13d80c30c0 tx=0x7f13d002bf50 comp rx=0 tx=0).stop 2024-03-24T10:52:31.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.429+0000 7f13de1bc640 1 -- 172.21.15.5:0/177947555 shutdown_connections 2024-03-24T10:52:31.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.429+0000 7f13de1bc640 1 --2- 172.21.15.5:0/177947555 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13d80cdfb0 0x7f13d806c910 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:31.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.429+0000 7f13de1bc640 1 --2- 172.21.15.5:0/177947555 >> v2:172.21.15.5:3300/0 conn(0x7f13d80c4300 0x7f13d80cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:31.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.429+0000 7f13de1bc640 1 -- 172.21.15.5:0/177947555 >> 172.21.15.5:0/177947555 conn(0x7f13d806f830 msgr2=0x7f13d806fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:31.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.429+0000 7f13de1bc640 1 -- 172.21.15.5:0/177947555 shutdown_connections 2024-03-24T10:52:31.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.429+0000 7f13de1bc640 1 -- 172.21.15.5:0/177947555 wait complete. 2024-03-24T10:52:31.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.431+0000 7f13de1bc640 1 Processor -- start 2024-03-24T10:52:31.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.431+0000 7f13de1bc640 1 -- start start 2024-03-24T10:52:31.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.431+0000 7f13de1bc640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f13d80c4300 0x7f13d814f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:31.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.431+0000 7f13de1bc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13d80cdfb0 0x7f13d81f0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:31.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.432+0000 7f13de1bc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13d8154220 0x7f13d81f0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:31.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.432+0000 7f13de1bc640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f13d80b01a0 con 0x7f13d80cdfb0 2024-03-24T10:52:31.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.432+0000 7f13de1bc640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f13d81efe00 con 0x7f13d8154220 2024-03-24T10:52:31.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.432+0000 7f13de1bc640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f13d81f3e20 con 0x7f13d80c4300 2024-03-24T10:52:31.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.432+0000 7f13d6ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13d80cdfb0 0x7f13d81f0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:31.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.432+0000 7f13d6ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13d80cdfb0 0x7f13d81f0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:53792/0 (socket says 172.21.15.5:53792) 2024-03-24T10:52:31.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.432+0000 7f13d6ffd640 1 -- 172.21.15.5:0/206075681 learned_addr learned my addr 172.21.15.5:0/206075681 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:31.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.432+0000 7f13d77fe640 1 --2- 172.21.15.5:0/206075681 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f13d80c4300 0x7f13d814f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:31.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.433+0000 7f13d7fff640 1 --2- 172.21.15.5:0/206075681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13d8154220 0x7f13d81f0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:31.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.436+0000 7f13d6ffd640 1 -- 172.21.15.5:0/206075681 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f13d80c4300 msgr2=0x7f13d814f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:31.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.436+0000 7f13d6ffd640 1 --2- 172.21.15.5:0/206075681 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f13d80c4300 0x7f13d814f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:31.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.436+0000 7f13d6ffd640 1 -- 172.21.15.5:0/206075681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13d8154220 msgr2=0x7f13d81f0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:31.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.436+0000 7f13d6ffd640 1 --2- 172.21.15.5:0/206075681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13d8154220 0x7f13d81f0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:31.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.436+0000 7f13d6ffd640 1 -- 172.21.15.5:0/206075681 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13d0009a70 con 0x7f13d80cdfb0 2024-03-24T10:52:31.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.438+0000 7f13d6ffd640 1 --2- 172.21.15.5:0/206075681 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13d80cdfb0 0x7f13d81f0300 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f13cc00e9b0 tx=0x7f13cc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:31.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.442+0000 7f13bffff640 1 -- 172.21.15.5:0/206075681 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc009670 con 0x7f13d80cdfb0 2024-03-24T10:52:31.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.442+0000 7f13bffff640 1 -- 172.21.15.5:0/206075681 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13cc00c9a0 con 0x7f13d80cdfb0 2024-03-24T10:52:31.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.442+0000 7f13bffff640 1 -- 172.21.15.5:0/206075681 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13cc00cd50 con 0x7f13d80cdfb0 2024-03-24T10:52:31.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.442+0000 7f13de1bc640 1 -- 172.21.15.5:0/206075681 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13d81f4100 con 0x7f13d80cdfb0 2024-03-24T10:52:31.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.442+0000 7f13de1bc640 1 -- 172.21.15.5:0/206075681 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13d80b06f0 con 0x7f13d80cdfb0 2024-03-24T10:52:31.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.444+0000 7f13de1bc640 1 -- 172.21.15.5:0/206075681 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13d80b08b0 con 0x7f13d80cdfb0 2024-03-24T10:52:31.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.712+0000 7f13bffff640 1 -- 172.21.15.5:0/206075681 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f13cc013470 con 0x7f13d80cdfb0 2024-03-24T10:52:31.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.712+0000 7f13bffff640 1 --2- 172.21.15.5:0/206075681 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f13c406b860 0x7f13c406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:31.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.712+0000 7f13d77fe640 1 --2- 172.21.15.5:0/206075681 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f13c406b860 0x7f13c406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:31.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.713+0000 7f13d77fe640 1 --2- 172.21.15.5:0/206075681 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f13c406b860 0x7f13c406dd20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f13c8007fd0 tx=0x7f13c8007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:31.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.726+0000 7f13bffff640 1 -- 172.21.15.5:0/206075681 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f13cc0591e0 con 0x7f13d80cdfb0 2024-03-24T10:52:31.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.726+0000 7f13bffff640 1 -- 172.21.15.5:0/206075681 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f13cc055480 con 0x7f13d80cdfb0 2024-03-24T10:52:31.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:31.987+0000 7f13de1bc640 1 -- 172.21.15.5:0/206075681 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13d80b08b0 con 0x7f13d80cdfb0 2024-03-24T10:52:32.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:32.036+0000 7f13bffff640 1 -- 172.21.15.5:0/206075681 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f13d80b08b0 con 0x7f13d80cdfb0 2024-03-24T10:52:32.077 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:32.078 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:32.080 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:32.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:32.039+0000 7f13de1bc640 1 -- 172.21.15.5:0/206075681 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f13c406b860 msgr2=0x7f13c406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:32.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:32.039+0000 7f13de1bc640 1 --2- 172.21.15.5:0/206075681 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f13c406b860 0x7f13c406dd20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f13c8007fd0 tx=0x7f13c8007e80 comp rx=0 tx=0).stop 2024-03-24T10:52:32.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:32.039+0000 7f13de1bc640 1 -- 172.21.15.5:0/206075681 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13d80cdfb0 msgr2=0x7f13d81f0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:32.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:32.039+0000 7f13de1bc640 1 --2- 172.21.15.5:0/206075681 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13d80cdfb0 0x7f13d81f0300 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f13cc00e9b0 tx=0x7f13cc00ee70 comp rx=0 tx=0).stop 2024-03-24T10:52:32.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:32.039+0000 7f13de1bc640 1 -- 172.21.15.5:0/206075681 shutdown_connections 2024-03-24T10:52:32.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:32.039+0000 7f13de1bc640 1 --2- 172.21.15.5:0/206075681 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f13c406b860 0x7f13c406dd20 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:32.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:32.039+0000 7f13de1bc640 1 --2- 172.21.15.5:0/206075681 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f13d80cdfb0 0x7f13d81f0300 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:32.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:32.039+0000 7f13de1bc640 1 --2- 172.21.15.5:0/206075681 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f13d8154220 0x7f13d81f0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:32.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:32.039+0000 7f13de1bc640 1 --2- 172.21.15.5:0/206075681 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f13d80c4300 0x7f13d814f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:32.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:32.039+0000 7f13de1bc640 1 -- 172.21.15.5:0/206075681 >> 172.21.15.5:0/206075681 conn(0x7f13d806f830 msgr2=0x7f13d8152430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:32.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:32.039+0000 7f13de1bc640 1 -- 172.21.15.5:0/206075681 shutdown_connections 2024-03-24T10:52:32.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:32.039+0000 7f13de1bc640 1 -- 172.21.15.5:0/206075681 wait complete. 2024-03-24T10:52:32.142 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:52:32.146 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:52:32.686 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:32.686 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:33.687 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:33.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.790+0000 7fcc58b01640 1 Processor -- start 2024-03-24T10:52:33.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.790+0000 7fcc58b01640 1 -- start start 2024-03-24T10:52:33.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.791+0000 7fcc58b01640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc540ca5a0 0x7fcc540caa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:33.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.791+0000 7fcc58b01640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc5406cd50 con 0x7fcc5415aac0 2024-03-24T10:52:33.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.791+0000 7fcc58b01640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc540caf70 con 0x7fcc5415aea0 2024-03-24T10:52:33.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.791+0000 7fcc58b01640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fcc540ce790 con 0x7fcc540ca5a0 2024-03-24T10:52:33.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.791+0000 7fcc5259c640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fcc5415aac0 0x7fcc5406c230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:51452/0 (socket says 172.21.15.5:51452) 2024-03-24T10:52:33.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.791+0000 7fcc5259c640 1 -- 172.21.15.5:0/359744559 learned_addr learned my addr 172.21.15.5:0/359744559 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:52:33.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.792+0000 7fcc52d9d640 1 --2- 172.21.15.5:0/359744559 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc540ca5a0 0x7fcc540caa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:33.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.793+0000 7fcc5159a640 1 -- 172.21.15.5:0/359744559 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2780917093 0 0) 0x7fcc5406cd50 con 0x7fcc5415aac0 2024-03-24T10:52:33.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.793+0000 7fcc5159a640 1 -- 172.21.15.5:0/359744559 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc3c003520 con 0x7fcc5415aac0 2024-03-24T10:52:33.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.794+0000 7fcc5159a640 1 -- 172.21.15.5:0/359744559 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1781403189 0 0) 0x7fcc540caf70 con 0x7fcc5415aea0 2024-03-24T10:52:33.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.794+0000 7fcc5159a640 1 -- 172.21.15.5:0/359744559 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc5406cd50 con 0x7fcc5415aea0 2024-03-24T10:52:33.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.796+0000 7fcc52d9d640 1 -- 172.21.15.5:0/359744559 >> v1:172.21.15.57:6789/0 conn(0x7fcc5415aea0 legacy=0x7fcc540cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:33.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.796+0000 7fcc52d9d640 1 -- 172.21.15.5:0/359744559 >> v1:172.21.15.5:6789/0 conn(0x7fcc5415aac0 legacy=0x7fcc5406c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:33.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.796+0000 7fcc52d9d640 1 -- 172.21.15.5:0/359744559 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc540cdb10 con 0x7fcc540ca5a0 2024-03-24T10:52:33.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.799+0000 7fcc52d9d640 1 --2- 172.21.15.5:0/359744559 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc540ca5a0 0x7fcc540caa30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c009dc0 tx=0x7fcc4c02bfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d4f234cf7f30829 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:33.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.803+0000 7fcc5159a640 1 -- 172.21.15.5:0/359744559 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c02cd60 con 0x7fcc540ca5a0 2024-03-24T10:52:33.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.803+0000 7fcc5159a640 1 -- 172.21.15.5:0/359744559 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc4c034440 con 0x7fcc540ca5a0 2024-03-24T10:52:33.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.803+0000 7fcc5159a640 1 -- 172.21.15.5:0/359744559 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c0347d0 con 0x7fcc540ca5a0 2024-03-24T10:52:33.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.804+0000 7fcc58b01640 1 -- 172.21.15.5:0/359744559 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc540ca5a0 msgr2=0x7fcc540caa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:33.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.804+0000 7fcc58b01640 1 --2- 172.21.15.5:0/359744559 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc540ca5a0 0x7fcc540caa30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c009dc0 tx=0x7fcc4c02bfb0 comp rx=0 tx=0).stop 2024-03-24T10:52:33.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.804+0000 7fcc58b01640 1 -- 172.21.15.5:0/359744559 shutdown_connections 2024-03-24T10:52:33.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.804+0000 7fcc58b01640 1 --2- 172.21.15.5:0/359744559 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc540ca5a0 0x7fcc540caa30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:33.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.804+0000 7fcc58b01640 1 -- 172.21.15.5:0/359744559 >> 172.21.15.5:0/359744559 conn(0x7fcc540cfb90 msgr2=0x7fcc540cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:33.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.804+0000 7fcc58b01640 1 -- 172.21.15.5:0/359744559 shutdown_connections 2024-03-24T10:52:33.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.804+0000 7fcc58b01640 1 -- 172.21.15.5:0/359744559 wait complete. 2024-03-24T10:52:33.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.806+0000 7fcc58b01640 1 Processor -- start 2024-03-24T10:52:33.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.806+0000 7fcc58b01640 1 -- start start 2024-03-24T10:52:33.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.806+0000 7fcc58b01640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc540ca5a0 0x7fcc541f0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:33.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.806+0000 7fcc58b01640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc5415aac0 0x7fcc54157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:33.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.806+0000 7fcc58b01640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc5415aea0 0x7fcc5414fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:33.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.806+0000 7fcc58b01640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fcc541e4cd0 con 0x7fcc540ca5a0 2024-03-24T10:52:33.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.806+0000 7fcc58b01640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fcc54155e10 con 0x7fcc5415aac0 2024-03-24T10:52:33.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.806+0000 7fcc58b01640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fcc541f0fb0 con 0x7fcc5415aea0 2024-03-24T10:52:33.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.807+0000 7fcc5259c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc540ca5a0 0x7fcc541f0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:33.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.807+0000 7fcc51d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc5415aac0 0x7fcc54157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:33.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.807+0000 7fcc52d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc5415aea0 0x7fcc5414fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:33.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.807+0000 7fcc5259c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc540ca5a0 0x7fcc541f0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:55250/0 (socket says 172.21.15.5:55250) 2024-03-24T10:52:33.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.807+0000 7fcc5259c640 1 -- 172.21.15.5:0/165410602 learned_addr learned my addr 172.21.15.5:0/165410602 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:33.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.814+0000 7fcc5259c640 1 -- 172.21.15.5:0/165410602 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc5415aea0 msgr2=0x7fcc5414fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:33.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.814+0000 7fcc5259c640 1 --2- 172.21.15.5:0/165410602 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc5415aea0 0x7fcc5414fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:33.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.814+0000 7fcc5259c640 1 -- 172.21.15.5:0/165410602 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc5415aac0 msgr2=0x7fcc54157200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:33.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.814+0000 7fcc5259c640 1 --2- 172.21.15.5:0/165410602 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc5415aac0 0x7fcc54157200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:33.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.814+0000 7fcc5259c640 1 -- 172.21.15.5:0/165410602 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc4c009a70 con 0x7fcc540ca5a0 2024-03-24T10:52:33.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.817+0000 7fcc5259c640 1 --2- 172.21.15.5:0/165410602 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc540ca5a0 0x7fcc541f0a70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcc4400c500 tx=0x7fcc4400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:33.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.821+0000 7fcc42ffd640 1 -- 172.21.15.5:0/165410602 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc44005ce0 con 0x7fcc540ca5a0 2024-03-24T10:52:33.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.821+0000 7fcc42ffd640 1 -- 172.21.15.5:0/165410602 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc44005e60 con 0x7fcc540ca5a0 2024-03-24T10:52:33.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.821+0000 7fcc58b01640 1 -- 172.21.15.5:0/165410602 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc54153e90 con 0x7fcc540ca5a0 2024-03-24T10:52:33.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.821+0000 7fcc42ffd640 1 -- 172.21.15.5:0/165410602 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc44007330 con 0x7fcc540ca5a0 2024-03-24T10:52:33.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.822+0000 7fcc58b01640 1 -- 172.21.15.5:0/165410602 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc54155120 con 0x7fcc540ca5a0 2024-03-24T10:52:33.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:33.822+0000 7fcc58b01640 1 -- 172.21.15.5:0/165410602 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc54155460 con 0x7fcc540ca5a0 2024-03-24T10:52:34.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:34.100+0000 7fcc42ffd640 1 -- 172.21.15.5:0/165410602 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcc44007490 con 0x7fcc540ca5a0 2024-03-24T10:52:34.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:34.100+0000 7fcc42ffd640 1 --2- 172.21.15.5:0/165410602 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcc3406b890 0x7fcc3406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:34.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:34.101+0000 7fcc51d9b640 1 --2- 172.21.15.5:0/165410602 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcc3406b890 0x7fcc3406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:34.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:34.101+0000 7fcc51d9b640 1 --2- 172.21.15.5:0/165410602 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcc3406b890 0x7fcc3406dd50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fcc48007fd0 tx=0x7fcc48009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:34.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:34.129+0000 7fcc42ffd640 1 -- 172.21.15.5:0/165410602 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fcc44058510 con 0x7fcc540ca5a0 2024-03-24T10:52:34.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:34.129+0000 7fcc42ffd640 1 -- 172.21.15.5:0/165410602 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcc440547b0 con 0x7fcc540ca5a0 2024-03-24T10:52:34.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:34.396+0000 7fcc58b01640 1 -- 172.21.15.5:0/165410602 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc540ca1e0 con 0x7fcc540ca5a0 2024-03-24T10:52:34.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:34.451+0000 7fcc42ffd640 1 -- 172.21.15.5:0/165410602 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fcc44017090 con 0x7fcc540ca5a0 2024-03-24T10:52:34.451 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:34.451 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:34.451 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:34.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:34.453+0000 7fcc58b01640 1 -- 172.21.15.5:0/165410602 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcc3406b890 msgr2=0x7fcc3406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:34.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:34.453+0000 7fcc58b01640 1 --2- 172.21.15.5:0/165410602 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcc3406b890 0x7fcc3406dd50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fcc48007fd0 tx=0x7fcc48009040 comp rx=0 tx=0).stop 2024-03-24T10:52:34.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:34.453+0000 7fcc58b01640 1 -- 172.21.15.5:0/165410602 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc540ca5a0 msgr2=0x7fcc541f0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:34.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:34.453+0000 7fcc58b01640 1 --2- 172.21.15.5:0/165410602 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc540ca5a0 0x7fcc541f0a70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcc4400c500 tx=0x7fcc4400c9c0 comp rx=0 tx=0).stop 2024-03-24T10:52:34.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:34.453+0000 7fcc58b01640 1 -- 172.21.15.5:0/165410602 shutdown_connections 2024-03-24T10:52:34.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:34.453+0000 7fcc58b01640 1 --2- 172.21.15.5:0/165410602 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcc3406b890 0x7fcc3406dd50 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:34.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:34.453+0000 7fcc58b01640 1 --2- 172.21.15.5:0/165410602 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc5415aea0 0x7fcc5414fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:34.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:34.453+0000 7fcc58b01640 1 --2- 172.21.15.5:0/165410602 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc5415aac0 0x7fcc54157200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:34.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:34.453+0000 7fcc58b01640 1 --2- 172.21.15.5:0/165410602 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc540ca5a0 0x7fcc541f0a70 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:34.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:34.453+0000 7fcc58b01640 1 -- 172.21.15.5:0/165410602 >> 172.21.15.5:0/165410602 conn(0x7fcc540cfb90 msgr2=0x7fcc540b1500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:34.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:34.453+0000 7fcc58b01640 1 -- 172.21.15.5:0/165410602 shutdown_connections 2024-03-24T10:52:34.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:34.453+0000 7fcc58b01640 1 -- 172.21.15.5:0/165410602 wait complete. 2024-03-24T10:52:35.068 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:35.068 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:36.069 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:36.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.172+0000 7feebf2d4640 1 Processor -- start 2024-03-24T10:52:36.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.173+0000 7feebf2d4640 1 -- start start 2024-03-24T10:52:36.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.174+0000 7feebf2d4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7feeb815ac00 0x7feeb80cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:36.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.174+0000 7feebf2d4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feeb80ccaa0 0x7feeb806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:36.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.174+0000 7feebf2d4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feeb80cb2f0 con 0x7feeb81543a0 2024-03-24T10:52:36.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.174+0000 7feebf2d4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7feeb80c6750 con 0x7feeb815ac00 2024-03-24T10:52:36.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.174+0000 7feebf2d4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7feeb80cceb0 con 0x7feeb80ccaa0 2024-03-24T10:52:36.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.175+0000 7feebd871640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feeb80ccaa0 0x7feeb806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:36.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.175+0000 7feebd070640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7feeb81543a0 0x7feeb806d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:42566/0 (socket says 172.21.15.5:42566) 2024-03-24T10:52:36.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.175+0000 7feebd070640 1 -- 172.21.15.5:0/73657034 learned_addr learned my addr 172.21.15.5:0/73657034 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:52:36.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.175+0000 7feebc86f640 1 --2- 172.21.15.5:0/73657034 >> v2:172.21.15.57:3300/0 conn(0x7feeb815ac00 0x7feeb80cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:36.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.178+0000 7feebd871640 1 -- 172.21.15.5:0/73657034 >> v2:172.21.15.57:3300/0 conn(0x7feeb815ac00 msgr2=0x7feeb80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:36.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.178+0000 7feebd871640 1 --2- 172.21.15.5:0/73657034 >> v2:172.21.15.57:3300/0 conn(0x7feeb815ac00 0x7feeb80cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:36.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.178+0000 7feebd871640 1 -- 172.21.15.5:0/73657034 >> v1:172.21.15.57:6789/0 conn(0x7feeb81543a0 legacy=0x7feeb806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:36.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.178+0000 7feebd871640 1 -- 172.21.15.5:0/73657034 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb815b070 con 0x7feeb80ccaa0 2024-03-24T10:52:36.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.181+0000 7feebd871640 1 --2- 172.21.15.5:0/73657034 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feeb80ccaa0 0x7feeb806bf50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7feeb0009d90 tx=0x7feeb002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=855c638953a634f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:36.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.186+0000 7feeaffff640 1 -- 172.21.15.5:0/73657034 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb002cc50 con 0x7feeb80ccaa0 2024-03-24T10:52:36.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.187+0000 7feeaffff640 1 -- 172.21.15.5:0/73657034 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feeb002cdb0 con 0x7feeb80ccaa0 2024-03-24T10:52:36.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.187+0000 7feeaffff640 1 -- 172.21.15.5:0/73657034 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb0034660 con 0x7feeb80ccaa0 2024-03-24T10:52:36.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.187+0000 7feebf2d4640 1 -- 172.21.15.5:0/73657034 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feeb80ccaa0 msgr2=0x7feeb806bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:36.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.187+0000 7feebf2d4640 1 --2- 172.21.15.5:0/73657034 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feeb80ccaa0 0x7feeb806bf50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7feeb0009d90 tx=0x7feeb002bea0 comp rx=0 tx=0).stop 2024-03-24T10:52:36.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.188+0000 7feebf2d4640 1 -- 172.21.15.5:0/73657034 shutdown_connections 2024-03-24T10:52:36.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.188+0000 7feebf2d4640 1 --2- 172.21.15.5:0/73657034 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feeb80ccaa0 0x7feeb806bf50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:36.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.188+0000 7feebf2d4640 1 --2- 172.21.15.5:0/73657034 >> v2:172.21.15.57:3300/0 conn(0x7feeb815ac00 0x7feeb80cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:36.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.188+0000 7feebf2d4640 1 -- 172.21.15.5:0/73657034 >> 172.21.15.5:0/73657034 conn(0x7feeb8072ef0 msgr2=0x7feeb8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:36.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.188+0000 7feebf2d4640 1 -- 172.21.15.5:0/73657034 shutdown_connections 2024-03-24T10:52:36.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.188+0000 7feebf2d4640 1 -- 172.21.15.5:0/73657034 wait complete. 2024-03-24T10:52:36.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.189+0000 7feebf2d4640 1 Processor -- start 2024-03-24T10:52:36.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.190+0000 7feebf2d4640 1 -- start start 2024-03-24T10:52:36.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.191+0000 7feebf2d4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feeb80ccaa0 0x7feeb81f4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:36.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.191+0000 7feebf2d4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feeb81543a0 0x7feeb81f0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:36.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.191+0000 7feebf2d4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feeb815ac00 0x7feeb815e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:36.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.191+0000 7feebf2d4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7feeb81f06b0 con 0x7feeb81543a0 2024-03-24T10:52:36.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.191+0000 7feebf2d4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7feeb81f3dd0 con 0x7feeb815ac00 2024-03-24T10:52:36.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.191+0000 7feebf2d4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7feeb81f1100 con 0x7feeb80ccaa0 2024-03-24T10:52:36.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.191+0000 7feebc86f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feeb81543a0 0x7feeb81f0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:36.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.191+0000 7feebd070640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feeb80ccaa0 0x7feeb81f4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:36.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.192+0000 7feebc86f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feeb81543a0 0x7feeb81f0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:55256/0 (socket says 172.21.15.5:55256) 2024-03-24T10:52:36.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.192+0000 7feebc86f640 1 -- 172.21.15.5:0/229893759 learned_addr learned my addr 172.21.15.5:0/229893759 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:36.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.192+0000 7feebd871640 1 --2- 172.21.15.5:0/229893759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feeb815ac00 0x7feeb815e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:36.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.195+0000 7feebd070640 1 -- 172.21.15.5:0/229893759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feeb815ac00 msgr2=0x7feeb815e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:36.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.195+0000 7feebd070640 1 --2- 172.21.15.5:0/229893759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feeb815ac00 0x7feeb815e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:36.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.195+0000 7feebd070640 1 -- 172.21.15.5:0/229893759 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feeb81543a0 msgr2=0x7feeb81f0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:36.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.195+0000 7feebd070640 1 --2- 172.21.15.5:0/229893759 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feeb81543a0 0x7feeb81f0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:36.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.195+0000 7feebd070640 1 -- 172.21.15.5:0/229893759 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb0009a70 con 0x7feeb80ccaa0 2024-03-24T10:52:36.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.198+0000 7feebd070640 1 --2- 172.21.15.5:0/229893759 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feeb80ccaa0 0x7feeb81f4260 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7feeb400c370 tx=0x7feeb400c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:36.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.202+0000 7feeadffb640 1 -- 172.21.15.5:0/229893759 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb4005ce0 con 0x7feeb80ccaa0 2024-03-24T10:52:36.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.202+0000 7feeadffb640 1 -- 172.21.15.5:0/229893759 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feeb4005e40 con 0x7feeb80ccaa0 2024-03-24T10:52:36.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.202+0000 7feebf2d4640 1 -- 172.21.15.5:0/229893759 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeb815e590 con 0x7feeb80ccaa0 2024-03-24T10:52:36.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.203+0000 7feeadffb640 1 -- 172.21.15.5:0/229893759 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb4007310 con 0x7feeb80ccaa0 2024-03-24T10:52:36.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.203+0000 7feebf2d4640 1 -- 172.21.15.5:0/229893759 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feeb8147f80 con 0x7feeb80ccaa0 2024-03-24T10:52:36.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.204+0000 7feebf2d4640 1 -- 172.21.15.5:0/229893759 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feeb80b5e70 con 0x7feeb80ccaa0 2024-03-24T10:52:36.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.237+0000 7feeadffb640 1 -- 172.21.15.5:0/229893759 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7feeb4007470 con 0x7feeb80ccaa0 2024-03-24T10:52:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.237+0000 7feeadffb640 1 --2- 172.21.15.5:0/229893759 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fee9c06b860 0x7fee9c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.237+0000 7feebc86f640 1 --2- 172.21.15.5:0/229893759 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fee9c06b860 0x7fee9c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.238+0000 7feebc86f640 1 --2- 172.21.15.5:0/229893759 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fee9c06b860 0x7fee9c06dd20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7feeb806cd10 tx=0x7feea8009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:36.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.241+0000 7feeadffb640 1 -- 172.21.15.5:0/229893759 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7feeb4007720 con 0x7feeb80ccaa0 2024-03-24T10:52:36.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.510+0000 7feeadffb640 1 -- 172.21.15.5:0/229893759 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7feeb40548e0 con 0x7feeb80ccaa0 2024-03-24T10:52:36.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.756+0000 7feebf2d4640 1 -- 172.21.15.5:0/229893759 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feeb806d060 con 0x7feeb80ccaa0 2024-03-24T10:52:36.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.798+0000 7feeadffb640 1 -- 172.21.15.5:0/229893759 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7feeb4017020 con 0x7feeb80ccaa0 2024-03-24T10:52:36.799 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:36.799 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:36.799 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:36.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.801+0000 7feebf2d4640 1 -- 172.21.15.5:0/229893759 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fee9c06b860 msgr2=0x7fee9c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:36.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.801+0000 7feebf2d4640 1 --2- 172.21.15.5:0/229893759 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fee9c06b860 0x7fee9c06dd20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7feeb806cd10 tx=0x7feea8009190 comp rx=0 tx=0).stop 2024-03-24T10:52:36.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.801+0000 7feebf2d4640 1 -- 172.21.15.5:0/229893759 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feeb80ccaa0 msgr2=0x7feeb81f4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:36.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.801+0000 7feebf2d4640 1 --2- 172.21.15.5:0/229893759 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feeb80ccaa0 0x7feeb81f4260 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7feeb400c370 tx=0x7feeb400c830 comp rx=0 tx=0).stop 2024-03-24T10:52:36.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.801+0000 7feebf2d4640 1 -- 172.21.15.5:0/229893759 shutdown_connections 2024-03-24T10:52:36.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.801+0000 7feebf2d4640 1 --2- 172.21.15.5:0/229893759 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fee9c06b860 0x7fee9c06dd20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:36.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.801+0000 7feebf2d4640 1 --2- 172.21.15.5:0/229893759 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7feeb81543a0 0x7feeb81f0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:36.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.801+0000 7feebf2d4640 1 --2- 172.21.15.5:0/229893759 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7feeb815ac00 0x7feeb815e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:36.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.801+0000 7feebf2d4640 1 --2- 172.21.15.5:0/229893759 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7feeb80ccaa0 0x7feeb81f4260 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:36.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.801+0000 7feebf2d4640 1 -- 172.21.15.5:0/229893759 >> 172.21.15.5:0/229893759 conn(0x7feeb8072ef0 msgr2=0x7feeb81511e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:36.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.801+0000 7feebf2d4640 1 -- 172.21.15.5:0/229893759 shutdown_connections 2024-03-24T10:52:36.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:36.801+0000 7feebf2d4640 1 -- 172.21.15.5:0/229893759 wait complete. 2024-03-24T10:52:37.416 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:37.417 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:38.418 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:38.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.520+0000 7f54b3ba5640 1 Processor -- start 2024-03-24T10:52:38.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.520+0000 7f54b3ba5640 1 -- start start 2024-03-24T10:52:38.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.521+0000 7f54b3ba5640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f54ac1543a0 0x7f54ac15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:38.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.521+0000 7f54b3ba5640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f54ac06d4f0 0x7f54ac0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:38.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.521+0000 7f54b3ba5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f54ac0ccaa0 0x7f54ac06bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:38.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.521+0000 7f54b3ba5640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f54ac0c6750 con 0x7f54ac06d4f0 2024-03-24T10:52:38.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.521+0000 7f54b3ba5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f54ac0cc380 con 0x7f54ac0ccaa0 2024-03-24T10:52:38.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.521+0000 7f54b3ba5640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f54ac0cb2f0 con 0x7f54ac1543a0 2024-03-24T10:52:38.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.521+0000 7f54b2142640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f54ac0ccaa0 0x7f54ac06bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:38.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.521+0000 7f54b1941640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f54ac1543a0 0x7f54ac15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:38.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.521+0000 7f54b1140640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f54ac06d4f0 0x7f54ac0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:38.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.522+0000 7f54b2142640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f54ac0ccaa0 0x7f54ac06bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:60180/0 (socket says 172.21.15.5:60180) 2024-03-24T10:52:38.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.522+0000 7f54b2142640 1 -- 172.21.15.5:0/4286943616 learned_addr learned my addr 172.21.15.5:0/4286943616 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:38.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.524+0000 7f54b2142640 1 -- 172.21.15.5:0/4286943616 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f54ac1543a0 msgr2=0x7f54ac15ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:38.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.524+0000 7f54b2142640 1 --2- 172.21.15.5:0/4286943616 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f54ac1543a0 0x7f54ac15ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:38.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.524+0000 7f54b2142640 1 -- 172.21.15.5:0/4286943616 >> v2:172.21.15.5:3300/0 conn(0x7f54ac06d4f0 msgr2=0x7f54ac0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:38.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.524+0000 7f54b2142640 1 --2- 172.21.15.5:0/4286943616 >> v2:172.21.15.5:3300/0 conn(0x7f54ac06d4f0 0x7f54ac0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:38.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.524+0000 7f54b2142640 1 -- 172.21.15.5:0/4286943616 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54ac0ca7e0 con 0x7f54ac0ccaa0 2024-03-24T10:52:38.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.527+0000 7f54b2142640 1 --2- 172.21.15.5:0/4286943616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54ac0ccaa0 0x7f54ac06bf50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f54ac06de60 tx=0x7f54a402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd86f0a256288caa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:38.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.531+0000 7f54b093f640 1 -- 172.21.15.5:0/4286943616 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a402cb00 con 0x7f54ac0ccaa0 2024-03-24T10:52:38.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.531+0000 7f54b093f640 1 -- 172.21.15.5:0/4286943616 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54a402cc60 con 0x7f54ac0ccaa0 2024-03-24T10:52:38.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.531+0000 7f54b093f640 1 -- 172.21.15.5:0/4286943616 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a40343f0 con 0x7f54ac0ccaa0 2024-03-24T10:52:38.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.532+0000 7f54b3ba5640 1 -- 172.21.15.5:0/4286943616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54ac0ccaa0 msgr2=0x7f54ac06bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:38.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.532+0000 7f54b3ba5640 1 --2- 172.21.15.5:0/4286943616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54ac0ccaa0 0x7f54ac06bf50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f54ac06de60 tx=0x7f54a402bf60 comp rx=0 tx=0).stop 2024-03-24T10:52:38.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.532+0000 7f54b3ba5640 1 -- 172.21.15.5:0/4286943616 shutdown_connections 2024-03-24T10:52:38.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.532+0000 7f54b3ba5640 1 --2- 172.21.15.5:0/4286943616 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54ac0ccaa0 0x7f54ac06bf50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:38.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.532+0000 7f54b3ba5640 1 --2- 172.21.15.5:0/4286943616 >> v2:172.21.15.5:3300/0 conn(0x7f54ac06d4f0 0x7f54ac0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:38.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.532+0000 7f54b3ba5640 1 --2- 172.21.15.5:0/4286943616 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f54ac1543a0 0x7f54ac15ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:38.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.532+0000 7f54b3ba5640 1 -- 172.21.15.5:0/4286943616 >> 172.21.15.5:0/4286943616 conn(0x7f54ac072ef0 msgr2=0x7f54ac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:38.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.533+0000 7f54b3ba5640 1 -- 172.21.15.5:0/4286943616 shutdown_connections 2024-03-24T10:52:38.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.533+0000 7f54b3ba5640 1 -- 172.21.15.5:0/4286943616 wait complete. 2024-03-24T10:52:38.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.534+0000 7f54b3ba5640 1 Processor -- start 2024-03-24T10:52:38.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.535+0000 7f54b3ba5640 1 -- start start 2024-03-24T10:52:38.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.536+0000 7f54b3ba5640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54ac06d4f0 0x7f54ac1f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:38.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.536+0000 7f54b3ba5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54ac0ccaa0 0x7f54ac0c4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:38.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.536+0000 7f54b3ba5640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f54ac1543a0 0x7f54ac144170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:38.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.536+0000 7f54b3ba5640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f54ac0b38b0 con 0x7f54ac06d4f0 2024-03-24T10:52:38.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.536+0000 7f54b3ba5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f54ac0cf870 con 0x7f54ac0ccaa0 2024-03-24T10:52:38.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.536+0000 7f54b3ba5640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f54ac0cf2b0 con 0x7f54ac1543a0 2024-03-24T10:52:38.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.536+0000 7f54b1941640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54ac06d4f0 0x7f54ac1f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:38.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.536+0000 7f54b1140640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54ac0ccaa0 0x7f54ac0c4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:38.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.536+0000 7f54b1941640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54ac06d4f0 0x7f54ac1f2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:55266/0 (socket says 172.21.15.5:55266) 2024-03-24T10:52:38.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.536+0000 7f54b1941640 1 -- 172.21.15.5:0/4006218731 learned_addr learned my addr 172.21.15.5:0/4006218731 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:38.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.537+0000 7f54b2142640 1 --2- 172.21.15.5:0/4006218731 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f54ac1543a0 0x7f54ac144170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:38.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.540+0000 7f54b1941640 1 -- 172.21.15.5:0/4006218731 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f54ac1543a0 msgr2=0x7f54ac144170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:38.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.540+0000 7f54b1941640 1 --2- 172.21.15.5:0/4006218731 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f54ac1543a0 0x7f54ac144170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:38.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.540+0000 7f54b1941640 1 -- 172.21.15.5:0/4006218731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54ac0ccaa0 msgr2=0x7f54ac0c4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:38.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.540+0000 7f54b1941640 1 --2- 172.21.15.5:0/4006218731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54ac0ccaa0 0x7f54ac0c4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:38.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.540+0000 7f54b1941640 1 -- 172.21.15.5:0/4006218731 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a4009b30 con 0x7f54ac06d4f0 2024-03-24T10:52:38.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.542+0000 7f54b1941640 1 --2- 172.21.15.5:0/4006218731 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54ac06d4f0 0x7f54ac1f2f20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f54a800e870 tx=0x7f54a800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:38.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.546+0000 7f54a27fc640 1 -- 172.21.15.5:0/4006218731 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a80094f0 con 0x7f54ac06d4f0 2024-03-24T10:52:38.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.546+0000 7f54a27fc640 1 -- 172.21.15.5:0/4006218731 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54a800c940 con 0x7f54ac06d4f0 2024-03-24T10:52:38.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.546+0000 7f54a27fc640 1 -- 172.21.15.5:0/4006218731 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a800ccf0 con 0x7f54ac06d4f0 2024-03-24T10:52:38.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.546+0000 7f54b3ba5640 1 -- 172.21.15.5:0/4006218731 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54ac0b32a0 con 0x7f54ac06d4f0 2024-03-24T10:52:38.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.546+0000 7f54b3ba5640 1 -- 172.21.15.5:0/4006218731 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54ac0cefd0 con 0x7f54ac06d4f0 2024-03-24T10:52:38.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.547+0000 7f54b3ba5640 1 -- 172.21.15.5:0/4006218731 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54ac0b5e70 con 0x7f54ac06d4f0 2024-03-24T10:52:38.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.828+0000 7f54a27fc640 1 -- 172.21.15.5:0/4006218731 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f54a8013470 con 0x7f54ac06d4f0 2024-03-24T10:52:38.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.828+0000 7f54a27fc640 1 --2- 172.21.15.5:0/4006218731 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f549006b860 0x7f549006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:38.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.828+0000 7f54b1140640 1 --2- 172.21.15.5:0/4006218731 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f549006b860 0x7f549006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:38.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.829+0000 7f54b1140640 1 --2- 172.21.15.5:0/4006218731 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f549006b860 0x7f549006dd20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f54ac14a640 tx=0x7f549c009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:38.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.853+0000 7f54a27fc640 1 -- 172.21.15.5:0/4006218731 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f54a8059170 con 0x7f54ac06d4f0 2024-03-24T10:52:38.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:38.853+0000 7f54a27fc640 1 -- 172.21.15.5:0/4006218731 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f54a8055410 con 0x7f54ac06d4f0 2024-03-24T10:52:39.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:39.076+0000 7f54b3ba5640 1 -- 172.21.15.5:0/4006218731 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54ac0681d0 con 0x7f54ac06d4f0 2024-03-24T10:52:39.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:39.090+0000 7f54a27fc640 1 -- 172.21.15.5:0/4006218731 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f54a8021090 con 0x7f54ac06d4f0 2024-03-24T10:52:39.090 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:39.090 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:39.091 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:39.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:39.092+0000 7f54b3ba5640 1 -- 172.21.15.5:0/4006218731 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f549006b860 msgr2=0x7f549006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:39.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:39.092+0000 7f54b3ba5640 1 --2- 172.21.15.5:0/4006218731 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f549006b860 0x7f549006dd20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f54ac14a640 tx=0x7f549c009190 comp rx=0 tx=0).stop 2024-03-24T10:52:39.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:39.092+0000 7f54b3ba5640 1 -- 172.21.15.5:0/4006218731 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54ac06d4f0 msgr2=0x7f54ac1f2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:39.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:39.092+0000 7f54b3ba5640 1 --2- 172.21.15.5:0/4006218731 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54ac06d4f0 0x7f54ac1f2f20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f54a800e870 tx=0x7f54a800ed30 comp rx=0 tx=0).stop 2024-03-24T10:52:39.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:39.092+0000 7f54b3ba5640 1 -- 172.21.15.5:0/4006218731 shutdown_connections 2024-03-24T10:52:39.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:39.092+0000 7f54b3ba5640 1 --2- 172.21.15.5:0/4006218731 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f549006b860 0x7f549006dd20 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:39.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:39.092+0000 7f54b3ba5640 1 --2- 172.21.15.5:0/4006218731 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f54ac1543a0 0x7f54ac144170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:39.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:39.092+0000 7f54b3ba5640 1 --2- 172.21.15.5:0/4006218731 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f54ac0ccaa0 0x7f54ac0c4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:39.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:39.092+0000 7f54b3ba5640 1 --2- 172.21.15.5:0/4006218731 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f54ac06d4f0 0x7f54ac1f2f20 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:39.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:39.092+0000 7f54b3ba5640 1 -- 172.21.15.5:0/4006218731 >> 172.21.15.5:0/4006218731 conn(0x7f54ac072ef0 msgr2=0x7f54ac061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:39.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:39.092+0000 7f54b3ba5640 1 -- 172.21.15.5:0/4006218731 shutdown_connections 2024-03-24T10:52:39.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:39.092+0000 7f54b3ba5640 1 -- 172.21.15.5:0/4006218731 wait complete. 2024-03-24T10:52:39.707 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:39.707 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:40.708 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:40.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.810+0000 7f9eab207640 1 Processor -- start 2024-03-24T10:52:40.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.811+0000 7f9eab207640 1 -- start start 2024-03-24T10:52:40.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.811+0000 7f9eab207640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9ea40cae90 0x7f9ea40cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:40.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.811+0000 7f9eab207640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9ea40cd960 0x7f9ea40cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:40.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.811+0000 7f9eab207640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ea40ccba0 con 0x7f9ea40cb270 2024-03-24T10:52:40.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.811+0000 7f9eab207640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f9ea40cc9b0 con 0x7f9ea40cd960 2024-03-24T10:52:40.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.811+0000 7f9eab207640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f9ea40cce40 con 0x7f9ea40cae90 2024-03-24T10:52:40.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.812+0000 7f9ea8fa3640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9ea40cae90 0x7f9ea40cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:40.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.812+0000 7f9ea3fff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f9ea40cb270 0x7f9ea40cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:51468/0 (socket says 172.21.15.5:51468) 2024-03-24T10:52:40.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.812+0000 7f9ea3fff640 1 -- 172.21.15.5:0/2726808823 learned_addr learned my addr 172.21.15.5:0/2726808823 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:52:40.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.813+0000 7f9ea97a4640 1 --2- 172.21.15.5:0/2726808823 >> v2:172.21.15.5:3300/0 conn(0x7f9ea40cd960 0x7f9ea40cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:40.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.815+0000 7f9ea8fa3640 1 -- 172.21.15.5:0/2726808823 >> v2:172.21.15.5:3300/0 conn(0x7f9ea40cd960 msgr2=0x7f9ea40cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:40.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.815+0000 7f9ea8fa3640 1 --2- 172.21.15.5:0/2726808823 >> v2:172.21.15.5:3300/0 conn(0x7f9ea40cd960 0x7f9ea40cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:40.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.815+0000 7f9ea8fa3640 1 -- 172.21.15.5:0/2726808823 >> v1:172.21.15.5:6789/0 conn(0x7f9ea40cb270 legacy=0x7f9ea40cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:40.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.815+0000 7f9ea8fa3640 1 -- 172.21.15.5:0/2726808823 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea415a590 con 0x7f9ea40cae90 2024-03-24T10:52:40.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.817+0000 7f9ea8fa3640 1 --2- 172.21.15.5:0/2726808823 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9ea40cae90 0x7f9ea40cf530 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9e980097a0 tx=0x7f9e9802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5817b19796bb3d40 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:40.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.821+0000 7f9ea37fe640 1 -- 172.21.15.5:0/2726808823 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e98030610 con 0x7f9ea40cae90 2024-03-24T10:52:40.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.821+0000 7f9ea37fe640 1 -- 172.21.15.5:0/2726808823 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e98030770 con 0x7f9ea40cae90 2024-03-24T10:52:40.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.821+0000 7f9ea37fe640 1 -- 172.21.15.5:0/2726808823 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e98030b00 con 0x7f9ea40cae90 2024-03-24T10:52:40.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.822+0000 7f9eab207640 1 -- 172.21.15.5:0/2726808823 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9ea40cae90 msgr2=0x7f9ea40cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:40.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.822+0000 7f9eab207640 1 --2- 172.21.15.5:0/2726808823 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9ea40cae90 0x7f9ea40cf530 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9e980097a0 tx=0x7f9e9802b960 comp rx=0 tx=0).stop 2024-03-24T10:52:40.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.822+0000 7f9eab207640 1 -- 172.21.15.5:0/2726808823 shutdown_connections 2024-03-24T10:52:40.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.822+0000 7f9eab207640 1 --2- 172.21.15.5:0/2726808823 >> v2:172.21.15.5:3300/0 conn(0x7f9ea40cd960 0x7f9ea40cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:40.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.822+0000 7f9eab207640 1 --2- 172.21.15.5:0/2726808823 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9ea40cae90 0x7f9ea40cf530 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:40.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.822+0000 7f9eab207640 1 -- 172.21.15.5:0/2726808823 >> 172.21.15.5:0/2726808823 conn(0x7f9ea40cfb80 msgr2=0x7f9ea40cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:40.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.822+0000 7f9eab207640 1 -- 172.21.15.5:0/2726808823 shutdown_connections 2024-03-24T10:52:40.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.822+0000 7f9eab207640 1 -- 172.21.15.5:0/2726808823 wait complete. 2024-03-24T10:52:40.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.823+0000 7f9eab207640 1 Processor -- start 2024-03-24T10:52:40.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.823+0000 7f9eab207640 1 -- start start 2024-03-24T10:52:40.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.824+0000 7f9eab207640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea40cae90 0x7f9ea40c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:40.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.824+0000 7f9eab207640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9ea40cb270 0x7f9ea414f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:40.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.824+0000 7f9eab207640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9ea40cd960 0x7f9ea40c37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:40.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.824+0000 7f9eab207640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea40c54e0 con 0x7f9ea40cb270 2024-03-24T10:52:40.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.824+0000 7f9eab207640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea41f0e10 con 0x7f9ea40cae90 2024-03-24T10:52:40.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.824+0000 7f9eab207640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f9ea41f0200 con 0x7f9ea40cd960 2024-03-24T10:52:40.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.825+0000 7f9ea97a4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9ea40cd960 0x7f9ea40c37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:40.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.825+0000 7f9ea3fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9ea40cb270 0x7f9ea414f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:40.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.825+0000 7f9ea8fa3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea40cae90 0x7f9ea40c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:40.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.825+0000 7f9ea3fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9ea40cb270 0x7f9ea414f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:55276/0 (socket says 172.21.15.5:55276) 2024-03-24T10:52:40.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.825+0000 7f9ea3fff640 1 -- 172.21.15.5:0/4247587532 learned_addr learned my addr 172.21.15.5:0/4247587532 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:40.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.829+0000 7f9ea3fff640 1 -- 172.21.15.5:0/4247587532 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9ea40cd960 msgr2=0x7f9ea40c37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:40.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.829+0000 7f9ea3fff640 1 --2- 172.21.15.5:0/4247587532 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9ea40cd960 0x7f9ea40c37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:40.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.829+0000 7f9ea3fff640 1 -- 172.21.15.5:0/4247587532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea40cae90 msgr2=0x7f9ea40c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:40.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.829+0000 7f9ea3fff640 1 --2- 172.21.15.5:0/4247587532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea40cae90 0x7f9ea40c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:40.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.829+0000 7f9ea3fff640 1 -- 172.21.15.5:0/4247587532 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e98009400 con 0x7f9ea40cb270 2024-03-24T10:52:40.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.831+0000 7f9ea3fff640 1 --2- 172.21.15.5:0/4247587532 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9ea40cb270 0x7f9ea414f770 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9e9000c400 tx=0x7f9e9000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:40.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.836+0000 7f9ea17fa640 1 -- 172.21.15.5:0/4247587532 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e90005ce0 con 0x7f9ea40cb270 2024-03-24T10:52:40.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.836+0000 7f9ea17fa640 1 -- 172.21.15.5:0/4247587532 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e90005e40 con 0x7f9ea40cb270 2024-03-24T10:52:40.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.836+0000 7f9ea17fa640 1 -- 172.21.15.5:0/4247587532 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e900072d0 con 0x7f9ea40cb270 2024-03-24T10:52:40.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.836+0000 7f9eab207640 1 -- 172.21.15.5:0/4247587532 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ea41f04e0 con 0x7f9ea40cb270 2024-03-24T10:52:40.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.836+0000 7f9eab207640 1 -- 172.21.15.5:0/4247587532 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ea41f0aa0 con 0x7f9ea40cb270 2024-03-24T10:52:40.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.837+0000 7f9eab207640 1 -- 172.21.15.5:0/4247587532 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ea40c34e0 con 0x7f9ea40cb270 2024-03-24T10:52:40.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.862+0000 7f9ea17fa640 1 -- 172.21.15.5:0/4247587532 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9e90007430 con 0x7f9ea40cb270 2024-03-24T10:52:40.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.862+0000 7f9ea17fa640 1 --2- 172.21.15.5:0/4247587532 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9e8806b890 0x7f9e8806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:40.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.863+0000 7f9ea8fa3640 1 --2- 172.21.15.5:0/4247587532 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9e8806b890 0x7f9e8806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:40.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:40.863+0000 7f9ea8fa3640 1 --2- 172.21.15.5:0/4247587532 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9e8806b890 0x7f9e8806dd50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9e98036000 tx=0x7f9e98008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:41.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:41.128+0000 7f9ea17fa640 1 -- 172.21.15.5:0/4247587532 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f9e90058230 con 0x7f9ea40cb270 2024-03-24T10:52:41.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:41.128+0000 7f9ea17fa640 1 -- 172.21.15.5:0/4247587532 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9e900544d0 con 0x7f9ea40cb270 2024-03-24T10:52:41.373 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:41.373+0000 7f9eab207640 1 -- 172.21.15.5:0/4247587532 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ea40c34e0 con 0x7f9ea40cb270 2024-03-24T10:52:41.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:41.386+0000 7f9ea17fa640 1 -- 172.21.15.5:0/4247587532 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f9ea40c34e0 con 0x7f9ea40cb270 2024-03-24T10:52:41.386 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:41.386 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:41.387 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:41.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:41.389+0000 7f9eab207640 1 -- 172.21.15.5:0/4247587532 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9e8806b890 msgr2=0x7f9e8806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:41.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:41.389+0000 7f9eab207640 1 --2- 172.21.15.5:0/4247587532 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9e8806b890 0x7f9e8806dd50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9e98036000 tx=0x7f9e98008030 comp rx=0 tx=0).stop 2024-03-24T10:52:41.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:41.389+0000 7f9eab207640 1 -- 172.21.15.5:0/4247587532 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9ea40cb270 msgr2=0x7f9ea414f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:41.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:41.389+0000 7f9eab207640 1 --2- 172.21.15.5:0/4247587532 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9ea40cb270 0x7f9ea414f770 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9e9000c400 tx=0x7f9e9000c8c0 comp rx=0 tx=0).stop 2024-03-24T10:52:41.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:41.389+0000 7f9eab207640 1 -- 172.21.15.5:0/4247587532 shutdown_connections 2024-03-24T10:52:41.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:41.389+0000 7f9eab207640 1 --2- 172.21.15.5:0/4247587532 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9e8806b890 0x7f9e8806dd50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:41.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:41.389+0000 7f9eab207640 1 --2- 172.21.15.5:0/4247587532 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9ea40cb270 0x7f9ea414f770 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:41.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:41.389+0000 7f9eab207640 1 --2- 172.21.15.5:0/4247587532 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9ea40cd960 0x7f9ea40c37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:41.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:41.389+0000 7f9eab207640 1 --2- 172.21.15.5:0/4247587532 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9ea40cae90 0x7f9ea40c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:41.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:41.389+0000 7f9eab207640 1 -- 172.21.15.5:0/4247587532 >> 172.21.15.5:0/4247587532 conn(0x7f9ea40cfb80 msgr2=0x7f9ea4159e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:41.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:41.389+0000 7f9eab207640 1 -- 172.21.15.5:0/4247587532 shutdown_connections 2024-03-24T10:52:41.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:41.389+0000 7f9eab207640 1 -- 172.21.15.5:0/4247587532 wait complete. 2024-03-24T10:52:42.005 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:42.006 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:43.006 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:43.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.109+0000 7f462ebf4640 1 Processor -- start 2024-03-24T10:52:43.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.109+0000 7f462ebf4640 1 -- start start 2024-03-24T10:52:43.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.110+0000 7f462ebf4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f46281543a0 0x7f462815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:43.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.110+0000 7f462ebf4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f462806d4f0 0x7f46280cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:43.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.110+0000 7f462ebf4640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f46280ccaa0 0x7f462806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:43.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.110+0000 7f462ebf4640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f46280c6750 con 0x7f46280ccaa0 2024-03-24T10:52:43.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.110+0000 7f462ebf4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f46280cc380 con 0x7f46281543a0 2024-03-24T10:52:43.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.110+0000 7f462ebf4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f46280cb2f0 con 0x7f462806d4f0 2024-03-24T10:52:43.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.111+0000 7f4627fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f462806d4f0 0x7f46280cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:43.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.111+0000 7f462c990640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f46281543a0 0x7f462815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:43.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.111+0000 7f462d191640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f46280ccaa0 0x7f462806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:43.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.111+0000 7f4627fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f462806d4f0 0x7f46280cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:37088/0 (socket says 172.21.15.5:37088) 2024-03-24T10:52:43.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.111+0000 7f4627fff640 1 -- 172.21.15.5:0/646998861 learned_addr learned my addr 172.21.15.5:0/646998861 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:43.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.115+0000 7f4627fff640 1 -- 172.21.15.5:0/646998861 >> v2:172.21.15.57:3300/0 conn(0x7f46281543a0 msgr2=0x7f462815ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:43.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.115+0000 7f4627fff640 1 --2- 172.21.15.5:0/646998861 >> v2:172.21.15.57:3300/0 conn(0x7f46281543a0 0x7f462815ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:43.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.115+0000 7f4627fff640 1 -- 172.21.15.5:0/646998861 >> v2:172.21.15.5:3300/0 conn(0x7f46280ccaa0 msgr2=0x7f462806bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:43.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.115+0000 7f4627fff640 1 --2- 172.21.15.5:0/646998861 >> v2:172.21.15.5:3300/0 conn(0x7f46280ccaa0 0x7f462806bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:43.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.115+0000 7f4627fff640 1 -- 172.21.15.5:0/646998861 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46280ca7e0 con 0x7f462806d4f0 2024-03-24T10:52:43.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.115+0000 7f462d191640 1 --2- 172.21.15.5:0/646998861 >> v2:172.21.15.5:3300/0 conn(0x7f46280ccaa0 0x7f462806bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:52:43.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.117+0000 7f4627fff640 1 --2- 172.21.15.5:0/646998861 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f462806d4f0 0x7f46280cc560 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f46280cd600 tx=0x7f461402b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f235e7ef995a57c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:43.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.123+0000 7f46277fe640 1 -- 172.21.15.5:0/646998861 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4614030860 con 0x7f462806d4f0 2024-03-24T10:52:43.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.123+0000 7f46277fe640 1 -- 172.21.15.5:0/646998861 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46140309c0 con 0x7f462806d4f0 2024-03-24T10:52:43.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.123+0000 7f46277fe640 1 -- 172.21.15.5:0/646998861 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4614030d50 con 0x7f462806d4f0 2024-03-24T10:52:43.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.123+0000 7f462ebf4640 1 -- 172.21.15.5:0/646998861 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f462806d4f0 msgr2=0x7f46280cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:43.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.123+0000 7f462ebf4640 1 --2- 172.21.15.5:0/646998861 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f462806d4f0 0x7f46280cc560 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f46280cd600 tx=0x7f461402b9a0 comp rx=0 tx=0).stop 2024-03-24T10:52:43.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.123+0000 7f462ebf4640 1 -- 172.21.15.5:0/646998861 shutdown_connections 2024-03-24T10:52:43.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.123+0000 7f462ebf4640 1 --2- 172.21.15.5:0/646998861 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f462806d4f0 0x7f46280cc560 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:43.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.123+0000 7f462ebf4640 1 --2- 172.21.15.5:0/646998861 >> v2:172.21.15.5:3300/0 conn(0x7f46280ccaa0 0x7f462806bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:43.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.123+0000 7f462ebf4640 1 --2- 172.21.15.5:0/646998861 >> v2:172.21.15.57:3300/0 conn(0x7f46281543a0 0x7f462815ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:43.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.123+0000 7f462ebf4640 1 -- 172.21.15.5:0/646998861 >> 172.21.15.5:0/646998861 conn(0x7f4628072ef0 msgr2=0x7f4628061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:43.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.123+0000 7f462ebf4640 1 -- 172.21.15.5:0/646998861 shutdown_connections 2024-03-24T10:52:43.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.123+0000 7f462ebf4640 1 -- 172.21.15.5:0/646998861 wait complete. 2024-03-24T10:52:43.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.125+0000 7f462ebf4640 1 Processor -- start 2024-03-24T10:52:43.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.125+0000 7f462ebf4640 1 -- start start 2024-03-24T10:52:43.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.125+0000 7f462ebf4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f462806d4f0 0x7f4628144a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:43.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.125+0000 7f462ebf4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46280ccaa0 0x7f46280c6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:43.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.125+0000 7f462ebf4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f46281543a0 0x7f46280cea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:43.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.125+0000 7f462ebf4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f4628158cf0 con 0x7f46280ccaa0 2024-03-24T10:52:43.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.125+0000 7f462ebf4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f46280c9700 con 0x7f462806d4f0 2024-03-24T10:52:43.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.125+0000 7f462ebf4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f46280c9f10 con 0x7f46281543a0 2024-03-24T10:52:43.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.126+0000 7f462d191640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f46281543a0 0x7f46280cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:43.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.126+0000 7f4627fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46280ccaa0 0x7f46280c6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:43.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.126+0000 7f462c990640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f462806d4f0 0x7f4628144a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:43.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.127+0000 7f4627fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46280ccaa0 0x7f46280c6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48842/0 (socket says 172.21.15.5:48842) 2024-03-24T10:52:43.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.127+0000 7f462d191640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f46281543a0 0x7f46280cea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:37096/0 (socket says 172.21.15.5:37096) 2024-03-24T10:52:43.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.127+0000 7f4627fff640 1 -- 172.21.15.5:0/979505315 learned_addr learned my addr 172.21.15.5:0/979505315 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:43.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.132+0000 7f4627fff640 1 -- 172.21.15.5:0/979505315 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f46281543a0 msgr2=0x7f46280cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:43.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.132+0000 7f4627fff640 1 --2- 172.21.15.5:0/979505315 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f46281543a0 0x7f46280cea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:43.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.132+0000 7f4627fff640 1 -- 172.21.15.5:0/979505315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f462806d4f0 msgr2=0x7f4628144a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:43.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.132+0000 7f4627fff640 1 --2- 172.21.15.5:0/979505315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f462806d4f0 0x7f4628144a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:43.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.132+0000 7f4627fff640 1 -- 172.21.15.5:0/979505315 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4620009a70 con 0x7f46280ccaa0 2024-03-24T10:52:43.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.132+0000 7f462d191640 1 --2- 172.21.15.5:0/979505315 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f46281543a0 0x7f46280cea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:52:43.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.136+0000 7f4627fff640 1 --2- 172.21.15.5:0/979505315 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46280ccaa0 0x7f46280c6fb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4614000c00 tx=0x7f4614004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:43.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.141+0000 7f46257fa640 1 -- 172.21.15.5:0/979505315 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4614039c90 con 0x7f46280ccaa0 2024-03-24T10:52:43.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.141+0000 7f46257fa640 1 -- 172.21.15.5:0/979505315 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4614039df0 con 0x7f46280ccaa0 2024-03-24T10:52:43.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.141+0000 7f46257fa640 1 -- 172.21.15.5:0/979505315 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4614043640 con 0x7f46280ccaa0 2024-03-24T10:52:43.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.141+0000 7f462ebf4640 1 -- 172.21.15.5:0/979505315 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46140094d0 con 0x7f46280ccaa0 2024-03-24T10:52:43.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.141+0000 7f462ebf4640 1 -- 172.21.15.5:0/979505315 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46280c9c30 con 0x7f46280ccaa0 2024-03-24T10:52:43.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.143+0000 7f462ebf4640 1 -- 172.21.15.5:0/979505315 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46280681d0 con 0x7f46280ccaa0 2024-03-24T10:52:43.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.426+0000 7f46257fa640 1 -- 172.21.15.5:0/979505315 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f46140437a0 con 0x7f46280ccaa0 2024-03-24T10:52:43.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.427+0000 7f46257fa640 1 --2- 172.21.15.5:0/979505315 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f460c06b860 0x7f460c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:43.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.427+0000 7f462c990640 1 --2- 172.21.15.5:0/979505315 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f460c06b860 0x7f460c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:43.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.427+0000 7f462c990640 1 --2- 172.21.15.5:0/979505315 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f460c06b860 0x7f460c06dd20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f461c009a20 tx=0x7f461c009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:43.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.449+0000 7f46257fa640 1 -- 172.21.15.5:0/979505315 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f461403e070 con 0x7f46280ccaa0 2024-03-24T10:52:43.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.449+0000 7f46257fa640 1 -- 172.21.15.5:0/979505315 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4614078620 con 0x7f46280ccaa0 2024-03-24T10:52:43.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.713+0000 7f462ebf4640 1 -- 172.21.15.5:0/979505315 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f462815aee0 con 0x7f46280ccaa0 2024-03-24T10:52:43.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.733+0000 7f46257fa640 1 -- 172.21.15.5:0/979505315 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f4614034070 con 0x7f46280ccaa0 2024-03-24T10:52:43.733 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:43.734 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:43.734 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:43.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.736+0000 7f462ebf4640 1 -- 172.21.15.5:0/979505315 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f460c06b860 msgr2=0x7f460c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:43.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.736+0000 7f462ebf4640 1 --2- 172.21.15.5:0/979505315 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f460c06b860 0x7f460c06dd20 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f461c009a20 tx=0x7f461c009190 comp rx=0 tx=0).stop 2024-03-24T10:52:43.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.736+0000 7f462ebf4640 1 -- 172.21.15.5:0/979505315 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46280ccaa0 msgr2=0x7f46280c6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:43.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.736+0000 7f462ebf4640 1 --2- 172.21.15.5:0/979505315 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46280ccaa0 0x7f46280c6fb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f4614000c00 tx=0x7f4614004730 comp rx=0 tx=0).stop 2024-03-24T10:52:43.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.736+0000 7f462ebf4640 1 -- 172.21.15.5:0/979505315 shutdown_connections 2024-03-24T10:52:43.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.736+0000 7f462ebf4640 1 --2- 172.21.15.5:0/979505315 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f460c06b860 0x7f460c06dd20 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:43.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.736+0000 7f462ebf4640 1 --2- 172.21.15.5:0/979505315 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f46280ccaa0 0x7f46280c6fb0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:43.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.736+0000 7f462ebf4640 1 --2- 172.21.15.5:0/979505315 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f46281543a0 0x7f46280cea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:43.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.736+0000 7f462ebf4640 1 --2- 172.21.15.5:0/979505315 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f462806d4f0 0x7f4628144a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:43.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.736+0000 7f462ebf4640 1 -- 172.21.15.5:0/979505315 >> 172.21.15.5:0/979505315 conn(0x7f4628072ef0 msgr2=0x7f46281535b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:43.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.736+0000 7f462ebf4640 1 -- 172.21.15.5:0/979505315 shutdown_connections 2024-03-24T10:52:43.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:43.736+0000 7f462ebf4640 1 -- 172.21.15.5:0/979505315 wait complete. 2024-03-24T10:52:44.354 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:44.355 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:45.355 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:45.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.457+0000 7f6b9946b640 1 Processor -- start 2024-03-24T10:52:45.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.457+0000 7f6b9946b640 1 -- start start 2024-03-24T10:52:45.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.458+0000 7f6b9946b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6b9415ac00 0x7f6b940cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:45.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.458+0000 7f6b9946b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6b940ccaa0 0x7f6b9406bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:45.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.458+0000 7f6b9946b640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b940cb2f0 con 0x7f6b941543a0 2024-03-24T10:52:45.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.458+0000 7f6b9946b640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f6b940c6750 con 0x7f6b9415ac00 2024-03-24T10:52:45.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.458+0000 7f6b9946b640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f6b940cceb0 con 0x7f6b940ccaa0 2024-03-24T10:52:45.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.459+0000 7f6b937fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6b940ccaa0 0x7f6b9406bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:45.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.459+0000 7f6b927fc640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6b9415ac00 0x7f6b940cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:45.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.459+0000 7f6b92ffd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f6b941543a0 0x7f6b9406d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36992/0 (socket says 172.21.15.5:36992) 2024-03-24T10:52:45.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.459+0000 7f6b92ffd640 1 -- 172.21.15.5:0/1593212301 learned_addr learned my addr 172.21.15.5:0/1593212301 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:52:45.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.463+0000 7f6b937fe640 1 -- 172.21.15.5:0/1593212301 >> v2:172.21.15.5:3300/0 conn(0x7f6b9415ac00 msgr2=0x7f6b940cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:45.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.463+0000 7f6b937fe640 1 --2- 172.21.15.5:0/1593212301 >> v2:172.21.15.5:3300/0 conn(0x7f6b9415ac00 0x7f6b940cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:45.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.463+0000 7f6b937fe640 1 -- 172.21.15.5:0/1593212301 >> v1:172.21.15.5:6789/0 conn(0x7f6b941543a0 legacy=0x7f6b9406d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:45.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.463+0000 7f6b937fe640 1 -- 172.21.15.5:0/1593212301 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9415b070 con 0x7f6b940ccaa0 2024-03-24T10:52:45.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.467+0000 7f6b937fe640 1 --2- 172.21.15.5:0/1593212301 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6b940ccaa0 0x7f6b9406bf50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c009d90 tx=0x7f6b8c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5f4b843330a43c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:45.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.474+0000 7f6b91ffb640 1 -- 172.21.15.5:0/1593212301 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c02cc50 con 0x7f6b940ccaa0 2024-03-24T10:52:45.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.474+0000 7f6b91ffb640 1 -- 172.21.15.5:0/1593212301 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b8c02cdb0 con 0x7f6b940ccaa0 2024-03-24T10:52:45.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.474+0000 7f6b91ffb640 1 -- 172.21.15.5:0/1593212301 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c034660 con 0x7f6b940ccaa0 2024-03-24T10:52:45.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.475+0000 7f6b9946b640 1 -- 172.21.15.5:0/1593212301 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6b940ccaa0 msgr2=0x7f6b9406bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:45.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.475+0000 7f6b9946b640 1 --2- 172.21.15.5:0/1593212301 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6b940ccaa0 0x7f6b9406bf50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c009d90 tx=0x7f6b8c02bea0 comp rx=0 tx=0).stop 2024-03-24T10:52:45.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.475+0000 7f6b9946b640 1 -- 172.21.15.5:0/1593212301 shutdown_connections 2024-03-24T10:52:45.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.475+0000 7f6b9946b640 1 --2- 172.21.15.5:0/1593212301 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6b940ccaa0 0x7f6b9406bf50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:45.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.475+0000 7f6b9946b640 1 --2- 172.21.15.5:0/1593212301 >> v2:172.21.15.5:3300/0 conn(0x7f6b9415ac00 0x7f6b940cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:45.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.475+0000 7f6b9946b640 1 -- 172.21.15.5:0/1593212301 >> 172.21.15.5:0/1593212301 conn(0x7f6b94072ef0 msgr2=0x7f6b94061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:45.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.475+0000 7f6b9946b640 1 -- 172.21.15.5:0/1593212301 shutdown_connections 2024-03-24T10:52:45.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.475+0000 7f6b9946b640 1 -- 172.21.15.5:0/1593212301 wait complete. 2024-03-24T10:52:45.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.476+0000 7f6b9946b640 1 Processor -- start 2024-03-24T10:52:45.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.476+0000 7f6b9946b640 1 -- start start 2024-03-24T10:52:45.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.477+0000 7f6b9946b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b940ccaa0 0x7f6b9414a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:45.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.477+0000 7f6b9946b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6b941543a0 0x7f6b941f03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:45.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.477+0000 7f6b9946b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b9415ac00 0x7f6b94157000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:45.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.477+0000 7f6b9946b640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9414ad20 con 0x7f6b940ccaa0 2024-03-24T10:52:45.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.477+0000 7f6b9946b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6b94155080 con 0x7f6b9415ac00 2024-03-24T10:52:45.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.477+0000 7f6b9946b640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f6b9415b430 con 0x7f6b941543a0 2024-03-24T10:52:45.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.478+0000 7f6b937fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b9415ac00 0x7f6b94157000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:45.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.478+0000 7f6b937fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b9415ac00 0x7f6b94157000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:59532/0 (socket says 172.21.15.5:59532) 2024-03-24T10:52:45.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.478+0000 7f6b937fe640 1 -- 172.21.15.5:0/3244814757 learned_addr learned my addr 172.21.15.5:0/3244814757 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:45.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.479+0000 7f6b927fc640 1 --2- 172.21.15.5:0/3244814757 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6b941543a0 0x7f6b941f03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:45.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.482+0000 7f6b937fe640 1 -- 172.21.15.5:0/3244814757 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6b941543a0 msgr2=0x7f6b941f03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:45.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.482+0000 7f6b937fe640 1 --2- 172.21.15.5:0/3244814757 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6b941543a0 0x7f6b941f03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:45.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.482+0000 7f6b937fe640 1 -- 172.21.15.5:0/3244814757 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b940ccaa0 msgr2=0x7f6b9414a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:45.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.482+0000 7f6b937fe640 1 --2- 172.21.15.5:0/3244814757 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b940ccaa0 0x7f6b9414a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:45.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.482+0000 7f6b937fe640 1 -- 172.21.15.5:0/3244814757 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8c009a70 con 0x7f6b9415ac00 2024-03-24T10:52:45.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.484+0000 7f6b937fe640 1 --2- 172.21.15.5:0/3244814757 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b9415ac00 0x7f6b94157000 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c009eb0 tx=0x7f6b8c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:45.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.489+0000 7f6b7b7fe640 1 -- 172.21.15.5:0/3244814757 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c03e610 con 0x7f6b9415ac00 2024-03-24T10:52:45.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.489+0000 7f6b7b7fe640 1 -- 172.21.15.5:0/3244814757 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b8c03e770 con 0x7f6b9415ac00 2024-03-24T10:52:45.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.489+0000 7f6b9946b640 1 -- 172.21.15.5:0/3244814757 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b9415b710 con 0x7f6b9415ac00 2024-03-24T10:52:45.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.490+0000 7f6b7b7fe640 1 -- 172.21.15.5:0/3244814757 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c03eb20 con 0x7f6b9415ac00 2024-03-24T10:52:45.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.490+0000 7f6b9946b640 1 -- 172.21.15.5:0/3244814757 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b9415a270 con 0x7f6b9415ac00 2024-03-24T10:52:45.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.491+0000 7f6b9946b640 1 -- 172.21.15.5:0/3244814757 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b941f00c0 con 0x7f6b9415ac00 2024-03-24T10:52:45.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.513+0000 7f6b7b7fe640 1 -- 172.21.15.5:0/3244814757 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6b8c03b050 con 0x7f6b9415ac00 2024-03-24T10:52:45.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.514+0000 7f6b7b7fe640 1 --2- 172.21.15.5:0/3244814757 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6b8006b810 0x7f6b8006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:45.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.514+0000 7f6b92ffd640 1 --2- 172.21.15.5:0/3244814757 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6b8006b810 0x7f6b8006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:45.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.514+0000 7f6b92ffd640 1 --2- 172.21.15.5:0/3244814757 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6b8006b810 0x7f6b8006dcd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6b84000f60 tx=0x7f6b84007a80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:45.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.785+0000 7f6b7b7fe640 1 -- 172.21.15.5:0/3244814757 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f6b8c07b510 con 0x7f6b9415ac00 2024-03-24T10:52:45.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:45.804+0000 7f6b7b7fe640 1 -- 172.21.15.5:0/3244814757 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6b8c0777b0 con 0x7f6b9415ac00 2024-03-24T10:52:46.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:46.076+0000 7f6b9946b640 1 -- 172.21.15.5:0/3244814757 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b941f00c0 con 0x7f6b9415ac00 2024-03-24T10:52:46.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:46.098+0000 7f6b7b7fe640 1 -- 172.21.15.5:0/3244814757 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f6b941f00c0 con 0x7f6b9415ac00 2024-03-24T10:52:46.099 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:46.099 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:46.099 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:46.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:46.100+0000 7f6b9946b640 1 -- 172.21.15.5:0/3244814757 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6b8006b810 msgr2=0x7f6b8006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:46.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:46.100+0000 7f6b9946b640 1 --2- 172.21.15.5:0/3244814757 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6b8006b810 0x7f6b8006dcd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6b84000f60 tx=0x7f6b84007a80 comp rx=0 tx=0).stop 2024-03-24T10:52:46.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:46.101+0000 7f6b9946b640 1 -- 172.21.15.5:0/3244814757 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b9415ac00 msgr2=0x7f6b94157000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:46.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:46.101+0000 7f6b9946b640 1 --2- 172.21.15.5:0/3244814757 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b9415ac00 0x7f6b94157000 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c009eb0 tx=0x7f6b8c004a40 comp rx=0 tx=0).stop 2024-03-24T10:52:46.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:46.101+0000 7f6b9946b640 1 -- 172.21.15.5:0/3244814757 shutdown_connections 2024-03-24T10:52:46.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:46.101+0000 7f6b9946b640 1 --2- 172.21.15.5:0/3244814757 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6b8006b810 0x7f6b8006dcd0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:46.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:46.101+0000 7f6b9946b640 1 --2- 172.21.15.5:0/3244814757 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6b9415ac00 0x7f6b94157000 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:46.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:46.101+0000 7f6b9946b640 1 --2- 172.21.15.5:0/3244814757 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6b941543a0 0x7f6b941f03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:46.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:46.101+0000 7f6b9946b640 1 --2- 172.21.15.5:0/3244814757 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6b940ccaa0 0x7f6b9414a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:46.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:46.101+0000 7f6b9946b640 1 -- 172.21.15.5:0/3244814757 >> 172.21.15.5:0/3244814757 conn(0x7f6b94072ef0 msgr2=0x7f6b94151050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:46.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:46.101+0000 7f6b9946b640 1 -- 172.21.15.5:0/3244814757 shutdown_connections 2024-03-24T10:52:46.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:46.101+0000 7f6b9946b640 1 -- 172.21.15.5:0/3244814757 wait complete. 2024-03-24T10:52:46.716 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:46.716 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:47.718 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:47.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.820+0000 7f3f91bc2640 1 Processor -- start 2024-03-24T10:52:47.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.820+0000 7f3f91bc2640 1 -- start start 2024-03-24T10:52:47.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.821+0000 7f3f91bc2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3f8c0c4300 0x7f3f8c0cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:47.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.821+0000 7f3f91bc2640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f8c0c4720 con 0x7f3f8c154220 2024-03-24T10:52:47.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.821+0000 7f3f91bc2640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f8c0c9050 con 0x7f3f8c0cdfb0 2024-03-24T10:52:47.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.821+0000 7f3f91bc2640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f3f8c0c8bf0 con 0x7f3f8c0c4300 2024-03-24T10:52:47.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.821+0000 7f3f8b7fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f3f8c154220 0x7f3f8c0c84e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:37006/0 (socket says 172.21.15.5:37006) 2024-03-24T10:52:47.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.821+0000 7f3f8b7fe640 1 -- 172.21.15.5:0/947417434 learned_addr learned my addr 172.21.15.5:0/947417434 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:52:47.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.822+0000 7f3f8affd640 1 --2- 172.21.15.5:0/947417434 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3f8c0c4300 0x7f3f8c0cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:47.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.824+0000 7f3f8a7fc640 1 -- 172.21.15.5:0/947417434 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2115570522 0 0) 0x7f3f8c0c9050 con 0x7f3f8c0cdfb0 2024-03-24T10:52:47.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.824+0000 7f3f8a7fc640 1 -- 172.21.15.5:0/947417434 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f74003520 con 0x7f3f8c0cdfb0 2024-03-24T10:52:47.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.825+0000 7f3f8a7fc640 1 -- 172.21.15.5:0/947417434 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 185883593 0 0) 0x7f3f8c0c4720 con 0x7f3f8c154220 2024-03-24T10:52:47.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.825+0000 7f3f8a7fc640 1 -- 172.21.15.5:0/947417434 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f8c0c9050 con 0x7f3f8c154220 2024-03-24T10:52:47.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.826+0000 7f3f8affd640 1 -- 172.21.15.5:0/947417434 >> v1:172.21.15.57:6789/0 conn(0x7f3f8c0cdfb0 legacy=0x7f3f8c06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:47.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.826+0000 7f3f8affd640 1 -- 172.21.15.5:0/947417434 >> v1:172.21.15.5:6789/0 conn(0x7f3f8c154220 legacy=0x7f3f8c0c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:47.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.826+0000 7f3f8affd640 1 -- 172.21.15.5:0/947417434 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f8c15a5f0 con 0x7f3f8c0c4300 2024-03-24T10:52:47.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.829+0000 7f3f8affd640 1 --2- 172.21.15.5:0/947417434 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3f8c0c4300 0x7f3f8c0cda70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3f800097d0 tx=0x7f3f8002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae235896ec8fa076 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:47.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.834+0000 7f3f8a7fc640 1 -- 172.21.15.5:0/947417434 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f80030880 con 0x7f3f8c0c4300 2024-03-24T10:52:47.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.834+0000 7f3f8a7fc640 1 -- 172.21.15.5:0/947417434 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f800309e0 con 0x7f3f8c0c4300 2024-03-24T10:52:47.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.834+0000 7f3f8a7fc640 1 -- 172.21.15.5:0/947417434 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f80030d70 con 0x7f3f8c0c4300 2024-03-24T10:52:47.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.835+0000 7f3f91bc2640 1 -- 172.21.15.5:0/947417434 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3f8c0c4300 msgr2=0x7f3f8c0cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:47.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.835+0000 7f3f91bc2640 1 --2- 172.21.15.5:0/947417434 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3f8c0c4300 0x7f3f8c0cda70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3f800097d0 tx=0x7f3f8002b9c0 comp rx=0 tx=0).stop 2024-03-24T10:52:47.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.835+0000 7f3f91bc2640 1 -- 172.21.15.5:0/947417434 shutdown_connections 2024-03-24T10:52:47.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.835+0000 7f3f91bc2640 1 --2- 172.21.15.5:0/947417434 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3f8c0c4300 0x7f3f8c0cda70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:47.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.835+0000 7f3f91bc2640 1 -- 172.21.15.5:0/947417434 >> 172.21.15.5:0/947417434 conn(0x7f3f8c06f830 msgr2=0x7f3f8c06fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:47.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.835+0000 7f3f91bc2640 1 -- 172.21.15.5:0/947417434 shutdown_connections 2024-03-24T10:52:47.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.835+0000 7f3f91bc2640 1 -- 172.21.15.5:0/947417434 wait complete. 2024-03-24T10:52:47.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.837+0000 7f3f91bc2640 1 Processor -- start 2024-03-24T10:52:47.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.837+0000 7f3f91bc2640 1 -- start start 2024-03-24T10:52:47.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.839+0000 7f3f91bc2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3f8c0c4300 0x7f3f8c1f12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:47.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.839+0000 7f3f91bc2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f8c0cdfb0 0x7f3f8c155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:47.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.839+0000 7f3f91bc2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3f8c154220 0x7f3f8c15dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:47.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.839+0000 7f3f91bc2640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8c1f0e20 con 0x7f3f8c154220 2024-03-24T10:52:47.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.839+0000 7f3f91bc2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3f8c1f08c0 con 0x7f3f8c0cdfb0 2024-03-24T10:52:47.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.839+0000 7f3f91bc2640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f3f8c154ff0 con 0x7f3f8c0c4300 2024-03-24T10:52:47.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.839+0000 7f3f8bfff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3f8c154220 0x7f3f8c15dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:47.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.839+0000 7f3f8bfff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3f8c154220 0x7f3f8c15dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48878/0 (socket says 172.21.15.5:48878) 2024-03-24T10:52:47.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.839+0000 7f3f8bfff640 1 -- 172.21.15.5:0/3499277540 learned_addr learned my addr 172.21.15.5:0/3499277540 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:47.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.840+0000 7f3f8affd640 1 --2- 172.21.15.5:0/3499277540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f8c0cdfb0 0x7f3f8c155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:47.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.840+0000 7f3f8b7fe640 1 --2- 172.21.15.5:0/3499277540 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3f8c0c4300 0x7f3f8c1f12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:47.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.843+0000 7f3f8bfff640 1 -- 172.21.15.5:0/3499277540 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3f8c0c4300 msgr2=0x7f3f8c1f12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:47.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.843+0000 7f3f8bfff640 1 --2- 172.21.15.5:0/3499277540 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3f8c0c4300 0x7f3f8c1f12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:47.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.843+0000 7f3f8bfff640 1 -- 172.21.15.5:0/3499277540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f8c0cdfb0 msgr2=0x7f3f8c155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:47.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.843+0000 7f3f8bfff640 1 --2- 172.21.15.5:0/3499277540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f8c0cdfb0 0x7f3f8c155d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:47.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.843+0000 7f3f8bfff640 1 -- 172.21.15.5:0/3499277540 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f80009400 con 0x7f3f8c154220 2024-03-24T10:52:47.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.847+0000 7f3f8bfff640 1 --2- 172.21.15.5:0/3499277540 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3f8c154220 0x7f3f8c15dad0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400c790 tx=0x7f3f8400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:47.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.852+0000 7f3f73fff640 1 -- 172.21.15.5:0/3499277540 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8400cea0 con 0x7f3f8c154220 2024-03-24T10:52:47.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.852+0000 7f3f73fff640 1 -- 172.21.15.5:0/3499277540 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f84003b70 con 0x7f3f8c154220 2024-03-24T10:52:47.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.852+0000 7f3f73fff640 1 -- 172.21.15.5:0/3499277540 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8400a850 con 0x7f3f8c154220 2024-03-24T10:52:47.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.852+0000 7f3f91bc2640 1 -- 172.21.15.5:0/3499277540 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f8c1f0b60 con 0x7f3f8c154220 2024-03-24T10:52:47.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.852+0000 7f3f91bc2640 1 -- 172.21.15.5:0/3499277540 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f8c155570 con 0x7f3f8c154220 2024-03-24T10:52:47.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:47.853+0000 7f3f91bc2640 1 -- 172.21.15.5:0/3499277540 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f8c0c2e40 con 0x7f3f8c154220 2024-03-24T10:52:48.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:48.189+0000 7f3f73fff640 1 -- 172.21.15.5:0/3499277540 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3f8400a9b0 con 0x7f3f8c154220 2024-03-24T10:52:48.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:48.189+0000 7f3f73fff640 1 --2- 172.21.15.5:0/3499277540 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3f7806b890 0x7f3f7806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:48.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:48.189+0000 7f3f8b7fe640 1 --2- 172.21.15.5:0/3499277540 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3f7806b890 0x7f3f7806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:48.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:48.190+0000 7f3f8b7fe640 1 --2- 172.21.15.5:0/3499277540 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3f7806b890 0x7f3f7806dd50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c005880 tx=0x7f3f7c009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:48.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:48.215+0000 7f3f73fff640 1 -- 172.21.15.5:0/3499277540 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f3f84059220 con 0x7f3f8c154220 2024-03-24T10:52:48.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:48.215+0000 7f3f73fff640 1 -- 172.21.15.5:0/3499277540 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3f840554c0 con 0x7f3f8c154220 2024-03-24T10:52:48.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:48.443+0000 7f3f91bc2640 1 -- 172.21.15.5:0/3499277540 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f8c06c240 con 0x7f3f8c154220 2024-03-24T10:52:48.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:48.457+0000 7f3f73fff640 1 -- 172.21.15.5:0/3499277540 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f3f84058b80 con 0x7f3f8c154220 2024-03-24T10:52:48.457 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:48.457 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:48.457 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:48.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:48.459+0000 7f3f91bc2640 1 -- 172.21.15.5:0/3499277540 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3f7806b890 msgr2=0x7f3f7806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:48.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:48.459+0000 7f3f91bc2640 1 --2- 172.21.15.5:0/3499277540 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3f7806b890 0x7f3f7806dd50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c005880 tx=0x7f3f7c009040 comp rx=0 tx=0).stop 2024-03-24T10:52:48.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:48.459+0000 7f3f91bc2640 1 -- 172.21.15.5:0/3499277540 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3f8c154220 msgr2=0x7f3f8c15dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:48.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:48.459+0000 7f3f91bc2640 1 --2- 172.21.15.5:0/3499277540 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3f8c154220 0x7f3f8c15dad0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3f8400c790 tx=0x7f3f8400cc50 comp rx=0 tx=0).stop 2024-03-24T10:52:48.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:48.459+0000 7f3f91bc2640 1 -- 172.21.15.5:0/3499277540 shutdown_connections 2024-03-24T10:52:48.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:48.459+0000 7f3f91bc2640 1 --2- 172.21.15.5:0/3499277540 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3f7806b890 0x7f3f7806dd50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:48.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:48.459+0000 7f3f91bc2640 1 --2- 172.21.15.5:0/3499277540 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3f8c154220 0x7f3f8c15dad0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:48.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:48.459+0000 7f3f91bc2640 1 --2- 172.21.15.5:0/3499277540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3f8c0cdfb0 0x7f3f8c155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:48.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:48.459+0000 7f3f91bc2640 1 --2- 172.21.15.5:0/3499277540 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3f8c0c4300 0x7f3f8c1f12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:48.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:48.459+0000 7f3f91bc2640 1 -- 172.21.15.5:0/3499277540 >> 172.21.15.5:0/3499277540 conn(0x7f3f8c06f830 msgr2=0x7f3f8c0c3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:48.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:48.459+0000 7f3f91bc2640 1 -- 172.21.15.5:0/3499277540 shutdown_connections 2024-03-24T10:52:48.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:48.459+0000 7f3f91bc2640 1 -- 172.21.15.5:0/3499277540 wait complete. 2024-03-24T10:52:49.077 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:49.077 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:50.078 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:50.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.179+0000 7f7f27cc9640 1 Processor -- start 2024-03-24T10:52:50.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.179+0000 7f7f27cc9640 1 -- start start 2024-03-24T10:52:50.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.180+0000 7f7f27cc9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f200cc8f0 0x7f7f2015b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:50.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.180+0000 7f7f27cc9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7f2006c580 0x7f7f201538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:50.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.180+0000 7f7f27cc9640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f200cd890 con 0x7f7f200cf110 2024-03-24T10:52:50.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.180+0000 7f7f27cc9640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f7f2006cae0 con 0x7f7f2006c580 2024-03-24T10:52:50.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.180+0000 7f7f27cc9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7f200cdb50 con 0x7f7f200cc8f0 2024-03-24T10:52:50.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.181+0000 7f7f25a65640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f200cc8f0 0x7f7f2015b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:50.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.181+0000 7f7f25264640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7f2006c580 0x7f7f201538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:50.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.181+0000 7f7f26266640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f7f200cf110 0x7f7f20153df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:37022/0 (socket says 172.21.15.5:37022) 2024-03-24T10:52:50.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.181+0000 7f7f26266640 1 -- 172.21.15.5:0/1758486765 learned_addr learned my addr 172.21.15.5:0/1758486765 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:52:50.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.184+0000 7f7f24a63640 1 -- 172.21.15.5:0/1758486765 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1368933553 0 0) 0x7f7f200cd890 con 0x7f7f200cf110 2024-03-24T10:52:50.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.184+0000 7f7f24a63640 1 -- 172.21.15.5:0/1758486765 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f0c003520 con 0x7f7f200cf110 2024-03-24T10:52:50.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.184+0000 7f7f25264640 1 -- 172.21.15.5:0/1758486765 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f200cc8f0 msgr2=0x7f7f2015b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:50.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.184+0000 7f7f25264640 1 --2- 172.21.15.5:0/1758486765 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f200cc8f0 0x7f7f2015b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:50.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.184+0000 7f7f25264640 1 -- 172.21.15.5:0/1758486765 >> v1:172.21.15.5:6789/0 conn(0x7f7f200cf110 legacy=0x7f7f20153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:50.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.184+0000 7f7f25264640 1 -- 172.21.15.5:0/1758486765 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f200cde30 con 0x7f7f2006c580 2024-03-24T10:52:50.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.186+0000 7f7f25264640 1 --2- 172.21.15.5:0/1758486765 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f2006c580 0x7f7f201538b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7f100097a0 tx=0x7f7f1002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfd3efd226ffc22 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:50.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.191+0000 7f7f24a63640 1 -- 172.21.15.5:0/1758486765 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f10031830 con 0x7f7f2006c580 2024-03-24T10:52:50.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.191+0000 7f7f24a63640 1 -- 172.21.15.5:0/1758486765 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f10031990 con 0x7f7f2006c580 2024-03-24T10:52:50.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.191+0000 7f7f24a63640 1 -- 172.21.15.5:0/1758486765 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f10031d20 con 0x7f7f2006c580 2024-03-24T10:52:50.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.192+0000 7f7f27cc9640 1 -- 172.21.15.5:0/1758486765 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f2006c580 msgr2=0x7f7f201538b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:50.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.192+0000 7f7f27cc9640 1 --2- 172.21.15.5:0/1758486765 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f2006c580 0x7f7f201538b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7f100097a0 tx=0x7f7f1002b8d0 comp rx=0 tx=0).stop 2024-03-24T10:52:50.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.192+0000 7f7f27cc9640 1 -- 172.21.15.5:0/1758486765 shutdown_connections 2024-03-24T10:52:50.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.192+0000 7f7f27cc9640 1 --2- 172.21.15.5:0/1758486765 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f2006c580 0x7f7f201538b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:50.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.192+0000 7f7f27cc9640 1 --2- 172.21.15.5:0/1758486765 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f200cc8f0 0x7f7f2015b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:50.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.192+0000 7f7f27cc9640 1 -- 172.21.15.5:0/1758486765 >> 172.21.15.5:0/1758486765 conn(0x7f7f200b1820 msgr2=0x7f7f200b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:50.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.192+0000 7f7f27cc9640 1 -- 172.21.15.5:0/1758486765 shutdown_connections 2024-03-24T10:52:50.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.192+0000 7f7f27cc9640 1 -- 172.21.15.5:0/1758486765 wait complete. 2024-03-24T10:52:50.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.193+0000 7f7f27cc9640 1 Processor -- start 2024-03-24T10:52:50.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.193+0000 7f7f27cc9640 1 -- start start 2024-03-24T10:52:50.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.194+0000 7f7f27cc9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f2006c580 0x7f7f201e6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:50.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.194+0000 7f7f27cc9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f200cc8f0 0x7f7f200c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:50.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.194+0000 7f7f27cc9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f200cf110 0x7f7f200c9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:50.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.194+0000 7f7f27cc9640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f7f200c9130 con 0x7f7f200cc8f0 2024-03-24T10:52:50.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.194+0000 7f7f27cc9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7f201e65f0 con 0x7f7f200cf110 2024-03-24T10:52:50.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.194+0000 7f7f27cc9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7f200c9fb0 con 0x7f7f2006c580 2024-03-24T10:52:50.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.195+0000 7f7f25264640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f200cc8f0 0x7f7f200c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:50.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.195+0000 7f7f25a65640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f2006c580 0x7f7f201e6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:50.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.195+0000 7f7f25264640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f200cc8f0 0x7f7f200c9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:48890/0 (socket says 172.21.15.5:48890) 2024-03-24T10:52:50.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.195+0000 7f7f25264640 1 -- 172.21.15.5:0/2908314404 learned_addr learned my addr 172.21.15.5:0/2908314404 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:50.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.195+0000 7f7f26266640 1 --2- 172.21.15.5:0/2908314404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f200cf110 0x7f7f200c9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:50.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.198+0000 7f7f25264640 1 -- 172.21.15.5:0/2908314404 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f2006c580 msgr2=0x7f7f201e6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:50.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.198+0000 7f7f25264640 1 --2- 172.21.15.5:0/2908314404 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f2006c580 0x7f7f201e6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:50.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.198+0000 7f7f25264640 1 -- 172.21.15.5:0/2908314404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f200cf110 msgr2=0x7f7f200c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:50.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.198+0000 7f7f25264640 1 --2- 172.21.15.5:0/2908314404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f200cf110 0x7f7f200c9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:50.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.199+0000 7f7f25264640 1 -- 172.21.15.5:0/2908314404 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f10009400 con 0x7f7f200cc8f0 2024-03-24T10:52:50.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.201+0000 7f7f25264640 1 --2- 172.21.15.5:0/2908314404 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f200cc8f0 0x7f7f200c9530 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7f100098c0 tx=0x7f7f100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:50.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.204+0000 7f7f167fc640 1 -- 172.21.15.5:0/2908314404 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1003a8f0 con 0x7f7f200cc8f0 2024-03-24T10:52:50.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.204+0000 7f7f27cc9640 1 -- 172.21.15.5:0/2908314404 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f200c6db0 con 0x7f7f200cc8f0 2024-03-24T10:52:50.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.204+0000 7f7f27cc9640 1 -- 172.21.15.5:0/2908314404 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f201f4380 con 0x7f7f200cc8f0 2024-03-24T10:52:50.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.205+0000 7f7f27cc9640 1 -- 172.21.15.5:0/2908314404 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f201e6780 con 0x7f7f200cc8f0 2024-03-24T10:52:50.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.205+0000 7f7f167fc640 1 -- 172.21.15.5:0/2908314404 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f1003aa50 con 0x7f7f200cc8f0 2024-03-24T10:52:50.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.205+0000 7f7f167fc640 1 -- 172.21.15.5:0/2908314404 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1003adc0 con 0x7f7f200cc8f0 2024-03-24T10:52:50.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.472+0000 7f7f167fc640 1 -- 172.21.15.5:0/2908314404 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7f1004b450 con 0x7f7f200cc8f0 2024-03-24T10:52:50.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.473+0000 7f7f167fc640 1 --2- 172.21.15.5:0/2908314404 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7f0406b840 0x7f7f0406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:50.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.473+0000 7f7f25a65640 1 --2- 172.21.15.5:0/2908314404 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7f0406b840 0x7f7f0406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:50.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.473+0000 7f7f25a65640 1 --2- 172.21.15.5:0/2908314404 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7f0406b840 0x7f7f0406dd00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c009b40 tx=0x7f7f1c009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:50.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.489+0000 7f7f167fc640 1 -- 172.21.15.5:0/2908314404 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f7f1007ab60 con 0x7f7f200cc8f0 2024-03-24T10:52:50.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.490+0000 7f7f167fc640 1 -- 172.21.15.5:0/2908314404 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7f10076e00 con 0x7f7f200cc8f0 2024-03-24T10:52:50.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.786+0000 7f7f27cc9640 1 -- 172.21.15.5:0/2908314404 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f200cccd0 con 0x7f7f200cc8f0 2024-03-24T10:52:50.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.800+0000 7f7f167fc640 1 -- 172.21.15.5:0/2908314404 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f7f10035070 con 0x7f7f200cc8f0 2024-03-24T10:52:50.800 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:50.800 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:50.801 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:50.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.802+0000 7f7f27cc9640 1 -- 172.21.15.5:0/2908314404 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7f0406b840 msgr2=0x7f7f0406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:50.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.802+0000 7f7f27cc9640 1 --2- 172.21.15.5:0/2908314404 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7f0406b840 0x7f7f0406dd00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c009b40 tx=0x7f7f1c009190 comp rx=0 tx=0).stop 2024-03-24T10:52:50.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.802+0000 7f7f27cc9640 1 -- 172.21.15.5:0/2908314404 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f200cc8f0 msgr2=0x7f7f200c9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:50.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.802+0000 7f7f27cc9640 1 --2- 172.21.15.5:0/2908314404 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f200cc8f0 0x7f7f200c9530 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7f100098c0 tx=0x7f7f100046c0 comp rx=0 tx=0).stop 2024-03-24T10:52:50.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.802+0000 7f7f27cc9640 1 -- 172.21.15.5:0/2908314404 shutdown_connections 2024-03-24T10:52:50.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.802+0000 7f7f27cc9640 1 --2- 172.21.15.5:0/2908314404 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7f0406b840 0x7f7f0406dd00 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:50.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.802+0000 7f7f27cc9640 1 --2- 172.21.15.5:0/2908314404 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f200cc8f0 0x7f7f200c9530 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:50.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.802+0000 7f7f27cc9640 1 --2- 172.21.15.5:0/2908314404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f200cf110 0x7f7f200c9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:50.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.802+0000 7f7f27cc9640 1 --2- 172.21.15.5:0/2908314404 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f2006c580 0x7f7f201e6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:50.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.802+0000 7f7f27cc9640 1 -- 172.21.15.5:0/2908314404 >> 172.21.15.5:0/2908314404 conn(0x7f7f200b1820 msgr2=0x7f7f2006bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:50.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.802+0000 7f7f27cc9640 1 -- 172.21.15.5:0/2908314404 shutdown_connections 2024-03-24T10:52:50.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:50.802+0000 7f7f27cc9640 1 -- 172.21.15.5:0/2908314404 wait complete. 2024-03-24T10:52:51.417 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:51.417 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:52.418 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:52.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.521+0000 7fd98f283640 1 Processor -- start 2024-03-24T10:52:52.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.522+0000 7fd98f283640 1 -- start start 2024-03-24T10:52:52.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.522+0000 7fd98f283640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd9880cae90 0x7fd9880cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:52.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.522+0000 7fd98f283640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd9880cb270 0x7fd9880cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:52.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.522+0000 7fd98f283640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9880ccba0 con 0x7fd9880cd960 2024-03-24T10:52:52.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.522+0000 7fd98f283640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fd9880cce40 con 0x7fd9880cae90 2024-03-24T10:52:52.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.522+0000 7fd98f283640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd98815a590 con 0x7fd9880cb270 2024-03-24T10:52:52.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.523+0000 7fd98d01f640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd9880cae90 0x7fd9880cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:52.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.523+0000 7fd98d820640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fd9880cd960 0x7fd9880cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:50832/0 (socket says 172.21.15.5:50832) 2024-03-24T10:52:52.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.523+0000 7fd98d820640 1 -- 172.21.15.5:0/153274720 learned_addr learned my addr 172.21.15.5:0/153274720 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:52:52.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.523+0000 7fd98c81e640 1 --2- 172.21.15.5:0/153274720 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd9880cb270 0x7fd9880cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:52.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.526+0000 7fd98d01f640 1 -- 172.21.15.5:0/153274720 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd9880cb270 msgr2=0x7fd9880cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:52.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.526+0000 7fd98d01f640 1 --2- 172.21.15.5:0/153274720 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd9880cb270 0x7fd9880cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:52.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.526+0000 7fd98d01f640 1 -- 172.21.15.5:0/153274720 >> v1:172.21.15.57:6789/0 conn(0x7fd9880cd960 legacy=0x7fd9880cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:52.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.526+0000 7fd98d01f640 1 -- 172.21.15.5:0/153274720 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98815a870 con 0x7fd9880cae90 2024-03-24T10:52:52.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.528+0000 7fd98d01f640 1 --2- 172.21.15.5:0/153274720 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9880cae90 0x7fd9880cf530 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd9840097a0 tx=0x7fd98402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba1124878ea8b1b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:52.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.530+0000 7fd97ffff640 1 -- 172.21.15.5:0/153274720 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984030610 con 0x7fd9880cae90 2024-03-24T10:52:52.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.532+0000 7fd97ffff640 1 -- 172.21.15.5:0/153274720 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd984030770 con 0x7fd9880cae90 2024-03-24T10:52:52.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.532+0000 7fd97ffff640 1 -- 172.21.15.5:0/153274720 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984030b00 con 0x7fd9880cae90 2024-03-24T10:52:52.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.532+0000 7fd98f283640 1 -- 172.21.15.5:0/153274720 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9880cae90 msgr2=0x7fd9880cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:52.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.532+0000 7fd98f283640 1 --2- 172.21.15.5:0/153274720 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9880cae90 0x7fd9880cf530 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd9840097a0 tx=0x7fd98402b960 comp rx=0 tx=0).stop 2024-03-24T10:52:52.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.532+0000 7fd98f283640 1 -- 172.21.15.5:0/153274720 shutdown_connections 2024-03-24T10:52:52.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.532+0000 7fd98f283640 1 --2- 172.21.15.5:0/153274720 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd9880cb270 0x7fd9880cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:52.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.532+0000 7fd98f283640 1 --2- 172.21.15.5:0/153274720 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9880cae90 0x7fd9880cf530 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:52.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.532+0000 7fd98f283640 1 -- 172.21.15.5:0/153274720 >> 172.21.15.5:0/153274720 conn(0x7fd9880cfb80 msgr2=0x7fd9880cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:52.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.532+0000 7fd98f283640 1 -- 172.21.15.5:0/153274720 shutdown_connections 2024-03-24T10:52:52.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.532+0000 7fd98f283640 1 -- 172.21.15.5:0/153274720 wait complete. 2024-03-24T10:52:52.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.534+0000 7fd98f283640 1 Processor -- start 2024-03-24T10:52:52.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.534+0000 7fd98f283640 1 -- start start 2024-03-24T10:52:52.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.535+0000 7fd98f283640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd9880cae90 0x7fd988150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:52.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.535+0000 7fd98f283640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9880cb270 0x7fd98815cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:52.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.535+0000 7fd98f283640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9880cd960 0x7fd9880b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:52.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.535+0000 7fd98f283640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd9880b4db0 con 0x7fd9880cd960 2024-03-24T10:52:52.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.535+0000 7fd98f283640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd9880b5410 con 0x7fd9880cb270 2024-03-24T10:52:52.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.535+0000 7fd98f283640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd9880b5150 con 0x7fd9880cae90 2024-03-24T10:52:52.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.535+0000 7fd98c81e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9880cb270 0x7fd98815cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:52.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.535+0000 7fd98c81e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9880cb270 0x7fd98815cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:40008/0 (socket says 172.21.15.5:40008) 2024-03-24T10:52:52.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.535+0000 7fd98c81e640 1 -- 172.21.15.5:0/1457589204 learned_addr learned my addr 172.21.15.5:0/1457589204 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:52.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.536+0000 7fd98d820640 1 --2- 172.21.15.5:0/1457589204 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9880cd960 0x7fd9880b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:52.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.537+0000 7fd98d01f640 1 --2- 172.21.15.5:0/1457589204 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd9880cae90 0x7fd988150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:52.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.540+0000 7fd98d820640 1 -- 172.21.15.5:0/1457589204 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd9880cae90 msgr2=0x7fd988150130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:52.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.540+0000 7fd98d820640 1 --2- 172.21.15.5:0/1457589204 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd9880cae90 0x7fd988150130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:52.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.540+0000 7fd98d820640 1 -- 172.21.15.5:0/1457589204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9880cb270 msgr2=0x7fd98815cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:52.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.540+0000 7fd98d820640 1 --2- 172.21.15.5:0/1457589204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9880cb270 0x7fd98815cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:52.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.540+0000 7fd98d820640 1 -- 172.21.15.5:0/1457589204 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd984009400 con 0x7fd9880cd960 2024-03-24T10:52:52.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.542+0000 7fd98d820640 1 --2- 172.21.15.5:0/1457589204 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9880cd960 0x7fd9880b4870 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd98000caa0 tx=0x7fd98000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:52.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.547+0000 7fd97dffb640 1 -- 172.21.15.5:0/1457589204 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98000ab40 con 0x7fd9880cd960 2024-03-24T10:52:52.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.547+0000 7fd97dffb640 1 -- 172.21.15.5:0/1457589204 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd98000aca0 con 0x7fd9880cd960 2024-03-24T10:52:52.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.547+0000 7fd98f283640 1 -- 172.21.15.5:0/1457589204 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9880b5620 con 0x7fd9880cd960 2024-03-24T10:52:52.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.547+0000 7fd97dffb640 1 -- 172.21.15.5:0/1457589204 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd980012600 con 0x7fd9880cd960 2024-03-24T10:52:52.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.547+0000 7fd98f283640 1 -- 172.21.15.5:0/1457589204 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9880b3a90 con 0x7fd9880cd960 2024-03-24T10:52:52.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.548+0000 7fd98f283640 1 -- 172.21.15.5:0/1457589204 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd98815c8e0 con 0x7fd9880cd960 2024-03-24T10:52:52.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.571+0000 7fd97dffb640 1 -- 172.21.15.5:0/1457589204 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd980007070 con 0x7fd9880cd960 2024-03-24T10:52:52.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.571+0000 7fd97dffb640 1 --2- 172.21.15.5:0/1457589204 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd96c06b890 0x7fd96c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:52.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.571+0000 7fd98d01f640 1 --2- 172.21.15.5:0/1457589204 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd96c06b890 0x7fd96c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:52.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.572+0000 7fd98d01f640 1 --2- 172.21.15.5:0/1457589204 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd96c06b890 0x7fd96c06dd50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd984036000 tx=0x7fd984008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:52.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.575+0000 7fd97dffb640 1 -- 172.21.15.5:0/1457589204 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fd980058ac0 con 0x7fd9880cd960 2024-03-24T10:52:52.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:52.845+0000 7fd97dffb640 1 -- 172.21.15.5:0/1457589204 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd980054d60 con 0x7fd9880cd960 2024-03-24T10:52:53.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:53.068+0000 7fd98f283640 1 -- 172.21.15.5:0/1457589204 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd98815c8e0 con 0x7fd9880cd960 2024-03-24T10:52:53.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:53.081+0000 7fd97dffb640 1 -- 172.21.15.5:0/1457589204 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fd98815c8e0 con 0x7fd9880cd960 2024-03-24T10:52:53.081 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:53.081 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:53.082 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:53.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:53.083+0000 7fd98f283640 1 -- 172.21.15.5:0/1457589204 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd96c06b890 msgr2=0x7fd96c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:53.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:53.083+0000 7fd98f283640 1 --2- 172.21.15.5:0/1457589204 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd96c06b890 0x7fd96c06dd50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd984036000 tx=0x7fd984008030 comp rx=0 tx=0).stop 2024-03-24T10:52:53.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:53.083+0000 7fd98f283640 1 -- 172.21.15.5:0/1457589204 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9880cd960 msgr2=0x7fd9880b4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:53.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:53.083+0000 7fd98f283640 1 --2- 172.21.15.5:0/1457589204 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9880cd960 0x7fd9880b4870 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd98000caa0 tx=0x7fd98000cf60 comp rx=0 tx=0).stop 2024-03-24T10:52:53.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:53.083+0000 7fd98f283640 1 -- 172.21.15.5:0/1457589204 shutdown_connections 2024-03-24T10:52:53.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:53.083+0000 7fd98f283640 1 --2- 172.21.15.5:0/1457589204 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd96c06b890 0x7fd96c06dd50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:53.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:53.083+0000 7fd98f283640 1 --2- 172.21.15.5:0/1457589204 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd9880cd960 0x7fd9880b4870 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:53.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:53.083+0000 7fd98f283640 1 --2- 172.21.15.5:0/1457589204 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd9880cb270 0x7fd98815cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:53.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:53.083+0000 7fd98f283640 1 --2- 172.21.15.5:0/1457589204 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd9880cae90 0x7fd988150130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:53.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:53.083+0000 7fd98f283640 1 -- 172.21.15.5:0/1457589204 >> 172.21.15.5:0/1457589204 conn(0x7fd9880cfb80 msgr2=0x7fd9880c8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:53.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:53.083+0000 7fd98f283640 1 -- 172.21.15.5:0/1457589204 shutdown_connections 2024-03-24T10:52:53.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:53.083+0000 7fd98f283640 1 -- 172.21.15.5:0/1457589204 wait complete. 2024-03-24T10:52:53.699 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:53.699 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:54.700 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:54.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.802+0000 7f61f91a1640 1 Processor -- start 2024-03-24T10:52:54.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.802+0000 7f61f91a1640 1 -- start start 2024-03-24T10:52:54.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.803+0000 7f61f91a1640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f61f41543a0 0x7f61f415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:54.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.803+0000 7f61f91a1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61f406d4f0 0x7f61f40cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:54.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.803+0000 7f61f91a1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f61f40ccaa0 0x7f61f406bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:54.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.803+0000 7f61f91a1640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f61f40c6750 con 0x7f61f41543a0 2024-03-24T10:52:54.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.803+0000 7f61f91a1640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f61f40cc380 con 0x7f61f40ccaa0 2024-03-24T10:52:54.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.803+0000 7f61f91a1640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f61f40cb2f0 con 0x7f61f406d4f0 2024-03-24T10:52:54.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.803+0000 7f61f2d9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f61f41543a0 0x7f61f415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:54.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.804+0000 7f61ebfff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61f406d4f0 0x7f61f40cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:54.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.804+0000 7f61f359e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f61f40ccaa0 0x7f61f406bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:54.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.804+0000 7f61f2d9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f61f41543a0 0x7f61f415ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35030/0 (socket says 172.21.15.5:35030) 2024-03-24T10:52:54.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.804+0000 7f61f2d9d640 1 -- 172.21.15.5:0/2538485825 learned_addr learned my addr 172.21.15.5:0/2538485825 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:54.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.807+0000 7f61f2d9d640 1 -- 172.21.15.5:0/2538485825 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61f406d4f0 msgr2=0x7f61f40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:54.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.807+0000 7f61f2d9d640 1 --2- 172.21.15.5:0/2538485825 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61f406d4f0 0x7f61f40cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:54.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.807+0000 7f61f2d9d640 1 -- 172.21.15.5:0/2538485825 >> v2:172.21.15.57:3300/0 conn(0x7f61f40ccaa0 msgr2=0x7f61f406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:54.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.807+0000 7f61f2d9d640 1 --2- 172.21.15.5:0/2538485825 >> v2:172.21.15.57:3300/0 conn(0x7f61f40ccaa0 0x7f61f406bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:54.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.807+0000 7f61f2d9d640 1 -- 172.21.15.5:0/2538485825 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f40ca7e0 con 0x7f61f41543a0 2024-03-24T10:52:54.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.810+0000 7f61f2d9d640 1 --2- 172.21.15.5:0/2538485825 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61f41543a0 0x7f61f415ac00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f61f40628b0 tx=0x7f61e402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=141eec37a3d104d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:54.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.815+0000 7f61f259c640 1 -- 172.21.15.5:0/2538485825 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e4030860 con 0x7f61f41543a0 2024-03-24T10:52:54.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.815+0000 7f61f259c640 1 -- 172.21.15.5:0/2538485825 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61e40309c0 con 0x7f61f41543a0 2024-03-24T10:52:54.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.815+0000 7f61f259c640 1 -- 172.21.15.5:0/2538485825 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e4030d50 con 0x7f61f41543a0 2024-03-24T10:52:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.815+0000 7f61f91a1640 1 -- 172.21.15.5:0/2538485825 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61f41543a0 msgr2=0x7f61f415ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.815+0000 7f61f91a1640 1 --2- 172.21.15.5:0/2538485825 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61f41543a0 0x7f61f415ac00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f61f40628b0 tx=0x7f61e402b9a0 comp rx=0 tx=0).stop 2024-03-24T10:52:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.816+0000 7f61f91a1640 1 -- 172.21.15.5:0/2538485825 shutdown_connections 2024-03-24T10:52:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.816+0000 7f61f91a1640 1 --2- 172.21.15.5:0/2538485825 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61f406d4f0 0x7f61f40cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.816+0000 7f61f91a1640 1 --2- 172.21.15.5:0/2538485825 >> v2:172.21.15.57:3300/0 conn(0x7f61f40ccaa0 0x7f61f406bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.816+0000 7f61f91a1640 1 --2- 172.21.15.5:0/2538485825 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61f41543a0 0x7f61f415ac00 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.816+0000 7f61f91a1640 1 -- 172.21.15.5:0/2538485825 >> 172.21.15.5:0/2538485825 conn(0x7f61f4072ef0 msgr2=0x7f61f4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.816+0000 7f61f91a1640 1 -- 172.21.15.5:0/2538485825 shutdown_connections 2024-03-24T10:52:54.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.816+0000 7f61f91a1640 1 -- 172.21.15.5:0/2538485825 wait complete. 2024-03-24T10:52:54.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.817+0000 7f61f91a1640 1 Processor -- start 2024-03-24T10:52:54.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.817+0000 7f61f91a1640 1 -- start start 2024-03-24T10:52:54.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.818+0000 7f61f91a1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61f406d4f0 0x7f61f4159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:54.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.818+0000 7f61f91a1640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61f40ccaa0 0x7f61f4144350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:54.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.818+0000 7f61f91a1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61f41543a0 0x7f61f4144890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:54.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.818+0000 7f61f91a1640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f61f40b4fe0 con 0x7f61f40ccaa0 2024-03-24T10:52:54.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.818+0000 7f61f91a1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f61f40b4a40 con 0x7f61f41543a0 2024-03-24T10:52:54.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.818+0000 7f61f91a1640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f61f40b4d00 con 0x7f61f406d4f0 2024-03-24T10:52:54.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.818+0000 7f61ebfff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61f40ccaa0 0x7f61f4144350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:54.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.818+0000 7f61f2d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61f406d4f0 0x7f61f4159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:54.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.818+0000 7f61f359e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61f41543a0 0x7f61f4144890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:54.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.818+0000 7f61ebfff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61f40ccaa0 0x7f61f4144350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35042/0 (socket says 172.21.15.5:35042) 2024-03-24T10:52:54.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.819+0000 7f61ebfff640 1 -- 172.21.15.5:0/2524236227 learned_addr learned my addr 172.21.15.5:0/2524236227 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:54.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.822+0000 7f61ebfff640 1 -- 172.21.15.5:0/2524236227 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61f406d4f0 msgr2=0x7f61f4159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:54.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.822+0000 7f61ebfff640 1 --2- 172.21.15.5:0/2524236227 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61f406d4f0 0x7f61f4159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:54.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.822+0000 7f61ebfff640 1 -- 172.21.15.5:0/2524236227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61f41543a0 msgr2=0x7f61f4144890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:54.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.822+0000 7f61ebfff640 1 --2- 172.21.15.5:0/2524236227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61f41543a0 0x7f61f4144890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:54.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.822+0000 7f61ebfff640 1 -- 172.21.15.5:0/2524236227 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61e40094d0 con 0x7f61f40ccaa0 2024-03-24T10:52:54.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.825+0000 7f61ebfff640 1 --2- 172.21.15.5:0/2524236227 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61f40ccaa0 0x7f61f4144350 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00e9b0 tx=0x7f61dc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:54.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.830+0000 7f61eb7fe640 1 -- 172.21.15.5:0/2524236227 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61dc009670 con 0x7f61f40ccaa0 2024-03-24T10:52:54.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.830+0000 7f61eb7fe640 1 -- 172.21.15.5:0/2524236227 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61dc00c9a0 con 0x7f61f40ccaa0 2024-03-24T10:52:54.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.830+0000 7f61f91a1640 1 -- 172.21.15.5:0/2524236227 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61f4154c00 con 0x7f61f40ccaa0 2024-03-24T10:52:54.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.830+0000 7f61eb7fe640 1 -- 172.21.15.5:0/2524236227 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61dc00cd50 con 0x7f61f40ccaa0 2024-03-24T10:52:54.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.830+0000 7f61f91a1640 1 -- 172.21.15.5:0/2524236227 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61f41560d0 con 0x7f61f40ccaa0 2024-03-24T10:52:54.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:54.831+0000 7f61f91a1640 1 -- 172.21.15.5:0/2524236227 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61f4154a60 con 0x7f61f40ccaa0 2024-03-24T10:52:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:55.061+0000 7f61eb7fe640 1 -- 172.21.15.5:0/2524236227 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f61dc013470 con 0x7f61f40ccaa0 2024-03-24T10:52:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:55.062+0000 7f61eb7fe640 1 --2- 172.21.15.5:0/2524236227 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f61d406b840 0x7f61d406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:55.062+0000 7f61f2d9d640 1 --2- 172.21.15.5:0/2524236227 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f61d406b840 0x7f61d406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:55.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:55.062+0000 7f61f2d9d640 1 --2- 172.21.15.5:0/2524236227 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f61d406b840 0x7f61d406dd00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f61e4036000 tx=0x7f61e4008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:55.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:55.133+0000 7f61eb7fe640 1 -- 172.21.15.5:0/2524236227 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f61dc0591e0 con 0x7f61f40ccaa0 2024-03-24T10:52:55.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:55.134+0000 7f61eb7fe640 1 -- 172.21.15.5:0/2524236227 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f61dc055480 con 0x7f61f40ccaa0 2024-03-24T10:52:55.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:55.403+0000 7f61f91a1640 1 -- 172.21.15.5:0/2524236227 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61f4154a60 con 0x7f61f40ccaa0 2024-03-24T10:52:55.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:55.418+0000 7f61eb7fe640 1 -- 172.21.15.5:0/2524236227 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f61f4154a60 con 0x7f61f40ccaa0 2024-03-24T10:52:55.419 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:55.419 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:55.419 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:55.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:55.420+0000 7f61f91a1640 1 -- 172.21.15.5:0/2524236227 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f61d406b840 msgr2=0x7f61d406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:55.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:55.420+0000 7f61f91a1640 1 --2- 172.21.15.5:0/2524236227 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f61d406b840 0x7f61d406dd00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f61e4036000 tx=0x7f61e4008030 comp rx=0 tx=0).stop 2024-03-24T10:52:55.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:55.420+0000 7f61f91a1640 1 -- 172.21.15.5:0/2524236227 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61f40ccaa0 msgr2=0x7f61f4144350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:55.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:55.420+0000 7f61f91a1640 1 --2- 172.21.15.5:0/2524236227 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61f40ccaa0 0x7f61f4144350 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00e9b0 tx=0x7f61dc00ee70 comp rx=0 tx=0).stop 2024-03-24T10:52:55.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:55.421+0000 7f61f91a1640 1 -- 172.21.15.5:0/2524236227 shutdown_connections 2024-03-24T10:52:55.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:55.421+0000 7f61f91a1640 1 --2- 172.21.15.5:0/2524236227 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f61d406b840 0x7f61d406dd00 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:55.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:55.421+0000 7f61f91a1640 1 --2- 172.21.15.5:0/2524236227 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f61f40ccaa0 0x7f61f4144350 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:55.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:55.421+0000 7f61f91a1640 1 --2- 172.21.15.5:0/2524236227 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f61f41543a0 0x7f61f4144890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:55.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:55.421+0000 7f61f91a1640 1 --2- 172.21.15.5:0/2524236227 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f61f406d4f0 0x7f61f4159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:55.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:55.421+0000 7f61f91a1640 1 -- 172.21.15.5:0/2524236227 >> 172.21.15.5:0/2524236227 conn(0x7f61f4072ef0 msgr2=0x7f61f406bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:55.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:55.421+0000 7f61f91a1640 1 -- 172.21.15.5:0/2524236227 shutdown_connections 2024-03-24T10:52:55.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:55.421+0000 7f61f91a1640 1 -- 172.21.15.5:0/2524236227 wait complete. 2024-03-24T10:52:56.037 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:56.038 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:57.038 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:57.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.141+0000 7fbbc0e40640 1 Processor -- start 2024-03-24T10:52:57.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.141+0000 7fbbc0e40640 1 -- start start 2024-03-24T10:52:57.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.143+0000 7fbbc0e40640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbbbc1543a0 0x7fbbbc15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:57.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.143+0000 7fbbc0e40640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbbbc06d4f0 0x7fbbbc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:57.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.143+0000 7fbbc0e40640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbbbc0ccaa0 0x7fbbbc06bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:57.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.143+0000 7fbbc0e40640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fbbbc0c6750 con 0x7fbbbc1543a0 2024-03-24T10:52:57.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.143+0000 7fbbc0e40640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fbbbc0cceb0 con 0x7fbbbc06d4f0 2024-03-24T10:52:57.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.143+0000 7fbbc0e40640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fbbbc0cc380 con 0x7fbbbc0ccaa0 2024-03-24T10:52:57.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.143+0000 7fbbba59c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbbbc1543a0 0x7fbbbc15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:57.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.143+0000 7fbbbad9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbbbc0ccaa0 0x7fbbbc06bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:57.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.143+0000 7fbbb9d9b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbbbc06d4f0 0x7fbbbc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:57.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.144+0000 7fbbba59c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbbbc1543a0 0x7fbbbc15ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35046/0 (socket says 172.21.15.5:35046) 2024-03-24T10:52:57.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.144+0000 7fbbba59c640 1 -- 172.21.15.5:0/1090253074 learned_addr learned my addr 172.21.15.5:0/1090253074 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:57.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.146+0000 7fbbba59c640 1 -- 172.21.15.5:0/1090253074 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbbbc0ccaa0 msgr2=0x7fbbbc06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:57.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.146+0000 7fbbba59c640 1 --2- 172.21.15.5:0/1090253074 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbbbc0ccaa0 0x7fbbbc06bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:57.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.146+0000 7fbbba59c640 1 -- 172.21.15.5:0/1090253074 >> v2:172.21.15.57:3300/0 conn(0x7fbbbc06d4f0 msgr2=0x7fbbbc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:57.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.146+0000 7fbbba59c640 1 --2- 172.21.15.5:0/1090253074 >> v2:172.21.15.57:3300/0 conn(0x7fbbbc06d4f0 0x7fbbbc0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:57.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.146+0000 7fbbba59c640 1 -- 172.21.15.5:0/1090253074 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbbc0cb2f0 con 0x7fbbbc1543a0 2024-03-24T10:52:57.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.148+0000 7fbbba59c640 1 --2- 172.21.15.5:0/1090253074 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbbc1543a0 0x7fbbbc15ac00 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc0c6990 tx=0x7fbbac02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c04eda29f0985b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:57.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.151+0000 7fbbb959a640 1 -- 172.21.15.5:0/1090253074 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac030860 con 0x7fbbbc1543a0 2024-03-24T10:52:57.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.151+0000 7fbbb959a640 1 -- 172.21.15.5:0/1090253074 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbac0309c0 con 0x7fbbbc1543a0 2024-03-24T10:52:57.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.151+0000 7fbbb959a640 1 -- 172.21.15.5:0/1090253074 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac030d50 con 0x7fbbbc1543a0 2024-03-24T10:52:57.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.152+0000 7fbbc0e40640 1 -- 172.21.15.5:0/1090253074 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbbc1543a0 msgr2=0x7fbbbc15ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:57.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.152+0000 7fbbc0e40640 1 --2- 172.21.15.5:0/1090253074 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbbc1543a0 0x7fbbbc15ac00 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc0c6990 tx=0x7fbbac02b9a0 comp rx=0 tx=0).stop 2024-03-24T10:52:57.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.153+0000 7fbbc0e40640 1 -- 172.21.15.5:0/1090253074 shutdown_connections 2024-03-24T10:52:57.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.153+0000 7fbbc0e40640 1 --2- 172.21.15.5:0/1090253074 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbbbc0ccaa0 0x7fbbbc06bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:57.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.153+0000 7fbbc0e40640 1 --2- 172.21.15.5:0/1090253074 >> v2:172.21.15.57:3300/0 conn(0x7fbbbc06d4f0 0x7fbbbc0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:57.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.153+0000 7fbbc0e40640 1 --2- 172.21.15.5:0/1090253074 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbbc1543a0 0x7fbbbc15ac00 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:57.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.153+0000 7fbbc0e40640 1 -- 172.21.15.5:0/1090253074 >> 172.21.15.5:0/1090253074 conn(0x7fbbbc072ef0 msgr2=0x7fbbbc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:57.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.153+0000 7fbbc0e40640 1 -- 172.21.15.5:0/1090253074 shutdown_connections 2024-03-24T10:52:57.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.153+0000 7fbbc0e40640 1 -- 172.21.15.5:0/1090253074 wait complete. 2024-03-24T10:52:57.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.155+0000 7fbbc0e40640 1 Processor -- start 2024-03-24T10:52:57.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.155+0000 7fbbc0e40640 1 -- start start 2024-03-24T10:52:57.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.156+0000 7fbbc0e40640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbbbc06d4f0 0x7fbbbc1554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:57.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.156+0000 7fbbc0e40640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbbc0ccaa0 0x7fbbbc0c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:57.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.156+0000 7fbbc0e40640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbbc1543a0 0x7fbbbc1f3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:57.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.156+0000 7fbbc0e40640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc154a60 con 0x7fbbbc0ccaa0 2024-03-24T10:52:57.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.156+0000 7fbbc0e40640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc1f31d0 con 0x7fbbbc1543a0 2024-03-24T10:52:57.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.156+0000 7fbbc0e40640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fbbbc154cb0 con 0x7fbbbc06d4f0 2024-03-24T10:52:57.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.156+0000 7fbbbad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbbc1543a0 0x7fbbbc1f3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:57.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.157+0000 7fbbba59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbbbc06d4f0 0x7fbbbc1554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:57.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.157+0000 7fbbb9d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbbc0ccaa0 0x7fbbbc0c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:57.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.157+0000 7fbbbad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbbc1543a0 0x7fbbbc1f3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:40030/0 (socket says 172.21.15.5:40030) 2024-03-24T10:52:57.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.157+0000 7fbbbad9d640 1 -- 172.21.15.5:0/1546754038 learned_addr learned my addr 172.21.15.5:0/1546754038 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:57.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.162+0000 7fbbbad9d640 1 -- 172.21.15.5:0/1546754038 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbbbc06d4f0 msgr2=0x7fbbbc1554b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:57.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.162+0000 7fbbbad9d640 1 --2- 172.21.15.5:0/1546754038 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbbbc06d4f0 0x7fbbbc1554b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:57.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.162+0000 7fbbbad9d640 1 -- 172.21.15.5:0/1546754038 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbbc0ccaa0 msgr2=0x7fbbbc0c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:57.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.162+0000 7fbbbad9d640 1 --2- 172.21.15.5:0/1546754038 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbbc0ccaa0 0x7fbbbc0c84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:57.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.162+0000 7fbbbad9d640 1 -- 172.21.15.5:0/1546754038 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbac0094d0 con 0x7fbbbc1543a0 2024-03-24T10:52:57.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.166+0000 7fbbbad9d640 1 --2- 172.21.15.5:0/1546754038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbbc1543a0 0x7fbbbc1f3660 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbbb400efc0 tx=0x7fbbb400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:57.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.170+0000 7fbbaaffd640 1 -- 172.21.15.5:0/1546754038 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4014480 con 0x7fbbbc1543a0 2024-03-24T10:52:57.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.170+0000 7fbbaaffd640 1 -- 172.21.15.5:0/1546754038 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbb400f040 con 0x7fbbbc1543a0 2024-03-24T10:52:57.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.170+0000 7fbbaaffd640 1 -- 172.21.15.5:0/1546754038 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4014820 con 0x7fbbbc1543a0 2024-03-24T10:52:57.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.170+0000 7fbbc0e40640 1 -- 172.21.15.5:0/1546754038 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbbc154f90 con 0x7fbbbc1543a0 2024-03-24T10:52:57.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.170+0000 7fbbc0e40640 1 -- 172.21.15.5:0/1546754038 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbbc1f2e60 con 0x7fbbbc1543a0 2024-03-24T10:52:57.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.171+0000 7fbbc0e40640 1 -- 172.21.15.5:0/1546754038 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbbc1f3360 con 0x7fbbbc1543a0 2024-03-24T10:52:57.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.453+0000 7fbbaaffd640 1 -- 172.21.15.5:0/1546754038 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbbb4014980 con 0x7fbbbc1543a0 2024-03-24T10:52:57.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.453+0000 7fbbaaffd640 1 --2- 172.21.15.5:0/1546754038 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbb9c06b890 0x7fbb9c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:57.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.453+0000 7fbbba59c640 1 --2- 172.21.15.5:0/1546754038 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbb9c06b890 0x7fbb9c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:57.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.454+0000 7fbbba59c640 1 --2- 172.21.15.5:0/1546754038 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbb9c06b890 0x7fbb9c06dd50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbbac036000 tx=0x7fbbac008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:57.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.458+0000 7fbbaaffd640 1 -- 172.21.15.5:0/1546754038 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fbbb4059cf0 con 0x7fbbbc1543a0 2024-03-24T10:52:57.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.469+0000 7fbbaaffd640 1 -- 172.21.15.5:0/1546754038 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbbb4055f90 con 0x7fbbbc1543a0 2024-03-24T10:52:57.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.732+0000 7fbbc0e40640 1 -- 172.21.15.5:0/1546754038 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbbbc1f2ff0 con 0x7fbbbc1543a0 2024-03-24T10:52:57.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.745+0000 7fbbaaffd640 1 -- 172.21.15.5:0/1546754038 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fbbb4018070 con 0x7fbbbc1543a0 2024-03-24T10:52:57.745 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:52:57.745 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:52:57.745 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:52:57.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.748+0000 7fbbc0e40640 1 -- 172.21.15.5:0/1546754038 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbb9c06b890 msgr2=0x7fbb9c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:57.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.748+0000 7fbbc0e40640 1 --2- 172.21.15.5:0/1546754038 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbb9c06b890 0x7fbb9c06dd50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbbac036000 tx=0x7fbbac008030 comp rx=0 tx=0).stop 2024-03-24T10:52:57.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.748+0000 7fbbc0e40640 1 -- 172.21.15.5:0/1546754038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbbc1543a0 msgr2=0x7fbbbc1f3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:57.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.748+0000 7fbbc0e40640 1 --2- 172.21.15.5:0/1546754038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbbc1543a0 0x7fbbbc1f3660 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbbb400efc0 tx=0x7fbbb400c490 comp rx=0 tx=0).stop 2024-03-24T10:52:57.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.748+0000 7fbbc0e40640 1 -- 172.21.15.5:0/1546754038 shutdown_connections 2024-03-24T10:52:57.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.748+0000 7fbbc0e40640 1 --2- 172.21.15.5:0/1546754038 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbb9c06b890 0x7fbb9c06dd50 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:57.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.748+0000 7fbbc0e40640 1 --2- 172.21.15.5:0/1546754038 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbbbc0ccaa0 0x7fbbbc0c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:57.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.748+0000 7fbbc0e40640 1 --2- 172.21.15.5:0/1546754038 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbbbc1543a0 0x7fbbbc1f3660 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:57.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.748+0000 7fbbc0e40640 1 --2- 172.21.15.5:0/1546754038 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbbbc06d4f0 0x7fbbbc1554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:57.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.748+0000 7fbbc0e40640 1 -- 172.21.15.5:0/1546754038 >> 172.21.15.5:0/1546754038 conn(0x7fbbbc072ef0 msgr2=0x7fbbbc0c5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:57.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.748+0000 7fbbc0e40640 1 -- 172.21.15.5:0/1546754038 shutdown_connections 2024-03-24T10:52:57.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:57.748+0000 7fbbc0e40640 1 -- 172.21.15.5:0/1546754038 wait complete. 2024-03-24T10:52:58.364 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:52:58.364 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:52:59.365 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:52:59.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.468+0000 7ff5a693e640 1 Processor -- start 2024-03-24T10:52:59.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.468+0000 7ff5a693e640 1 -- start start 2024-03-24T10:52:59.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.469+0000 7ff5a693e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff5a00c2ea0 0x7ff5a00caab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:59.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.469+0000 7ff5a693e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff5a00c58c0 0x7ff5a00c3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:59.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.469+0000 7ff5a693e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5a0156fc0 con 0x7ff5a00c9610 2024-03-24T10:52:59.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.469+0000 7ff5a693e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7ff5a00cb210 con 0x7ff5a00c58c0 2024-03-24T10:52:59.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.469+0000 7ff5a693e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7ff5a0156db0 con 0x7ff5a00c2ea0 2024-03-24T10:52:59.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.469+0000 7ff59ffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff5a00c2ea0 0x7ff5a00caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:59.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.470+0000 7ff59f7fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff5a00c58c0 0x7ff5a00c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:59.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.470+0000 7ff5a4edb640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7ff5a00c9610 0x7ff5a0150840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:58752/0 (socket says 172.21.15.5:58752) 2024-03-24T10:52:59.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.470+0000 7ff5a4edb640 1 -- 172.21.15.5:0/3049113905 learned_addr learned my addr 172.21.15.5:0/3049113905 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:52:59.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.473+0000 7ff59ffff640 1 -- 172.21.15.5:0/3049113905 >> v2:172.21.15.5:3300/0 conn(0x7ff5a00c58c0 msgr2=0x7ff5a00c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:52:59.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.473+0000 7ff59ffff640 1 --2- 172.21.15.5:0/3049113905 >> v2:172.21.15.5:3300/0 conn(0x7ff5a00c58c0 0x7ff5a00c3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:59.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.473+0000 7ff59ffff640 1 -- 172.21.15.5:0/3049113905 >> v1:172.21.15.5:6789/0 conn(0x7ff5a00c9610 legacy=0x7ff5a0150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:59.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.473+0000 7ff59ffff640 1 -- 172.21.15.5:0/3049113905 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a01572a0 con 0x7ff5a00c2ea0 2024-03-24T10:52:59.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.479+0000 7ff59ffff640 1 --2- 172.21.15.5:0/3049113905 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5a00c2ea0 0x7ff5a00caab0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff5a00715e0 tx=0x7ff59002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f962d00fa71edef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:59.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.483+0000 7ff59effd640 1 -- 172.21.15.5:0/3049113905 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff590030610 con 0x7ff5a00c2ea0 2024-03-24T10:52:59.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.484+0000 7ff59effd640 1 -- 172.21.15.5:0/3049113905 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff590030770 con 0x7ff5a00c2ea0 2024-03-24T10:52:59.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.484+0000 7ff59effd640 1 -- 172.21.15.5:0/3049113905 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff590030b00 con 0x7ff5a00c2ea0 2024-03-24T10:52:59.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.484+0000 7ff5a693e640 1 -- 172.21.15.5:0/3049113905 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5a00c2ea0 msgr2=0x7ff5a00caab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:59.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.484+0000 7ff5a693e640 1 --2- 172.21.15.5:0/3049113905 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5a00c2ea0 0x7ff5a00caab0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff5a00715e0 tx=0x7ff59002b960 comp rx=0 tx=0).stop 2024-03-24T10:52:59.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.484+0000 7ff5a693e640 1 -- 172.21.15.5:0/3049113905 shutdown_connections 2024-03-24T10:52:59.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.484+0000 7ff5a693e640 1 --2- 172.21.15.5:0/3049113905 >> v2:172.21.15.5:3300/0 conn(0x7ff5a00c58c0 0x7ff5a00c3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:59.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.484+0000 7ff5a693e640 1 --2- 172.21.15.5:0/3049113905 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5a00c2ea0 0x7ff5a00caab0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:59.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.484+0000 7ff5a693e640 1 -- 172.21.15.5:0/3049113905 >> 172.21.15.5:0/3049113905 conn(0x7ff5a006b130 msgr2=0x7ff5a006b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:52:59.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.484+0000 7ff5a693e640 1 -- 172.21.15.5:0/3049113905 shutdown_connections 2024-03-24T10:52:59.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.484+0000 7ff5a693e640 1 -- 172.21.15.5:0/3049113905 wait complete. 2024-03-24T10:52:59.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.486+0000 7ff5a693e640 1 Processor -- start 2024-03-24T10:52:59.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.486+0000 7ff5a693e640 1 -- start start 2024-03-24T10:52:59.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.486+0000 7ff5a693e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5a00c2ea0 0x7ff5a00c7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:59.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.486+0000 7ff5a693e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5a00c58c0 0x7ff5a0145000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:59.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.486+0000 7ff5a693e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff5a00c9610 0x7ff5a0145540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:59.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.487+0000 7ff5a693e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a015a360 con 0x7ff5a00c58c0 2024-03-24T10:52:59.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.487+0000 7ff5a693e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a01f1f90 con 0x7ff5a00c2ea0 2024-03-24T10:52:59.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.487+0000 7ff5a693e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff5a01f10a0 con 0x7ff5a00c9610 2024-03-24T10:52:59.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.487+0000 7ff59ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5a00c2ea0 0x7ff5a00c7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:59.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.487+0000 7ff5a4edb640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff5a00c9610 0x7ff5a0145540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:59.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.487+0000 7ff59ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5a00c2ea0 0x7ff5a00c7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:40044/0 (socket says 172.21.15.5:40044) 2024-03-24T10:52:59.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.487+0000 7ff59ffff640 1 -- 172.21.15.5:0/2856292780 learned_addr learned my addr 172.21.15.5:0/2856292780 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:52:59.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.492+0000 7ff59ffff640 1 -- 172.21.15.5:0/2856292780 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff5a00c9610 msgr2=0x7ff5a0145540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:59.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.492+0000 7ff59ffff640 1 --2- 172.21.15.5:0/2856292780 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff5a00c9610 0x7ff5a0145540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:59.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.492+0000 7ff59ffff640 1 -- 172.21.15.5:0/2856292780 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5a00c58c0 msgr2=0x7ff5a0145000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:52:59.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.492+0000 7ff59ffff640 1 --2- 172.21.15.5:0/2856292780 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5a00c58c0 0x7ff5a0145000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:52:59.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.492+0000 7ff59ffff640 1 -- 172.21.15.5:0/2856292780 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff590009400 con 0x7ff5a00c2ea0 2024-03-24T10:52:59.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.496+0000 7ff59ffff640 1 --2- 172.21.15.5:0/2856292780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5a00c2ea0 0x7ff5a00c7fb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff590009890 tx=0x7ff5900046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:59.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.500+0000 7ff59cff9640 1 -- 172.21.15.5:0/2856292780 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59003e070 con 0x7ff5a00c2ea0 2024-03-24T10:52:59.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.500+0000 7ff59cff9640 1 -- 172.21.15.5:0/2856292780 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff590039910 con 0x7ff5a00c2ea0 2024-03-24T10:52:59.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.500+0000 7ff59cff9640 1 -- 172.21.15.5:0/2856292780 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff590039cc0 con 0x7ff5a00c2ea0 2024-03-24T10:52:59.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.500+0000 7ff5a693e640 1 -- 172.21.15.5:0/2856292780 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5a01f1380 con 0x7ff5a00c2ea0 2024-03-24T10:52:59.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.500+0000 7ff5a693e640 1 -- 172.21.15.5:0/2856292780 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5a01f1940 con 0x7ff5a00c2ea0 2024-03-24T10:52:59.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.501+0000 7ff5a693e640 1 -- 172.21.15.5:0/2856292780 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5a0062a00 con 0x7ff5a00c2ea0 2024-03-24T10:52:59.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.535+0000 7ff59cff9640 1 -- 172.21.15.5:0/2856292780 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff5900423f0 con 0x7ff5a00c2ea0 2024-03-24T10:52:59.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.535+0000 7ff59cff9640 1 --2- 172.21.15.5:0/2856292780 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff58c06b890 0x7ff58c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:52:59.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.535+0000 7ff59f7fe640 1 --2- 172.21.15.5:0/2856292780 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff58c06b890 0x7ff58c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:52:59.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.536+0000 7ff59f7fe640 1 --2- 172.21.15.5:0/2856292780 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff58c06b890 0x7ff58c06dd50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff5a00c24d0 tx=0x7ff594008040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:52:59.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.540+0000 7ff59cff9640 1 -- 172.21.15.5:0/2856292780 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7ff590039e20 con 0x7ff5a00c2ea0 2024-03-24T10:52:59.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:52:59.814+0000 7ff59cff9640 1 -- 172.21.15.5:0/2856292780 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff590076980 con 0x7ff5a00c2ea0 2024-03-24T10:53:00.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:00.038+0000 7ff5a693e640 1 -- 172.21.15.5:0/2856292780 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5a01eda90 con 0x7ff5a00c2ea0 2024-03-24T10:53:00.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:00.055+0000 7ff59cff9640 1 -- 172.21.15.5:0/2856292780 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7ff590034090 con 0x7ff5a00c2ea0 2024-03-24T10:53:00.055 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:00.055 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:00.056 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:00.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:00.057+0000 7ff5a693e640 1 -- 172.21.15.5:0/2856292780 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff58c06b890 msgr2=0x7ff58c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:00.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:00.057+0000 7ff5a693e640 1 --2- 172.21.15.5:0/2856292780 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff58c06b890 0x7ff58c06dd50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff5a00c24d0 tx=0x7ff594008040 comp rx=0 tx=0).stop 2024-03-24T10:53:00.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:00.057+0000 7ff5a693e640 1 -- 172.21.15.5:0/2856292780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5a00c2ea0 msgr2=0x7ff5a00c7fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:00.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:00.057+0000 7ff5a693e640 1 --2- 172.21.15.5:0/2856292780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5a00c2ea0 0x7ff5a00c7fb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff590009890 tx=0x7ff5900046c0 comp rx=0 tx=0).stop 2024-03-24T10:53:00.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:00.057+0000 7ff5a693e640 1 -- 172.21.15.5:0/2856292780 shutdown_connections 2024-03-24T10:53:00.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:00.057+0000 7ff5a693e640 1 --2- 172.21.15.5:0/2856292780 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff58c06b890 0x7ff58c06dd50 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:00.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:00.057+0000 7ff5a693e640 1 --2- 172.21.15.5:0/2856292780 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff5a00c58c0 0x7ff5a0145000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:00.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:00.057+0000 7ff5a693e640 1 --2- 172.21.15.5:0/2856292780 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff5a00c9610 0x7ff5a0145540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:00.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:00.057+0000 7ff5a693e640 1 --2- 172.21.15.5:0/2856292780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff5a00c2ea0 0x7ff5a00c7fb0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:00.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:00.057+0000 7ff5a693e640 1 -- 172.21.15.5:0/2856292780 >> 172.21.15.5:0/2856292780 conn(0x7ff5a006b130 msgr2=0x7ff5a006ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:00.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:00.057+0000 7ff5a693e640 1 -- 172.21.15.5:0/2856292780 shutdown_connections 2024-03-24T10:53:00.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:00.057+0000 7ff5a693e640 1 -- 172.21.15.5:0/2856292780 wait complete. 2024-03-24T10:53:00.672 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:00.673 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:01.674 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:01.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.777+0000 7f2a5955c640 1 Processor -- start 2024-03-24T10:53:01.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.777+0000 7f2a5955c640 1 -- start start 2024-03-24T10:53:01.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.778+0000 7f2a5955c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2a540c70e0 0x7f2a5415e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:01.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.778+0000 7f2a5955c640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a5415b1f0 con 0x7f2a54154470 2024-03-24T10:53:01.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.778+0000 7f2a5955c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a5415a5e0 con 0x7f2a5406c0a0 2024-03-24T10:53:01.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.778+0000 7f2a5955c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f2a5415e810 con 0x7f2a540c70e0 2024-03-24T10:53:01.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.779+0000 7f2a52ffd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f2a54154470 0x7f2a5406f770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:58754/0 (socket says 172.21.15.5:58754) 2024-03-24T10:53:01.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.779+0000 7f2a52ffd640 1 -- 172.21.15.5:0/1392600780 learned_addr learned my addr 172.21.15.5:0/1392600780 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:53:01.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.779+0000 7f2a537fe640 1 --2- 172.21.15.5:0/1392600780 >> v2:172.21.15.57:3300/0 conn(0x7f2a540c70e0 0x7f2a5415e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:01.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.782+0000 7f2a51ffb640 1 -- 172.21.15.5:0/1392600780 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3102139644 0 0) 0x7f2a5415b1f0 con 0x7f2a54154470 2024-03-24T10:53:01.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.782+0000 7f2a51ffb640 1 -- 172.21.15.5:0/1392600780 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a3c003520 con 0x7f2a54154470 2024-03-24T10:53:01.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.783+0000 7f2a537fe640 1 -- 172.21.15.5:0/1392600780 >> v1:172.21.15.57:6789/0 conn(0x7f2a5406c0a0 legacy=0x7f2a540c69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:01.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.783+0000 7f2a537fe640 1 -- 172.21.15.5:0/1392600780 >> v1:172.21.15.5:6789/0 conn(0x7f2a54154470 legacy=0x7f2a5406f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:01.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.783+0000 7f2a537fe640 1 -- 172.21.15.5:0/1392600780 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a5415a8c0 con 0x7f2a540c70e0 2024-03-24T10:53:01.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.788+0000 7f2a537fe640 1 --2- 172.21.15.5:0/1392600780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a540c70e0 0x7f2a5415e2d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c009d90 tx=0x7f2a4c02bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffbb511ec6adf40 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:01.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.793+0000 7f2a51ffb640 1 -- 172.21.15.5:0/1392600780 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c02cd80 con 0x7f2a540c70e0 2024-03-24T10:53:01.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.793+0000 7f2a51ffb640 1 -- 172.21.15.5:0/1392600780 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a4c034440 con 0x7f2a540c70e0 2024-03-24T10:53:01.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.793+0000 7f2a51ffb640 1 -- 172.21.15.5:0/1392600780 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c0347d0 con 0x7f2a540c70e0 2024-03-24T10:53:01.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.794+0000 7f2a5955c640 1 -- 172.21.15.5:0/1392600780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a540c70e0 msgr2=0x7f2a5415e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:01.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.794+0000 7f2a5955c640 1 --2- 172.21.15.5:0/1392600780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a540c70e0 0x7f2a5415e2d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c009d90 tx=0x7f2a4c02bfd0 comp rx=0 tx=0).stop 2024-03-24T10:53:01.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.794+0000 7f2a5955c640 1 -- 172.21.15.5:0/1392600780 shutdown_connections 2024-03-24T10:53:01.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.794+0000 7f2a5955c640 1 --2- 172.21.15.5:0/1392600780 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a540c70e0 0x7f2a5415e2d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:01.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.794+0000 7f2a5955c640 1 -- 172.21.15.5:0/1392600780 >> 172.21.15.5:0/1392600780 conn(0x7f2a5406d790 msgr2=0x7f2a5406dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:01.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.794+0000 7f2a5955c640 1 -- 172.21.15.5:0/1392600780 shutdown_connections 2024-03-24T10:53:01.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.794+0000 7f2a5955c640 1 -- 172.21.15.5:0/1392600780 wait complete. 2024-03-24T10:53:01.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.795+0000 7f2a5955c640 1 Processor -- start 2024-03-24T10:53:01.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.795+0000 7f2a5955c640 1 -- start start 2024-03-24T10:53:01.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.796+0000 7f2a5955c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a5406c0a0 0x7f2a5415c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:01.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.796+0000 7f2a5955c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a540c70e0 0x7f2a540cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:01.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.796+0000 7f2a5955c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2a54154470 0x7f2a54156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:01.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.796+0000 7f2a5955c640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f2a540ccaa0 con 0x7f2a5406c0a0 2024-03-24T10:53:01.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.796+0000 7f2a5955c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2a541f4500 con 0x7f2a540c70e0 2024-03-24T10:53:01.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.796+0000 7f2a5955c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f2a541f4070 con 0x7f2a54154470 2024-03-24T10:53:01.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.796+0000 7f2a52ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a5406c0a0 0x7f2a5415c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:01.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.796+0000 7f2a527fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a540c70e0 0x7f2a540cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:01.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.797+0000 7f2a52ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a5406c0a0 0x7f2a5415c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35064/0 (socket says 172.21.15.5:35064) 2024-03-24T10:53:01.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.797+0000 7f2a52ffd640 1 -- 172.21.15.5:0/923024086 learned_addr learned my addr 172.21.15.5:0/923024086 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:01.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.797+0000 7f2a537fe640 1 --2- 172.21.15.5:0/923024086 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2a54154470 0x7f2a54156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:01.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.800+0000 7f2a52ffd640 1 -- 172.21.15.5:0/923024086 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2a54154470 msgr2=0x7f2a54156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:01.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.800+0000 7f2a52ffd640 1 --2- 172.21.15.5:0/923024086 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2a54154470 0x7f2a54156ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:01.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.800+0000 7f2a52ffd640 1 -- 172.21.15.5:0/923024086 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a540c70e0 msgr2=0x7f2a540cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:01.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.800+0000 7f2a52ffd640 1 --2- 172.21.15.5:0/923024086 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a540c70e0 0x7f2a540cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:01.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.800+0000 7f2a52ffd640 1 -- 172.21.15.5:0/923024086 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a4c009a70 con 0x7f2a5406c0a0 2024-03-24T10:53:01.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.802+0000 7f2a52ffd640 1 --2- 172.21.15.5:0/923024086 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a5406c0a0 0x7f2a5415c9e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400c410 tx=0x7f2a4400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:01.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.807+0000 7f2a3b7fe640 1 -- 172.21.15.5:0/923024086 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a44005ce0 con 0x7f2a5406c0a0 2024-03-24T10:53:01.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.807+0000 7f2a3b7fe640 1 -- 172.21.15.5:0/923024086 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a44005e40 con 0x7f2a5406c0a0 2024-03-24T10:53:01.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.807+0000 7f2a3b7fe640 1 -- 172.21.15.5:0/923024086 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a440072f0 con 0x7f2a5406c0a0 2024-03-24T10:53:01.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.807+0000 7f2a5955c640 1 -- 172.21.15.5:0/923024086 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a540cc7c0 con 0x7f2a5406c0a0 2024-03-24T10:53:01.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.807+0000 7f2a5955c640 1 -- 172.21.15.5:0/923024086 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a541f3a20 con 0x7f2a5406c0a0 2024-03-24T10:53:01.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.808+0000 7f2a5955c640 1 -- 172.21.15.5:0/923024086 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a54154b80 con 0x7f2a5406c0a0 2024-03-24T10:53:01.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.828+0000 7f2a3b7fe640 1 -- 172.21.15.5:0/923024086 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2a44007510 con 0x7f2a5406c0a0 2024-03-24T10:53:01.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.829+0000 7f2a3b7fe640 1 --2- 172.21.15.5:0/923024086 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2a4006b860 0x7f2a4006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:01.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.829+0000 7f2a527fc640 1 --2- 172.21.15.5:0/923024086 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2a4006b860 0x7f2a4006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:01.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:01.829+0000 7f2a527fc640 1 --2- 172.21.15.5:0/923024086 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2a4006b860 0x7f2a4006dd20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2a5406e1f0 tx=0x7f2a48009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:02.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:02.094+0000 7f2a3b7fe640 1 -- 172.21.15.5:0/923024086 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f2a44058510 con 0x7f2a5406c0a0 2024-03-24T10:53:02.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:02.094+0000 7f2a3b7fe640 1 -- 172.21.15.5:0/923024086 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2a440547b0 con 0x7f2a5406c0a0 2024-03-24T10:53:02.205 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:53:02.208 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:53:02.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:02.409+0000 7f2a5955c640 1 -- 172.21.15.5:0/923024086 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a540681d0 con 0x7f2a5406c0a0 2024-03-24T10:53:02.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:02.430+0000 7f2a3b7fe640 1 -- 172.21.15.5:0/923024086 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f2a44017090 con 0x7f2a5406c0a0 2024-03-24T10:53:02.431 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:02.431 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:02.431 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:02.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:02.432+0000 7f2a5955c640 1 -- 172.21.15.5:0/923024086 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2a4006b860 msgr2=0x7f2a4006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:02.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:02.432+0000 7f2a5955c640 1 --2- 172.21.15.5:0/923024086 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2a4006b860 0x7f2a4006dd20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2a5406e1f0 tx=0x7f2a48009040 comp rx=0 tx=0).stop 2024-03-24T10:53:02.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:02.432+0000 7f2a5955c640 1 -- 172.21.15.5:0/923024086 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a5406c0a0 msgr2=0x7f2a5415c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:02.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:02.432+0000 7f2a5955c640 1 --2- 172.21.15.5:0/923024086 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a5406c0a0 0x7f2a5415c9e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400c410 tx=0x7f2a4400c8d0 comp rx=0 tx=0).stop 2024-03-24T10:53:02.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:02.432+0000 7f2a5955c640 1 -- 172.21.15.5:0/923024086 shutdown_connections 2024-03-24T10:53:02.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:02.432+0000 7f2a5955c640 1 --2- 172.21.15.5:0/923024086 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2a4006b860 0x7f2a4006dd20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:02.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:02.432+0000 7f2a5955c640 1 --2- 172.21.15.5:0/923024086 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2a54154470 0x7f2a54156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:02.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:02.432+0000 7f2a5955c640 1 --2- 172.21.15.5:0/923024086 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2a540c70e0 0x7f2a540cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:02.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:02.432+0000 7f2a5955c640 1 --2- 172.21.15.5:0/923024086 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2a5406c0a0 0x7f2a5415c9e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:02.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:02.432+0000 7f2a5955c640 1 -- 172.21.15.5:0/923024086 >> 172.21.15.5:0/923024086 conn(0x7f2a5406d790 msgr2=0x7f2a54072530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:02.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:02.433+0000 7f2a5955c640 1 -- 172.21.15.5:0/923024086 shutdown_connections 2024-03-24T10:53:02.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:02.433+0000 7f2a5955c640 1 -- 172.21.15.5:0/923024086 wait complete. 2024-03-24T10:53:03.049 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:03.049 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:04.050 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:04.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.152+0000 7f26c6284640 1 Processor -- start 2024-03-24T10:53:04.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.152+0000 7f26c6284640 1 -- start start 2024-03-24T10:53:04.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.153+0000 7f26c6284640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26c00cb270 0x7f26c00cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:04.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.153+0000 7f26c6284640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f26c00cd960 0x7f26c00cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:04.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.153+0000 7f26c6284640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26c00ccba0 con 0x7f26c00cae90 2024-03-24T10:53:04.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.153+0000 7f26c6284640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f26c00cc9b0 con 0x7f26c00cd960 2024-03-24T10:53:04.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.153+0000 7f26c6284640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f26c00cce40 con 0x7f26c00cb270 2024-03-24T10:53:04.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.154+0000 7f26bf7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26c00cb270 0x7f26c00cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:04.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.154+0000 7f26c4821640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f26c00cd960 0x7f26c00cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:04.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.154+0000 7f26bf7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26c00cb270 0x7f26c00cf530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:33336/0 (socket says 172.21.15.5:33336) 2024-03-24T10:53:04.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.154+0000 7f26bf7fe640 1 -- 172.21.15.5:0/438671591 learned_addr learned my addr 172.21.15.5:0/438671591 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:04.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.157+0000 7f26bf7fe640 1 -- 172.21.15.5:0/438671591 >> v2:172.21.15.57:3300/0 conn(0x7f26c00cd960 msgr2=0x7f26c00cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:04.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.157+0000 7f26bf7fe640 1 --2- 172.21.15.5:0/438671591 >> v2:172.21.15.57:3300/0 conn(0x7f26c00cd960 0x7f26c00cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:04.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.157+0000 7f26bf7fe640 1 -- 172.21.15.5:0/438671591 >> v1:172.21.15.57:6789/0 conn(0x7f26c00cae90 legacy=0x7f26c00cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:04.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.157+0000 7f26bf7fe640 1 -- 172.21.15.5:0/438671591 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26c015a590 con 0x7f26c00cb270 2024-03-24T10:53:04.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.159+0000 7f26bf7fe640 1 --2- 172.21.15.5:0/438671591 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26c00cb270 0x7f26c00cf530 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f26b4009870 tx=0x7f26b402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af52206f3458ee1c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:04.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.163+0000 7f26beffd640 1 -- 172.21.15.5:0/438671591 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4030840 con 0x7f26c00cb270 2024-03-24T10:53:04.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.163+0000 7f26beffd640 1 -- 172.21.15.5:0/438671591 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26b40309a0 con 0x7f26c00cb270 2024-03-24T10:53:04.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.163+0000 7f26beffd640 1 -- 172.21.15.5:0/438671591 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b4030d30 con 0x7f26c00cb270 2024-03-24T10:53:04.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.164+0000 7f26c6284640 1 -- 172.21.15.5:0/438671591 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26c00cb270 msgr2=0x7f26c00cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:04.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.164+0000 7f26c6284640 1 --2- 172.21.15.5:0/438671591 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26c00cb270 0x7f26c00cf530 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f26b4009870 tx=0x7f26b402b980 comp rx=0 tx=0).stop 2024-03-24T10:53:04.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.164+0000 7f26c6284640 1 -- 172.21.15.5:0/438671591 shutdown_connections 2024-03-24T10:53:04.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.164+0000 7f26c6284640 1 --2- 172.21.15.5:0/438671591 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26c00cb270 0x7f26c00cf530 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:04.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.164+0000 7f26c6284640 1 --2- 172.21.15.5:0/438671591 >> v2:172.21.15.57:3300/0 conn(0x7f26c00cd960 0x7f26c00cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:04.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.164+0000 7f26c6284640 1 -- 172.21.15.5:0/438671591 >> 172.21.15.5:0/438671591 conn(0x7f26c00cfb80 msgr2=0x7f26c00cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:04.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.164+0000 7f26c6284640 1 -- 172.21.15.5:0/438671591 shutdown_connections 2024-03-24T10:53:04.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.164+0000 7f26c6284640 1 -- 172.21.15.5:0/438671591 wait complete. 2024-03-24T10:53:04.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.165+0000 7f26c6284640 1 Processor -- start 2024-03-24T10:53:04.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.165+0000 7f26c6284640 1 -- start start 2024-03-24T10:53:04.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.166+0000 7f26c6284640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26c00cae90 0x7f26c00c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:04.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.166+0000 7f26c6284640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26c00cb270 0x7f26c01efc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:04.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.166+0000 7f26c6284640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26c00cd960 0x7f26c00ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:04.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.166+0000 7f26c6284640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f26c01f0170 con 0x7f26c00cae90 2024-03-24T10:53:04.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.166+0000 7f26c6284640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f26c00c36a0 con 0x7f26c00cd960 2024-03-24T10:53:04.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.166+0000 7f26c6284640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f26c00c3940 con 0x7f26c00cb270 2024-03-24T10:53:04.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.167+0000 7f26bffff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26c00cae90 0x7f26c00c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:04.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.167+0000 7f26c4821640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26c00cd960 0x7f26c00ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:04.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.167+0000 7f26bf7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26c00cb270 0x7f26c01efc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:04.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.167+0000 7f26bffff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26c00cae90 0x7f26c00c6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:41740/0 (socket says 172.21.15.5:41740) 2024-03-24T10:53:04.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.167+0000 7f26bffff640 1 -- 172.21.15.5:0/1215745437 learned_addr learned my addr 172.21.15.5:0/1215745437 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:04.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.170+0000 7f26bffff640 1 -- 172.21.15.5:0/1215745437 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26c00cb270 msgr2=0x7f26c01efc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:04.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.170+0000 7f26bffff640 1 --2- 172.21.15.5:0/1215745437 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26c00cb270 0x7f26c01efc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:04.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.170+0000 7f26bffff640 1 -- 172.21.15.5:0/1215745437 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26c00cd960 msgr2=0x7f26c00ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:04.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.170+0000 7f26bffff640 1 --2- 172.21.15.5:0/1215745437 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26c00cd960 0x7f26c00ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:04.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.170+0000 7f26bffff640 1 -- 172.21.15.5:0/1215745437 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26b40094d0 con 0x7f26c00cae90 2024-03-24T10:53:04.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.173+0000 7f26bffff640 1 --2- 172.21.15.5:0/1215745437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26c00cae90 0x7f26c00c6100 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f26b000c430 tx=0x7f26b000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:04.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.177+0000 7f26bcff9640 1 -- 172.21.15.5:0/1215745437 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b0011790 con 0x7f26c00cae90 2024-03-24T10:53:04.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.177+0000 7f26bcff9640 1 -- 172.21.15.5:0/1215745437 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26b00118f0 con 0x7f26c00cae90 2024-03-24T10:53:04.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.177+0000 7f26c6284640 1 -- 172.21.15.5:0/1215745437 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26c00c3c20 con 0x7f26c00cae90 2024-03-24T10:53:04.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.177+0000 7f26bcff9640 1 -- 172.21.15.5:0/1215745437 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b0011ce0 con 0x7f26c00cae90 2024-03-24T10:53:04.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.177+0000 7f26c6284640 1 -- 172.21.15.5:0/1215745437 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26c00c5be0 con 0x7f26c00cae90 2024-03-24T10:53:04.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.179+0000 7f26c6284640 1 -- 172.21.15.5:0/1215745437 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26c00681d0 con 0x7f26c00cae90 2024-03-24T10:53:04.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.475+0000 7f26bcff9640 1 -- 172.21.15.5:0/1215745437 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f26b001c460 con 0x7f26c00cae90 2024-03-24T10:53:04.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.476+0000 7f26bcff9640 1 --2- 172.21.15.5:0/1215745437 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f26ac06b840 0x7f26ac06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:04.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.476+0000 7f26bf7fe640 1 --2- 172.21.15.5:0/1215745437 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f26ac06b840 0x7f26ac06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:04.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.476+0000 7f26bf7fe640 1 --2- 172.21.15.5:0/1215745437 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f26ac06b840 0x7f26ac06dd00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f26b4036000 tx=0x7f26b4008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:04.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.494+0000 7f26bcff9640 1 -- 172.21.15.5:0/1215745437 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f26b0011e40 con 0x7f26c00cae90 2024-03-24T10:53:04.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.494+0000 7f26bcff9640 1 -- 172.21.15.5:0/1215745437 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f26b00555b0 con 0x7f26c00cae90 2024-03-24T10:53:04.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.732+0000 7f26c6284640 1 -- 172.21.15.5:0/1215745437 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f26c00b5490 con 0x7f26c00cae90 2024-03-24T10:53:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.755+0000 7f26bcff9640 1 -- 172.21.15.5:0/1215745437 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f26b0058c70 con 0x7f26c00cae90 2024-03-24T10:53:04.755 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:04.755 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:04.755 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:04.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.757+0000 7f26c6284640 1 -- 172.21.15.5:0/1215745437 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f26ac06b840 msgr2=0x7f26ac06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:04.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.757+0000 7f26c6284640 1 --2- 172.21.15.5:0/1215745437 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f26ac06b840 0x7f26ac06dd00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f26b4036000 tx=0x7f26b4008030 comp rx=0 tx=0).stop 2024-03-24T10:53:04.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.757+0000 7f26c6284640 1 -- 172.21.15.5:0/1215745437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26c00cae90 msgr2=0x7f26c00c6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:04.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.757+0000 7f26c6284640 1 --2- 172.21.15.5:0/1215745437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26c00cae90 0x7f26c00c6100 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f26b000c430 tx=0x7f26b000c8f0 comp rx=0 tx=0).stop 2024-03-24T10:53:04.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.757+0000 7f26c6284640 1 -- 172.21.15.5:0/1215745437 shutdown_connections 2024-03-24T10:53:04.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.757+0000 7f26c6284640 1 --2- 172.21.15.5:0/1215745437 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f26ac06b840 0x7f26ac06dd00 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:04.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.757+0000 7f26c6284640 1 --2- 172.21.15.5:0/1215745437 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26c00cd960 0x7f26c00ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:04.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.757+0000 7f26c6284640 1 --2- 172.21.15.5:0/1215745437 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26c00cb270 0x7f26c01efc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:04.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.757+0000 7f26c6284640 1 --2- 172.21.15.5:0/1215745437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26c00cae90 0x7f26c00c6100 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:04.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.757+0000 7f26c6284640 1 -- 172.21.15.5:0/1215745437 >> 172.21.15.5:0/1215745437 conn(0x7f26c00cfb80 msgr2=0x7f26c00c7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:04.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.757+0000 7f26c6284640 1 -- 172.21.15.5:0/1215745437 shutdown_connections 2024-03-24T10:53:04.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:04.757+0000 7f26c6284640 1 -- 172.21.15.5:0/1215745437 wait complete. 2024-03-24T10:53:05.373 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:05.373 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:06.374 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:06.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.476+0000 7f0941e81640 1 Processor -- start 2024-03-24T10:53:06.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.476+0000 7f0941e81640 1 -- start start 2024-03-24T10:53:06.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.477+0000 7f0941e81640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f093c0cace0 0x7f093c1432d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:06.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.477+0000 7f0941e81640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f093c15a310 0x7f093c143810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:06.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.477+0000 7f0941e81640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f093c0cb0c0 con 0x7f093c06c240 2024-03-24T10:53:06.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.477+0000 7f0941e81640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f093c06f110 con 0x7f093c15a310 2024-03-24T10:53:06.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.477+0000 7f0941e81640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f093c15b320 con 0x7f093c0cace0 2024-03-24T10:53:06.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.478+0000 7f093b7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f093c0cace0 0x7f093c1432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:06.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.478+0000 7f093affd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f093c15a310 0x7f093c143810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:06.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.478+0000 7f093affd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f093c15a310 0x7f093c143810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:41744/0 (socket says 172.21.15.5:41744) 2024-03-24T10:53:06.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.478+0000 7f093affd640 1 -- 172.21.15.5:0/736075054 learned_addr learned my addr 172.21.15.5:0/736075054 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:06.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.481+0000 7f093affd640 1 -- 172.21.15.5:0/736075054 >> v2:172.21.15.57:3300/0 conn(0x7f093c0cace0 msgr2=0x7f093c1432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:06.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.481+0000 7f093affd640 1 --2- 172.21.15.5:0/736075054 >> v2:172.21.15.57:3300/0 conn(0x7f093c0cace0 0x7f093c1432d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:06.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.481+0000 7f093affd640 1 -- 172.21.15.5:0/736075054 >> v1:172.21.15.57:6789/0 conn(0x7f093c06c240 legacy=0x7f093c06f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:06.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.481+0000 7f093affd640 1 -- 172.21.15.5:0/736075054 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f093c06cf70 con 0x7f093c15a310 2024-03-24T10:53:06.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.483+0000 7f093affd640 1 --2- 172.21.15.5:0/736075054 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f093c15a310 0x7f093c143810 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f093c0cb320 tx=0x7f093002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a733b6cf4b817d88 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:06.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.488+0000 7f093a7fc640 1 -- 172.21.15.5:0/736075054 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093002f850 con 0x7f093c15a310 2024-03-24T10:53:06.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.488+0000 7f093a7fc640 1 -- 172.21.15.5:0/736075054 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f093002f9b0 con 0x7f093c15a310 2024-03-24T10:53:06.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.488+0000 7f093a7fc640 1 -- 172.21.15.5:0/736075054 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093002fd40 con 0x7f093c15a310 2024-03-24T10:53:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.488+0000 7f0941e81640 1 -- 172.21.15.5:0/736075054 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f093c15a310 msgr2=0x7f093c143810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.488+0000 7f0941e81640 1 --2- 172.21.15.5:0/736075054 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f093c15a310 0x7f093c143810 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f093c0cb320 tx=0x7f093002ba30 comp rx=0 tx=0).stop 2024-03-24T10:53:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.488+0000 7f0941e81640 1 -- 172.21.15.5:0/736075054 shutdown_connections 2024-03-24T10:53:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.488+0000 7f0941e81640 1 --2- 172.21.15.5:0/736075054 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f093c15a310 0x7f093c143810 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.488+0000 7f0941e81640 1 --2- 172.21.15.5:0/736075054 >> v2:172.21.15.57:3300/0 conn(0x7f093c0cace0 0x7f093c1432d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.488+0000 7f0941e81640 1 -- 172.21.15.5:0/736075054 >> 172.21.15.5:0/736075054 conn(0x7f093c071240 msgr2=0x7f093c071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.488+0000 7f0941e81640 1 -- 172.21.15.5:0/736075054 shutdown_connections 2024-03-24T10:53:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.489+0000 7f0941e81640 1 -- 172.21.15.5:0/736075054 wait complete. 2024-03-24T10:53:06.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.490+0000 7f0941e81640 1 Processor -- start 2024-03-24T10:53:06.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.490+0000 7f0941e81640 1 -- start start 2024-03-24T10:53:06.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.491+0000 7f0941e81640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f093c06c240 0x7f093c13db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:06.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.491+0000 7f0941e81640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f093c0cace0 0x7f093c1f0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:06.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.491+0000 7f0941e81640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f093c0ae890 0x7f093c13b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:06.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.491+0000 7f0941e81640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f093c0b0590 con 0x7f093c0cace0 2024-03-24T10:53:06.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.491+0000 7f0941e81640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f093c1f3f70 con 0x7f093c06c240 2024-03-24T10:53:06.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.491+0000 7f0941e81640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f093c157700 con 0x7f093c0ae890 2024-03-24T10:53:06.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.491+0000 7f093affd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f093c0cace0 0x7f093c1f0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:06.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.491+0000 7f093b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f093c06c240 0x7f093c13db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:06.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.491+0000 7f093affd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f093c0cace0 0x7f093c1f0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:41758/0 (socket says 172.21.15.5:41758) 2024-03-24T10:53:06.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.491+0000 7f093affd640 1 -- 172.21.15.5:0/3974970459 learned_addr learned my addr 172.21.15.5:0/3974970459 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:06.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.491+0000 7f093bfff640 1 --2- 172.21.15.5:0/3974970459 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f093c0ae890 0x7f093c13b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:06.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.495+0000 7f093affd640 1 -- 172.21.15.5:0/3974970459 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f093c0ae890 msgr2=0x7f093c13b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:06.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.495+0000 7f093affd640 1 --2- 172.21.15.5:0/3974970459 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f093c0ae890 0x7f093c13b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:06.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.495+0000 7f093affd640 1 -- 172.21.15.5:0/3974970459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f093c06c240 msgr2=0x7f093c13db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:06.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.495+0000 7f093affd640 1 --2- 172.21.15.5:0/3974970459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f093c06c240 0x7f093c13db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:06.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.495+0000 7f093affd640 1 -- 172.21.15.5:0/3974970459 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09300094d0 con 0x7f093c0cace0 2024-03-24T10:53:06.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.498+0000 7f093affd640 1 --2- 172.21.15.5:0/3974970459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f093c0cace0 0x7f093c1f0030 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f093002ba70 tx=0x7f0930004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.503+0000 7f0923fff640 1 -- 172.21.15.5:0/3974970459 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093003c040 con 0x7f093c0cace0 2024-03-24T10:53:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.503+0000 7f0923fff640 1 -- 172.21.15.5:0/3974970459 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0930037be0 con 0x7f093c0cace0 2024-03-24T10:53:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.503+0000 7f0941e81640 1 -- 172.21.15.5:0/3974970459 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f093c1579a0 con 0x7f093c0cace0 2024-03-24T10:53:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.503+0000 7f0923fff640 1 -- 172.21.15.5:0/3974970459 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09300423f0 con 0x7f093c0cace0 2024-03-24T10:53:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.503+0000 7f0941e81640 1 -- 172.21.15.5:0/3974970459 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f093c1f4400 con 0x7f093c0cace0 2024-03-24T10:53:06.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.504+0000 7f0941e81640 1 -- 172.21.15.5:0/3974970459 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f093c0681d0 con 0x7f093c0cace0 2024-03-24T10:53:06.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.529+0000 7f0923fff640 1 -- 172.21.15.5:0/3974970459 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f093002d3e0 con 0x7f093c0cace0 2024-03-24T10:53:06.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.529+0000 7f0923fff640 1 --2- 172.21.15.5:0/3974970459 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f092806b860 0x7f092806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:06.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.529+0000 7f093b7fe640 1 --2- 172.21.15.5:0/3974970459 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f092806b860 0x7f092806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:06.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.530+0000 7f093b7fe640 1 --2- 172.21.15.5:0/3974970459 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f092806b860 0x7f092806dd20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f092c009b40 tx=0x7f092c009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:06.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.790+0000 7f0923fff640 1 -- 172.21.15.5:0/3974970459 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f093007b190 con 0x7f093c0cace0 2024-03-24T10:53:06.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:06.790+0000 7f0923fff640 1 -- 172.21.15.5:0/3974970459 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0930077430 con 0x7f093c0cace0 2024-03-24T10:53:07.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:07.039+0000 7f0941e81640 1 -- 172.21.15.5:0/3974970459 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f093c070a60 con 0x7f093c0cace0 2024-03-24T10:53:07.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:07.054+0000 7f0923fff640 1 -- 172.21.15.5:0/3974970459 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f0930047020 con 0x7f093c0cace0 2024-03-24T10:53:07.055 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:07.055 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:07.055 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:07.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:07.056+0000 7f0941e81640 1 -- 172.21.15.5:0/3974970459 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f092806b860 msgr2=0x7f092806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:07.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:07.056+0000 7f0941e81640 1 --2- 172.21.15.5:0/3974970459 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f092806b860 0x7f092806dd20 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f092c009b40 tx=0x7f092c009190 comp rx=0 tx=0).stop 2024-03-24T10:53:07.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:07.056+0000 7f0941e81640 1 -- 172.21.15.5:0/3974970459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f093c0cace0 msgr2=0x7f093c1f0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:07.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:07.056+0000 7f0941e81640 1 --2- 172.21.15.5:0/3974970459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f093c0cace0 0x7f093c1f0030 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f093002ba70 tx=0x7f0930004730 comp rx=0 tx=0).stop 2024-03-24T10:53:07.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:07.056+0000 7f0941e81640 1 -- 172.21.15.5:0/3974970459 shutdown_connections 2024-03-24T10:53:07.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:07.056+0000 7f0941e81640 1 --2- 172.21.15.5:0/3974970459 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f092806b860 0x7f092806dd20 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:07.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:07.057+0000 7f0941e81640 1 --2- 172.21.15.5:0/3974970459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f093c0cace0 0x7f093c1f0030 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:07.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:07.057+0000 7f0941e81640 1 --2- 172.21.15.5:0/3974970459 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f093c0ae890 0x7f093c13b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:07.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:07.057+0000 7f0941e81640 1 --2- 172.21.15.5:0/3974970459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f093c06c240 0x7f093c13db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:07.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:07.057+0000 7f0941e81640 1 -- 172.21.15.5:0/3974970459 >> 172.21.15.5:0/3974970459 conn(0x7f093c071240 msgr2=0x7f093c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:07.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:07.058+0000 7f0941e81640 1 -- 172.21.15.5:0/3974970459 shutdown_connections 2024-03-24T10:53:07.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:07.058+0000 7f0941e81640 1 -- 172.21.15.5:0/3974970459 wait complete. 2024-03-24T10:53:07.672 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:07.672 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:08.674 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:08.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.779+0000 7fea0186f640 1 Processor -- start 2024-03-24T10:53:08.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.779+0000 7fea0186f640 1 -- start start 2024-03-24T10:53:08.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.779+0000 7fea0186f640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe9fc1543a0 0x7fe9fc15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:08.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.779+0000 7fea0186f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9fc06d4f0 0x7fe9fc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:08.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.779+0000 7fea0186f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe9fc0ccaa0 0x7fe9fc06bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:08.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.779+0000 7fea0186f640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fe9fc0c6750 con 0x7fe9fc1543a0 2024-03-24T10:53:08.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.779+0000 7fea0186f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fe9fc0cc380 con 0x7fe9fc0ccaa0 2024-03-24T10:53:08.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.779+0000 7fea0186f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe9fc0cb2f0 con 0x7fe9fc06d4f0 2024-03-24T10:53:08.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.780+0000 7fe9faffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe9fc1543a0 0x7fe9fc15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:08.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.780+0000 7fe9fa7fc640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9fc06d4f0 0x7fe9fc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:08.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.780+0000 7fe9fb7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe9fc0ccaa0 0x7fe9fc06bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:08.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.780+0000 7fe9faffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe9fc1543a0 0x7fe9fc15ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:41774/0 (socket says 172.21.15.5:41774) 2024-03-24T10:53:08.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.781+0000 7fe9faffd640 1 -- 172.21.15.5:0/3277953890 learned_addr learned my addr 172.21.15.5:0/3277953890 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:08.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.783+0000 7fe9faffd640 1 -- 172.21.15.5:0/3277953890 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9fc06d4f0 msgr2=0x7fe9fc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:08.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.783+0000 7fe9faffd640 1 --2- 172.21.15.5:0/3277953890 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9fc06d4f0 0x7fe9fc0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:08.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.783+0000 7fe9faffd640 1 -- 172.21.15.5:0/3277953890 >> v2:172.21.15.57:3300/0 conn(0x7fe9fc0ccaa0 msgr2=0x7fe9fc06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:08.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.783+0000 7fe9faffd640 1 --2- 172.21.15.5:0/3277953890 >> v2:172.21.15.57:3300/0 conn(0x7fe9fc0ccaa0 0x7fe9fc06bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:08.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.783+0000 7fe9faffd640 1 -- 172.21.15.5:0/3277953890 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9fc0ca7e0 con 0x7fe9fc1543a0 2024-03-24T10:53:08.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.785+0000 7fe9faffd640 1 --2- 172.21.15.5:0/3277953890 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9fc1543a0 0x7fe9fc15ac00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc0628b0 tx=0x7fe9ec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f399b56b92483f95 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:08.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.790+0000 7fe9f9ffb640 1 -- 172.21.15.5:0/3277953890 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec030860 con 0x7fe9fc1543a0 2024-03-24T10:53:08.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.790+0000 7fe9f9ffb640 1 -- 172.21.15.5:0/3277953890 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9ec0309c0 con 0x7fe9fc1543a0 2024-03-24T10:53:08.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.790+0000 7fe9f9ffb640 1 -- 172.21.15.5:0/3277953890 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ec030d50 con 0x7fe9fc1543a0 2024-03-24T10:53:08.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.790+0000 7fea0186f640 1 -- 172.21.15.5:0/3277953890 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9fc1543a0 msgr2=0x7fe9fc15ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:08.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.790+0000 7fea0186f640 1 --2- 172.21.15.5:0/3277953890 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9fc1543a0 0x7fe9fc15ac00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc0628b0 tx=0x7fe9ec02b9a0 comp rx=0 tx=0).stop 2024-03-24T10:53:08.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.790+0000 7fea0186f640 1 -- 172.21.15.5:0/3277953890 shutdown_connections 2024-03-24T10:53:08.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.790+0000 7fea0186f640 1 --2- 172.21.15.5:0/3277953890 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9fc06d4f0 0x7fe9fc0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:08.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.790+0000 7fea0186f640 1 --2- 172.21.15.5:0/3277953890 >> v2:172.21.15.57:3300/0 conn(0x7fe9fc0ccaa0 0x7fe9fc06bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:08.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.790+0000 7fea0186f640 1 --2- 172.21.15.5:0/3277953890 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9fc1543a0 0x7fe9fc15ac00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:08.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.790+0000 7fea0186f640 1 -- 172.21.15.5:0/3277953890 >> 172.21.15.5:0/3277953890 conn(0x7fe9fc072ef0 msgr2=0x7fe9fc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:08.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.790+0000 7fea0186f640 1 -- 172.21.15.5:0/3277953890 shutdown_connections 2024-03-24T10:53:08.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.790+0000 7fea0186f640 1 -- 172.21.15.5:0/3277953890 wait complete. 2024-03-24T10:53:08.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.792+0000 7fea0186f640 1 Processor -- start 2024-03-24T10:53:08.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.792+0000 7fea0186f640 1 -- start start 2024-03-24T10:53:08.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.793+0000 7fea0186f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9fc06d4f0 0x7fe9fc159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:08.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.793+0000 7fea0186f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9fc0ccaa0 0x7fe9fc144350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:08.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.793+0000 7fea0186f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9fc1543a0 0x7fe9fc144890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:08.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.793+0000 7fea0186f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc0b4fe0 con 0x7fe9fc0ccaa0 2024-03-24T10:53:08.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.793+0000 7fea0186f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc0b4a40 con 0x7fe9fc1543a0 2024-03-24T10:53:08.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.793+0000 7fea0186f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe9fc0b4d00 con 0x7fe9fc06d4f0 2024-03-24T10:53:08.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.794+0000 7fe9fa7fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9fc0ccaa0 0x7fe9fc144350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:08.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.794+0000 7fe9fb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9fc1543a0 0x7fe9fc144890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:08.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.794+0000 7fe9fa7fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9fc0ccaa0 0x7fe9fc144350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:41782/0 (socket says 172.21.15.5:41782) 2024-03-24T10:53:08.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.794+0000 7fe9fa7fc640 1 -- 172.21.15.5:0/1321643035 learned_addr learned my addr 172.21.15.5:0/1321643035 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:08.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.795+0000 7fe9faffd640 1 --2- 172.21.15.5:0/1321643035 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9fc06d4f0 0x7fe9fc159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:08.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.797+0000 7fe9fa7fc640 1 -- 172.21.15.5:0/1321643035 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9fc06d4f0 msgr2=0x7fe9fc159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:08.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.797+0000 7fe9fa7fc640 1 --2- 172.21.15.5:0/1321643035 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9fc06d4f0 0x7fe9fc159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:08.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.797+0000 7fe9fa7fc640 1 -- 172.21.15.5:0/1321643035 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9fc1543a0 msgr2=0x7fe9fc144890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:08.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.797+0000 7fe9fa7fc640 1 --2- 172.21.15.5:0/1321643035 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9fc1543a0 0x7fe9fc144890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:08.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.797+0000 7fe9fa7fc640 1 -- 172.21.15.5:0/1321643035 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9ec0094d0 con 0x7fe9fc0ccaa0 2024-03-24T10:53:08.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.800+0000 7fe9fa7fc640 1 --2- 172.21.15.5:0/1321643035 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9fc0ccaa0 0x7fe9fc144350 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe9f000e9b0 tx=0x7fe9f000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.805+0000 7fe9e3fff640 1 -- 172.21.15.5:0/1321643035 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f0009670 con 0x7fe9fc0ccaa0 2024-03-24T10:53:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.805+0000 7fe9e3fff640 1 -- 172.21.15.5:0/1321643035 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9f000c9a0 con 0x7fe9fc0ccaa0 2024-03-24T10:53:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.805+0000 7fea0186f640 1 -- 172.21.15.5:0/1321643035 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9fc154c00 con 0x7fe9fc0ccaa0 2024-03-24T10:53:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.805+0000 7fe9e3fff640 1 -- 172.21.15.5:0/1321643035 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f000cd50 con 0x7fe9fc0ccaa0 2024-03-24T10:53:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.805+0000 7fea0186f640 1 -- 172.21.15.5:0/1321643035 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9fc1560d0 con 0x7fe9fc0ccaa0 2024-03-24T10:53:08.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:08.806+0000 7fea0186f640 1 -- 172.21.15.5:0/1321643035 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9fc154a60 con 0x7fe9fc0ccaa0 2024-03-24T10:53:09.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:09.086+0000 7fe9e3fff640 1 -- 172.21.15.5:0/1321643035 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe9f0013470 con 0x7fe9fc0ccaa0 2024-03-24T10:53:09.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:09.086+0000 7fe9e3fff640 1 --2- 172.21.15.5:0/1321643035 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe9e806b840 0x7fe9e806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:09.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:09.086+0000 7fe9faffd640 1 --2- 172.21.15.5:0/1321643035 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe9e806b840 0x7fe9e806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:09.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:09.087+0000 7fe9faffd640 1 --2- 172.21.15.5:0/1321643035 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe9e806b840 0x7fe9e806dd00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec036000 tx=0x7fe9ec008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:09.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:09.109+0000 7fe9e3fff640 1 -- 172.21.15.5:0/1321643035 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fe9f0059200 con 0x7fe9fc0ccaa0 2024-03-24T10:53:09.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:09.109+0000 7fe9e3fff640 1 -- 172.21.15.5:0/1321643035 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe9f00554a0 con 0x7fe9fc0ccaa0 2024-03-24T10:53:09.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:09.332+0000 7fea0186f640 1 -- 172.21.15.5:0/1321643035 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9fc154a60 con 0x7fe9fc0ccaa0 2024-03-24T10:53:09.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:09.345+0000 7fe9e3fff640 1 -- 172.21.15.5:0/1321643035 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fe9fc154a60 con 0x7fe9fc0ccaa0 2024-03-24T10:53:09.345 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:09.345 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:09.345 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:09.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:09.347+0000 7fea0186f640 1 -- 172.21.15.5:0/1321643035 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe9e806b840 msgr2=0x7fe9e806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:09.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:09.347+0000 7fea0186f640 1 --2- 172.21.15.5:0/1321643035 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe9e806b840 0x7fe9e806dd00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec036000 tx=0x7fe9ec008030 comp rx=0 tx=0).stop 2024-03-24T10:53:09.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:09.347+0000 7fea0186f640 1 -- 172.21.15.5:0/1321643035 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9fc0ccaa0 msgr2=0x7fe9fc144350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:09.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:09.347+0000 7fea0186f640 1 --2- 172.21.15.5:0/1321643035 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9fc0ccaa0 0x7fe9fc144350 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe9f000e9b0 tx=0x7fe9f000ee70 comp rx=0 tx=0).stop 2024-03-24T10:53:09.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:09.347+0000 7fea0186f640 1 -- 172.21.15.5:0/1321643035 shutdown_connections 2024-03-24T10:53:09.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:09.347+0000 7fea0186f640 1 --2- 172.21.15.5:0/1321643035 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe9e806b840 0x7fe9e806dd00 secure :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec036000 tx=0x7fe9ec008030 comp rx=0 tx=0).stop 2024-03-24T10:53:09.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:09.347+0000 7fea0186f640 1 --2- 172.21.15.5:0/1321643035 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9fc0ccaa0 0x7fe9fc144350 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:09.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:09.347+0000 7fea0186f640 1 --2- 172.21.15.5:0/1321643035 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9fc1543a0 0x7fe9fc144890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:09.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:09.347+0000 7fea0186f640 1 --2- 172.21.15.5:0/1321643035 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9fc06d4f0 0x7fe9fc159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:09.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:09.347+0000 7fea0186f640 1 -- 172.21.15.5:0/1321643035 >> 172.21.15.5:0/1321643035 conn(0x7fe9fc072ef0 msgr2=0x7fe9fc06bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:09.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:09.347+0000 7fea0186f640 1 -- 172.21.15.5:0/1321643035 shutdown_connections 2024-03-24T10:53:09.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:09.347+0000 7fea0186f640 1 -- 172.21.15.5:0/1321643035 wait complete. 2024-03-24T10:53:09.963 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:09.963 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:10.964 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:11.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.066+0000 7fc45a8eb640 1 Processor -- start 2024-03-24T10:53:11.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.066+0000 7fc45a8eb640 1 -- start start 2024-03-24T10:53:11.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.067+0000 7fc45a8eb640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4541543a0 0x7fc45415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:11.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.067+0000 7fc45a8eb640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc45406d4f0 0x7fc4540cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:11.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.067+0000 7fc45a8eb640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4540cb2f0 con 0x7fc4540ccaa0 2024-03-24T10:53:11.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.067+0000 7fc45a8eb640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fc4540c6750 con 0x7fc45406d4f0 2024-03-24T10:53:11.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.067+0000 7fc45a8eb640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc4540cc380 con 0x7fc4541543a0 2024-03-24T10:53:11.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.068+0000 7fc4537fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc45406d4f0 0x7fc4540cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:11.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.068+0000 7fc453fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4541543a0 0x7fc45415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:11.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.068+0000 7fc453fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4541543a0 0x7fc45415ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:33386/0 (socket says 172.21.15.5:33386) 2024-03-24T10:53:11.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.068+0000 7fc453fff640 1 -- 172.21.15.5:0/2785306667 learned_addr learned my addr 172.21.15.5:0/2785306667 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:11.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.072+0000 7fc453fff640 1 -- 172.21.15.5:0/2785306667 >> v2:172.21.15.5:3300/0 conn(0x7fc45406d4f0 msgr2=0x7fc4540cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:11.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.072+0000 7fc453fff640 1 --2- 172.21.15.5:0/2785306667 >> v2:172.21.15.5:3300/0 conn(0x7fc45406d4f0 0x7fc4540cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:11.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.072+0000 7fc453fff640 1 -- 172.21.15.5:0/2785306667 >> v1:172.21.15.5:6789/0 conn(0x7fc4540ccaa0 legacy=0x7fc45406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:11.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.072+0000 7fc453fff640 1 -- 172.21.15.5:0/2785306667 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4540ca7e0 con 0x7fc4541543a0 2024-03-24T10:53:11.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.075+0000 7fc453fff640 1 --2- 172.21.15.5:0/2785306667 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4541543a0 0x7fc45415ac00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc444009870 tx=0x7fc44402ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=673b5bb6371856be server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:11.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.080+0000 7fc452ffd640 1 -- 172.21.15.5:0/2785306667 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44402f840 con 0x7fc4541543a0 2024-03-24T10:53:11.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.080+0000 7fc452ffd640 1 -- 172.21.15.5:0/2785306667 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc44402f9a0 con 0x7fc4541543a0 2024-03-24T10:53:11.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.080+0000 7fc452ffd640 1 -- 172.21.15.5:0/2785306667 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44402fd30 con 0x7fc4541543a0 2024-03-24T10:53:11.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.080+0000 7fc45a8eb640 1 -- 172.21.15.5:0/2785306667 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4541543a0 msgr2=0x7fc45415ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:11.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.080+0000 7fc45a8eb640 1 --2- 172.21.15.5:0/2785306667 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4541543a0 0x7fc45415ac00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc444009870 tx=0x7fc44402ba20 comp rx=0 tx=0).stop 2024-03-24T10:53:11.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.080+0000 7fc45a8eb640 1 -- 172.21.15.5:0/2785306667 shutdown_connections 2024-03-24T10:53:11.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.080+0000 7fc45a8eb640 1 --2- 172.21.15.5:0/2785306667 >> v2:172.21.15.5:3300/0 conn(0x7fc45406d4f0 0x7fc4540cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:11.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.080+0000 7fc45a8eb640 1 --2- 172.21.15.5:0/2785306667 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4541543a0 0x7fc45415ac00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:11.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.080+0000 7fc45a8eb640 1 -- 172.21.15.5:0/2785306667 >> 172.21.15.5:0/2785306667 conn(0x7fc454072ef0 msgr2=0x7fc454061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:11.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.080+0000 7fc45a8eb640 1 -- 172.21.15.5:0/2785306667 shutdown_connections 2024-03-24T10:53:11.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.081+0000 7fc45a8eb640 1 -- 172.21.15.5:0/2785306667 wait complete. 2024-03-24T10:53:11.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.082+0000 7fc45a8eb640 1 Processor -- start 2024-03-24T10:53:11.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.082+0000 7fc45a8eb640 1 -- start start 2024-03-24T10:53:11.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.082+0000 7fc45a8eb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc45406d4f0 0x7fc4540b3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:11.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.082+0000 7fc45a8eb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4540ccaa0 0x7fc4541effe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:11.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.083+0000 7fc45a8eb640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4541543a0 0x7fc4541f0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:11.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.083+0000 7fc45a8eb640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fc45415bdd0 con 0x7fc45406d4f0 2024-03-24T10:53:11.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.083+0000 7fc45a8eb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc4540b3c20 con 0x7fc4540ccaa0 2024-03-24T10:53:11.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.083+0000 7fc45a8eb640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc45415b350 con 0x7fc4541543a0 2024-03-24T10:53:11.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.083+0000 7fc458e88640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4541543a0 0x7fc4541f0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:11.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.084+0000 7fc458e88640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4541543a0 0x7fc4541f0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:33392/0 (socket says 172.21.15.5:33392) 2024-03-24T10:53:11.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.084+0000 7fc458e88640 1 -- 172.21.15.5:0/421828688 learned_addr learned my addr 172.21.15.5:0/421828688 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:11.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.084+0000 7fc453fff640 1 --2- 172.21.15.5:0/421828688 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc45406d4f0 0x7fc4540b3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:11.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.084+0000 7fc4537fe640 1 --2- 172.21.15.5:0/421828688 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4540ccaa0 0x7fc4541effe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:11.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.089+0000 7fc4537fe640 1 -- 172.21.15.5:0/421828688 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4541543a0 msgr2=0x7fc4541f0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:11.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.089+0000 7fc4537fe640 1 --2- 172.21.15.5:0/421828688 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4541543a0 0x7fc4541f0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:11.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.089+0000 7fc4537fe640 1 -- 172.21.15.5:0/421828688 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc45406d4f0 msgr2=0x7fc4540b3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:11.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.089+0000 7fc4537fe640 1 --2- 172.21.15.5:0/421828688 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc45406d4f0 0x7fc4540b3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:11.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.089+0000 7fc4537fe640 1 -- 172.21.15.5:0/421828688 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4440094d0 con 0x7fc4540ccaa0 2024-03-24T10:53:11.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.089+0000 7fc458e88640 1 --2- 172.21.15.5:0/421828688 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4541543a0 0x7fc4541f0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:53:11.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.092+0000 7fc4537fe640 1 --2- 172.21.15.5:0/421828688 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4540ccaa0 0x7fc4541effe0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc44800e8a0 tx=0x7fc44800ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.100+0000 7fc450ff9640 1 -- 172.21.15.5:0/421828688 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc448009e60 con 0x7fc4540ccaa0 2024-03-24T10:53:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.100+0000 7fc450ff9640 1 -- 172.21.15.5:0/421828688 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc44800c940 con 0x7fc4540ccaa0 2024-03-24T10:53:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.100+0000 7fc45a8eb640 1 -- 172.21.15.5:0/421828688 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc45415b630 con 0x7fc4540ccaa0 2024-03-24T10:53:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.101+0000 7fc450ff9640 1 -- 172.21.15.5:0/421828688 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44800ccd0 con 0x7fc4540ccaa0 2024-03-24T10:53:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.101+0000 7fc45a8eb640 1 -- 172.21.15.5:0/421828688 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4540b3f00 con 0x7fc4540ccaa0 2024-03-24T10:53:11.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.102+0000 7fc45a8eb640 1 -- 172.21.15.5:0/421828688 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4540681d0 con 0x7fc4540ccaa0 2024-03-24T10:53:11.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.381+0000 7fc450ff9640 1 -- 172.21.15.5:0/421828688 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc448013470 con 0x7fc4540ccaa0 2024-03-24T10:53:11.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.382+0000 7fc450ff9640 1 --2- 172.21.15.5:0/421828688 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc44006b810 0x7fc44006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:11.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.382+0000 7fc453fff640 1 --2- 172.21.15.5:0/421828688 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc44006b810 0x7fc44006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:11.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.382+0000 7fc453fff640 1 --2- 172.21.15.5:0/421828688 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc44006b810 0x7fc44006dcd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc444036000 tx=0x7fc444008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:11.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.384+0000 7fc450ff9640 1 -- 172.21.15.5:0/421828688 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fc4480591c0 con 0x7fc4540ccaa0 2024-03-24T10:53:11.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.395+0000 7fc450ff9640 1 -- 172.21.15.5:0/421828688 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc448055460 con 0x7fc4540ccaa0 2024-03-24T10:53:11.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.646+0000 7fc45a8eb640 1 -- 172.21.15.5:0/421828688 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc454152ef0 con 0x7fc4540ccaa0 2024-03-24T10:53:11.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.704+0000 7fc450ff9640 1 -- 172.21.15.5:0/421828688 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fc448021090 con 0x7fc4540ccaa0 2024-03-24T10:53:11.704 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:11.704 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:11.705 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:11.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.706+0000 7fc45a8eb640 1 -- 172.21.15.5:0/421828688 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc44006b810 msgr2=0x7fc44006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:11.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.706+0000 7fc45a8eb640 1 --2- 172.21.15.5:0/421828688 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc44006b810 0x7fc44006dcd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc444036000 tx=0x7fc444008030 comp rx=0 tx=0).stop 2024-03-24T10:53:11.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.706+0000 7fc45a8eb640 1 -- 172.21.15.5:0/421828688 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4540ccaa0 msgr2=0x7fc4541effe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:11.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.706+0000 7fc45a8eb640 1 --2- 172.21.15.5:0/421828688 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4540ccaa0 0x7fc4541effe0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc44800e8a0 tx=0x7fc44800ed60 comp rx=0 tx=0).stop 2024-03-24T10:53:11.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.706+0000 7fc45a8eb640 1 -- 172.21.15.5:0/421828688 shutdown_connections 2024-03-24T10:53:11.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.706+0000 7fc45a8eb640 1 --2- 172.21.15.5:0/421828688 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc44006b810 0x7fc44006dcd0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:11.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.706+0000 7fc45a8eb640 1 --2- 172.21.15.5:0/421828688 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4541543a0 0x7fc4541f0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:11.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.706+0000 7fc45a8eb640 1 --2- 172.21.15.5:0/421828688 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4540ccaa0 0x7fc4541effe0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:11.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.706+0000 7fc45a8eb640 1 --2- 172.21.15.5:0/421828688 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc45406d4f0 0x7fc4540b3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:11.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.706+0000 7fc45a8eb640 1 -- 172.21.15.5:0/421828688 >> 172.21.15.5:0/421828688 conn(0x7fc454072ef0 msgr2=0x7fc45406be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:11.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.706+0000 7fc45a8eb640 1 -- 172.21.15.5:0/421828688 shutdown_connections 2024-03-24T10:53:11.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:11.706+0000 7fc45a8eb640 1 -- 172.21.15.5:0/421828688 wait complete. 2024-03-24T10:53:12.321 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:12.321 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:13.322 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:13.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.424+0000 7fbf50eaf640 1 Processor -- start 2024-03-24T10:53:13.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.424+0000 7fbf50eaf640 1 -- start start 2024-03-24T10:53:13.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.425+0000 7fbf50eaf640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf4c0cb270 0x7fbf4c0cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:13.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.425+0000 7fbf50eaf640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbf4c0cd960 0x7fbf4c0cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:13.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.425+0000 7fbf50eaf640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf4c0ccba0 con 0x7fbf4c0cae90 2024-03-24T10:53:13.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.425+0000 7fbf50eaf640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fbf4c0cc9b0 con 0x7fbf4c0cd960 2024-03-24T10:53:13.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.425+0000 7fbf50eaf640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fbf4c0cce40 con 0x7fbf4c0cb270 2024-03-24T10:53:13.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.425+0000 7fbf4ad9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbf4c0cd960 0x7fbf4c0cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:13.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.426+0000 7fbf49d9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf4c0cb270 0x7fbf4c0cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:13.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.426+0000 7fbf4ad9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbf4c0cd960 0x7fbf4c0cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52296/0 (socket says 172.21.15.5:52296) 2024-03-24T10:53:13.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.426+0000 7fbf4ad9d640 1 -- 172.21.15.5:0/2255628522 learned_addr learned my addr 172.21.15.5:0/2255628522 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:13.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.429+0000 7fbf4ad9d640 1 -- 172.21.15.5:0/2255628522 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf4c0cb270 msgr2=0x7fbf4c0cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:13.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.429+0000 7fbf4ad9d640 1 --2- 172.21.15.5:0/2255628522 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf4c0cb270 0x7fbf4c0cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:13.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.429+0000 7fbf4ad9d640 1 -- 172.21.15.5:0/2255628522 >> v1:172.21.15.5:6789/0 conn(0x7fbf4c0cae90 legacy=0x7fbf4c0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:13.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.429+0000 7fbf4ad9d640 1 -- 172.21.15.5:0/2255628522 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf4c15a590 con 0x7fbf4c0cd960 2024-03-24T10:53:13.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.434+0000 7fbf4ad9d640 1 --2- 172.21.15.5:0/2255628522 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf4c0cd960 0x7fbf4c0cb6f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fbf44009e50 tx=0x7fbf4402bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32316ef6ce132f2f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:13.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.437+0000 7fbf4959a640 1 -- 172.21.15.5:0/2255628522 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4402cb90 con 0x7fbf4c0cd960 2024-03-24T10:53:13.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.439+0000 7fbf4959a640 1 -- 172.21.15.5:0/2255628522 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf4402ccf0 con 0x7fbf4c0cd960 2024-03-24T10:53:13.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.439+0000 7fbf4959a640 1 -- 172.21.15.5:0/2255628522 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf44034610 con 0x7fbf4c0cd960 2024-03-24T10:53:13.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.440+0000 7fbf50eaf640 1 -- 172.21.15.5:0/2255628522 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf4c0cd960 msgr2=0x7fbf4c0cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:13.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.440+0000 7fbf50eaf640 1 --2- 172.21.15.5:0/2255628522 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf4c0cd960 0x7fbf4c0cb6f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fbf44009e50 tx=0x7fbf4402bff0 comp rx=0 tx=0).stop 2024-03-24T10:53:13.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.440+0000 7fbf50eaf640 1 -- 172.21.15.5:0/2255628522 shutdown_connections 2024-03-24T10:53:13.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.440+0000 7fbf50eaf640 1 --2- 172.21.15.5:0/2255628522 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf4c0cd960 0x7fbf4c0cb6f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:13.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.440+0000 7fbf50eaf640 1 --2- 172.21.15.5:0/2255628522 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf4c0cb270 0x7fbf4c0cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:13.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.440+0000 7fbf50eaf640 1 -- 172.21.15.5:0/2255628522 >> 172.21.15.5:0/2255628522 conn(0x7fbf4c0cfb80 msgr2=0x7fbf4c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:13.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.440+0000 7fbf50eaf640 1 -- 172.21.15.5:0/2255628522 shutdown_connections 2024-03-24T10:53:13.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.440+0000 7fbf50eaf640 1 -- 172.21.15.5:0/2255628522 wait complete. 2024-03-24T10:53:13.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.442+0000 7fbf50eaf640 1 Processor -- start 2024-03-24T10:53:13.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.442+0000 7fbf50eaf640 1 -- start start 2024-03-24T10:53:13.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.444+0000 7fbf50eaf640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf4c0cae90 0x7fbf4c157820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:13.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.444+0000 7fbf50eaf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf4c0cb270 0x7fbf4c1f2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:13.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.444+0000 7fbf50eaf640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf4c0cd960 0x7fbf4c0b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:13.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.444+0000 7fbf50eaf640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4c157d60 con 0x7fbf4c0cae90 2024-03-24T10:53:13.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.444+0000 7fbf50eaf640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4c155d60 con 0x7fbf4c0cb270 2024-03-24T10:53:13.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.444+0000 7fbf50eaf640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fbf4c1f3920 con 0x7fbf4c0cd960 2024-03-24T10:53:13.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.444+0000 7fbf4a59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf4c0cae90 0x7fbf4c157820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:13.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.444+0000 7fbf4ad9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf4c0cd960 0x7fbf4c0b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:13.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.444+0000 7fbf49d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf4c0cb270 0x7fbf4c1f2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:13.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.444+0000 7fbf4a59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf4c0cae90 0x7fbf4c157820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52308/0 (socket says 172.21.15.5:52308) 2024-03-24T10:53:13.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.444+0000 7fbf4a59c640 1 -- 172.21.15.5:0/1845247831 learned_addr learned my addr 172.21.15.5:0/1845247831 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:13.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.448+0000 7fbf4a59c640 1 -- 172.21.15.5:0/1845247831 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf4c0cd960 msgr2=0x7fbf4c0b15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:13.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.448+0000 7fbf4a59c640 1 --2- 172.21.15.5:0/1845247831 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf4c0cd960 0x7fbf4c0b15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:13.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.448+0000 7fbf4a59c640 1 -- 172.21.15.5:0/1845247831 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf4c0cb270 msgr2=0x7fbf4c1f2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:13.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.448+0000 7fbf4a59c640 1 --2- 172.21.15.5:0/1845247831 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf4c0cb270 0x7fbf4c1f2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:13.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.448+0000 7fbf4a59c640 1 -- 172.21.15.5:0/1845247831 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf44009b30 con 0x7fbf4c0cae90 2024-03-24T10:53:13.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.451+0000 7fbf4a59c640 1 --2- 172.21.15.5:0/1845247831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf4c0cae90 0x7fbf4c157820 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c00c3c0 tx=0x7fbf3c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:13.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.455+0000 7fbf3affd640 1 -- 172.21.15.5:0/1845247831 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c0052d0 con 0x7fbf4c0cae90 2024-03-24T10:53:13.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.455+0000 7fbf3affd640 1 -- 172.21.15.5:0/1845247831 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf3c005450 con 0x7fbf4c0cae90 2024-03-24T10:53:13.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.455+0000 7fbf50eaf640 1 -- 172.21.15.5:0/1845247831 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf4c1f3be0 con 0x7fbf4c0cae90 2024-03-24T10:53:13.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.455+0000 7fbf3affd640 1 -- 172.21.15.5:0/1845247831 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf3c007260 con 0x7fbf4c0cae90 2024-03-24T10:53:13.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.455+0000 7fbf50eaf640 1 -- 172.21.15.5:0/1845247831 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf4c155a80 con 0x7fbf4c0cae90 2024-03-24T10:53:13.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.456+0000 7fbf50eaf640 1 -- 172.21.15.5:0/1845247831 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf4c0681d0 con 0x7fbf4c0cae90 2024-03-24T10:53:13.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.746+0000 7fbf3affd640 1 -- 172.21.15.5:0/1845247831 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbf3c0073c0 con 0x7fbf4c0cae90 2024-03-24T10:53:13.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.747+0000 7fbf3affd640 1 --2- 172.21.15.5:0/1845247831 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbf2c06b860 0x7fbf2c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:13.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.747+0000 7fbf49d9b640 1 --2- 172.21.15.5:0/1845247831 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbf2c06b860 0x7fbf2c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:13.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.748+0000 7fbf49d9b640 1 --2- 172.21.15.5:0/1845247831 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbf2c06b860 0x7fbf2c06dd20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c0c7af0 tx=0x7fbf40009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:13.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.771+0000 7fbf3affd640 1 -- 172.21.15.5:0/1845247831 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fbf3c007670 con 0x7fbf4c0cae90 2024-03-24T10:53:13.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:13.771+0000 7fbf3affd640 1 -- 172.21.15.5:0/1845247831 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbf3c054810 con 0x7fbf4c0cae90 2024-03-24T10:53:14.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:14.035+0000 7fbf50eaf640 1 -- 172.21.15.5:0/1845247831 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf4c0d02e0 con 0x7fbf4c0cae90 2024-03-24T10:53:14.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:14.063+0000 7fbf3affd640 1 -- 172.21.15.5:0/1845247831 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fbf3c020020 con 0x7fbf4c0cae90 2024-03-24T10:53:14.063 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:14.063 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:14.063 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:14.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:14.065+0000 7fbf50eaf640 1 -- 172.21.15.5:0/1845247831 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbf2c06b860 msgr2=0x7fbf2c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:14.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:14.065+0000 7fbf50eaf640 1 --2- 172.21.15.5:0/1845247831 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbf2c06b860 0x7fbf2c06dd20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c0c7af0 tx=0x7fbf40009190 comp rx=0 tx=0).stop 2024-03-24T10:53:14.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:14.065+0000 7fbf50eaf640 1 -- 172.21.15.5:0/1845247831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf4c0cae90 msgr2=0x7fbf4c157820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:14.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:14.065+0000 7fbf50eaf640 1 --2- 172.21.15.5:0/1845247831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf4c0cae90 0x7fbf4c157820 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c00c3c0 tx=0x7fbf3c00c880 comp rx=0 tx=0).stop 2024-03-24T10:53:14.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:14.065+0000 7fbf50eaf640 1 -- 172.21.15.5:0/1845247831 shutdown_connections 2024-03-24T10:53:14.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:14.065+0000 7fbf50eaf640 1 --2- 172.21.15.5:0/1845247831 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbf2c06b860 0x7fbf2c06dd20 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:14.065+0000 7fbf50eaf640 1 --2- 172.21.15.5:0/1845247831 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbf4c0cd960 0x7fbf4c0b15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:14.065+0000 7fbf50eaf640 1 --2- 172.21.15.5:0/1845247831 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbf4c0cb270 0x7fbf4c1f2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:14.065+0000 7fbf50eaf640 1 --2- 172.21.15.5:0/1845247831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbf4c0cae90 0x7fbf4c157820 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:14.065+0000 7fbf50eaf640 1 -- 172.21.15.5:0/1845247831 >> 172.21.15.5:0/1845247831 conn(0x7fbf4c0cfb80 msgr2=0x7fbf4c0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:14.065+0000 7fbf50eaf640 1 -- 172.21.15.5:0/1845247831 shutdown_connections 2024-03-24T10:53:14.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:14.065+0000 7fbf50eaf640 1 -- 172.21.15.5:0/1845247831 wait complete. 2024-03-24T10:53:14.681 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:14.681 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:14.816 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:09:06.317 36946== Warning: unimplemented fcntl command: 1036 2024-03-24T10:53:14.878 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:09:06.379 36946== Warning: unimplemented fcntl command: 1036 2024-03-24T10:53:14.953 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:09:06.417 36958== Warning: unimplemented fcntl command: 1036 2024-03-24T10:53:15.001 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:09:06.465 36957== Warning: unimplemented fcntl command: 1036 2024-03-24T10:53:15.042 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:09:06.506 36958== Warning: unimplemented fcntl command: 1036 2024-03-24T10:53:15.071 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:09:06.535 36957== Warning: unimplemented fcntl command: 1036 2024-03-24T10:53:15.243 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:09:06.744 36946== Warning: unimplemented fcntl command: 1036 2024-03-24T10:53:15.488 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:09:06.952 36957== Warning: unimplemented fcntl command: 1036 2024-03-24T10:53:15.603 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:09:07.067 36958== Warning: unimplemented fcntl command: 1036 2024-03-24T10:53:15.681 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:15.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.784+0000 7f271a9d0640 1 Processor -- start 2024-03-24T10:53:15.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.784+0000 7f271a9d0640 1 -- start start 2024-03-24T10:53:15.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.785+0000 7f271a9d0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f27140cae90 0x7f27140cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:15.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.785+0000 7f271a9d0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f27140cb270 0x7f27140cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:15.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.785+0000 7f271a9d0640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27140ccba0 con 0x7f27140cd960 2024-03-24T10:53:15.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.785+0000 7f271a9d0640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f27140cce40 con 0x7f27140cb270 2024-03-24T10:53:15.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.785+0000 7f271a9d0640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f271415a590 con 0x7f27140cae90 2024-03-24T10:53:15.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.786+0000 7f27137fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f27140cb270 0x7f27140cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:15.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.786+0000 7f27137fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f27140cb270 0x7f27140cc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52310/0 (socket says 172.21.15.5:52310) 2024-03-24T10:53:15.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.786+0000 7f27137fe640 1 -- 172.21.15.5:0/795992747 learned_addr learned my addr 172.21.15.5:0/795992747 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:15.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.790+0000 7f27137fe640 1 -- 172.21.15.5:0/795992747 >> v2:172.21.15.57:3300/0 conn(0x7f27140cae90 msgr2=0x7f27140cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:15.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.790+0000 7f27137fe640 1 --2- 172.21.15.5:0/795992747 >> v2:172.21.15.57:3300/0 conn(0x7f27140cae90 0x7f27140cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:15.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.790+0000 7f27137fe640 1 -- 172.21.15.5:0/795992747 >> v1:172.21.15.57:6789/0 conn(0x7f27140cd960 legacy=0x7f27140cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:15.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.790+0000 7f27137fe640 1 -- 172.21.15.5:0/795992747 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f271415a870 con 0x7f27140cb270 2024-03-24T10:53:15.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.792+0000 7f27137fe640 1 --2- 172.21.15.5:0/795992747 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27140cb270 0x7f27140cc2a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f27140b26a0 tx=0x7f270802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be2c2cc393b00ceb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:15.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.796+0000 7f2712ffd640 1 -- 172.21.15.5:0/795992747 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708030860 con 0x7f27140cb270 2024-03-24T10:53:15.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.796+0000 7f2712ffd640 1 -- 172.21.15.5:0/795992747 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27080309c0 con 0x7f27140cb270 2024-03-24T10:53:15.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.796+0000 7f2712ffd640 1 -- 172.21.15.5:0/795992747 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708030d50 con 0x7f27140cb270 2024-03-24T10:53:15.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.797+0000 7f271a9d0640 1 -- 172.21.15.5:0/795992747 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27140cb270 msgr2=0x7f27140cc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:15.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.797+0000 7f271a9d0640 1 --2- 172.21.15.5:0/795992747 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27140cb270 0x7f27140cc2a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f27140b26a0 tx=0x7f270802b9a0 comp rx=0 tx=0).stop 2024-03-24T10:53:15.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.797+0000 7f271a9d0640 1 -- 172.21.15.5:0/795992747 shutdown_connections 2024-03-24T10:53:15.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.797+0000 7f271a9d0640 1 --2- 172.21.15.5:0/795992747 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27140cb270 0x7f27140cc2a0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:15.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.797+0000 7f271a9d0640 1 --2- 172.21.15.5:0/795992747 >> v2:172.21.15.57:3300/0 conn(0x7f27140cae90 0x7f27140cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:15.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.797+0000 7f271a9d0640 1 -- 172.21.15.5:0/795992747 >> 172.21.15.5:0/795992747 conn(0x7f27140cfb80 msgr2=0x7f27140cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:15.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.797+0000 7f271a9d0640 1 -- 172.21.15.5:0/795992747 shutdown_connections 2024-03-24T10:53:15.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.797+0000 7f271a9d0640 1 -- 172.21.15.5:0/795992747 wait complete. 2024-03-24T10:53:15.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.799+0000 7f271a9d0640 1 Processor -- start 2024-03-24T10:53:15.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.799+0000 7f271a9d0640 1 -- start start 2024-03-24T10:53:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.800+0000 7f271a9d0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27140cae90 0x7f27140c7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.800+0000 7f271a9d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27140cb270 0x7f271415bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.800+0000 7f271a9d0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f27140cd960 0x7f271415c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.800+0000 7f271a9d0640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f27141f3eb0 con 0x7f27140cae90 2024-03-24T10:53:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.800+0000 7f271a9d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f27141f3330 con 0x7f27140cb270 2024-03-24T10:53:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.800+0000 7f271a9d0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f27141f3610 con 0x7f27140cd960 2024-03-24T10:53:15.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.801+0000 7f2713fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27140cae90 0x7f27140c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:15.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.801+0000 7f2713fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27140cae90 0x7f27140c7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52320/0 (socket says 172.21.15.5:52320) 2024-03-24T10:53:15.803 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.801+0000 7f2713fff640 1 -- 172.21.15.5:0/3710346521 learned_addr learned my addr 172.21.15.5:0/3710346521 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:15.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.804+0000 7f27137fe640 1 --2- 172.21.15.5:0/3710346521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27140cb270 0x7f271415bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:15.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.805+0000 7f2713fff640 1 -- 172.21.15.5:0/3710346521 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f27140cd960 msgr2=0x7f271415c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:15.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.805+0000 7f2713fff640 1 --2- 172.21.15.5:0/3710346521 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f27140cd960 0x7f271415c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:15.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.805+0000 7f2713fff640 1 -- 172.21.15.5:0/3710346521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27140cb270 msgr2=0x7f271415bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:15.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.805+0000 7f2713fff640 1 --2- 172.21.15.5:0/3710346521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27140cb270 0x7f271415bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:15.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.805+0000 7f2713fff640 1 -- 172.21.15.5:0/3710346521 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27080094d0 con 0x7f27140cae90 2024-03-24T10:53:15.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.808+0000 7f2713fff640 1 --2- 172.21.15.5:0/3710346521 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27140cae90 0x7f27140c7620 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f270400b4a0 tx=0x7f270400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:15.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.812+0000 7f2710ff9640 1 -- 172.21.15.5:0/3710346521 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2704010610 con 0x7f27140cae90 2024-03-24T10:53:15.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.812+0000 7f2710ff9640 1 -- 172.21.15.5:0/3710346521 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2704010770 con 0x7f27140cae90 2024-03-24T10:53:15.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.812+0000 7f271a9d0640 1 -- 172.21.15.5:0/3710346521 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27141f38f0 con 0x7f27140cae90 2024-03-24T10:53:15.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.812+0000 7f2710ff9640 1 -- 172.21.15.5:0/3710346521 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2704010b20 con 0x7f27140cae90 2024-03-24T10:53:15.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.812+0000 7f271a9d0640 1 -- 172.21.15.5:0/3710346521 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f271415b780 con 0x7f27140cae90 2024-03-24T10:53:15.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.813+0000 7f271a9d0640 1 -- 172.21.15.5:0/3710346521 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f271415b5e0 con 0x7f27140cae90 2024-03-24T10:53:15.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.835+0000 7f2710ff9640 1 -- 172.21.15.5:0/3710346521 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2704010c80 con 0x7f27140cae90 2024-03-24T10:53:15.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.835+0000 7f2710ff9640 1 --2- 172.21.15.5:0/3710346521 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f270006b810 0x7f270006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:15.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.835+0000 7f27137fe640 1 --2- 172.21.15.5:0/3710346521 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f270006b810 0x7f270006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:15.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.836+0000 7f27137fe640 1 --2- 172.21.15.5:0/3710346521 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f270006b810 0x7f270006dcd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2708036000 tx=0x7f270803b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:15.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:15.838+0000 7f2710ff9640 1 -- 172.21.15.5:0/3710346521 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f2704059020 con 0x7f27140cae90 2024-03-24T10:53:16.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:16.114+0000 7f2710ff9640 1 -- 172.21.15.5:0/3710346521 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2704054bc0 con 0x7f27140cae90 2024-03-24T10:53:16.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:16.371+0000 7f271a9d0640 1 -- 172.21.15.5:0/3710346521 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f271415b5e0 con 0x7f27140cae90 2024-03-24T10:53:16.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:16.384+0000 7f2710ff9640 1 -- 172.21.15.5:0/3710346521 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f271415b5e0 con 0x7f27140cae90 2024-03-24T10:53:16.385 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:16.385 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:16.385 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:16.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:16.386+0000 7f271a9d0640 1 -- 172.21.15.5:0/3710346521 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f270006b810 msgr2=0x7f270006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:16.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:16.386+0000 7f271a9d0640 1 --2- 172.21.15.5:0/3710346521 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f270006b810 0x7f270006dcd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2708036000 tx=0x7f270803b040 comp rx=0 tx=0).stop 2024-03-24T10:53:16.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:16.386+0000 7f271a9d0640 1 -- 172.21.15.5:0/3710346521 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27140cae90 msgr2=0x7f27140c7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:16.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:16.386+0000 7f271a9d0640 1 --2- 172.21.15.5:0/3710346521 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27140cae90 0x7f27140c7620 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f270400b4a0 tx=0x7f270400b960 comp rx=0 tx=0).stop 2024-03-24T10:53:16.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:16.386+0000 7f271a9d0640 1 -- 172.21.15.5:0/3710346521 shutdown_connections 2024-03-24T10:53:16.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:16.386+0000 7f271a9d0640 1 --2- 172.21.15.5:0/3710346521 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f270006b810 0x7f270006dcd0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:16.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:16.386+0000 7f271a9d0640 1 --2- 172.21.15.5:0/3710346521 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f27140cd960 0x7f271415c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:16.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:16.386+0000 7f271a9d0640 1 --2- 172.21.15.5:0/3710346521 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f27140cb270 0x7f271415bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:16.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:16.386+0000 7f271a9d0640 1 --2- 172.21.15.5:0/3710346521 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f27140cae90 0x7f27140c7620 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:16.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:16.386+0000 7f271a9d0640 1 -- 172.21.15.5:0/3710346521 >> 172.21.15.5:0/3710346521 conn(0x7f27140cfb80 msgr2=0x7f2714159ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:16.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:16.386+0000 7f271a9d0640 1 -- 172.21.15.5:0/3710346521 shutdown_connections 2024-03-24T10:53:16.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:16.386+0000 7f271a9d0640 1 -- 172.21.15.5:0/3710346521 wait complete. 2024-03-24T10:53:17.002 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:17.003 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:18.003 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:18.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.105+0000 7fda44513640 1 Processor -- start 2024-03-24T10:53:18.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.105+0000 7fda44513640 1 -- start start 2024-03-24T10:53:18.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.107+0000 7fda44513640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fda3c15ac00 0x7fda3c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:18.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.107+0000 7fda44513640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fda3c0ccaa0 0x7fda3c06bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:18.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.107+0000 7fda44513640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda3c0cb2f0 con 0x7fda3c1543a0 2024-03-24T10:53:18.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.107+0000 7fda44513640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fda3c0c6750 con 0x7fda3c0ccaa0 2024-03-24T10:53:18.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.107+0000 7fda44513640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fda3c0cc380 con 0x7fda3c15ac00 2024-03-24T10:53:18.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.107+0000 7fda42ab0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fda3c0ccaa0 0x7fda3c06bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:18.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.107+0000 7fda41aae640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fda3c15ac00 0x7fda3c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:18.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.108+0000 7fda41aae640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fda3c15ac00 0x7fda3c0cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:45064/0 (socket says 172.21.15.5:45064) 2024-03-24T10:53:18.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.108+0000 7fda41aae640 1 -- 172.21.15.5:0/3989165223 learned_addr learned my addr 172.21.15.5:0/3989165223 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:18.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.111+0000 7fda41aae640 1 -- 172.21.15.5:0/3989165223 >> v2:172.21.15.5:3300/0 conn(0x7fda3c0ccaa0 msgr2=0x7fda3c06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:18.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.111+0000 7fda41aae640 1 --2- 172.21.15.5:0/3989165223 >> v2:172.21.15.5:3300/0 conn(0x7fda3c0ccaa0 0x7fda3c06bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:18.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.111+0000 7fda41aae640 1 -- 172.21.15.5:0/3989165223 >> v1:172.21.15.5:6789/0 conn(0x7fda3c1543a0 legacy=0x7fda3c06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:18.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.111+0000 7fda41aae640 1 -- 172.21.15.5:0/3989165223 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda3c15b070 con 0x7fda3c15ac00 2024-03-24T10:53:18.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.114+0000 7fda41aae640 1 --2- 172.21.15.5:0/3989165223 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda3c15ac00 0x7fda3c0cc560 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fda3c0c6990 tx=0x7fda2c02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9dd0c3cdf28d70f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:18.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.118+0000 7fda412ad640 1 -- 172.21.15.5:0/3989165223 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2c030860 con 0x7fda3c15ac00 2024-03-24T10:53:18.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.118+0000 7fda412ad640 1 -- 172.21.15.5:0/3989165223 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda2c0309c0 con 0x7fda3c15ac00 2024-03-24T10:53:18.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.118+0000 7fda412ad640 1 -- 172.21.15.5:0/3989165223 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2c030d50 con 0x7fda3c15ac00 2024-03-24T10:53:18.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.119+0000 7fda44513640 1 -- 172.21.15.5:0/3989165223 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda3c15ac00 msgr2=0x7fda3c0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:18.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.119+0000 7fda44513640 1 --2- 172.21.15.5:0/3989165223 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda3c15ac00 0x7fda3c0cc560 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fda3c0c6990 tx=0x7fda2c02b9a0 comp rx=0 tx=0).stop 2024-03-24T10:53:18.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.119+0000 7fda44513640 1 -- 172.21.15.5:0/3989165223 shutdown_connections 2024-03-24T10:53:18.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.119+0000 7fda44513640 1 --2- 172.21.15.5:0/3989165223 >> v2:172.21.15.5:3300/0 conn(0x7fda3c0ccaa0 0x7fda3c06bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:18.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.119+0000 7fda44513640 1 --2- 172.21.15.5:0/3989165223 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda3c15ac00 0x7fda3c0cc560 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:18.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.119+0000 7fda44513640 1 -- 172.21.15.5:0/3989165223 >> 172.21.15.5:0/3989165223 conn(0x7fda3c072ef0 msgr2=0x7fda3c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:18.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.119+0000 7fda44513640 1 -- 172.21.15.5:0/3989165223 shutdown_connections 2024-03-24T10:53:18.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.119+0000 7fda44513640 1 -- 172.21.15.5:0/3989165223 wait complete. 2024-03-24T10:53:18.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.121+0000 7fda44513640 1 Processor -- start 2024-03-24T10:53:18.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.121+0000 7fda44513640 1 -- start start 2024-03-24T10:53:18.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.122+0000 7fda44513640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda3c0ccaa0 0x7fda3c0ce920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:18.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.123+0000 7fda44513640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda3c1543a0 0x7fda3c1f0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:18.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.123+0000 7fda44513640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda3c15ac00 0x7fda3c14f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:18.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.123+0000 7fda44513640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c1f0dd0 con 0x7fda3c0ccaa0 2024-03-24T10:53:18.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.123+0000 7fda44513640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c0b3980 con 0x7fda3c1543a0 2024-03-24T10:53:18.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.123+0000 7fda44513640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fda3c0cef40 con 0x7fda3c15ac00 2024-03-24T10:53:18.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.123+0000 7fda422af640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda3c0ccaa0 0x7fda3c0ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:18.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.123+0000 7fda41aae640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda3c1543a0 0x7fda3c1f0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:18.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.123+0000 7fda422af640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda3c0ccaa0 0x7fda3c0ce920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52344/0 (socket says 172.21.15.5:52344) 2024-03-24T10:53:18.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.123+0000 7fda422af640 1 -- 172.21.15.5:0/172882407 learned_addr learned my addr 172.21.15.5:0/172882407 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:18.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.128+0000 7fda42ab0640 1 --2- 172.21.15.5:0/172882407 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda3c15ac00 0x7fda3c14f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:18.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.128+0000 7fda422af640 1 -- 172.21.15.5:0/172882407 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda3c15ac00 msgr2=0x7fda3c14f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:18.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.128+0000 7fda422af640 1 --2- 172.21.15.5:0/172882407 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda3c15ac00 0x7fda3c14f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:18.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.128+0000 7fda422af640 1 -- 172.21.15.5:0/172882407 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda3c1543a0 msgr2=0x7fda3c1f0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:18.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.128+0000 7fda422af640 1 --2- 172.21.15.5:0/172882407 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda3c1543a0 0x7fda3c1f0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:18.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.128+0000 7fda422af640 1 -- 172.21.15.5:0/172882407 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda2c0094d0 con 0x7fda3c0ccaa0 2024-03-24T10:53:18.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.145+0000 7fda422af640 1 --2- 172.21.15.5:0/172882407 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda3c0ccaa0 0x7fda3c0ce920 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fda3800c400 tx=0x7fda3800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:18.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.153+0000 7fda32ffd640 1 -- 172.21.15.5:0/172882407 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda380052d0 con 0x7fda3c0ccaa0 2024-03-24T10:53:18.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.153+0000 7fda32ffd640 1 -- 172.21.15.5:0/172882407 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda38005450 con 0x7fda3c0ccaa0 2024-03-24T10:53:18.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.153+0000 7fda32ffd640 1 -- 172.21.15.5:0/172882407 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda38007260 con 0x7fda3c0ccaa0 2024-03-24T10:53:18.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.153+0000 7fda44513640 1 -- 172.21.15.5:0/172882407 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda3c0cf220 con 0x7fda3c0ccaa0 2024-03-24T10:53:18.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.153+0000 7fda44513640 1 -- 172.21.15.5:0/172882407 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda3c0b3330 con 0x7fda3c0ccaa0 2024-03-24T10:53:18.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.154+0000 7fda44513640 1 -- 172.21.15.5:0/172882407 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda3c0681d0 con 0x7fda3c0ccaa0 2024-03-24T10:53:18.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.431+0000 7fda32ffd640 1 -- 172.21.15.5:0/172882407 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fda380073c0 con 0x7fda3c0ccaa0 2024-03-24T10:53:18.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.431+0000 7fda32ffd640 1 --2- 172.21.15.5:0/172882407 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda2006b810 0x7fda2006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:18.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.431+0000 7fda41aae640 1 --2- 172.21.15.5:0/172882407 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda2006b810 0x7fda2006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:18.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.431+0000 7fda41aae640 1 --2- 172.21.15.5:0/172882407 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda2006b810 0x7fda2006dcd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fda2c036000 tx=0x7fda2c008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:18.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.452+0000 7fda32ffd640 1 -- 172.21.15.5:0/172882407 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fda38058300 con 0x7fda3c0ccaa0 2024-03-24T10:53:18.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.452+0000 7fda32ffd640 1 -- 172.21.15.5:0/172882407 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fda380545a0 con 0x7fda3c0ccaa0 2024-03-24T10:53:18.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.676+0000 7fda44513640 1 -- 172.21.15.5:0/172882407 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda3c157b30 con 0x7fda3c0ccaa0 2024-03-24T10:53:18.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.696+0000 7fda32ffd640 1 -- 172.21.15.5:0/172882407 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fda38020070 con 0x7fda3c0ccaa0 2024-03-24T10:53:18.698 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:18.698 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:18.698 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:18.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.699+0000 7fda44513640 1 -- 172.21.15.5:0/172882407 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda2006b810 msgr2=0x7fda2006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:18.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.699+0000 7fda44513640 1 --2- 172.21.15.5:0/172882407 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda2006b810 0x7fda2006dcd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fda2c036000 tx=0x7fda2c008030 comp rx=0 tx=0).stop 2024-03-24T10:53:18.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.699+0000 7fda44513640 1 -- 172.21.15.5:0/172882407 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda3c0ccaa0 msgr2=0x7fda3c0ce920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:18.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.699+0000 7fda44513640 1 --2- 172.21.15.5:0/172882407 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda3c0ccaa0 0x7fda3c0ce920 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fda3800c400 tx=0x7fda3800c8c0 comp rx=0 tx=0).stop 2024-03-24T10:53:18.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.699+0000 7fda44513640 1 -- 172.21.15.5:0/172882407 shutdown_connections 2024-03-24T10:53:18.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.699+0000 7fda44513640 1 --2- 172.21.15.5:0/172882407 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda2006b810 0x7fda2006dcd0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:18.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.699+0000 7fda44513640 1 --2- 172.21.15.5:0/172882407 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda3c15ac00 0x7fda3c14f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:18.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.699+0000 7fda44513640 1 --2- 172.21.15.5:0/172882407 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda3c1543a0 0x7fda3c1f0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:18.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.699+0000 7fda44513640 1 --2- 172.21.15.5:0/172882407 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda3c0ccaa0 0x7fda3c0ce920 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:18.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.699+0000 7fda44513640 1 -- 172.21.15.5:0/172882407 >> 172.21.15.5:0/172882407 conn(0x7fda3c072ef0 msgr2=0x7fda3c158a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:18.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.699+0000 7fda44513640 1 -- 172.21.15.5:0/172882407 shutdown_connections 2024-03-24T10:53:18.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:18.699+0000 7fda44513640 1 -- 172.21.15.5:0/172882407 wait complete. 2024-03-24T10:53:19.314 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:19.315 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:20.315 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:20.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.417+0000 7f53062d4640 1 Processor -- start 2024-03-24T10:53:20.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.417+0000 7f53062d4640 1 -- start start 2024-03-24T10:53:20.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.418+0000 7f53062d4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f530015ac00 0x7f53000cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:20.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.418+0000 7f53062d4640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f53000ccaa0 0x7f530006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:20.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.418+0000 7f53062d4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53000cb2f0 con 0x7f53001543a0 2024-03-24T10:53:20.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.418+0000 7f53062d4640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f53000c6750 con 0x7f53000ccaa0 2024-03-24T10:53:20.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.418+0000 7f53062d4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f53000cc380 con 0x7f530015ac00 2024-03-24T10:53:20.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.419+0000 7f5304871640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f53000ccaa0 0x7f530006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:20.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.419+0000 7f52ff7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f530015ac00 0x7f53000cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:20.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.419+0000 7f52fffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f53001543a0 0x7f530006d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:58062/0 (socket says 172.21.15.5:58062) 2024-03-24T10:53:20.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.419+0000 7f52fffff640 1 -- 172.21.15.5:0/2280118254 learned_addr learned my addr 172.21.15.5:0/2280118254 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:53:20.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.422+0000 7f5304871640 1 -- 172.21.15.5:0/2280118254 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f530015ac00 msgr2=0x7f53000cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:20.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.422+0000 7f5304871640 1 --2- 172.21.15.5:0/2280118254 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f530015ac00 0x7f53000cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:20.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.422+0000 7f5304871640 1 -- 172.21.15.5:0/2280118254 >> v1:172.21.15.57:6789/0 conn(0x7f53001543a0 legacy=0x7f530006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:20.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.422+0000 7f5304871640 1 -- 172.21.15.5:0/2280118254 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530015b070 con 0x7f53000ccaa0 2024-03-24T10:53:20.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.424+0000 7f5304871640 1 --2- 172.21.15.5:0/2280118254 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53000ccaa0 0x7f530006bf50 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009d90 tx=0x7f52f802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c83aef7e099a124c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:20.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.429+0000 7f52feffd640 1 -- 172.21.15.5:0/2280118254 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f802cc50 con 0x7f53000ccaa0 2024-03-24T10:53:20.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.429+0000 7f52feffd640 1 -- 172.21.15.5:0/2280118254 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52f802cdb0 con 0x7f53000ccaa0 2024-03-24T10:53:20.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.429+0000 7f52feffd640 1 -- 172.21.15.5:0/2280118254 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f8034660 con 0x7f53000ccaa0 2024-03-24T10:53:20.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.430+0000 7f53062d4640 1 -- 172.21.15.5:0/2280118254 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53000ccaa0 msgr2=0x7f530006bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:20.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.430+0000 7f53062d4640 1 --2- 172.21.15.5:0/2280118254 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53000ccaa0 0x7f530006bf50 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009d90 tx=0x7f52f802bea0 comp rx=0 tx=0).stop 2024-03-24T10:53:20.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.430+0000 7f53062d4640 1 -- 172.21.15.5:0/2280118254 shutdown_connections 2024-03-24T10:53:20.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.430+0000 7f53062d4640 1 --2- 172.21.15.5:0/2280118254 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f530015ac00 0x7f53000cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:20.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.430+0000 7f53062d4640 1 --2- 172.21.15.5:0/2280118254 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53000ccaa0 0x7f530006bf50 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:20.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.430+0000 7f53062d4640 1 -- 172.21.15.5:0/2280118254 >> 172.21.15.5:0/2280118254 conn(0x7f5300072ef0 msgr2=0x7f5300061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:20.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.430+0000 7f53062d4640 1 -- 172.21.15.5:0/2280118254 shutdown_connections 2024-03-24T10:53:20.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.430+0000 7f53062d4640 1 -- 172.21.15.5:0/2280118254 wait complete. 2024-03-24T10:53:20.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.432+0000 7f53062d4640 1 Processor -- start 2024-03-24T10:53:20.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.432+0000 7f53062d4640 1 -- start start 2024-03-24T10:53:20.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.433+0000 7f53062d4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f53000ccaa0 0x7f53000cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:20.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.433+0000 7f53062d4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53001543a0 0x7f5300146390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:20.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.433+0000 7f53062d4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f530015ac00 0x7f53001468d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:20.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.433+0000 7f53062d4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f53001f3de0 con 0x7f530015ac00 2024-03-24T10:53:20.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.433+0000 7f53062d4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f53000c71a0 con 0x7f53001543a0 2024-03-24T10:53:20.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.433+0000 7f53062d4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f5300151900 con 0x7f53000ccaa0 2024-03-24T10:53:20.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.433+0000 7f5304871640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f530015ac00 0x7f53001468d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:20.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.434+0000 7f52fffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f53000ccaa0 0x7f53000cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:20.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.434+0000 7f52ff7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53001543a0 0x7f5300146390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:20.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.434+0000 7f5304871640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f530015ac00 0x7f53001468d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52370/0 (socket says 172.21.15.5:52370) 2024-03-24T10:53:20.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.434+0000 7f5304871640 1 -- 172.21.15.5:0/4209143850 learned_addr learned my addr 172.21.15.5:0/4209143850 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:20.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.437+0000 7f5304871640 1 -- 172.21.15.5:0/4209143850 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f53000ccaa0 msgr2=0x7f53000cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:20.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.437+0000 7f5304871640 1 --2- 172.21.15.5:0/4209143850 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f53000ccaa0 0x7f53000cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:20.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.437+0000 7f5304871640 1 -- 172.21.15.5:0/4209143850 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53001543a0 msgr2=0x7f5300146390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:20.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.437+0000 7f5304871640 1 --2- 172.21.15.5:0/4209143850 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53001543a0 0x7f5300146390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:20.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.437+0000 7f5304871640 1 -- 172.21.15.5:0/4209143850 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f8009a70 con 0x7f530015ac00 2024-03-24T10:53:20.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.440+0000 7f5304871640 1 --2- 172.21.15.5:0/4209143850 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f530015ac00 0x7f53001468d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009eb0 tx=0x7f52f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:20.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.445+0000 7f52fcff9640 1 -- 172.21.15.5:0/4209143850 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f803e610 con 0x7f530015ac00 2024-03-24T10:53:20.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.445+0000 7f52fcff9640 1 -- 172.21.15.5:0/4209143850 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52f803e770 con 0x7f530015ac00 2024-03-24T10:53:20.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.445+0000 7f52fcff9640 1 -- 172.21.15.5:0/4209143850 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f803eb20 con 0x7f530015ac00 2024-03-24T10:53:20.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.445+0000 7f53062d4640 1 -- 172.21.15.5:0/4209143850 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53001f2ee0 con 0x7f530015ac00 2024-03-24T10:53:20.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.445+0000 7f53062d4640 1 -- 172.21.15.5:0/4209143850 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53001f3160 con 0x7f530015ac00 2024-03-24T10:53:20.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.446+0000 7f53062d4640 1 -- 172.21.15.5:0/4209143850 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53000d05c0 con 0x7f530015ac00 2024-03-24T10:53:20.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.469+0000 7f52fcff9640 1 -- 172.21.15.5:0/4209143850 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f52f803b050 con 0x7f530015ac00 2024-03-24T10:53:20.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.469+0000 7f52fcff9640 1 --2- 172.21.15.5:0/4209143850 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f52ec06b810 0x7f52ec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:20.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.470+0000 7f52fffff640 1 --2- 172.21.15.5:0/4209143850 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f52ec06b810 0x7f52ec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:20.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.470+0000 7f52fffff640 1 --2- 172.21.15.5:0/4209143850 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f52ec06b810 0x7f52ec06dcd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f52f0005880 tx=0x7f52f0009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:20.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.741+0000 7f52fcff9640 1 -- 172.21.15.5:0/4209143850 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f52f807b980 con 0x7f530015ac00 2024-03-24T10:53:20.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.741+0000 7f52fcff9640 1 -- 172.21.15.5:0/4209143850 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f52f8077c20 con 0x7f530015ac00 2024-03-24T10:53:20.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.965+0000 7f53062d4640 1 -- 172.21.15.5:0/4209143850 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53000681d0 con 0x7f530015ac00 2024-03-24T10:53:20.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.978+0000 7f52fcff9640 1 -- 172.21.15.5:0/4209143850 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f52f8039090 con 0x7f530015ac00 2024-03-24T10:53:20.979 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:20.980 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:20.980 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:20.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.981+0000 7f53062d4640 1 -- 172.21.15.5:0/4209143850 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f52ec06b810 msgr2=0x7f52ec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:20.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.981+0000 7f53062d4640 1 --2- 172.21.15.5:0/4209143850 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f52ec06b810 0x7f52ec06dcd0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f52f0005880 tx=0x7f52f0009040 comp rx=0 tx=0).stop 2024-03-24T10:53:20.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.981+0000 7f53062d4640 1 -- 172.21.15.5:0/4209143850 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f530015ac00 msgr2=0x7f53001468d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:20.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.981+0000 7f53062d4640 1 --2- 172.21.15.5:0/4209143850 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f530015ac00 0x7f53001468d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f52f8009eb0 tx=0x7f52f8004a40 comp rx=0 tx=0).stop 2024-03-24T10:53:20.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.981+0000 7f53062d4640 1 -- 172.21.15.5:0/4209143850 shutdown_connections 2024-03-24T10:53:20.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.981+0000 7f53062d4640 1 --2- 172.21.15.5:0/4209143850 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f52ec06b810 0x7f52ec06dcd0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:20.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.981+0000 7f53062d4640 1 --2- 172.21.15.5:0/4209143850 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f530015ac00 0x7f53001468d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:20.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.981+0000 7f53062d4640 1 --2- 172.21.15.5:0/4209143850 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53001543a0 0x7f5300146390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:20.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.981+0000 7f53062d4640 1 --2- 172.21.15.5:0/4209143850 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f53000ccaa0 0x7f53000cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:20.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.981+0000 7f53062d4640 1 -- 172.21.15.5:0/4209143850 >> 172.21.15.5:0/4209143850 conn(0x7f5300072ef0 msgr2=0x7f5300061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:20.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.982+0000 7f53062d4640 1 -- 172.21.15.5:0/4209143850 shutdown_connections 2024-03-24T10:53:20.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:20.982+0000 7f53062d4640 1 -- 172.21.15.5:0/4209143850 wait complete. 2024-03-24T10:53:21.600 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:21.601 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:22.602 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:22.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.703+0000 7fb88892e640 1 Processor -- start 2024-03-24T10:53:22.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.703+0000 7fb88892e640 1 -- start start 2024-03-24T10:53:22.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.704+0000 7fb88892e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb88415aea0 0x7fb8840cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:22.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.704+0000 7fb88892e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb8840cf100 0x7fb8840ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:22.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.704+0000 7fb88892e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb88406cd50 con 0x7fb88415aac0 2024-03-24T10:53:22.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.704+0000 7fb88892e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fb8840ce790 con 0x7fb8840cf100 2024-03-24T10:53:22.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.704+0000 7fb88892e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fb8840cdb10 con 0x7fb88415aea0 2024-03-24T10:53:22.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.705+0000 7fb882d9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb8840cf100 0x7fb8840ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:22.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.705+0000 7fb87bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb88415aea0 0x7fb8840cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:22.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.705+0000 7fb88259c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb88415aac0 0x7fb88406c230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:45998/0 (socket says 172.21.15.5:45998) 2024-03-24T10:53:22.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.705+0000 7fb88259c640 1 -- 172.21.15.5:0/449668629 learned_addr learned my addr 172.21.15.5:0/449668629 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:53:22.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.709+0000 7fb882d9d640 1 -- 172.21.15.5:0/449668629 >> v2:172.21.15.57:3300/0 conn(0x7fb88415aea0 msgr2=0x7fb8840cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:22.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.709+0000 7fb882d9d640 1 --2- 172.21.15.5:0/449668629 >> v2:172.21.15.57:3300/0 conn(0x7fb88415aea0 0x7fb8840cebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:22.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.709+0000 7fb882d9d640 1 -- 172.21.15.5:0/449668629 >> v1:172.21.15.57:6789/0 conn(0x7fb88415aac0 legacy=0x7fb88406c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:22.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.709+0000 7fb882d9d640 1 -- 172.21.15.5:0/449668629 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8840cddf0 con 0x7fb8840cf100 2024-03-24T10:53:22.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.711+0000 7fb882d9d640 1 --2- 172.21.15.5:0/449668629 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8840cf100 0x7fb8840ca5a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb8840c9930 tx=0x7fb87c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6746fd92ae48f607 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:22.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.715+0000 7fb881d9b640 1 -- 172.21.15.5:0/449668629 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c02cd00 con 0x7fb8840cf100 2024-03-24T10:53:22.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.715+0000 7fb881d9b640 1 -- 172.21.15.5:0/449668629 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb87c02ce60 con 0x7fb8840cf100 2024-03-24T10:53:22.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.715+0000 7fb881d9b640 1 -- 172.21.15.5:0/449668629 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c034660 con 0x7fb8840cf100 2024-03-24T10:53:22.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.716+0000 7fb88892e640 1 -- 172.21.15.5:0/449668629 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8840cf100 msgr2=0x7fb8840ca5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:22.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.716+0000 7fb88892e640 1 --2- 172.21.15.5:0/449668629 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8840cf100 0x7fb8840ca5a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb8840c9930 tx=0x7fb87c02bf50 comp rx=0 tx=0).stop 2024-03-24T10:53:22.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.716+0000 7fb88892e640 1 -- 172.21.15.5:0/449668629 shutdown_connections 2024-03-24T10:53:22.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.716+0000 7fb88892e640 1 --2- 172.21.15.5:0/449668629 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb8840cf100 0x7fb8840ca5a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:22.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.716+0000 7fb88892e640 1 --2- 172.21.15.5:0/449668629 >> v2:172.21.15.57:3300/0 conn(0x7fb88415aea0 0x7fb8840cebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:22.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.716+0000 7fb88892e640 1 -- 172.21.15.5:0/449668629 >> 172.21.15.5:0/449668629 conn(0x7fb8840cfb90 msgr2=0x7fb8840cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:22.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.716+0000 7fb88892e640 1 -- 172.21.15.5:0/449668629 shutdown_connections 2024-03-24T10:53:22.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.716+0000 7fb88892e640 1 -- 172.21.15.5:0/449668629 wait complete. 2024-03-24T10:53:22.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.718+0000 7fb88892e640 1 Processor -- start 2024-03-24T10:53:22.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.718+0000 7fb88892e640 1 -- start start 2024-03-24T10:53:22.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.719+0000 7fb88892e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8840cf100 0x7fb884153990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:22.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.719+0000 7fb88892e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb88415aac0 0x7fb8840b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:22.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.719+0000 7fb88892e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb88415aea0 0x7fb8840b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:22.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.719+0000 7fb88892e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fb8841f1090 con 0x7fb88415aea0 2024-03-24T10:53:22.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.719+0000 7fb88892e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb8840c4ef0 con 0x7fb8840cf100 2024-03-24T10:53:22.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.719+0000 7fb88892e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fb8840b1270 con 0x7fb88415aac0 2024-03-24T10:53:22.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.720+0000 7fb882d9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb88415aea0 0x7fb8840b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:22.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.720+0000 7fb88259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8840cf100 0x7fb884153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:22.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.720+0000 7fb87bfff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb88415aac0 0x7fb8840b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:22.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.720+0000 7fb88259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8840cf100 0x7fb884153990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:59400/0 (socket says 172.21.15.5:59400) 2024-03-24T10:53:22.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.720+0000 7fb88259c640 1 -- 172.21.15.5:0/3689525844 learned_addr learned my addr 172.21.15.5:0/3689525844 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:22.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.724+0000 7fb882d9d640 1 -- 172.21.15.5:0/3689525844 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb88415aac0 msgr2=0x7fb8840b47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:22.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.724+0000 7fb882d9d640 1 --2- 172.21.15.5:0/3689525844 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb88415aac0 0x7fb8840b47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:22.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.724+0000 7fb882d9d640 1 -- 172.21.15.5:0/3689525844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8840cf100 msgr2=0x7fb884153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:22.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.724+0000 7fb882d9d640 1 --2- 172.21.15.5:0/3689525844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8840cf100 0x7fb884153990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:22.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.724+0000 7fb882d9d640 1 -- 172.21.15.5:0/3689525844 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87c009a70 con 0x7fb88415aea0 2024-03-24T10:53:22.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.727+0000 7fb882d9d640 1 --2- 172.21.15.5:0/3689525844 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb88415aea0 0x7fb8840b4d00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb88408cca0 tx=0x7fb87c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:22.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.731+0000 7fb87affd640 1 -- 172.21.15.5:0/3689525844 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c03d600 con 0x7fb88415aea0 2024-03-24T10:53:22.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.731+0000 7fb87affd640 1 -- 172.21.15.5:0/3689525844 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb87c03d780 con 0x7fb88415aea0 2024-03-24T10:53:22.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.731+0000 7fb88892e640 1 -- 172.21.15.5:0/3689525844 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8840c5110 con 0x7fb88415aea0 2024-03-24T10:53:22.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.731+0000 7fb87affd640 1 -- 172.21.15.5:0/3689525844 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c03db30 con 0x7fb88415aea0 2024-03-24T10:53:22.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.731+0000 7fb88892e640 1 -- 172.21.15.5:0/3689525844 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8841f1270 con 0x7fb88415aea0 2024-03-24T10:53:22.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.733+0000 7fb88892e640 1 -- 172.21.15.5:0/3689525844 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8840c5630 con 0x7fb88415aea0 2024-03-24T10:53:22.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.753+0000 7fb87affd640 1 -- 172.21.15.5:0/3689525844 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb87c03b040 con 0x7fb88415aea0 2024-03-24T10:53:22.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.753+0000 7fb87affd640 1 --2- 172.21.15.5:0/3689525844 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb86406b840 0x7fb86406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:22.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.753+0000 7fb88259c640 1 --2- 172.21.15.5:0/3689525844 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb86406b840 0x7fb86406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:22.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.754+0000 7fb88259c640 1 --2- 172.21.15.5:0/3689525844 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb86406b840 0x7fb86406dd00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb874004e40 tx=0x7fb874007f10 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:22.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:22.757+0000 7fb87affd640 1 -- 172.21.15.5:0/3689525844 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fb87c07aba0 con 0x7fb88415aea0 2024-03-24T10:53:23.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:23.021+0000 7fb87affd640 1 -- 172.21.15.5:0/3689525844 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb87c076e40 con 0x7fb88415aea0 2024-03-24T10:53:23.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:23.240+0000 7fb88892e640 1 -- 172.21.15.5:0/3689525844 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8840681d0 con 0x7fb88415aea0 2024-03-24T10:53:23.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:23.254+0000 7fb87affd640 1 -- 172.21.15.5:0/3689525844 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fb87c039030 con 0x7fb88415aea0 2024-03-24T10:53:23.254 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:23.255 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:23.255 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:23.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:23.257+0000 7fb88892e640 1 -- 172.21.15.5:0/3689525844 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb86406b840 msgr2=0x7fb86406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:23.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:23.257+0000 7fb88892e640 1 --2- 172.21.15.5:0/3689525844 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb86406b840 0x7fb86406dd00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb874004e40 tx=0x7fb874007f10 comp rx=0 tx=0).stop 2024-03-24T10:53:23.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:23.257+0000 7fb88892e640 1 -- 172.21.15.5:0/3689525844 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb88415aea0 msgr2=0x7fb8840b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:23.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:23.257+0000 7fb88892e640 1 --2- 172.21.15.5:0/3689525844 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb88415aea0 0x7fb8840b4d00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb88408cca0 tx=0x7fb87c004a40 comp rx=0 tx=0).stop 2024-03-24T10:53:23.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:23.257+0000 7fb88892e640 1 -- 172.21.15.5:0/3689525844 shutdown_connections 2024-03-24T10:53:23.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:23.257+0000 7fb88892e640 1 --2- 172.21.15.5:0/3689525844 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb86406b840 0x7fb86406dd00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:23.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:23.257+0000 7fb88892e640 1 --2- 172.21.15.5:0/3689525844 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb88415aea0 0x7fb8840b4d00 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:23.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:23.257+0000 7fb88892e640 1 --2- 172.21.15.5:0/3689525844 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb88415aac0 0x7fb8840b47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:23.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:23.257+0000 7fb88892e640 1 --2- 172.21.15.5:0/3689525844 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8840cf100 0x7fb884153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:23.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:23.257+0000 7fb88892e640 1 -- 172.21.15.5:0/3689525844 >> 172.21.15.5:0/3689525844 conn(0x7fb8840cfb90 msgr2=0x7fb8840cca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:23.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:23.257+0000 7fb88892e640 1 -- 172.21.15.5:0/3689525844 shutdown_connections 2024-03-24T10:53:23.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:23.257+0000 7fb88892e640 1 -- 172.21.15.5:0/3689525844 wait complete. 2024-03-24T10:53:23.872 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:23.872 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:24.873 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:24.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.975+0000 7fb960b3e640 1 Processor -- start 2024-03-24T10:53:24.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.975+0000 7fb960b3e640 1 -- start start 2024-03-24T10:53:24.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.976+0000 7fb960b3e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb95c1543a0 0x7fb95c15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:24.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.976+0000 7fb960b3e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb95c06bf50 0x7fb95c06c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:24.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.976+0000 7fb960b3e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb95c0cb2f0 con 0x7fb95c06d4f0 2024-03-24T10:53:24.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.976+0000 7fb960b3e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fb95c0c6750 con 0x7fb95c06bf50 2024-03-24T10:53:24.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.976+0000 7fb960b3e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fb95c0cc380 con 0x7fb95c1543a0 2024-03-24T10:53:24.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.976+0000 7fb95ad9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb95c06bf50 0x7fb95c06c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:24.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.977+0000 7fb95ad9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb95c06bf50 0x7fb95c06c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:53704/0 (socket says 172.21.15.5:53704) 2024-03-24T10:53:24.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.977+0000 7fb95ad9d640 1 -- 172.21.15.5:0/1785368223 learned_addr learned my addr 172.21.15.5:0/1785368223 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:24.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.977+0000 7fb95a59c640 1 --2- 172.21.15.5:0/1785368223 >> v2:172.21.15.57:3300/0 conn(0x7fb95c1543a0 0x7fb95c15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:24.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.979+0000 7fb95ad9d640 1 -- 172.21.15.5:0/1785368223 >> v2:172.21.15.57:3300/0 conn(0x7fb95c1543a0 msgr2=0x7fb95c15ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:24.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.979+0000 7fb95ad9d640 1 --2- 172.21.15.5:0/1785368223 >> v2:172.21.15.57:3300/0 conn(0x7fb95c1543a0 0x7fb95c15ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:24.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.979+0000 7fb95ad9d640 1 -- 172.21.15.5:0/1785368223 >> v1:172.21.15.57:6789/0 conn(0x7fb95c06d4f0 legacy=0x7fb95c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:24.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.979+0000 7fb95ad9d640 1 -- 172.21.15.5:0/1785368223 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb95c0ca7e0 con 0x7fb95c06bf50 2024-03-24T10:53:24.981 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.981+0000 7fb95ad9d640 1 --2- 172.21.15.5:0/1785368223 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb95c06bf50 0x7fb95c06c330 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0cd600 tx=0x7fb95402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a506ef4bc07784ca server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:24.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.986+0000 7fb95959a640 1 -- 172.21.15.5:0/1785368223 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb95402cbb0 con 0x7fb95c06bf50 2024-03-24T10:53:24.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.986+0000 7fb95959a640 1 -- 172.21.15.5:0/1785368223 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb95402cd10 con 0x7fb95c06bf50 2024-03-24T10:53:24.986 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.986+0000 7fb95959a640 1 -- 172.21.15.5:0/1785368223 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb954034610 con 0x7fb95c06bf50 2024-03-24T10:53:24.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.987+0000 7fb960b3e640 1 -- 172.21.15.5:0/1785368223 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb95c06bf50 msgr2=0x7fb95c06c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:24.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.987+0000 7fb960b3e640 1 --2- 172.21.15.5:0/1785368223 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb95c06bf50 0x7fb95c06c330 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0cd600 tx=0x7fb95402c010 comp rx=0 tx=0).stop 2024-03-24T10:53:24.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.987+0000 7fb960b3e640 1 -- 172.21.15.5:0/1785368223 shutdown_connections 2024-03-24T10:53:24.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.987+0000 7fb960b3e640 1 --2- 172.21.15.5:0/1785368223 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb95c06bf50 0x7fb95c06c330 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:24.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.987+0000 7fb960b3e640 1 --2- 172.21.15.5:0/1785368223 >> v2:172.21.15.57:3300/0 conn(0x7fb95c1543a0 0x7fb95c15ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:24.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.987+0000 7fb960b3e640 1 -- 172.21.15.5:0/1785368223 >> 172.21.15.5:0/1785368223 conn(0x7fb95c072ef0 msgr2=0x7fb95c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:24.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.987+0000 7fb960b3e640 1 -- 172.21.15.5:0/1785368223 shutdown_connections 2024-03-24T10:53:24.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.987+0000 7fb960b3e640 1 -- 172.21.15.5:0/1785368223 wait complete. 2024-03-24T10:53:24.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.989+0000 7fb960b3e640 1 Processor -- start 2024-03-24T10:53:24.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.989+0000 7fb960b3e640 1 -- start start 2024-03-24T10:53:24.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.990+0000 7fb960b3e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb95c06bf50 0x7fb95c0cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:24.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.990+0000 7fb960b3e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb95c06d4f0 0x7fb95c0b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:24.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.990+0000 7fb960b3e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb95c1543a0 0x7fb95c14ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:24.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.990+0000 7fb960b3e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c1f2ab0 con 0x7fb95c06d4f0 2024-03-24T10:53:24.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.990+0000 7fb960b3e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c1f2ce0 con 0x7fb95c06bf50 2024-03-24T10:53:24.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.990+0000 7fb960b3e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fb95c1f2f80 con 0x7fb95c1543a0 2024-03-24T10:53:24.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.990+0000 7fb959d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb95c06d4f0 0x7fb95c0b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:24.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.991+0000 7fb95a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb95c06bf50 0x7fb95c0cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:24.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.991+0000 7fb95ad9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb95c1543a0 0x7fb95c14ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:24.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.991+0000 7fb959d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb95c06d4f0 0x7fb95c0b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:53708/0 (socket says 172.21.15.5:53708) 2024-03-24T10:53:24.991 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.991+0000 7fb959d9b640 1 -- 172.21.15.5:0/2335284708 learned_addr learned my addr 172.21.15.5:0/2335284708 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:24.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.994+0000 7fb959d9b640 1 -- 172.21.15.5:0/2335284708 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb95c1543a0 msgr2=0x7fb95c14ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:24.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.994+0000 7fb959d9b640 1 --2- 172.21.15.5:0/2335284708 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb95c1543a0 0x7fb95c14ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:24.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.994+0000 7fb959d9b640 1 -- 172.21.15.5:0/2335284708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb95c06bf50 msgr2=0x7fb95c0cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:24.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.994+0000 7fb959d9b640 1 --2- 172.21.15.5:0/2335284708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb95c06bf50 0x7fb95c0cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:24.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.994+0000 7fb959d9b640 1 -- 172.21.15.5:0/2335284708 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb954009b30 con 0x7fb95c06d4f0 2024-03-24T10:53:24.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:24.997+0000 7fb959d9b640 1 --2- 172.21.15.5:0/2335284708 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb95c06d4f0 0x7fb95c0b41c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb95000c400 tx=0x7fb95000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:25.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.002+0000 7fb94affd640 1 -- 172.21.15.5:0/2335284708 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9500052d0 con 0x7fb95c06d4f0 2024-03-24T10:53:25.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.002+0000 7fb94affd640 1 -- 172.21.15.5:0/2335284708 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb950005450 con 0x7fb95c06d4f0 2024-03-24T10:53:25.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.002+0000 7fb960b3e640 1 -- 172.21.15.5:0/2335284708 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb95c1f3260 con 0x7fb95c06d4f0 2024-03-24T10:53:25.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.002+0000 7fb94affd640 1 -- 172.21.15.5:0/2335284708 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb950007260 con 0x7fb95c06d4f0 2024-03-24T10:53:25.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.002+0000 7fb960b3e640 1 -- 172.21.15.5:0/2335284708 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb95c1f3820 con 0x7fb95c06d4f0 2024-03-24T10:53:25.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.003+0000 7fb960b3e640 1 -- 172.21.15.5:0/2335284708 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb95c0681d0 con 0x7fb95c06d4f0 2024-03-24T10:53:25.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.030+0000 7fb94affd640 1 -- 172.21.15.5:0/2335284708 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb9500073c0 con 0x7fb95c06d4f0 2024-03-24T10:53:25.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.030+0000 7fb94affd640 1 --2- 172.21.15.5:0/2335284708 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb93c06b810 0x7fb93c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:25.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.030+0000 7fb95a59c640 1 --2- 172.21.15.5:0/2335284708 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb93c06b810 0x7fb93c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:25.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.031+0000 7fb95a59c640 1 --2- 172.21.15.5:0/2335284708 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb93c06b810 0x7fb93c06dcd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb94c009aa0 tx=0x7fb94c009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:25.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.304+0000 7fb94affd640 1 -- 172.21.15.5:0/2335284708 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fb950007670 con 0x7fb95c06d4f0 2024-03-24T10:53:25.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.304+0000 7fb94affd640 1 -- 172.21.15.5:0/2335284708 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb950054560 con 0x7fb95c06d4f0 2024-03-24T10:53:25.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.526+0000 7fb960b3e640 1 -- 172.21.15.5:0/2335284708 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb95c061ed0 con 0x7fb95c06d4f0 2024-03-24T10:53:25.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.541+0000 7fb94affd640 1 -- 172.21.15.5:0/2335284708 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fb950020020 con 0x7fb95c06d4f0 2024-03-24T10:53:25.541 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:25.541 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:25.541 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:25.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.544+0000 7fb960b3e640 1 -- 172.21.15.5:0/2335284708 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb93c06b810 msgr2=0x7fb93c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:25.544 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.544+0000 7fb960b3e640 1 --2- 172.21.15.5:0/2335284708 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb93c06b810 0x7fb93c06dcd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb94c009aa0 tx=0x7fb94c009190 comp rx=0 tx=0).stop 2024-03-24T10:53:25.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.544+0000 7fb960b3e640 1 -- 172.21.15.5:0/2335284708 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb95c06d4f0 msgr2=0x7fb95c0b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:25.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.544+0000 7fb960b3e640 1 --2- 172.21.15.5:0/2335284708 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb95c06d4f0 0x7fb95c0b41c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb95000c400 tx=0x7fb95000c8c0 comp rx=0 tx=0).stop 2024-03-24T10:53:25.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.544+0000 7fb960b3e640 1 -- 172.21.15.5:0/2335284708 shutdown_connections 2024-03-24T10:53:25.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.544+0000 7fb960b3e640 1 --2- 172.21.15.5:0/2335284708 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb93c06b810 0x7fb93c06dcd0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:25.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.544+0000 7fb960b3e640 1 --2- 172.21.15.5:0/2335284708 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb95c06d4f0 0x7fb95c0b41c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:25.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.544+0000 7fb960b3e640 1 --2- 172.21.15.5:0/2335284708 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb95c1543a0 0x7fb95c14ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:25.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.544+0000 7fb960b3e640 1 --2- 172.21.15.5:0/2335284708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb95c06bf50 0x7fb95c0cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:25.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.544+0000 7fb960b3e640 1 -- 172.21.15.5:0/2335284708 >> 172.21.15.5:0/2335284708 conn(0x7fb95c072ef0 msgr2=0x7fb95c151bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:25.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.544+0000 7fb960b3e640 1 -- 172.21.15.5:0/2335284708 shutdown_connections 2024-03-24T10:53:25.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:25.544+0000 7fb960b3e640 1 -- 172.21.15.5:0/2335284708 wait complete. 2024-03-24T10:53:26.159 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:26.159 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:27.161 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:27.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.264+0000 7f9c57237640 1 Processor -- start 2024-03-24T10:53:27.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.264+0000 7f9c57237640 1 -- start start 2024-03-24T10:53:27.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.265+0000 7f9c57237640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9c500cb270 0x7f9c500cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:27.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.265+0000 7f9c57237640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9c500cd960 0x7f9c500cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:27.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.265+0000 7f9c57237640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c500ccba0 con 0x7f9c500cae90 2024-03-24T10:53:27.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.265+0000 7f9c57237640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f9c500cc9b0 con 0x7f9c500cd960 2024-03-24T10:53:27.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.265+0000 7f9c57237640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f9c500cce40 con 0x7f9c500cb270 2024-03-24T10:53:27.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.265+0000 7f9c54fd3640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f9c500cae90 0x7f9c500cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:38776/0 (socket says 172.21.15.5:38776) 2024-03-24T10:53:27.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.265+0000 7f9c54fd3640 1 -- 172.21.15.5:0/404473838 learned_addr learned my addr 172.21.15.5:0/404473838 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:53:27.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.265+0000 7f9c4ffff640 1 --2- 172.21.15.5:0/404473838 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9c500cb270 0x7f9c500cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:27.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.265+0000 7f9c557d4640 1 --2- 172.21.15.5:0/404473838 >> v2:172.21.15.57:3300/0 conn(0x7f9c500cd960 0x7f9c500cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:27.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.268+0000 7f9c4f7fe640 1 -- 172.21.15.5:0/404473838 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2556139843 0 0) 0x7f9c500ccba0 con 0x7f9c500cae90 2024-03-24T10:53:27.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.268+0000 7f9c4ffff640 1 -- 172.21.15.5:0/404473838 >> v2:172.21.15.57:3300/0 conn(0x7f9c500cd960 msgr2=0x7f9c500cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:27.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.268+0000 7f9c4ffff640 1 --2- 172.21.15.5:0/404473838 >> v2:172.21.15.57:3300/0 conn(0x7f9c500cd960 0x7f9c500cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:27.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.268+0000 7f9c4ffff640 1 -- 172.21.15.5:0/404473838 >> v1:172.21.15.5:6789/0 conn(0x7f9c500cae90 legacy=0x7f9c500cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:27.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.268+0000 7f9c4ffff640 1 -- 172.21.15.5:0/404473838 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c5015a590 con 0x7f9c500cb270 2024-03-24T10:53:27.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.270+0000 7f9c4ffff640 1 --2- 172.21.15.5:0/404473838 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9c500cb270 0x7f9c500cf530 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c0097a0 tx=0x7f9c3c02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0531a83de99a627 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:27.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.274+0000 7f9c4f7fe640 1 -- 172.21.15.5:0/404473838 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c030610 con 0x7f9c500cb270 2024-03-24T10:53:27.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.274+0000 7f9c4f7fe640 1 -- 172.21.15.5:0/404473838 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c3c030770 con 0x7f9c500cb270 2024-03-24T10:53:27.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.274+0000 7f9c4f7fe640 1 -- 172.21.15.5:0/404473838 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c030b00 con 0x7f9c500cb270 2024-03-24T10:53:27.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.275+0000 7f9c57237640 1 -- 172.21.15.5:0/404473838 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9c500cb270 msgr2=0x7f9c500cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:27.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.275+0000 7f9c57237640 1 --2- 172.21.15.5:0/404473838 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9c500cb270 0x7f9c500cf530 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c0097a0 tx=0x7f9c3c02b960 comp rx=0 tx=0).stop 2024-03-24T10:53:27.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.275+0000 7f9c57237640 1 -- 172.21.15.5:0/404473838 shutdown_connections 2024-03-24T10:53:27.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.275+0000 7f9c57237640 1 --2- 172.21.15.5:0/404473838 >> v2:172.21.15.57:3300/0 conn(0x7f9c500cd960 0x7f9c500cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:27.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.275+0000 7f9c57237640 1 --2- 172.21.15.5:0/404473838 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9c500cb270 0x7f9c500cf530 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:27.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.275+0000 7f9c57237640 1 -- 172.21.15.5:0/404473838 >> 172.21.15.5:0/404473838 conn(0x7f9c500cfb80 msgr2=0x7f9c500cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:27.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.275+0000 7f9c57237640 1 -- 172.21.15.5:0/404473838 shutdown_connections 2024-03-24T10:53:27.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.275+0000 7f9c57237640 1 -- 172.21.15.5:0/404473838 wait complete. 2024-03-24T10:53:27.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.276+0000 7f9c57237640 1 Processor -- start 2024-03-24T10:53:27.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.276+0000 7f9c57237640 1 -- start start 2024-03-24T10:53:27.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.277+0000 7f9c57237640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c500cae90 0x7f9c50157ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:27.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.277+0000 7f9c57237640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c500cb270 0x7f9c500b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:27.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.277+0000 7f9c57237640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9c500cd960 0x7f9c500b4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:27.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.277+0000 7f9c57237640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f9c50155ae0 con 0x7f9c500cb270 2024-03-24T10:53:27.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.277+0000 7f9c57237640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9c50156500 con 0x7f9c500cae90 2024-03-24T10:53:27.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.277+0000 7f9c57237640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f9c501567c0 con 0x7f9c500cd960 2024-03-24T10:53:27.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.278+0000 7f9c4ffff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c500cb270 0x7f9c500b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:27.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.278+0000 7f9c557d4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9c500cd960 0x7f9c500b4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:27.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.278+0000 7f9c54fd3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c500cae90 0x7f9c50157ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:27.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.278+0000 7f9c4ffff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c500cb270 0x7f9c500b2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:53710/0 (socket says 172.21.15.5:53710) 2024-03-24T10:53:27.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.278+0000 7f9c4ffff640 1 -- 172.21.15.5:0/4135723551 learned_addr learned my addr 172.21.15.5:0/4135723551 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:27.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.282+0000 7f9c557d4640 1 -- 172.21.15.5:0/4135723551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c500cae90 msgr2=0x7f9c50157ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:27.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.282+0000 7f9c557d4640 1 --2- 172.21.15.5:0/4135723551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c500cae90 0x7f9c50157ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:27.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.282+0000 7f9c557d4640 1 -- 172.21.15.5:0/4135723551 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c500cb270 msgr2=0x7f9c500b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:27.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.282+0000 7f9c557d4640 1 --2- 172.21.15.5:0/4135723551 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c500cb270 0x7f9c500b2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:27.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.282+0000 7f9c557d4640 1 -- 172.21.15.5:0/4135723551 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3c009400 con 0x7f9c500cd960 2024-03-24T10:53:27.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.285+0000 7f9c557d4640 1 --2- 172.21.15.5:0/4135723551 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9c500cd960 0x7f9c500b4790 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9c4800eba0 tx=0x7f9c4800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:27.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.289+0000 7f9c4d7fa640 1 -- 172.21.15.5:0/4135723551 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c4800cd20 con 0x7f9c500cd960 2024-03-24T10:53:27.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.289+0000 7f9c4d7fa640 1 -- 172.21.15.5:0/4135723551 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c4800f040 con 0x7f9c500cd960 2024-03-24T10:53:27.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.289+0000 7f9c4d7fa640 1 -- 172.21.15.5:0/4135723551 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c480146c0 con 0x7f9c500cd960 2024-03-24T10:53:27.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.289+0000 7f9c57237640 1 -- 172.21.15.5:0/4135723551 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c50156aa0 con 0x7f9c500cd960 2024-03-24T10:53:27.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.289+0000 7f9c57237640 1 -- 172.21.15.5:0/4135723551 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c50155770 con 0x7f9c500cd960 2024-03-24T10:53:27.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.290+0000 7f9c57237640 1 -- 172.21.15.5:0/4135723551 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c500681d0 con 0x7f9c500cd960 2024-03-24T10:53:27.380 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.380+0000 7f9c4d7fa640 1 -- 172.21.15.5:0/4135723551 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9c48014820 con 0x7f9c500cd960 2024-03-24T10:53:27.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.381+0000 7f9c4d7fa640 1 --2- 172.21.15.5:0/4135723551 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9c3406b860 0x7f9c3406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:27.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.381+0000 7f9c54fd3640 1 --2- 172.21.15.5:0/4135723551 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9c3406b860 0x7f9c3406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:27.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.381+0000 7f9c54fd3640 1 --2- 172.21.15.5:0/4135723551 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9c3406b860 0x7f9c3406dd20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9c44000f60 tx=0x7f9c44007d50 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:27.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.560+0000 7f9c4d7fa640 1 -- 172.21.15.5:0/4135723551 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f9c48059ba0 con 0x7f9c500cd960 2024-03-24T10:53:27.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.572+0000 7f9c4d7fa640 1 -- 172.21.15.5:0/4135723551 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9c48055e40 con 0x7f9c500cd960 2024-03-24T10:53:27.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.800+0000 7f9c57237640 1 -- 172.21.15.5:0/4135723551 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c50155c20 con 0x7f9c500cd960 2024-03-24T10:53:27.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.886+0000 7f9c4d7fa640 1 -- 172.21.15.5:0/4135723551 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f9c48018090 con 0x7f9c500cd960 2024-03-24T10:53:27.886 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:27.887 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:27.887 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:27.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.888+0000 7f9c57237640 1 -- 172.21.15.5:0/4135723551 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9c3406b860 msgr2=0x7f9c3406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:27.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.888+0000 7f9c57237640 1 --2- 172.21.15.5:0/4135723551 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9c3406b860 0x7f9c3406dd20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9c44000f60 tx=0x7f9c44007d50 comp rx=0 tx=0).stop 2024-03-24T10:53:27.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.888+0000 7f9c57237640 1 -- 172.21.15.5:0/4135723551 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9c500cd960 msgr2=0x7f9c500b4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:27.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.888+0000 7f9c57237640 1 --2- 172.21.15.5:0/4135723551 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9c500cd960 0x7f9c500b4790 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9c4800eba0 tx=0x7f9c4800c490 comp rx=0 tx=0).stop 2024-03-24T10:53:27.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.888+0000 7f9c57237640 1 -- 172.21.15.5:0/4135723551 shutdown_connections 2024-03-24T10:53:27.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.888+0000 7f9c57237640 1 --2- 172.21.15.5:0/4135723551 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9c3406b860 0x7f9c3406dd20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:27.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.888+0000 7f9c57237640 1 --2- 172.21.15.5:0/4135723551 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9c500cb270 0x7f9c500b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:27.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.888+0000 7f9c57237640 1 --2- 172.21.15.5:0/4135723551 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9c500cd960 0x7f9c500b4790 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:27.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.888+0000 7f9c57237640 1 --2- 172.21.15.5:0/4135723551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9c500cae90 0x7f9c50157ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:27.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.888+0000 7f9c57237640 1 -- 172.21.15.5:0/4135723551 >> 172.21.15.5:0/4135723551 conn(0x7f9c500cfb80 msgr2=0x7f9c500c73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:27.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.888+0000 7f9c57237640 1 -- 172.21.15.5:0/4135723551 shutdown_connections 2024-03-24T10:53:27.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:27.888+0000 7f9c57237640 1 -- 172.21.15.5:0/4135723551 wait complete. 2024-03-24T10:53:28.507 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:28.507 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:29.509 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:29.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.611+0000 7f0630a23640 1 Processor -- start 2024-03-24T10:53:29.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.611+0000 7f0630a23640 1 -- start start 2024-03-24T10:53:29.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.612+0000 7f0630a23640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f062c15ac00 0x7f062c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:29.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.612+0000 7f0630a23640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f062c0ccaa0 0x7f062c06bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:29.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.612+0000 7f0630a23640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f062c0cb2f0 con 0x7f062c1543a0 2024-03-24T10:53:29.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.612+0000 7f0630a23640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f062c0c6750 con 0x7f062c0ccaa0 2024-03-24T10:53:29.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.612+0000 7f0630a23640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f062c0cc380 con 0x7f062c15ac00 2024-03-24T10:53:29.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.612+0000 7f062ad9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f062c0ccaa0 0x7f062c06bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:29.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.612+0000 7f0629d9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f062c15ac00 0x7f062c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:29.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.613+0000 7f0629d9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f062c15ac00 0x7f062c0cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:56142/0 (socket says 172.21.15.5:56142) 2024-03-24T10:53:29.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.613+0000 7f0629d9b640 1 -- 172.21.15.5:0/3592893115 learned_addr learned my addr 172.21.15.5:0/3592893115 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:29.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.616+0000 7f062ad9d640 1 -- 172.21.15.5:0/3592893115 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f062c15ac00 msgr2=0x7f062c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:29.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.616+0000 7f062ad9d640 1 --2- 172.21.15.5:0/3592893115 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f062c15ac00 0x7f062c0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:29.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.616+0000 7f062ad9d640 1 -- 172.21.15.5:0/3592893115 >> v1:172.21.15.5:6789/0 conn(0x7f062c1543a0 legacy=0x7f062c06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:29.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.616+0000 7f062ad9d640 1 -- 172.21.15.5:0/3592893115 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f062c15b070 con 0x7f062c0ccaa0 2024-03-24T10:53:29.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.621+0000 7f062ad9d640 1 --2- 172.21.15.5:0/3592893115 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f062c0ccaa0 0x7f062c06bf50 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0624009d90 tx=0x7f062402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96dc9417fb8d7d3a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:29.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.626+0000 7f062959a640 1 -- 172.21.15.5:0/3592893115 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062402cc50 con 0x7f062c0ccaa0 2024-03-24T10:53:29.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.626+0000 7f062959a640 1 -- 172.21.15.5:0/3592893115 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f062402cdb0 con 0x7f062c0ccaa0 2024-03-24T10:53:29.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.626+0000 7f062959a640 1 -- 172.21.15.5:0/3592893115 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0624034660 con 0x7f062c0ccaa0 2024-03-24T10:53:29.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.627+0000 7f0630a23640 1 -- 172.21.15.5:0/3592893115 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f062c0ccaa0 msgr2=0x7f062c06bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:29.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.627+0000 7f0630a23640 1 --2- 172.21.15.5:0/3592893115 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f062c0ccaa0 0x7f062c06bf50 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0624009d90 tx=0x7f062402bea0 comp rx=0 tx=0).stop 2024-03-24T10:53:29.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.627+0000 7f0630a23640 1 -- 172.21.15.5:0/3592893115 shutdown_connections 2024-03-24T10:53:29.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.627+0000 7f0630a23640 1 --2- 172.21.15.5:0/3592893115 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f062c0ccaa0 0x7f062c06bf50 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:29.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.627+0000 7f0630a23640 1 --2- 172.21.15.5:0/3592893115 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f062c15ac00 0x7f062c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:29.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.627+0000 7f0630a23640 1 -- 172.21.15.5:0/3592893115 >> 172.21.15.5:0/3592893115 conn(0x7f062c072ef0 msgr2=0x7f062c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:29.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.627+0000 7f0630a23640 1 -- 172.21.15.5:0/3592893115 shutdown_connections 2024-03-24T10:53:29.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.627+0000 7f0630a23640 1 -- 172.21.15.5:0/3592893115 wait complete. 2024-03-24T10:53:29.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.629+0000 7f0630a23640 1 Processor -- start 2024-03-24T10:53:29.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.629+0000 7f0630a23640 1 -- start start 2024-03-24T10:53:29.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.630+0000 7f0630a23640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f062c0ccaa0 0x7f062c14f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:29.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.630+0000 7f0630a23640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f062c1543a0 0x7f062c14a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:29.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.630+0000 7f0630a23640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f062c15ac00 0x7f062c14a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:29.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.630+0000 7f0630a23640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f062c0cf3e0 con 0x7f062c15ac00 2024-03-24T10:53:29.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.630+0000 7f0630a23640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f062c1f3af0 con 0x7f062c1543a0 2024-03-24T10:53:29.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.630+0000 7f0630a23640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f062c0cf890 con 0x7f062c0ccaa0 2024-03-24T10:53:29.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.630+0000 7f062ad9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f062c15ac00 0x7f062c14a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:29.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.630+0000 7f062ad9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f062c15ac00 0x7f062c14a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:53722/0 (socket says 172.21.15.5:53722) 2024-03-24T10:53:29.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.630+0000 7f062ad9d640 1 -- 172.21.15.5:0/1609475174 learned_addr learned my addr 172.21.15.5:0/1609475174 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:29.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.630+0000 7f0629d9b640 1 --2- 172.21.15.5:0/1609475174 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f062c1543a0 0x7f062c14a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:29.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.631+0000 7f062a59c640 1 --2- 172.21.15.5:0/1609475174 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f062c0ccaa0 0x7f062c14f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:29.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.634+0000 7f062ad9d640 1 -- 172.21.15.5:0/1609475174 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f062c0ccaa0 msgr2=0x7f062c14f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:29.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.634+0000 7f062ad9d640 1 --2- 172.21.15.5:0/1609475174 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f062c0ccaa0 0x7f062c14f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:29.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.634+0000 7f062ad9d640 1 -- 172.21.15.5:0/1609475174 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f062c1543a0 msgr2=0x7f062c14a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:29.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.634+0000 7f062ad9d640 1 --2- 172.21.15.5:0/1609475174 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f062c1543a0 0x7f062c14a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:29.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.634+0000 7f062ad9d640 1 -- 172.21.15.5:0/1609475174 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0624009a70 con 0x7f062c15ac00 2024-03-24T10:53:29.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.636+0000 7f062ad9d640 1 --2- 172.21.15.5:0/1609475174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f062c15ac00 0x7f062c14a9b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f062402c4b0 tx=0x7f0624004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:29.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.638+0000 7f061affd640 1 -- 172.21.15.5:0/1609475174 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062403e600 con 0x7f062c15ac00 2024-03-24T10:53:29.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.638+0000 7f0630a23640 1 -- 172.21.15.5:0/1609475174 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f062c0cf5f0 con 0x7f062c15ac00 2024-03-24T10:53:29.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.638+0000 7f0630a23640 1 -- 172.21.15.5:0/1609475174 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f062c0cecd0 con 0x7f062c15ac00 2024-03-24T10:53:29.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.639+0000 7f0630a23640 1 -- 172.21.15.5:0/1609475174 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f062c14f2c0 con 0x7f062c15ac00 2024-03-24T10:53:29.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.641+0000 7f061affd640 1 -- 172.21.15.5:0/1609475174 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0624009450 con 0x7f062c15ac00 2024-03-24T10:53:29.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.641+0000 7f061affd640 1 -- 172.21.15.5:0/1609475174 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062403e960 con 0x7f062c15ac00 2024-03-24T10:53:29.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.954+0000 7f061affd640 1 -- 172.21.15.5:0/1609475174 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f062403eac0 con 0x7f062c15ac00 2024-03-24T10:53:29.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.954+0000 7f061affd640 1 --2- 172.21.15.5:0/1609475174 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f060c06b840 0x7f060c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:29.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.954+0000 7f062a59c640 1 --2- 172.21.15.5:0/1609475174 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f060c06b840 0x7f060c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:29.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.954+0000 7f062a59c640 1 --2- 172.21.15.5:0/1609475174 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f060c06b840 0x7f060c06dd00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f061c001780 tx=0x7f061c009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:29.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.978+0000 7f061affd640 1 -- 172.21.15.5:0/1609475174 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f062407ba70 con 0x7f062c15ac00 2024-03-24T10:53:29.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:29.978+0000 7f061affd640 1 -- 172.21.15.5:0/1609475174 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0624077d10 con 0x7f062c15ac00 2024-03-24T10:53:30.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:30.206+0000 7f0630a23640 1 -- 172.21.15.5:0/1609475174 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f062c0681d0 con 0x7f062c15ac00 2024-03-24T10:53:30.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:30.226+0000 7f061affd640 1 -- 172.21.15.5:0/1609475174 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f062404c090 con 0x7f062c15ac00 2024-03-24T10:53:30.227 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:30.227 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:30.227 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:30.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:30.229+0000 7f0630a23640 1 -- 172.21.15.5:0/1609475174 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f060c06b840 msgr2=0x7f060c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:30.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:30.229+0000 7f0630a23640 1 --2- 172.21.15.5:0/1609475174 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f060c06b840 0x7f060c06dd00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f061c001780 tx=0x7f061c009040 comp rx=0 tx=0).stop 2024-03-24T10:53:30.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:30.229+0000 7f0630a23640 1 -- 172.21.15.5:0/1609475174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f062c15ac00 msgr2=0x7f062c14a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:30.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:30.229+0000 7f0630a23640 1 --2- 172.21.15.5:0/1609475174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f062c15ac00 0x7f062c14a9b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f062402c4b0 tx=0x7f0624004a40 comp rx=0 tx=0).stop 2024-03-24T10:53:30.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:30.229+0000 7f0630a23640 1 -- 172.21.15.5:0/1609475174 shutdown_connections 2024-03-24T10:53:30.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:30.229+0000 7f0630a23640 1 --2- 172.21.15.5:0/1609475174 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f060c06b840 0x7f060c06dd00 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:30.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:30.229+0000 7f0630a23640 1 --2- 172.21.15.5:0/1609475174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f062c15ac00 0x7f062c14a9b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:30.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:30.229+0000 7f0630a23640 1 --2- 172.21.15.5:0/1609475174 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f062c1543a0 0x7f062c14a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:30.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:30.229+0000 7f0630a23640 1 --2- 172.21.15.5:0/1609475174 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f062c0ccaa0 0x7f062c14f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:30.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:30.229+0000 7f0630a23640 1 -- 172.21.15.5:0/1609475174 >> 172.21.15.5:0/1609475174 conn(0x7f062c072ef0 msgr2=0x7f062c06cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:30.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:30.229+0000 7f0630a23640 1 -- 172.21.15.5:0/1609475174 shutdown_connections 2024-03-24T10:53:30.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:30.229+0000 7f0630a23640 1 -- 172.21.15.5:0/1609475174 wait complete. 2024-03-24T10:53:30.844 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:30.845 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:31.845 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:31.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.948+0000 7fe14509d640 1 Processor -- start 2024-03-24T10:53:31.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.948+0000 7fe14509d640 1 -- start start 2024-03-24T10:53:31.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.949+0000 7fe14509d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe1400c58c0 0x7fe1400caab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:31.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.949+0000 7fe14509d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe1400c9610 0x7fe1400c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:31.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.949+0000 7fe14509d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1400ca000 con 0x7fe1400c2ea0 2024-03-24T10:53:31.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.949+0000 7fe14509d640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fe1400cb210 con 0x7fe1400c9610 2024-03-24T10:53:31.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.949+0000 7fe14509d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fe140156db0 con 0x7fe1400c58c0 2024-03-24T10:53:31.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.949+0000 7fe13f59e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe1400c9610 0x7fe1400c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:31.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.950+0000 7fe13e59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe1400c58c0 0x7fe1400caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:31.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.950+0000 7fe13f59e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe1400c9610 0x7fe1400c9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:53738/0 (socket says 172.21.15.5:53738) 2024-03-24T10:53:31.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.950+0000 7fe13f59e640 1 -- 172.21.15.5:0/378381997 learned_addr learned my addr 172.21.15.5:0/378381997 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:31.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.952+0000 7fe13f59e640 1 -- 172.21.15.5:0/378381997 >> v2:172.21.15.57:3300/0 conn(0x7fe1400c58c0 msgr2=0x7fe1400caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:31.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.952+0000 7fe13f59e640 1 --2- 172.21.15.5:0/378381997 >> v2:172.21.15.57:3300/0 conn(0x7fe1400c58c0 0x7fe1400caab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:31.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.952+0000 7fe13f59e640 1 -- 172.21.15.5:0/378381997 >> v1:172.21.15.57:6789/0 conn(0x7fe1400c2ea0 legacy=0x7fe1400c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:31.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.952+0000 7fe13f59e640 1 -- 172.21.15.5:0/378381997 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe140156fc0 con 0x7fe1400c9610 2024-03-24T10:53:31.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.954+0000 7fe13f59e640 1 --2- 172.21.15.5:0/378381997 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1400c9610 0x7fe1400c9ac0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe140151640 tx=0x7fe13802e180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e849e8aef77b89b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:31.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.958+0000 7fe13dd9b640 1 -- 172.21.15.5:0/378381997 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13800b0d0 con 0x7fe1400c9610 2024-03-24T10:53:31.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.959+0000 7fe13dd9b640 1 -- 172.21.15.5:0/378381997 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe13800b230 con 0x7fe1400c9610 2024-03-24T10:53:31.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.959+0000 7fe13dd9b640 1 -- 172.21.15.5:0/378381997 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe138009600 con 0x7fe1400c9610 2024-03-24T10:53:31.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.959+0000 7fe14509d640 1 -- 172.21.15.5:0/378381997 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1400c9610 msgr2=0x7fe1400c9ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:31.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.959+0000 7fe14509d640 1 --2- 172.21.15.5:0/378381997 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1400c9610 0x7fe1400c9ac0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe140151640 tx=0x7fe13802e180 comp rx=0 tx=0).stop 2024-03-24T10:53:31.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.959+0000 7fe14509d640 1 -- 172.21.15.5:0/378381997 shutdown_connections 2024-03-24T10:53:31.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.959+0000 7fe14509d640 1 --2- 172.21.15.5:0/378381997 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1400c9610 0x7fe1400c9ac0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:31.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.959+0000 7fe14509d640 1 --2- 172.21.15.5:0/378381997 >> v2:172.21.15.57:3300/0 conn(0x7fe1400c58c0 0x7fe1400caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:31.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.959+0000 7fe14509d640 1 -- 172.21.15.5:0/378381997 >> 172.21.15.5:0/378381997 conn(0x7fe14006b130 msgr2=0x7fe14006b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:31.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.959+0000 7fe14509d640 1 -- 172.21.15.5:0/378381997 shutdown_connections 2024-03-24T10:53:31.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.959+0000 7fe14509d640 1 -- 172.21.15.5:0/378381997 wait complete. 2024-03-24T10:53:31.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.961+0000 7fe14509d640 1 Processor -- start 2024-03-24T10:53:31.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.961+0000 7fe14509d640 1 -- start start 2024-03-24T10:53:31.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.962+0000 7fe14509d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe1400c2ea0 0x7fe1401f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:31.962 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.962+0000 7fe14509d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1400c58c0 0x7fe1401f07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:31.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.962+0000 7fe14509d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1400c9610 0x7fe1400c84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:31.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.962+0000 7fe14509d640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fe1401f44c0 con 0x7fe1400c58c0 2024-03-24T10:53:31.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.962+0000 7fe14509d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe1401f0360 con 0x7fe1400c9610 2024-03-24T10:53:31.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.962+0000 7fe14509d640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe140143c30 con 0x7fe1400c2ea0 2024-03-24T10:53:31.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.962+0000 7fe13e59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1400c58c0 0x7fe1401f07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:31.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.963+0000 7fe13f59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1400c9610 0x7fe1400c84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:31.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.963+0000 7fe13e59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1400c58c0 0x7fe1401f07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:53748/0 (socket says 172.21.15.5:53748) 2024-03-24T10:53:31.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.963+0000 7fe13e59c640 1 -- 172.21.15.5:0/3077955176 learned_addr learned my addr 172.21.15.5:0/3077955176 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:31.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.963+0000 7fe13ed9d640 1 --2- 172.21.15.5:0/3077955176 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe1400c2ea0 0x7fe1401f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:31.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.966+0000 7fe13e59c640 1 -- 172.21.15.5:0/3077955176 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe1400c2ea0 msgr2=0x7fe1401f3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:31.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.966+0000 7fe13e59c640 1 --2- 172.21.15.5:0/3077955176 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe1400c2ea0 0x7fe1401f3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:31.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.966+0000 7fe13e59c640 1 -- 172.21.15.5:0/3077955176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1400c9610 msgr2=0x7fe1400c84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:31.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.966+0000 7fe13e59c640 1 --2- 172.21.15.5:0/3077955176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1400c9610 0x7fe1400c84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:31.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.966+0000 7fe13e59c640 1 -- 172.21.15.5:0/3077955176 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe13800bd40 con 0x7fe1400c58c0 2024-03-24T10:53:31.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.969+0000 7fe13e59c640 1 --2- 172.21.15.5:0/3077955176 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1400c58c0 0x7fe1401f07f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe13400e8a0 tx=0x7fe13400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:31.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.974+0000 7fe1277fe640 1 -- 172.21.15.5:0/3077955176 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1340094f0 con 0x7fe1400c58c0 2024-03-24T10:53:31.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.974+0000 7fe1277fe640 1 -- 172.21.15.5:0/3077955176 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe13400c940 con 0x7fe1400c58c0 2024-03-24T10:53:31.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.974+0000 7fe14509d640 1 -- 172.21.15.5:0/3077955176 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe140158f50 con 0x7fe1400c58c0 2024-03-24T10:53:31.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.974+0000 7fe1277fe640 1 -- 172.21.15.5:0/3077955176 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe13400ccf0 con 0x7fe1400c58c0 2024-03-24T10:53:31.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.974+0000 7fe14509d640 1 -- 172.21.15.5:0/3077955176 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1401594f0 con 0x7fe1400c58c0 2024-03-24T10:53:31.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.975+0000 7fe14509d640 1 -- 172.21.15.5:0/3077955176 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe140062a00 con 0x7fe1400c58c0 2024-03-24T10:53:31.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.998+0000 7fe1277fe640 1 -- 172.21.15.5:0/3077955176 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe134013470 con 0x7fe1400c58c0 2024-03-24T10:53:31.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.999+0000 7fe1277fe640 1 --2- 172.21.15.5:0/3077955176 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe12c06b860 0x7fe12c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:31.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.999+0000 7fe13ed9d640 1 --2- 172.21.15.5:0/3077955176 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe12c06b860 0x7fe12c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:31.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:31.999+0000 7fe13ed9d640 1 --2- 172.21.15.5:0/3077955176 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe12c06b860 0x7fe12c06dd20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe130006fd0 tx=0x7fe130009060 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:32.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:32.262+0000 7fe1277fe640 1 -- 172.21.15.5:0/3077955176 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fe134059260 con 0x7fe1400c58c0 2024-03-24T10:53:32.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:32.262+0000 7fe1277fe640 1 -- 172.21.15.5:0/3077955176 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe134055500 con 0x7fe1400c58c0 2024-03-24T10:53:32.278 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:53:32.280 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:53:32.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:32.525+0000 7fe14509d640 1 -- 172.21.15.5:0/3077955176 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe14006e690 con 0x7fe1400c58c0 2024-03-24T10:53:32.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:32.538+0000 7fe1277fe640 1 -- 172.21.15.5:0/3077955176 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fe134021090 con 0x7fe1400c58c0 2024-03-24T10:53:32.538 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:32.539 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:32.539 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:32.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:32.541+0000 7fe14509d640 1 -- 172.21.15.5:0/3077955176 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe12c06b860 msgr2=0x7fe12c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:32.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:32.541+0000 7fe14509d640 1 --2- 172.21.15.5:0/3077955176 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe12c06b860 0x7fe12c06dd20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe130006fd0 tx=0x7fe130009060 comp rx=0 tx=0).stop 2024-03-24T10:53:32.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:32.541+0000 7fe14509d640 1 -- 172.21.15.5:0/3077955176 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1400c58c0 msgr2=0x7fe1401f07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:32.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:32.541+0000 7fe14509d640 1 --2- 172.21.15.5:0/3077955176 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1400c58c0 0x7fe1401f07f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe13400e8a0 tx=0x7fe13400ed60 comp rx=0 tx=0).stop 2024-03-24T10:53:32.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:32.541+0000 7fe14509d640 1 -- 172.21.15.5:0/3077955176 shutdown_connections 2024-03-24T10:53:32.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:32.541+0000 7fe14509d640 1 --2- 172.21.15.5:0/3077955176 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe12c06b860 0x7fe12c06dd20 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:32.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:32.541+0000 7fe14509d640 1 --2- 172.21.15.5:0/3077955176 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe1400c58c0 0x7fe1401f07f0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:32.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:32.541+0000 7fe14509d640 1 --2- 172.21.15.5:0/3077955176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe1400c9610 0x7fe1400c84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:32.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:32.541+0000 7fe14509d640 1 --2- 172.21.15.5:0/3077955176 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe1400c2ea0 0x7fe1401f3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:32.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:32.541+0000 7fe14509d640 1 -- 172.21.15.5:0/3077955176 >> 172.21.15.5:0/3077955176 conn(0x7fe14006b130 msgr2=0x7fe140150dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:32.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:32.541+0000 7fe14509d640 1 -- 172.21.15.5:0/3077955176 shutdown_connections 2024-03-24T10:53:32.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:32.541+0000 7fe14509d640 1 -- 172.21.15.5:0/3077955176 wait complete. 2024-03-24T10:53:33.155 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:33.156 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:34.156 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:34.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.271+0000 7fc472d3e640 1 Processor -- start 2024-03-24T10:53:34.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.271+0000 7fc472d3e640 1 -- start start 2024-03-24T10:53:34.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.272+0000 7fc472d3e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc46c15ac00 0x7fc46c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:34.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.272+0000 7fc472d3e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc46c0cb2f0 con 0x7fc46c1543a0 2024-03-24T10:53:34.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.272+0000 7fc472d3e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc46c15b070 con 0x7fc46c0ccaa0 2024-03-24T10:53:34.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.272+0000 7fc472d3e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fc46c0c6750 con 0x7fc46c15ac00 2024-03-24T10:53:34.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.273+0000 7fc470ada640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fc46c1543a0 0x7fc46c06d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:44656/0 (socket says 172.21.15.5:44656) 2024-03-24T10:53:34.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.273+0000 7fc470ada640 1 -- 172.21.15.5:0/2062375753 learned_addr learned my addr 172.21.15.5:0/2062375753 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:53:34.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.273+0000 7fc46bfff640 1 --2- 172.21.15.5:0/2062375753 >> v2:172.21.15.57:3300/0 conn(0x7fc46c15ac00 0x7fc46c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:34.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.275+0000 7fc46b7fe640 1 -- 172.21.15.5:0/2062375753 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4244539311 0 0) 0x7fc46c0cb2f0 con 0x7fc46c1543a0 2024-03-24T10:53:34.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.275+0000 7fc46b7fe640 1 -- 172.21.15.5:0/2062375753 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc45c003520 con 0x7fc46c1543a0 2024-03-24T10:53:34.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.277+0000 7fc46bfff640 1 -- 172.21.15.5:0/2062375753 >> v1:172.21.15.57:6789/0 conn(0x7fc46c0ccaa0 legacy=0x7fc46c06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:34.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.277+0000 7fc46bfff640 1 -- 172.21.15.5:0/2062375753 >> v1:172.21.15.5:6789/0 conn(0x7fc46c1543a0 legacy=0x7fc46c06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:34.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.277+0000 7fc46bfff640 1 -- 172.21.15.5:0/2062375753 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc46c0ca7e0 con 0x7fc46c15ac00 2024-03-24T10:53:34.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.284+0000 7fc46bfff640 1 --2- 172.21.15.5:0/2062375753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc46c15ac00 0x7fc46c0cc560 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc4580097a0 tx=0x7fc45802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=386c3b19b18820b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:34.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.290+0000 7fc46b7fe640 1 -- 172.21.15.5:0/2062375753 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc458030610 con 0x7fc46c15ac00 2024-03-24T10:53:34.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.290+0000 7fc46b7fe640 1 -- 172.21.15.5:0/2062375753 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc458030770 con 0x7fc46c15ac00 2024-03-24T10:53:34.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.290+0000 7fc46b7fe640 1 -- 172.21.15.5:0/2062375753 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc458030b00 con 0x7fc46c15ac00 2024-03-24T10:53:34.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.290+0000 7fc472d3e640 1 -- 172.21.15.5:0/2062375753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc46c15ac00 msgr2=0x7fc46c0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:34.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.291+0000 7fc472d3e640 1 --2- 172.21.15.5:0/2062375753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc46c15ac00 0x7fc46c0cc560 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc4580097a0 tx=0x7fc45802b960 comp rx=0 tx=0).stop 2024-03-24T10:53:34.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.291+0000 7fc472d3e640 1 -- 172.21.15.5:0/2062375753 shutdown_connections 2024-03-24T10:53:34.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.291+0000 7fc472d3e640 1 --2- 172.21.15.5:0/2062375753 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc46c15ac00 0x7fc46c0cc560 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:34.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.291+0000 7fc472d3e640 1 -- 172.21.15.5:0/2062375753 >> 172.21.15.5:0/2062375753 conn(0x7fc46c072ef0 msgr2=0x7fc46c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:34.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.291+0000 7fc472d3e640 1 -- 172.21.15.5:0/2062375753 shutdown_connections 2024-03-24T10:53:34.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.291+0000 7fc472d3e640 1 -- 172.21.15.5:0/2062375753 wait complete. 2024-03-24T10:53:34.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.292+0000 7fc472d3e640 1 Processor -- start 2024-03-24T10:53:34.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.293+0000 7fc472d3e640 1 -- start start 2024-03-24T10:53:34.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.294+0000 7fc472d3e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc46c0ccaa0 0x7fc46c145460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:34.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.294+0000 7fc472d3e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc46c1543a0 0x7fc46c1444f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:34.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.294+0000 7fc472d3e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc46c15ac00 0x7fc46c144a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:34.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.294+0000 7fc472d3e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fc46c144f70 con 0x7fc46c0ccaa0 2024-03-24T10:53:34.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.294+0000 7fc472d3e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc46c1efca0 con 0x7fc46c1543a0 2024-03-24T10:53:34.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.294+0000 7fc472d3e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc46c157360 con 0x7fc46c15ac00 2024-03-24T10:53:34.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.295+0000 7fc46bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc46c1543a0 0x7fc46c1444f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:34.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.295+0000 7fc4712db640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc46c15ac00 0x7fc46c144a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:34.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.295+0000 7fc46bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc46c1543a0 0x7fc46c1444f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:46984/0 (socket says 172.21.15.5:46984) 2024-03-24T10:53:34.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.295+0000 7fc46bfff640 1 -- 172.21.15.5:0/3863660951 learned_addr learned my addr 172.21.15.5:0/3863660951 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:34.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.298+0000 7fc46bfff640 1 -- 172.21.15.5:0/3863660951 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc46c15ac00 msgr2=0x7fc46c144a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:34.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.298+0000 7fc46bfff640 1 --2- 172.21.15.5:0/3863660951 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc46c15ac00 0x7fc46c144a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:34.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.298+0000 7fc46bfff640 1 -- 172.21.15.5:0/3863660951 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc46c0ccaa0 msgr2=0x7fc46c145460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:34.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.298+0000 7fc46bfff640 1 --2- 172.21.15.5:0/3863660951 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc46c0ccaa0 0x7fc46c145460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:34.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.299+0000 7fc46bfff640 1 -- 172.21.15.5:0/3863660951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc458009400 con 0x7fc46c1543a0 2024-03-24T10:53:34.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.301+0000 7fc46bfff640 1 --2- 172.21.15.5:0/3863660951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc46c1543a0 0x7fc46c1444f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc458009890 tx=0x7fc4580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:34.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.305+0000 7fc4697fa640 1 -- 172.21.15.5:0/3863660951 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc45803e070 con 0x7fc46c1543a0 2024-03-24T10:53:34.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.305+0000 7fc4697fa640 1 -- 172.21.15.5:0/3863660951 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc458030e10 con 0x7fc46c1543a0 2024-03-24T10:53:34.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.305+0000 7fc472d3e640 1 -- 172.21.15.5:0/3863660951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc46c157640 con 0x7fc46c1543a0 2024-03-24T10:53:34.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.305+0000 7fc4697fa640 1 -- 172.21.15.5:0/3863660951 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc458039c60 con 0x7fc46c1543a0 2024-03-24T10:53:34.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.305+0000 7fc472d3e640 1 -- 172.21.15.5:0/3863660951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc46c1f01d0 con 0x7fc46c1543a0 2024-03-24T10:53:34.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.306+0000 7fc472d3e640 1 -- 172.21.15.5:0/3863660951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc46c146a30 con 0x7fc46c1543a0 2024-03-24T10:53:34.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.578+0000 7fc4697fa640 1 -- 172.21.15.5:0/3863660951 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc4580423f0 con 0x7fc46c1543a0 2024-03-24T10:53:34.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.578+0000 7fc4697fa640 1 --2- 172.21.15.5:0/3863660951 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc45006b860 0x7fc45006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:34.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.578+0000 7fc470ada640 1 --2- 172.21.15.5:0/3863660951 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc45006b860 0x7fc45006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:34.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.579+0000 7fc470ada640 1 --2- 172.21.15.5:0/3863660951 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc45006b860 0x7fc45006dd20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc460001610 tx=0x7fc460007930 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:34.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.584+0000 7fc4697fa640 1 -- 172.21.15.5:0/3863660951 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fc45807a720 con 0x7fc46c1543a0 2024-03-24T10:53:34.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.595+0000 7fc4697fa640 1 -- 172.21.15.5:0/3863660951 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc4580769c0 con 0x7fc46c1543a0 2024-03-24T10:53:34.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.816+0000 7fc472d3e640 1 -- 172.21.15.5:0/3863660951 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc46c146a30 con 0x7fc46c1543a0 2024-03-24T10:53:34.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.847+0000 7fc4697fa640 1 -- 172.21.15.5:0/3863660951 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fc46c146a30 con 0x7fc46c1543a0 2024-03-24T10:53:34.847 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:34.849 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:34.850 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:34.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.849+0000 7fc472d3e640 1 -- 172.21.15.5:0/3863660951 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc45006b860 msgr2=0x7fc45006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:34.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.849+0000 7fc472d3e640 1 --2- 172.21.15.5:0/3863660951 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc45006b860 0x7fc45006dd20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc460001610 tx=0x7fc460007930 comp rx=0 tx=0).stop 2024-03-24T10:53:34.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.849+0000 7fc472d3e640 1 -- 172.21.15.5:0/3863660951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc46c1543a0 msgr2=0x7fc46c1444f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:34.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.849+0000 7fc472d3e640 1 --2- 172.21.15.5:0/3863660951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc46c1543a0 0x7fc46c1444f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc458009890 tx=0x7fc4580046c0 comp rx=0 tx=0).stop 2024-03-24T10:53:34.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.849+0000 7fc472d3e640 1 -- 172.21.15.5:0/3863660951 shutdown_connections 2024-03-24T10:53:34.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.849+0000 7fc472d3e640 1 --2- 172.21.15.5:0/3863660951 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc45006b860 0x7fc45006dd20 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:34.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.849+0000 7fc472d3e640 1 --2- 172.21.15.5:0/3863660951 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc46c15ac00 0x7fc46c144a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:34.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.849+0000 7fc472d3e640 1 --2- 172.21.15.5:0/3863660951 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc46c1543a0 0x7fc46c1444f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:34.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.849+0000 7fc472d3e640 1 --2- 172.21.15.5:0/3863660951 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc46c0ccaa0 0x7fc46c145460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:34.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.849+0000 7fc472d3e640 1 -- 172.21.15.5:0/3863660951 >> 172.21.15.5:0/3863660951 conn(0x7fc46c072ef0 msgr2=0x7fc46c0cd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:34.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.849+0000 7fc472d3e640 1 -- 172.21.15.5:0/3863660951 shutdown_connections 2024-03-24T10:53:34.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:34.849+0000 7fc472d3e640 1 -- 172.21.15.5:0/3863660951 wait complete. 2024-03-24T10:53:35.464 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:35.464 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:36.465 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:36.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.568+0000 7f8b4013e640 1 Processor -- start 2024-03-24T10:53:36.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.568+0000 7f8b4013e640 1 -- start start 2024-03-24T10:53:36.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.569+0000 7f8b4013e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8b3815ac00 0x7f8b380cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:36.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.569+0000 7f8b4013e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8b380ccaa0 0x7f8b3806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:36.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.569+0000 7f8b4013e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b380cb2f0 con 0x7f8b381543a0 2024-03-24T10:53:36.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.569+0000 7f8b4013e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f8b380c6750 con 0x7f8b3815ac00 2024-03-24T10:53:36.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.569+0000 7f8b4013e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f8b380cceb0 con 0x7f8b380ccaa0 2024-03-24T10:53:36.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.570+0000 7f8b3e6db640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8b380ccaa0 0x7f8b3806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:36.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.570+0000 7f8b3deda640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f8b381543a0 0x7f8b3806d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:44658/0 (socket says 172.21.15.5:44658) 2024-03-24T10:53:36.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.570+0000 7f8b3deda640 1 -- 172.21.15.5:0/4292595601 learned_addr learned my addr 172.21.15.5:0/4292595601 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:53:36.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.570+0000 7f8b3d6d9640 1 --2- 172.21.15.5:0/4292595601 >> v2:172.21.15.5:3300/0 conn(0x7f8b3815ac00 0x7f8b380cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:36.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.573+0000 7f8b3e6db640 1 -- 172.21.15.5:0/4292595601 >> v2:172.21.15.5:3300/0 conn(0x7f8b3815ac00 msgr2=0x7f8b380cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:36.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.573+0000 7f8b3e6db640 1 --2- 172.21.15.5:0/4292595601 >> v2:172.21.15.5:3300/0 conn(0x7f8b3815ac00 0x7f8b380cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:36.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.573+0000 7f8b3e6db640 1 -- 172.21.15.5:0/4292595601 >> v1:172.21.15.5:6789/0 conn(0x7f8b381543a0 legacy=0x7f8b3806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:36.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.573+0000 7f8b3e6db640 1 -- 172.21.15.5:0/4292595601 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b3815b070 con 0x7f8b380ccaa0 2024-03-24T10:53:36.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.576+0000 7f8b3e6db640 1 --2- 172.21.15.5:0/4292595601 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8b380ccaa0 0x7f8b3806bf50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8b30009d90 tx=0x7f8b3002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c65c310183bd4ae4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:36.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.580+0000 7f8b3ced8640 1 -- 172.21.15.5:0/4292595601 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b3002cc50 con 0x7f8b380ccaa0 2024-03-24T10:53:36.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.580+0000 7f8b3ced8640 1 -- 172.21.15.5:0/4292595601 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b3002cdb0 con 0x7f8b380ccaa0 2024-03-24T10:53:36.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.580+0000 7f8b3ced8640 1 -- 172.21.15.5:0/4292595601 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b30034660 con 0x7f8b380ccaa0 2024-03-24T10:53:36.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.581+0000 7f8b4013e640 1 -- 172.21.15.5:0/4292595601 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8b380ccaa0 msgr2=0x7f8b3806bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:36.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.581+0000 7f8b4013e640 1 --2- 172.21.15.5:0/4292595601 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8b380ccaa0 0x7f8b3806bf50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8b30009d90 tx=0x7f8b3002bea0 comp rx=0 tx=0).stop 2024-03-24T10:53:36.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.581+0000 7f8b4013e640 1 -- 172.21.15.5:0/4292595601 shutdown_connections 2024-03-24T10:53:36.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.581+0000 7f8b4013e640 1 --2- 172.21.15.5:0/4292595601 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8b380ccaa0 0x7f8b3806bf50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:36.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.581+0000 7f8b4013e640 1 --2- 172.21.15.5:0/4292595601 >> v2:172.21.15.5:3300/0 conn(0x7f8b3815ac00 0x7f8b380cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:36.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.581+0000 7f8b4013e640 1 -- 172.21.15.5:0/4292595601 >> 172.21.15.5:0/4292595601 conn(0x7f8b38072ef0 msgr2=0x7f8b38061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:36.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.581+0000 7f8b4013e640 1 -- 172.21.15.5:0/4292595601 shutdown_connections 2024-03-24T10:53:36.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.581+0000 7f8b4013e640 1 -- 172.21.15.5:0/4292595601 wait complete. 2024-03-24T10:53:36.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.582+0000 7f8b4013e640 1 Processor -- start 2024-03-24T10:53:36.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.583+0000 7f8b4013e640 1 -- start start 2024-03-24T10:53:36.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.583+0000 7f8b4013e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b380ccaa0 0x7f8b380c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:36.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.583+0000 7f8b4013e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b381543a0 0x7f8b38158970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:36.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.583+0000 7f8b4013e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8b3815ac00 0x7f8b38158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:36.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.583+0000 7f8b4013e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f8b38157f60 con 0x7f8b381543a0 2024-03-24T10:53:36.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.583+0000 7f8b4013e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8b380ce940 con 0x7f8b380ccaa0 2024-03-24T10:53:36.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.583+0000 7f8b4013e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f8b380cec20 con 0x7f8b3815ac00 2024-03-24T10:53:36.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.584+0000 7f8b3d6d9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b381543a0 0x7f8b38158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:36.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.584+0000 7f8b3e6db640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8b3815ac00 0x7f8b38158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:36.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.584+0000 7f8b3d6d9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b381543a0 0x7f8b38158970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59618/0 (socket says 172.21.15.5:59618) 2024-03-24T10:53:36.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.584+0000 7f8b3d6d9640 1 -- 172.21.15.5:0/3892330142 learned_addr learned my addr 172.21.15.5:0/3892330142 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:36.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.584+0000 7f8b3deda640 1 --2- 172.21.15.5:0/3892330142 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b380ccaa0 0x7f8b380c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:36.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.587+0000 7f8b3d6d9640 1 -- 172.21.15.5:0/3892330142 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8b3815ac00 msgr2=0x7f8b38158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:36.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.587+0000 7f8b3d6d9640 1 --2- 172.21.15.5:0/3892330142 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8b3815ac00 0x7f8b38158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:36.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.587+0000 7f8b3d6d9640 1 -- 172.21.15.5:0/3892330142 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b380ccaa0 msgr2=0x7f8b380c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:36.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.587+0000 7f8b3d6d9640 1 --2- 172.21.15.5:0/3892330142 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b380ccaa0 0x7f8b380c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:36.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.587+0000 7f8b3d6d9640 1 -- 172.21.15.5:0/3892330142 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b30009a70 con 0x7f8b381543a0 2024-03-24T10:53:36.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.590+0000 7f8b3d6d9640 1 --2- 172.21.15.5:0/3892330142 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b381543a0 0x7f8b38158970 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8b2800e870 tx=0x7f8b2800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:36.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.594+0000 7f8b2e7fc640 1 -- 172.21.15.5:0/3892330142 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b280094f0 con 0x7f8b381543a0 2024-03-24T10:53:36.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.594+0000 7f8b2e7fc640 1 -- 172.21.15.5:0/3892330142 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b2800c940 con 0x7f8b381543a0 2024-03-24T10:53:36.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.594+0000 7f8b2e7fc640 1 -- 172.21.15.5:0/3892330142 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b2800ccf0 con 0x7f8b381543a0 2024-03-24T10:53:36.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.594+0000 7f8b4013e640 1 -- 172.21.15.5:0/3892330142 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b380cef00 con 0x7f8b381543a0 2024-03-24T10:53:36.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.594+0000 7f8b4013e640 1 -- 172.21.15.5:0/3892330142 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b380b4dd0 con 0x7f8b381543a0 2024-03-24T10:53:36.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.595+0000 7f8b4013e640 1 -- 172.21.15.5:0/3892330142 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b3806d060 con 0x7f8b381543a0 2024-03-24T10:53:36.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.862+0000 7f8b2e7fc640 1 -- 172.21.15.5:0/3892330142 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8b28013470 con 0x7f8b381543a0 2024-03-24T10:53:36.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.862+0000 7f8b2e7fc640 1 --2- 172.21.15.5:0/3892330142 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8b1c06b810 0x7f8b1c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:36.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.862+0000 7f8b3deda640 1 --2- 172.21.15.5:0/3892330142 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8b1c06b810 0x7f8b1c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:36.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.863+0000 7f8b3deda640 1 --2- 172.21.15.5:0/3892330142 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8b1c06b810 0x7f8b1c06dcd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8b34007fd0 tx=0x7f8b34007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:36.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.876+0000 7f8b2e7fc640 1 -- 172.21.15.5:0/3892330142 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f8b28059210 con 0x7f8b381543a0 2024-03-24T10:53:36.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:36.876+0000 7f8b2e7fc640 1 -- 172.21.15.5:0/3892330142 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8b280554b0 con 0x7f8b381543a0 2024-03-24T10:53:37.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:37.094+0000 7f8b4013e640 1 -- 172.21.15.5:0/3892330142 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b380681d0 con 0x7f8b381543a0 2024-03-24T10:53:37.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:37.117+0000 7f8b2e7fc640 1 -- 172.21.15.5:0/3892330142 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f8b28021090 con 0x7f8b381543a0 2024-03-24T10:53:37.117 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:37.117 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:37.118 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:37.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:37.119+0000 7f8b4013e640 1 -- 172.21.15.5:0/3892330142 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8b1c06b810 msgr2=0x7f8b1c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:37.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:37.119+0000 7f8b4013e640 1 --2- 172.21.15.5:0/3892330142 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8b1c06b810 0x7f8b1c06dcd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8b34007fd0 tx=0x7f8b34007e80 comp rx=0 tx=0).stop 2024-03-24T10:53:37.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:37.119+0000 7f8b4013e640 1 -- 172.21.15.5:0/3892330142 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b381543a0 msgr2=0x7f8b38158970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:37.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:37.119+0000 7f8b4013e640 1 --2- 172.21.15.5:0/3892330142 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b381543a0 0x7f8b38158970 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f8b2800e870 tx=0x7f8b2800ed30 comp rx=0 tx=0).stop 2024-03-24T10:53:37.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:37.119+0000 7f8b4013e640 1 -- 172.21.15.5:0/3892330142 shutdown_connections 2024-03-24T10:53:37.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:37.119+0000 7f8b4013e640 1 --2- 172.21.15.5:0/3892330142 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8b1c06b810 0x7f8b1c06dcd0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:37.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:37.119+0000 7f8b4013e640 1 --2- 172.21.15.5:0/3892330142 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8b381543a0 0x7f8b38158970 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:37.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:37.119+0000 7f8b4013e640 1 --2- 172.21.15.5:0/3892330142 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8b3815ac00 0x7f8b38158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:37.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:37.119+0000 7f8b4013e640 1 --2- 172.21.15.5:0/3892330142 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8b380ccaa0 0x7f8b380c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:37.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:37.119+0000 7f8b4013e640 1 -- 172.21.15.5:0/3892330142 >> 172.21.15.5:0/3892330142 conn(0x7f8b38072ef0 msgr2=0x7f8b3806c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:37.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:37.119+0000 7f8b4013e640 1 -- 172.21.15.5:0/3892330142 shutdown_connections 2024-03-24T10:53:37.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:37.119+0000 7f8b4013e640 1 -- 172.21.15.5:0/3892330142 wait complete. 2024-03-24T10:53:37.737 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:37.737 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:38.738 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:38.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.840+0000 7f4bef0e1640 1 Processor -- start 2024-03-24T10:53:38.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.840+0000 7f4bef0e1640 1 -- start start 2024-03-24T10:53:38.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.841+0000 7f4bef0e1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4be81513b0 0x7f4be80c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:38.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.841+0000 7f4bef0e1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4be815e590 0x7f4be806bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:38.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.841+0000 7f4bef0e1640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4be806ca40 con 0x7f4be8150fd0 2024-03-24T10:53:38.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.841+0000 7f4bef0e1640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f4be815e100 con 0x7f4be81513b0 2024-03-24T10:53:38.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.841+0000 7f4bef0e1640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4be806c230 con 0x7f4be815e590 2024-03-24T10:53:38.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.842+0000 7f4bece7d640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f4be8150fd0 0x7f4be815aea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:44662/0 (socket says 172.21.15.5:44662) 2024-03-24T10:53:38.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.842+0000 7f4bece7d640 1 -- 172.21.15.5:0/972831610 learned_addr learned my addr 172.21.15.5:0/972831610 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:53:38.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.842+0000 7f4bed67e640 1 --2- 172.21.15.5:0/972831610 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4be815e590 0x7f4be806bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:38.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.842+0000 7f4be7fff640 1 --2- 172.21.15.5:0/972831610 >> v2:172.21.15.57:3300/0 conn(0x7f4be81513b0 0x7f4be80c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:38.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.844+0000 7f4be77fe640 1 -- 172.21.15.5:0/972831610 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 912291983 0 0) 0x7f4be806ca40 con 0x7f4be8150fd0 2024-03-24T10:53:38.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.844+0000 7f4be77fe640 1 -- 172.21.15.5:0/972831610 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4bd8003520 con 0x7f4be8150fd0 2024-03-24T10:53:38.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.845+0000 7f4bed67e640 1 -- 172.21.15.5:0/972831610 >> v2:172.21.15.57:3300/0 conn(0x7f4be81513b0 msgr2=0x7f4be80c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:38.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.845+0000 7f4bed67e640 1 --2- 172.21.15.5:0/972831610 >> v2:172.21.15.57:3300/0 conn(0x7f4be81513b0 0x7f4be80c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:38.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.845+0000 7f4bed67e640 1 -- 172.21.15.5:0/972831610 >> v1:172.21.15.5:6789/0 conn(0x7f4be8150fd0 legacy=0x7f4be815aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:38.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.845+0000 7f4bed67e640 1 -- 172.21.15.5:0/972831610 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be806b240 con 0x7f4be815e590 2024-03-24T10:53:38.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.847+0000 7f4bed67e640 1 --2- 172.21.15.5:0/972831610 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4be815e590 0x7f4be806bcf0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4be0009d90 tx=0x7f4be002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=292748e37fa829c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:38.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.852+0000 7f4be77fe640 1 -- 172.21.15.5:0/972831610 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be002cc50 con 0x7f4be815e590 2024-03-24T10:53:38.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.852+0000 7f4be77fe640 1 -- 172.21.15.5:0/972831610 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4be002cdb0 con 0x7f4be815e590 2024-03-24T10:53:38.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.852+0000 7f4be77fe640 1 -- 172.21.15.5:0/972831610 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be0034660 con 0x7f4be815e590 2024-03-24T10:53:38.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.853+0000 7f4bef0e1640 1 -- 172.21.15.5:0/972831610 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4be815e590 msgr2=0x7f4be806bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:38.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.853+0000 7f4bef0e1640 1 --2- 172.21.15.5:0/972831610 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4be815e590 0x7f4be806bcf0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4be0009d90 tx=0x7f4be002bea0 comp rx=0 tx=0).stop 2024-03-24T10:53:38.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.853+0000 7f4bef0e1640 1 -- 172.21.15.5:0/972831610 shutdown_connections 2024-03-24T10:53:38.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.853+0000 7f4bef0e1640 1 --2- 172.21.15.5:0/972831610 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4be815e590 0x7f4be806bcf0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:38.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.853+0000 7f4bef0e1640 1 --2- 172.21.15.5:0/972831610 >> v2:172.21.15.57:3300/0 conn(0x7f4be81513b0 0x7f4be80c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:38.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.853+0000 7f4bef0e1640 1 -- 172.21.15.5:0/972831610 >> 172.21.15.5:0/972831610 conn(0x7f4be806f8b0 msgr2=0x7f4be806fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:38.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.853+0000 7f4bef0e1640 1 -- 172.21.15.5:0/972831610 shutdown_connections 2024-03-24T10:53:38.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.853+0000 7f4bef0e1640 1 -- 172.21.15.5:0/972831610 wait complete. 2024-03-24T10:53:38.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.854+0000 7f4bef0e1640 1 Processor -- start 2024-03-24T10:53:38.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.854+0000 7f4bef0e1640 1 -- start start 2024-03-24T10:53:38.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.855+0000 7f4bef0e1640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4be8150fd0 0x7f4be8155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:38.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.855+0000 7f4bef0e1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4be81513b0 0x7f4be8148d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:38.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.855+0000 7f4bef0e1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4be815e590 0x7f4be81478b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:38.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.855+0000 7f4bef0e1640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f4be81f0840 con 0x7f4be8150fd0 2024-03-24T10:53:38.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.855+0000 7f4bef0e1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4be814c3d0 con 0x7f4be815e590 2024-03-24T10:53:38.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.855+0000 7f4bef0e1640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4be8155670 con 0x7f4be81513b0 2024-03-24T10:53:38.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.855+0000 7f4bece7d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4be8150fd0 0x7f4be8155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:38.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.855+0000 7f4be7fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4be81513b0 0x7f4be8148d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:38.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.855+0000 7f4bed67e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4be815e590 0x7f4be81478b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:38.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.856+0000 7f4be7fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4be81513b0 0x7f4be8148d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:48472/0 (socket says 172.21.15.5:48472) 2024-03-24T10:53:38.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.856+0000 7f4be7fff640 1 -- 172.21.15.5:0/1201537916 learned_addr learned my addr 172.21.15.5:0/1201537916 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:38.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.859+0000 7f4be7fff640 1 -- 172.21.15.5:0/1201537916 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4be815e590 msgr2=0x7f4be81478b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:38.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.859+0000 7f4be7fff640 1 --2- 172.21.15.5:0/1201537916 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4be815e590 0x7f4be81478b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:38.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.859+0000 7f4be7fff640 1 -- 172.21.15.5:0/1201537916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4be8150fd0 msgr2=0x7f4be8155130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:38.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.859+0000 7f4be7fff640 1 --2- 172.21.15.5:0/1201537916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4be8150fd0 0x7f4be8155130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:38.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.859+0000 7f4be7fff640 1 -- 172.21.15.5:0/1201537916 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be0009a70 con 0x7f4be81513b0 2024-03-24T10:53:38.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.862+0000 7f4be7fff640 1 --2- 172.21.15.5:0/1201537916 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4be81513b0 0x7f4be8148d50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400e9b0 tx=0x7f4bd400ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:38.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.867+0000 7f4be57fa640 1 -- 172.21.15.5:0/1201537916 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4009670 con 0x7f4be81513b0 2024-03-24T10:53:38.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.867+0000 7f4be57fa640 1 -- 172.21.15.5:0/1201537916 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bd400c9a0 con 0x7f4be81513b0 2024-03-24T10:53:38.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.867+0000 7f4be57fa640 1 -- 172.21.15.5:0/1201537916 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd400cd50 con 0x7f4be81513b0 2024-03-24T10:53:38.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.867+0000 7f4bef0e1640 1 -- 172.21.15.5:0/1201537916 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4be814c600 con 0x7f4be81513b0 2024-03-24T10:53:38.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.867+0000 7f4bef0e1640 1 -- 172.21.15.5:0/1201537916 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4be814c8c0 con 0x7f4be81513b0 2024-03-24T10:53:38.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.868+0000 7f4bef0e1640 1 -- 172.21.15.5:0/1201537916 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4be81f0ab0 con 0x7f4be81513b0 2024-03-24T10:53:38.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.892+0000 7f4be57fa640 1 -- 172.21.15.5:0/1201537916 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4bd4013470 con 0x7f4be81513b0 2024-03-24T10:53:38.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.892+0000 7f4be57fa640 1 --2- 172.21.15.5:0/1201537916 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4bcc06b860 0x7f4bcc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:38.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.892+0000 7f4bece7d640 1 --2- 172.21.15.5:0/1201537916 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4bcc06b860 0x7f4bcc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:38.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.892+0000 7f4bece7d640 1 --2- 172.21.15.5:0/1201537916 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4bcc06b860 0x7f4bcc06dd20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc005880 tx=0x7f4bdc009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:38.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:38.895+0000 7f4be57fa640 1 -- 172.21.15.5:0/1201537916 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f4bd40593c0 con 0x7f4be81513b0 2024-03-24T10:53:39.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:39.157+0000 7f4be57fa640 1 -- 172.21.15.5:0/1201537916 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4bd4055660 con 0x7f4be81513b0 2024-03-24T10:53:39.379 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:39.379+0000 7f4bef0e1640 1 -- 172.21.15.5:0/1201537916 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4be80681d0 con 0x7f4be81513b0 2024-03-24T10:53:39.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:39.400+0000 7f4be57fa640 1 -- 172.21.15.5:0/1201537916 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f4bd4017090 con 0x7f4be81513b0 2024-03-24T10:53:39.401 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:39.401 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:39.401 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:39.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:39.402+0000 7f4bef0e1640 1 -- 172.21.15.5:0/1201537916 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4bcc06b860 msgr2=0x7f4bcc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:39.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:39.402+0000 7f4bef0e1640 1 --2- 172.21.15.5:0/1201537916 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4bcc06b860 0x7f4bcc06dd20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc005880 tx=0x7f4bdc009040 comp rx=0 tx=0).stop 2024-03-24T10:53:39.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:39.402+0000 7f4bef0e1640 1 -- 172.21.15.5:0/1201537916 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4be81513b0 msgr2=0x7f4be8148d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:39.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:39.402+0000 7f4bef0e1640 1 --2- 172.21.15.5:0/1201537916 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4be81513b0 0x7f4be8148d50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400e9b0 tx=0x7f4bd400ee70 comp rx=0 tx=0).stop 2024-03-24T10:53:39.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:39.402+0000 7f4bef0e1640 1 -- 172.21.15.5:0/1201537916 shutdown_connections 2024-03-24T10:53:39.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:39.402+0000 7f4bef0e1640 1 --2- 172.21.15.5:0/1201537916 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4bcc06b860 0x7f4bcc06dd20 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:39.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:39.402+0000 7f4bef0e1640 1 --2- 172.21.15.5:0/1201537916 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4be815e590 0x7f4be81478b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:39.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:39.402+0000 7f4bef0e1640 1 --2- 172.21.15.5:0/1201537916 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4be81513b0 0x7f4be8148d50 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:39.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:39.402+0000 7f4bef0e1640 1 --2- 172.21.15.5:0/1201537916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4be8150fd0 0x7f4be8155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:39.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:39.402+0000 7f4bef0e1640 1 -- 172.21.15.5:0/1201537916 >> 172.21.15.5:0/1201537916 conn(0x7f4be806f8b0 msgr2=0x7f4be814e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:39.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:39.403+0000 7f4bef0e1640 1 -- 172.21.15.5:0/1201537916 shutdown_connections 2024-03-24T10:53:39.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:39.403+0000 7f4bef0e1640 1 -- 172.21.15.5:0/1201537916 wait complete. 2024-03-24T10:53:40.021 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:40.021 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:41.021 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:41.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.124+0000 7f0228b56640 1 Processor -- start 2024-03-24T10:53:41.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.124+0000 7f0228b56640 1 -- start start 2024-03-24T10:53:41.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.125+0000 7f0228b56640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f022415ac00 0x7f02240cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:41.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.125+0000 7f0228b56640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f02240ccaa0 0x7f022406bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:41.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.125+0000 7f0228b56640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02240cb2f0 con 0x7f02241543a0 2024-03-24T10:53:41.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.125+0000 7f0228b56640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f02240c6750 con 0x7f02240ccaa0 2024-03-24T10:53:41.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.125+0000 7f0228b56640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f02240cc380 con 0x7f022415ac00 2024-03-24T10:53:41.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.126+0000 7f0221d9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f022415ac00 0x7f02240cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:41.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.126+0000 7f0221d9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f022415ac00 0x7f02240cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:48476/0 (socket says 172.21.15.5:48476) 2024-03-24T10:53:41.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.126+0000 7f0221d9b640 1 -- 172.21.15.5:0/1192806089 learned_addr learned my addr 172.21.15.5:0/1192806089 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:41.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.127+0000 7f0222d9d640 1 --2- 172.21.15.5:0/1192806089 >> v2:172.21.15.5:3300/0 conn(0x7f02240ccaa0 0x7f022406bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:41.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.130+0000 7f0221d9b640 1 -- 172.21.15.5:0/1192806089 >> v2:172.21.15.5:3300/0 conn(0x7f02240ccaa0 msgr2=0x7f022406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:41.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.130+0000 7f0221d9b640 1 --2- 172.21.15.5:0/1192806089 >> v2:172.21.15.5:3300/0 conn(0x7f02240ccaa0 0x7f022406bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:41.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.130+0000 7f0221d9b640 1 -- 172.21.15.5:0/1192806089 >> v1:172.21.15.5:6789/0 conn(0x7f02241543a0 legacy=0x7f022406d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:41.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.130+0000 7f0221d9b640 1 -- 172.21.15.5:0/1192806089 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f022415b070 con 0x7f022415ac00 2024-03-24T10:53:41.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.133+0000 7f0221d9b640 1 --2- 172.21.15.5:0/1192806089 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f022415ac00 0x7f02240cc560 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0218009870 tx=0x7f021802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cce3629a1532b12b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:41.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.137+0000 7f022159a640 1 -- 172.21.15.5:0/1192806089 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0218030840 con 0x7f022415ac00 2024-03-24T10:53:41.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.137+0000 7f022159a640 1 -- 172.21.15.5:0/1192806089 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02180309a0 con 0x7f022415ac00 2024-03-24T10:53:41.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.137+0000 7f022159a640 1 -- 172.21.15.5:0/1192806089 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0218030d30 con 0x7f022415ac00 2024-03-24T10:53:41.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.138+0000 7f0228b56640 1 -- 172.21.15.5:0/1192806089 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f022415ac00 msgr2=0x7f02240cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:41.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.138+0000 7f0228b56640 1 --2- 172.21.15.5:0/1192806089 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f022415ac00 0x7f02240cc560 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0218009870 tx=0x7f021802b980 comp rx=0 tx=0).stop 2024-03-24T10:53:41.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.138+0000 7f0228b56640 1 -- 172.21.15.5:0/1192806089 shutdown_connections 2024-03-24T10:53:41.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.138+0000 7f0228b56640 1 --2- 172.21.15.5:0/1192806089 >> v2:172.21.15.5:3300/0 conn(0x7f02240ccaa0 0x7f022406bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:41.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.138+0000 7f0228b56640 1 --2- 172.21.15.5:0/1192806089 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f022415ac00 0x7f02240cc560 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:41.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.138+0000 7f0228b56640 1 -- 172.21.15.5:0/1192806089 >> 172.21.15.5:0/1192806089 conn(0x7f0224072ef0 msgr2=0x7f0224061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:41.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.138+0000 7f0228b56640 1 -- 172.21.15.5:0/1192806089 shutdown_connections 2024-03-24T10:53:41.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.138+0000 7f0228b56640 1 -- 172.21.15.5:0/1192806089 wait complete. 2024-03-24T10:53:41.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.139+0000 7f0228b56640 1 Processor -- start 2024-03-24T10:53:41.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.139+0000 7f0228b56640 1 -- start start 2024-03-24T10:53:41.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.140+0000 7f0228b56640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02240ccaa0 0x7f02240cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:41.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.140+0000 7f0228b56640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02241543a0 0x7f022414a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:41.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.140+0000 7f0228b56640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f022415ac00 0x7f022414a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:41.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.140+0000 7f0228b56640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f0224155c50 con 0x7f02241543a0 2024-03-24T10:53:41.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.140+0000 7f0228b56640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0224157720 con 0x7f02240ccaa0 2024-03-24T10:53:41.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.140+0000 7f0228b56640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0224155ea0 con 0x7f022415ac00 2024-03-24T10:53:41.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.141+0000 7f0221d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02241543a0 0x7f022414a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:41.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.141+0000 7f0222d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f022415ac00 0x7f022414a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:41.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.141+0000 7f022259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02240ccaa0 0x7f02240cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:41.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.141+0000 7f0221d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02241543a0 0x7f022414a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59648/0 (socket says 172.21.15.5:59648) 2024-03-24T10:53:41.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.141+0000 7f0221d9b640 1 -- 172.21.15.5:0/1993351271 learned_addr learned my addr 172.21.15.5:0/1993351271 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:41.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.144+0000 7f0221d9b640 1 -- 172.21.15.5:0/1993351271 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f022415ac00 msgr2=0x7f022414a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:41.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.144+0000 7f0221d9b640 1 --2- 172.21.15.5:0/1993351271 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f022415ac00 0x7f022414a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:41.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.144+0000 7f0221d9b640 1 -- 172.21.15.5:0/1993351271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02240ccaa0 msgr2=0x7f02240cf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:41.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.144+0000 7f0221d9b640 1 --2- 172.21.15.5:0/1993351271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02240ccaa0 0x7f02240cf0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:41.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.144+0000 7f0221d9b640 1 -- 172.21.15.5:0/1993351271 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02180094d0 con 0x7f02241543a0 2024-03-24T10:53:41.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.146+0000 7f0221d9b640 1 --2- 172.21.15.5:0/1993351271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02241543a0 0x7f022414a470 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f021802b9c0 tx=0x7f0218004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:41.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.151+0000 7f0212ffd640 1 -- 172.21.15.5:0/1993351271 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0218039a90 con 0x7f02241543a0 2024-03-24T10:53:41.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.151+0000 7f0212ffd640 1 -- 172.21.15.5:0/1993351271 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0218039bf0 con 0x7f02241543a0 2024-03-24T10:53:41.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.151+0000 7f0228b56640 1 -- 172.21.15.5:0/1993351271 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0224156df0 con 0x7f02241543a0 2024-03-24T10:53:41.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.151+0000 7f0212ffd640 1 -- 172.21.15.5:0/1993351271 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0218043410 con 0x7f02241543a0 2024-03-24T10:53:41.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.151+0000 7f0228b56640 1 -- 172.21.15.5:0/1993351271 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02241573b0 con 0x7f02241543a0 2024-03-24T10:53:41.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.152+0000 7f0228b56640 1 -- 172.21.15.5:0/1993351271 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0224157540 con 0x7f02241543a0 2024-03-24T10:53:41.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.172+0000 7f0212ffd640 1 -- 172.21.15.5:0/1993351271 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f021804c3b0 con 0x7f02241543a0 2024-03-24T10:53:41.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.172+0000 7f0212ffd640 1 --2- 172.21.15.5:0/1993351271 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f020406b890 0x7f020406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:41.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.173+0000 7f022259c640 1 --2- 172.21.15.5:0/1993351271 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f020406b890 0x7f020406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:41.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.173+0000 7f022259c640 1 --2- 172.21.15.5:0/1993351271 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f020406b890 0x7f020406dd50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0214000f60 tx=0x7f0214009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:41.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.175+0000 7f0212ffd640 1 -- 172.21.15.5:0/1993351271 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f021807b800 con 0x7f02241543a0 2024-03-24T10:53:41.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.438+0000 7f0212ffd640 1 -- 172.21.15.5:0/1993351271 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0218077aa0 con 0x7f02241543a0 2024-03-24T10:53:41.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.680+0000 7f0228b56640 1 -- 172.21.15.5:0/1993351271 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02240681d0 con 0x7f02241543a0 2024-03-24T10:53:41.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.729+0000 7f0212ffd640 1 -- 172.21.15.5:0/1993351271 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f0218034090 con 0x7f02241543a0 2024-03-24T10:53:41.729 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:41.729 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:41.729 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:41.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.731+0000 7f0228b56640 1 -- 172.21.15.5:0/1993351271 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f020406b890 msgr2=0x7f020406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:41.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.731+0000 7f0228b56640 1 --2- 172.21.15.5:0/1993351271 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f020406b890 0x7f020406dd50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0214000f60 tx=0x7f0214009040 comp rx=0 tx=0).stop 2024-03-24T10:53:41.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.731+0000 7f0228b56640 1 -- 172.21.15.5:0/1993351271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02241543a0 msgr2=0x7f022414a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:41.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.731+0000 7f0228b56640 1 --2- 172.21.15.5:0/1993351271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02241543a0 0x7f022414a470 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f021802b9c0 tx=0x7f0218004730 comp rx=0 tx=0).stop 2024-03-24T10:53:41.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.731+0000 7f0228b56640 1 -- 172.21.15.5:0/1993351271 shutdown_connections 2024-03-24T10:53:41.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.731+0000 7f0228b56640 1 --2- 172.21.15.5:0/1993351271 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f020406b890 0x7f020406dd50 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:41.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.731+0000 7f0228b56640 1 --2- 172.21.15.5:0/1993351271 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02241543a0 0x7f022414a470 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:41.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.731+0000 7f0228b56640 1 --2- 172.21.15.5:0/1993351271 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f022415ac00 0x7f022414a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:41.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.731+0000 7f0228b56640 1 --2- 172.21.15.5:0/1993351271 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02240ccaa0 0x7f02240cf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:41.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.731+0000 7f0228b56640 1 -- 172.21.15.5:0/1993351271 >> 172.21.15.5:0/1993351271 conn(0x7f0224072ef0 msgr2=0x7f022406ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:41.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.731+0000 7f0228b56640 1 -- 172.21.15.5:0/1993351271 shutdown_connections 2024-03-24T10:53:41.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:41.731+0000 7f0228b56640 1 -- 172.21.15.5:0/1993351271 wait complete. 2024-03-24T10:53:42.345 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:42.345 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:43.346 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:43.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.449+0000 7f6f3884e640 1 Processor -- start 2024-03-24T10:53:43.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.449+0000 7f6f3884e640 1 -- start start 2024-03-24T10:53:43.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.451+0000 7f6f3884e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6f341543a0 0x7f6f3415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:43.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.451+0000 7f6f3884e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6f3406bf50 0x7f6f3406c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:43.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.451+0000 7f6f3884e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f340cb2f0 con 0x7f6f3406d4f0 2024-03-24T10:53:43.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.451+0000 7f6f3884e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f6f340c6750 con 0x7f6f341543a0 2024-03-24T10:53:43.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.451+0000 7f6f3884e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f6f340cc380 con 0x7f6f3406bf50 2024-03-24T10:53:43.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.451+0000 7f6f32d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6f3406bf50 0x7f6f3406c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:43.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.452+0000 7f6f32d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6f3406bf50 0x7f6f3406c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:33434/0 (socket says 172.21.15.5:33434) 2024-03-24T10:53:43.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.452+0000 7f6f32d9d640 1 -- 172.21.15.5:0/1874842292 learned_addr learned my addr 172.21.15.5:0/1874842292 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:43.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.452+0000 7f6f3259c640 1 --2- 172.21.15.5:0/1874842292 >> v2:172.21.15.5:3300/0 conn(0x7f6f341543a0 0x7f6f3415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:43.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.454+0000 7f6f32d9d640 1 -- 172.21.15.5:0/1874842292 >> v2:172.21.15.5:3300/0 conn(0x7f6f341543a0 msgr2=0x7f6f3415ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:43.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.454+0000 7f6f32d9d640 1 --2- 172.21.15.5:0/1874842292 >> v2:172.21.15.5:3300/0 conn(0x7f6f341543a0 0x7f6f3415ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:43.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.454+0000 7f6f32d9d640 1 -- 172.21.15.5:0/1874842292 >> v1:172.21.15.5:6789/0 conn(0x7f6f3406d4f0 legacy=0x7f6f340cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:43.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.454+0000 7f6f32d9d640 1 -- 172.21.15.5:0/1874842292 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f340ca7e0 con 0x7f6f3406bf50 2024-03-24T10:53:43.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.456+0000 7f6f32d9d640 1 --2- 172.21.15.5:0/1874842292 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6f3406bf50 0x7f6f3406c360 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c009e50 tx=0x7f6f2c02bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=726e297e731577b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:43.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.460+0000 7f6f3159a640 1 -- 172.21.15.5:0/1874842292 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c02cb90 con 0x7f6f3406bf50 2024-03-24T10:53:43.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.460+0000 7f6f3159a640 1 -- 172.21.15.5:0/1874842292 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f2c02ccf0 con 0x7f6f3406bf50 2024-03-24T10:53:43.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.460+0000 7f6f3159a640 1 -- 172.21.15.5:0/1874842292 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c034610 con 0x7f6f3406bf50 2024-03-24T10:53:43.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.461+0000 7f6f3884e640 1 -- 172.21.15.5:0/1874842292 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6f3406bf50 msgr2=0x7f6f3406c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:43.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.461+0000 7f6f3884e640 1 --2- 172.21.15.5:0/1874842292 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6f3406bf50 0x7f6f3406c360 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c009e50 tx=0x7f6f2c02bff0 comp rx=0 tx=0).stop 2024-03-24T10:53:43.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.461+0000 7f6f3884e640 1 -- 172.21.15.5:0/1874842292 shutdown_connections 2024-03-24T10:53:43.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.461+0000 7f6f3884e640 1 --2- 172.21.15.5:0/1874842292 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6f3406bf50 0x7f6f3406c360 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:43.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.461+0000 7f6f3884e640 1 --2- 172.21.15.5:0/1874842292 >> v2:172.21.15.5:3300/0 conn(0x7f6f341543a0 0x7f6f3415ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:43.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.461+0000 7f6f3884e640 1 -- 172.21.15.5:0/1874842292 >> 172.21.15.5:0/1874842292 conn(0x7f6f34072ef0 msgr2=0x7f6f34061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:43.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.461+0000 7f6f3884e640 1 -- 172.21.15.5:0/1874842292 shutdown_connections 2024-03-24T10:53:43.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.461+0000 7f6f3884e640 1 -- 172.21.15.5:0/1874842292 wait complete. 2024-03-24T10:53:43.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.463+0000 7f6f3884e640 1 Processor -- start 2024-03-24T10:53:43.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.463+0000 7f6f3884e640 1 -- start start 2024-03-24T10:53:43.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.463+0000 7f6f3884e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6f3406bf50 0x7f6f340c7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:43.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.463+0000 7f6f3884e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f3406d4f0 0x7f6f341f0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:43.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.463+0000 7f6f3884e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6f341543a0 0x7f6f3414a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:43.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.463+0000 7f6f3884e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f6f341efe00 con 0x7f6f3406bf50 2024-03-24T10:53:43.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.463+0000 7f6f3884e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6f341525c0 con 0x7f6f3406d4f0 2024-03-24T10:53:43.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.463+0000 7f6f3884e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f6f340c8090 con 0x7f6f341543a0 2024-03-24T10:53:43.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.464+0000 7f6f32d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6f341543a0 0x7f6f3414a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:43.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.464+0000 7f6f3259c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6f3406bf50 0x7f6f340c7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:43.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.464+0000 7f6f31d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f3406d4f0 0x7f6f341f0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:43.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.464+0000 7f6f32d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6f341543a0 0x7f6f3414a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:33438/0 (socket says 172.21.15.5:33438) 2024-03-24T10:53:43.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.464+0000 7f6f32d9d640 1 -- 172.21.15.5:0/3961503034 learned_addr learned my addr 172.21.15.5:0/3961503034 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:43.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.468+0000 7f6f3259c640 1 -- 172.21.15.5:0/3961503034 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6f341543a0 msgr2=0x7f6f3414a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:43.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.468+0000 7f6f3259c640 1 --2- 172.21.15.5:0/3961503034 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6f341543a0 0x7f6f3414a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:43.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.468+0000 7f6f3259c640 1 -- 172.21.15.5:0/3961503034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f3406d4f0 msgr2=0x7f6f341f0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:43.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.468+0000 7f6f3259c640 1 --2- 172.21.15.5:0/3961503034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f3406d4f0 0x7f6f341f0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:43.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.468+0000 7f6f3259c640 1 -- 172.21.15.5:0/3961503034 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f2c009b30 con 0x7f6f3406bf50 2024-03-24T10:53:43.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.471+0000 7f6f3259c640 1 --2- 172.21.15.5:0/3961503034 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6f3406bf50 0x7f6f340c7100 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6f2400e8a0 tx=0x7f6f2400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:43.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.475+0000 7f6f22ffd640 1 -- 172.21.15.5:0/3961503034 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f24009e60 con 0x7f6f3406bf50 2024-03-24T10:53:43.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.475+0000 7f6f22ffd640 1 -- 172.21.15.5:0/3961503034 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f2400c940 con 0x7f6f3406bf50 2024-03-24T10:53:43.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.475+0000 7f6f3884e640 1 -- 172.21.15.5:0/3961503034 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f340c8330 con 0x7f6f3406bf50 2024-03-24T10:53:43.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.475+0000 7f6f22ffd640 1 -- 172.21.15.5:0/3961503034 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2400ccd0 con 0x7f6f3406bf50 2024-03-24T10:53:43.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.475+0000 7f6f3884e640 1 -- 172.21.15.5:0/3961503034 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f340c88f0 con 0x7f6f3406bf50 2024-03-24T10:53:43.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.476+0000 7f6f3884e640 1 -- 172.21.15.5:0/3961503034 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f340681d0 con 0x7f6f3406bf50 2024-03-24T10:53:43.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.495+0000 7f6f22ffd640 1 -- 172.21.15.5:0/3961503034 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6f24013470 con 0x7f6f3406bf50 2024-03-24T10:53:43.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.496+0000 7f6f22ffd640 1 --2- 172.21.15.5:0/3961503034 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6f1406b890 0x7f6f1406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:43.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.496+0000 7f6f31d9b640 1 --2- 172.21.15.5:0/3961503034 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6f1406b890 0x7f6f1406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:43.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.497+0000 7f6f31d9b640 1 --2- 172.21.15.5:0/3961503034 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6f1406b890 0x7f6f1406dd50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6f340b2460 tx=0x7f6f28009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:43.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.754+0000 7f6f22ffd640 1 -- 172.21.15.5:0/3961503034 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f6f240590d0 con 0x7f6f3406bf50 2024-03-24T10:53:43.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.754+0000 7f6f22ffd640 1 -- 172.21.15.5:0/3961503034 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6f24055370 con 0x7f6f3406bf50 2024-03-24T10:53:43.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.982+0000 7f6f3884e640 1 -- 172.21.15.5:0/3961503034 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f34151080 con 0x7f6f3406bf50 2024-03-24T10:53:43.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.994+0000 7f6f22ffd640 1 -- 172.21.15.5:0/3961503034 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f6f24021090 con 0x7f6f3406bf50 2024-03-24T10:53:43.995 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:43.995 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:43.995 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:43.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.997+0000 7f6f3884e640 1 -- 172.21.15.5:0/3961503034 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6f1406b890 msgr2=0x7f6f1406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:43.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.997+0000 7f6f3884e640 1 --2- 172.21.15.5:0/3961503034 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6f1406b890 0x7f6f1406dd50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6f340b2460 tx=0x7f6f28009040 comp rx=0 tx=0).stop 2024-03-24T10:53:43.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.997+0000 7f6f3884e640 1 -- 172.21.15.5:0/3961503034 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6f3406bf50 msgr2=0x7f6f340c7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:43.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.997+0000 7f6f3884e640 1 --2- 172.21.15.5:0/3961503034 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6f3406bf50 0x7f6f340c7100 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6f2400e8a0 tx=0x7f6f2400ed60 comp rx=0 tx=0).stop 2024-03-24T10:53:43.997 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.997+0000 7f6f3884e640 1 -- 172.21.15.5:0/3961503034 shutdown_connections 2024-03-24T10:53:43.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.997+0000 7f6f3884e640 1 --2- 172.21.15.5:0/3961503034 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6f1406b890 0x7f6f1406dd50 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:43.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.997+0000 7f6f3884e640 1 --2- 172.21.15.5:0/3961503034 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6f341543a0 0x7f6f3414a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:43.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.997+0000 7f6f3884e640 1 --2- 172.21.15.5:0/3961503034 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f3406d4f0 0x7f6f341f0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:43.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.997+0000 7f6f3884e640 1 --2- 172.21.15.5:0/3961503034 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6f3406bf50 0x7f6f340c7100 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:43.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.997+0000 7f6f3884e640 1 -- 172.21.15.5:0/3961503034 >> 172.21.15.5:0/3961503034 conn(0x7f6f34072ef0 msgr2=0x7f6f3414e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:43.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.997+0000 7f6f3884e640 1 -- 172.21.15.5:0/3961503034 shutdown_connections 2024-03-24T10:53:43.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:43.997+0000 7f6f3884e640 1 -- 172.21.15.5:0/3961503034 wait complete. 2024-03-24T10:53:44.614 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:44.614 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:45.615 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:45.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.719+0000 7f2bd66eb640 1 Processor -- start 2024-03-24T10:53:45.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.719+0000 7f2bd66eb640 1 -- start start 2024-03-24T10:53:45.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.720+0000 7f2bd66eb640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2bd01543a0 0x7f2bd015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:45.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.720+0000 7f2bd66eb640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2bd006d4f0 0x7f2bd00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:45.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.720+0000 7f2bd66eb640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bd00cb2f0 con 0x7f2bd00ccaa0 2024-03-24T10:53:45.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.720+0000 7f2bd66eb640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f2bd00c6750 con 0x7f2bd01543a0 2024-03-24T10:53:45.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.720+0000 7f2bd66eb640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f2bd00cc380 con 0x7f2bd006d4f0 2024-03-24T10:53:45.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.720+0000 7f2bcf7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2bd006d4f0 0x7f2bd00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:45.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.721+0000 7f2bcffff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2bd01543a0 0x7f2bd015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:45.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.721+0000 7f2bd4c88640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f2bd00ccaa0 0x7f2bd006bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:60782/0 (socket says 172.21.15.5:60782) 2024-03-24T10:53:45.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.721+0000 7f2bd4c88640 1 -- 172.21.15.5:0/1833489351 learned_addr learned my addr 172.21.15.5:0/1833489351 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:53:45.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.724+0000 7f2bceffd640 1 -- 172.21.15.5:0/1833489351 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3513045557 0 0) 0x7f2bd00cb2f0 con 0x7f2bd00ccaa0 2024-03-24T10:53:45.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.724+0000 7f2bceffd640 1 -- 172.21.15.5:0/1833489351 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bb8003520 con 0x7f2bd00ccaa0 2024-03-24T10:53:45.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.724+0000 7f2bcf7fe640 1 -- 172.21.15.5:0/1833489351 >> v2:172.21.15.5:3300/0 conn(0x7f2bd01543a0 msgr2=0x7f2bd015ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:45.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.724+0000 7f2bcf7fe640 1 --2- 172.21.15.5:0/1833489351 >> v2:172.21.15.5:3300/0 conn(0x7f2bd01543a0 0x7f2bd015ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:45.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.724+0000 7f2bcf7fe640 1 -- 172.21.15.5:0/1833489351 >> v1:172.21.15.57:6789/0 conn(0x7f2bd00ccaa0 legacy=0x7f2bd006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:45.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.724+0000 7f2bcf7fe640 1 -- 172.21.15.5:0/1833489351 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bd00ca7e0 con 0x7f2bd006d4f0 2024-03-24T10:53:45.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.726+0000 7f2bcf7fe640 1 --2- 172.21.15.5:0/1833489351 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2bd006d4f0 0x7f2bd00cc560 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2bc40097a0 tx=0x7f2bc402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=605b19e87898e024 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:45.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.731+0000 7f2bceffd640 1 -- 172.21.15.5:0/1833489351 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4030610 con 0x7f2bd006d4f0 2024-03-24T10:53:45.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.731+0000 7f2bceffd640 1 -- 172.21.15.5:0/1833489351 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bc4030770 con 0x7f2bd006d4f0 2024-03-24T10:53:45.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.731+0000 7f2bceffd640 1 -- 172.21.15.5:0/1833489351 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4030b00 con 0x7f2bd006d4f0 2024-03-24T10:53:45.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.731+0000 7f2bd66eb640 1 -- 172.21.15.5:0/1833489351 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2bd006d4f0 msgr2=0x7f2bd00cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:45.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.731+0000 7f2bd66eb640 1 --2- 172.21.15.5:0/1833489351 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2bd006d4f0 0x7f2bd00cc560 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2bc40097a0 tx=0x7f2bc402b960 comp rx=0 tx=0).stop 2024-03-24T10:53:45.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.732+0000 7f2bd66eb640 1 -- 172.21.15.5:0/1833489351 shutdown_connections 2024-03-24T10:53:45.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.732+0000 7f2bd66eb640 1 --2- 172.21.15.5:0/1833489351 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2bd006d4f0 0x7f2bd00cc560 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:45.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.732+0000 7f2bd66eb640 1 --2- 172.21.15.5:0/1833489351 >> v2:172.21.15.5:3300/0 conn(0x7f2bd01543a0 0x7f2bd015ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:45.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.732+0000 7f2bd66eb640 1 -- 172.21.15.5:0/1833489351 >> 172.21.15.5:0/1833489351 conn(0x7f2bd0072ef0 msgr2=0x7f2bd0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:45.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.732+0000 7f2bd66eb640 1 -- 172.21.15.5:0/1833489351 shutdown_connections 2024-03-24T10:53:45.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.732+0000 7f2bd66eb640 1 -- 172.21.15.5:0/1833489351 wait complete. 2024-03-24T10:53:45.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.733+0000 7f2bd66eb640 1 Processor -- start 2024-03-24T10:53:45.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.733+0000 7f2bd66eb640 1 -- start start 2024-03-24T10:53:45.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.734+0000 7f2bd66eb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bd006d4f0 0x7f2bd0159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:45.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.734+0000 7f2bd66eb640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2bd00ccaa0 0x7f2bd014a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:45.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.734+0000 7f2bd66eb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2bd01543a0 0x7f2bd006c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:45.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.734+0000 7f2bd66eb640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd01577d0 con 0x7f2bd006d4f0 2024-03-24T10:53:45.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.734+0000 7f2bd66eb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd0154e10 con 0x7f2bd01543a0 2024-03-24T10:53:45.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.734+0000 7f2bd66eb640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f2bd0156bc0 con 0x7f2bd00ccaa0 2024-03-24T10:53:45.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.734+0000 7f2bcf7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2bd00ccaa0 0x7f2bd014a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:45.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.734+0000 7f2bcffff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bd006d4f0 0x7f2bd0159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:45.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.734+0000 7f2bd4c88640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2bd01543a0 0x7f2bd006c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:45.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.734+0000 7f2bcf7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2bd00ccaa0 0x7f2bd014a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:33456/0 (socket says 172.21.15.5:33456) 2024-03-24T10:53:45.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.734+0000 7f2bcf7fe640 1 -- 172.21.15.5:0/1008109431 learned_addr learned my addr 172.21.15.5:0/1008109431 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:45.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.739+0000 7f2bcf7fe640 1 -- 172.21.15.5:0/1008109431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2bd01543a0 msgr2=0x7f2bd006c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:45.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.739+0000 7f2bcf7fe640 1 --2- 172.21.15.5:0/1008109431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2bd01543a0 0x7f2bd006c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:45.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.739+0000 7f2bcf7fe640 1 -- 172.21.15.5:0/1008109431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bd006d4f0 msgr2=0x7f2bd0159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:45.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.739+0000 7f2bcf7fe640 1 --2- 172.21.15.5:0/1008109431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bd006d4f0 0x7f2bd0159de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:45.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.739+0000 7f2bcf7fe640 1 -- 172.21.15.5:0/1008109431 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc4009400 con 0x7f2bd00ccaa0 2024-03-24T10:53:45.743 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.743+0000 7f2bcf7fe640 1 --2- 172.21.15.5:0/1008109431 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2bd00ccaa0 0x7f2bd014a870 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2bc402bf70 tx=0x7f2bc4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:45.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.745+0000 7f2bccff9640 1 -- 172.21.15.5:0/1008109431 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc403e040 con 0x7f2bd00ccaa0 2024-03-24T10:53:45.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.746+0000 7f2bd66eb640 1 -- 172.21.15.5:0/1008109431 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bd0156ea0 con 0x7f2bd00ccaa0 2024-03-24T10:53:45.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.746+0000 7f2bd66eb640 1 -- 172.21.15.5:0/1008109431 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bd0157460 con 0x7f2bd00ccaa0 2024-03-24T10:53:45.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.746+0000 7f2bd66eb640 1 -- 172.21.15.5:0/1008109431 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bd0157980 con 0x7f2bd00ccaa0 2024-03-24T10:53:45.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.748+0000 7f2bccff9640 1 -- 172.21.15.5:0/1008109431 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bc4039a20 con 0x7f2bd00ccaa0 2024-03-24T10:53:45.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.748+0000 7f2bccff9640 1 -- 172.21.15.5:0/1008109431 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4039d90 con 0x7f2bd00ccaa0 2024-03-24T10:53:45.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.772+0000 7f2bccff9640 1 -- 172.21.15.5:0/1008109431 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2bc404d3b0 con 0x7f2bd00ccaa0 2024-03-24T10:53:45.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.772+0000 7f2bccff9640 1 --2- 172.21.15.5:0/1008109431 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2bbc06b840 0x7f2bbc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:45.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.772+0000 7f2bcffff640 1 --2- 172.21.15.5:0/1008109431 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2bbc06b840 0x7f2bbc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:45.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:45.773+0000 7f2bcffff640 1 --2- 172.21.15.5:0/1008109431 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2bbc06b840 0x7f2bbc06dd00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0009aa0 tx=0x7f2bc0009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:46.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:46.091+0000 7f2bccff9640 1 -- 172.21.15.5:0/1008109431 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f2bc407b120 con 0x7f2bd00ccaa0 2024-03-24T10:53:46.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:46.103+0000 7f2bccff9640 1 -- 172.21.15.5:0/1008109431 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2bc40773c0 con 0x7f2bd00ccaa0 2024-03-24T10:53:46.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:46.328+0000 7f2bd66eb640 1 -- 172.21.15.5:0/1008109431 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2bd0157980 con 0x7f2bd00ccaa0 2024-03-24T10:53:46.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:46.374+0000 7f2bccff9640 1 -- 172.21.15.5:0/1008109431 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f2bd0157980 con 0x7f2bd00ccaa0 2024-03-24T10:53:46.375 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:46.375 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:46.375 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:46.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:46.376+0000 7f2bd66eb640 1 -- 172.21.15.5:0/1008109431 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2bbc06b840 msgr2=0x7f2bbc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:46.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:46.376+0000 7f2bd66eb640 1 --2- 172.21.15.5:0/1008109431 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2bbc06b840 0x7f2bbc06dd00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0009aa0 tx=0x7f2bc0009190 comp rx=0 tx=0).stop 2024-03-24T10:53:46.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:46.377+0000 7f2bd66eb640 1 -- 172.21.15.5:0/1008109431 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2bd00ccaa0 msgr2=0x7f2bd014a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:46.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:46.377+0000 7f2bd66eb640 1 --2- 172.21.15.5:0/1008109431 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2bd00ccaa0 0x7f2bd014a870 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2bc402bf70 tx=0x7f2bc4004730 comp rx=0 tx=0).stop 2024-03-24T10:53:46.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:46.377+0000 7f2bd66eb640 1 -- 172.21.15.5:0/1008109431 shutdown_connections 2024-03-24T10:53:46.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:46.377+0000 7f2bd66eb640 1 --2- 172.21.15.5:0/1008109431 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2bbc06b840 0x7f2bbc06dd00 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:46.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:46.377+0000 7f2bd66eb640 1 --2- 172.21.15.5:0/1008109431 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2bd01543a0 0x7f2bd006c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:46.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:46.377+0000 7f2bd66eb640 1 --2- 172.21.15.5:0/1008109431 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2bd00ccaa0 0x7f2bd014a870 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:46.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:46.377+0000 7f2bd66eb640 1 --2- 172.21.15.5:0/1008109431 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bd006d4f0 0x7f2bd0159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:46.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:46.377+0000 7f2bd66eb640 1 -- 172.21.15.5:0/1008109431 >> 172.21.15.5:0/1008109431 conn(0x7f2bd0072ef0 msgr2=0x7f2bd014ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:46.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:46.377+0000 7f2bd66eb640 1 -- 172.21.15.5:0/1008109431 shutdown_connections 2024-03-24T10:53:46.378 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:46.377+0000 7f2bd66eb640 1 -- 172.21.15.5:0/1008109431 wait complete. 2024-03-24T10:53:46.993 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:46.993 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:47.994 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:48.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.097+0000 7f88938ed640 1 Processor -- start 2024-03-24T10:53:48.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.097+0000 7f88938ed640 1 -- start start 2024-03-24T10:53:48.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.098+0000 7f88938ed640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f888c154220 0x7f888c0c4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:48.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.098+0000 7f88938ed640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f888c0c8900 0x7f888c06c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:48.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.098+0000 7f88938ed640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f888c0c9050 con 0x7f888c0c84e0 2024-03-24T10:53:48.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.098+0000 7f88938ed640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f888c15a5f0 con 0x7f888c0c8900 2024-03-24T10:53:48.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.098+0000 7f88938ed640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f888c15a8d0 con 0x7f888c154220 2024-03-24T10:53:48.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.098+0000 7f8891e8a640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f888c0c8900 0x7f888c06c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:48.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.099+0000 7f8891689640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f888c154220 0x7f888c0c4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:48.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.099+0000 7f8891689640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f888c154220 0x7f888c0c4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:33466/0 (socket says 172.21.15.5:33466) 2024-03-24T10:53:48.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.099+0000 7f8891689640 1 -- 172.21.15.5:0/951911772 learned_addr learned my addr 172.21.15.5:0/951911772 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:48.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.101+0000 7f8891689640 1 -- 172.21.15.5:0/951911772 >> v2:172.21.15.57:3300/0 conn(0x7f888c0c8900 msgr2=0x7f888c06c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:48.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.101+0000 7f8891689640 1 --2- 172.21.15.5:0/951911772 >> v2:172.21.15.57:3300/0 conn(0x7f888c0c8900 0x7f888c06c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:48.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.101+0000 7f8891689640 1 -- 172.21.15.5:0/951911772 >> v1:172.21.15.57:6789/0 conn(0x7f888c0c84e0 legacy=0x7f888c0cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:48.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.101+0000 7f8891689640 1 -- 172.21.15.5:0/951911772 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f888c15abb0 con 0x7f888c154220 2024-03-24T10:53:48.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.103+0000 7f8891689640 1 --2- 172.21.15.5:0/951911772 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f888c154220 0x7f888c0c4300 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8888009870 tx=0x7f888802ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7633d48930bf8dbd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:48.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.108+0000 7f8883fff640 1 -- 172.21.15.5:0/951911772 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888802f840 con 0x7f888c154220 2024-03-24T10:53:48.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.108+0000 7f8883fff640 1 -- 172.21.15.5:0/951911772 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f888802f9a0 con 0x7f888c154220 2024-03-24T10:53:48.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.108+0000 7f8883fff640 1 -- 172.21.15.5:0/951911772 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888802fd30 con 0x7f888c154220 2024-03-24T10:53:48.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.108+0000 7f88938ed640 1 -- 172.21.15.5:0/951911772 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f888c154220 msgr2=0x7f888c0c4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:48.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.108+0000 7f88938ed640 1 --2- 172.21.15.5:0/951911772 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f888c154220 0x7f888c0c4300 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8888009870 tx=0x7f888802ba20 comp rx=0 tx=0).stop 2024-03-24T10:53:48.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.108+0000 7f88938ed640 1 -- 172.21.15.5:0/951911772 shutdown_connections 2024-03-24T10:53:48.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.108+0000 7f88938ed640 1 --2- 172.21.15.5:0/951911772 >> v2:172.21.15.57:3300/0 conn(0x7f888c0c8900 0x7f888c06c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:48.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.108+0000 7f88938ed640 1 --2- 172.21.15.5:0/951911772 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f888c154220 0x7f888c0c4300 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:48.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.108+0000 7f88938ed640 1 -- 172.21.15.5:0/951911772 >> 172.21.15.5:0/951911772 conn(0x7f888c06f830 msgr2=0x7f888c06fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:48.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.108+0000 7f88938ed640 1 -- 172.21.15.5:0/951911772 shutdown_connections 2024-03-24T10:53:48.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.108+0000 7f88938ed640 1 -- 172.21.15.5:0/951911772 wait complete. 2024-03-24T10:53:48.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.110+0000 7f88938ed640 1 Processor -- start 2024-03-24T10:53:48.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.110+0000 7f88938ed640 1 -- start start 2024-03-24T10:53:48.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.110+0000 7f88938ed640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f888c0c84e0 0x7f888c0cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:48.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.110+0000 7f88938ed640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f888c0c8900 0x7f888c154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:48.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.110+0000 7f88938ed640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f888c154220 0x7f888c14d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:48.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.110+0000 7f88938ed640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f888c156200 con 0x7f888c0c8900 2024-03-24T10:53:48.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.110+0000 7f88938ed640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f888c1555f0 con 0x7f888c0c84e0 2024-03-24T10:53:48.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.110+0000 7f88938ed640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f888c14ddf0 con 0x7f888c154220 2024-03-24T10:53:48.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.111+0000 7f8891e8a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f888c154220 0x7f888c14d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:48.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.111+0000 7f8890e88640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f888c0c8900 0x7f888c154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:48.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.111+0000 7f8891689640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f888c0c84e0 0x7f888c0cafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:48.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.111+0000 7f8891e8a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f888c154220 0x7f888c14d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:33472/0 (socket says 172.21.15.5:33472) 2024-03-24T10:53:48.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.111+0000 7f8891e8a640 1 -- 172.21.15.5:0/822179386 learned_addr learned my addr 172.21.15.5:0/822179386 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:48.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.115+0000 7f8891e8a640 1 -- 172.21.15.5:0/822179386 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f888c0c84e0 msgr2=0x7f888c0cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:48.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.115+0000 7f8891e8a640 1 --2- 172.21.15.5:0/822179386 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f888c0c84e0 0x7f888c0cafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:48.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.115+0000 7f8891e8a640 1 -- 172.21.15.5:0/822179386 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f888c0c8900 msgr2=0x7f888c154bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:48.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.115+0000 7f8891e8a640 1 --2- 172.21.15.5:0/822179386 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f888c0c8900 0x7f888c154bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:48.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.115+0000 7f8891e8a640 1 -- 172.21.15.5:0/822179386 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88880094d0 con 0x7f888c154220 2024-03-24T10:53:48.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.117+0000 7f8891e8a640 1 --2- 172.21.15.5:0/822179386 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f888c154220 0x7f888c14d8b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f888400ef90 tx=0x7f888400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:48.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.121+0000 7f8881ffb640 1 -- 172.21.15.5:0/822179386 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8884009ca0 con 0x7f888c154220 2024-03-24T10:53:48.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.121+0000 7f8881ffb640 1 -- 172.21.15.5:0/822179386 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f888400f040 con 0x7f888c154220 2024-03-24T10:53:48.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.121+0000 7f8881ffb640 1 -- 172.21.15.5:0/822179386 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88840146c0 con 0x7f888c154220 2024-03-24T10:53:48.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.121+0000 7f88938ed640 1 -- 172.21.15.5:0/822179386 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f888c0ca920 con 0x7f888c154220 2024-03-24T10:53:48.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.122+0000 7f88938ed640 1 -- 172.21.15.5:0/822179386 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f888c0cc190 con 0x7f888c154220 2024-03-24T10:53:48.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.123+0000 7f88938ed640 1 -- 172.21.15.5:0/822179386 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f888c0681d0 con 0x7f888c154220 2024-03-24T10:53:48.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.147+0000 7f8881ffb640 1 -- 172.21.15.5:0/822179386 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8884014820 con 0x7f888c154220 2024-03-24T10:53:48.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.147+0000 7f8881ffb640 1 --2- 172.21.15.5:0/822179386 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f887006b860 0x7f887006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:48.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.147+0000 7f8891689640 1 --2- 172.21.15.5:0/822179386 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f887006b860 0x7f887006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:48.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.148+0000 7f8891689640 1 --2- 172.21.15.5:0/822179386 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f887006b860 0x7f887006dd20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8888036000 tx=0x7f8888008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:48.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.423+0000 7f8881ffb640 1 -- 172.21.15.5:0/822179386 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f8884059dd0 con 0x7f888c154220 2024-03-24T10:53:48.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.438+0000 7f8881ffb640 1 -- 172.21.15.5:0/822179386 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8884056070 con 0x7f888c154220 2024-03-24T10:53:48.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.691+0000 7f88938ed640 1 -- 172.21.15.5:0/822179386 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f888c0b3e70 con 0x7f888c154220 2024-03-24T10:53:48.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.779+0000 7f8881ffb640 1 -- 172.21.15.5:0/822179386 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f8884018090 con 0x7f888c154220 2024-03-24T10:53:48.779 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:48.780 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:48.780 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:48.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.781+0000 7f88938ed640 1 -- 172.21.15.5:0/822179386 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f887006b860 msgr2=0x7f887006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:48.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.781+0000 7f88938ed640 1 --2- 172.21.15.5:0/822179386 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f887006b860 0x7f887006dd20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8888036000 tx=0x7f8888008030 comp rx=0 tx=0).stop 2024-03-24T10:53:48.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.781+0000 7f88938ed640 1 -- 172.21.15.5:0/822179386 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f888c154220 msgr2=0x7f888c14d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:48.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.781+0000 7f88938ed640 1 --2- 172.21.15.5:0/822179386 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f888c154220 0x7f888c14d8b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f888400ef90 tx=0x7f888400c490 comp rx=0 tx=0).stop 2024-03-24T10:53:48.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.781+0000 7f88938ed640 1 -- 172.21.15.5:0/822179386 shutdown_connections 2024-03-24T10:53:48.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.781+0000 7f88938ed640 1 --2- 172.21.15.5:0/822179386 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f887006b860 0x7f887006dd20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:48.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.781+0000 7f88938ed640 1 --2- 172.21.15.5:0/822179386 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f888c0c8900 0x7f888c154bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:48.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.781+0000 7f88938ed640 1 --2- 172.21.15.5:0/822179386 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f888c154220 0x7f888c14d8b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:48.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.781+0000 7f88938ed640 1 --2- 172.21.15.5:0/822179386 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f888c0c84e0 0x7f888c0cafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:48.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.781+0000 7f88938ed640 1 -- 172.21.15.5:0/822179386 >> 172.21.15.5:0/822179386 conn(0x7f888c06f830 msgr2=0x7f888c1516a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:48.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.781+0000 7f88938ed640 1 -- 172.21.15.5:0/822179386 shutdown_connections 2024-03-24T10:53:48.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:48.781+0000 7f88938ed640 1 -- 172.21.15.5:0/822179386 wait complete. 2024-03-24T10:53:49.398 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:49.399 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:50.399 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:50.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.501+0000 7f56558af640 1 Processor -- start 2024-03-24T10:53:50.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.502+0000 7f56558af640 1 -- start start 2024-03-24T10:53:50.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.502+0000 7f56558af640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56501543a0 0x7f565015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:50.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.502+0000 7f56558af640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f565006bf50 0x7f565006c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:50.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.502+0000 7f56558af640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56500cb2f0 con 0x7f565006d4f0 2024-03-24T10:53:50.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.502+0000 7f56558af640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f56500c6750 con 0x7f565006bf50 2024-03-24T10:53:50.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.502+0000 7f56558af640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f56500cc380 con 0x7f56501543a0 2024-03-24T10:53:50.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.503+0000 7f564f7fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f565006bf50 0x7f565006c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:50.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.503+0000 7f564effd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56501543a0 0x7f565015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:50.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.503+0000 7f564e7fc640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f565006d4f0 0x7f56500cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:53226/0 (socket says 172.21.15.5:53226) 2024-03-24T10:53:50.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.503+0000 7f564e7fc640 1 -- 172.21.15.5:0/2961240863 learned_addr learned my addr 172.21.15.5:0/2961240863 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:53:50.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.507+0000 7f564effd640 1 -- 172.21.15.5:0/2961240863 >> v2:172.21.15.5:3300/0 conn(0x7f565006bf50 msgr2=0x7f565006c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:50.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.507+0000 7f564effd640 1 --2- 172.21.15.5:0/2961240863 >> v2:172.21.15.5:3300/0 conn(0x7f565006bf50 0x7f565006c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:50.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.507+0000 7f564effd640 1 -- 172.21.15.5:0/2961240863 >> v1:172.21.15.5:6789/0 conn(0x7f565006d4f0 legacy=0x7f56500cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:50.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.507+0000 7f564effd640 1 -- 172.21.15.5:0/2961240863 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56500ca7e0 con 0x7f56501543a0 2024-03-24T10:53:50.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.509+0000 7f564effd640 1 --2- 172.21.15.5:0/2961240863 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56501543a0 0x7f565015ac00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f56400097a0 tx=0x7f564002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b94b022af1d4179 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:50.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.514+0000 7f564dffb640 1 -- 172.21.15.5:0/2961240863 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5640030610 con 0x7f56501543a0 2024-03-24T10:53:50.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.514+0000 7f564dffb640 1 -- 172.21.15.5:0/2961240863 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5640030770 con 0x7f56501543a0 2024-03-24T10:53:50.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.514+0000 7f564dffb640 1 -- 172.21.15.5:0/2961240863 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5640030b00 con 0x7f56501543a0 2024-03-24T10:53:50.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.515+0000 7f56558af640 1 -- 172.21.15.5:0/2961240863 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56501543a0 msgr2=0x7f565015ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:50.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.515+0000 7f56558af640 1 --2- 172.21.15.5:0/2961240863 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56501543a0 0x7f565015ac00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f56400097a0 tx=0x7f564002b960 comp rx=0 tx=0).stop 2024-03-24T10:53:50.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.515+0000 7f56558af640 1 -- 172.21.15.5:0/2961240863 shutdown_connections 2024-03-24T10:53:50.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.515+0000 7f56558af640 1 --2- 172.21.15.5:0/2961240863 >> v2:172.21.15.5:3300/0 conn(0x7f565006bf50 0x7f565006c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:50.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.515+0000 7f56558af640 1 --2- 172.21.15.5:0/2961240863 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f56501543a0 0x7f565015ac00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:50.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.515+0000 7f56558af640 1 -- 172.21.15.5:0/2961240863 >> 172.21.15.5:0/2961240863 conn(0x7f5650072ef0 msgr2=0x7f5650061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:50.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.515+0000 7f56558af640 1 -- 172.21.15.5:0/2961240863 shutdown_connections 2024-03-24T10:53:50.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.515+0000 7f56558af640 1 -- 172.21.15.5:0/2961240863 wait complete. 2024-03-24T10:53:50.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.516+0000 7f56558af640 1 Processor -- start 2024-03-24T10:53:50.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.516+0000 7f56558af640 1 -- start start 2024-03-24T10:53:50.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.517+0000 7f56558af640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f565006bf50 0x7f565014b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:50.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.517+0000 7f56558af640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f565006d4f0 0x7f565014fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:50.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.517+0000 7f56558af640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56501543a0 0x7f565014a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:50.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.517+0000 7f56558af640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f565014bf40 con 0x7f565006d4f0 2024-03-24T10:53:50.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.517+0000 7f56558af640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f565014c990 con 0x7f56501543a0 2024-03-24T10:53:50.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.517+0000 7f56558af640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f56500ce990 con 0x7f565006bf50 2024-03-24T10:53:50.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.518+0000 7f564e7fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f565006d4f0 0x7f565014fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:50.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.518+0000 7f564effd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f565006bf50 0x7f565014b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:50.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.518+0000 7f564f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56501543a0 0x7f565014a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:50.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.518+0000 7f564e7fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f565006d4f0 0x7f565014fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:59432/0 (socket says 172.21.15.5:59432) 2024-03-24T10:53:50.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.518+0000 7f564e7fc640 1 -- 172.21.15.5:0/155810053 learned_addr learned my addr 172.21.15.5:0/155810053 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:50.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.521+0000 7f564e7fc640 1 -- 172.21.15.5:0/155810053 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f565006bf50 msgr2=0x7f565014b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:50.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.521+0000 7f564e7fc640 1 --2- 172.21.15.5:0/155810053 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f565006bf50 0x7f565014b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:50.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.521+0000 7f564e7fc640 1 -- 172.21.15.5:0/155810053 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56501543a0 msgr2=0x7f565014a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:50.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.521+0000 7f564e7fc640 1 --2- 172.21.15.5:0/155810053 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56501543a0 0x7f565014a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:50.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.521+0000 7f564e7fc640 1 -- 172.21.15.5:0/155810053 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5640009400 con 0x7f565006d4f0 2024-03-24T10:53:50.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.524+0000 7f564e7fc640 1 --2- 172.21.15.5:0/155810053 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f565006d4f0 0x7f565014fd70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f564400c400 tx=0x7f564400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:50.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.528+0000 7f5637fff640 1 -- 172.21.15.5:0/155810053 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5644005ce0 con 0x7f565006d4f0 2024-03-24T10:53:50.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.528+0000 7f5637fff640 1 -- 172.21.15.5:0/155810053 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5644005e40 con 0x7f565006d4f0 2024-03-24T10:53:50.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.528+0000 7f56558af640 1 -- 172.21.15.5:0/155810053 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5650157500 con 0x7f565006d4f0 2024-03-24T10:53:50.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.528+0000 7f5637fff640 1 -- 172.21.15.5:0/155810053 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56440072d0 con 0x7f565006d4f0 2024-03-24T10:53:50.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.528+0000 7f56558af640 1 -- 172.21.15.5:0/155810053 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56500b47c0 con 0x7f565006d4f0 2024-03-24T10:53:50.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.529+0000 7f56558af640 1 -- 172.21.15.5:0/155810053 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56500b4620 con 0x7f565006d4f0 2024-03-24T10:53:50.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.566+0000 7f5637fff640 1 -- 172.21.15.5:0/155810053 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f56440074f0 con 0x7f565006d4f0 2024-03-24T10:53:50.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.566+0000 7f5637fff640 1 --2- 172.21.15.5:0/155810053 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f563c06b860 0x7f563c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:50.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.566+0000 7f564effd640 1 --2- 172.21.15.5:0/155810053 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f563c06b860 0x7f563c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:50.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.567+0000 7f564effd640 1 --2- 172.21.15.5:0/155810053 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f563c06b860 0x7f563c06dd20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5640036000 tx=0x7f5640008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:50.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.832+0000 7f5637fff640 1 -- 172.21.15.5:0/155810053 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f5644058290 con 0x7f565006d4f0 2024-03-24T10:53:50.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:50.832+0000 7f5637fff640 1 -- 172.21.15.5:0/155810053 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5644054530 con 0x7f565006d4f0 2024-03-24T10:53:51.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:51.058+0000 7f56558af640 1 -- 172.21.15.5:0/155810053 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56500b4620 con 0x7f565006d4f0 2024-03-24T10:53:51.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:51.080+0000 7f5637fff640 1 -- 172.21.15.5:0/155810053 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f56500b4620 con 0x7f565006d4f0 2024-03-24T10:53:51.080 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:51.080 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:51.081 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:51.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:51.082+0000 7f56558af640 1 -- 172.21.15.5:0/155810053 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f563c06b860 msgr2=0x7f563c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:51.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:51.082+0000 7f56558af640 1 --2- 172.21.15.5:0/155810053 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f563c06b860 0x7f563c06dd20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5640036000 tx=0x7f5640008030 comp rx=0 tx=0).stop 2024-03-24T10:53:51.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:51.082+0000 7f56558af640 1 -- 172.21.15.5:0/155810053 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f565006d4f0 msgr2=0x7f565014fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:51.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:51.082+0000 7f56558af640 1 --2- 172.21.15.5:0/155810053 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f565006d4f0 0x7f565014fd70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f564400c400 tx=0x7f564400c8c0 comp rx=0 tx=0).stop 2024-03-24T10:53:51.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:51.082+0000 7f56558af640 1 -- 172.21.15.5:0/155810053 shutdown_connections 2024-03-24T10:53:51.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:51.082+0000 7f56558af640 1 --2- 172.21.15.5:0/155810053 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f563c06b860 0x7f563c06dd20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:51.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:51.082+0000 7f56558af640 1 --2- 172.21.15.5:0/155810053 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f565006d4f0 0x7f565014fd70 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:51.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:51.082+0000 7f56558af640 1 --2- 172.21.15.5:0/155810053 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f56501543a0 0x7f565014a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:51.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:51.082+0000 7f56558af640 1 --2- 172.21.15.5:0/155810053 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f565006bf50 0x7f565014b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:51.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:51.082+0000 7f56558af640 1 -- 172.21.15.5:0/155810053 >> 172.21.15.5:0/155810053 conn(0x7f5650072ef0 msgr2=0x7f5650151b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:51.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:51.082+0000 7f56558af640 1 -- 172.21.15.5:0/155810053 shutdown_connections 2024-03-24T10:53:51.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:51.082+0000 7f56558af640 1 -- 172.21.15.5:0/155810053 wait complete. 2024-03-24T10:53:51.702 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:51.702 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:52.703 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:52.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.805+0000 7fa6c8c4e640 1 Processor -- start 2024-03-24T10:53:52.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.805+0000 7fa6c8c4e640 1 -- start start 2024-03-24T10:53:52.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.806+0000 7fa6c8c4e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa6c406c580 0x7fa6c41538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:52.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.806+0000 7fa6c8c4e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa6c40cf110 0x7fa6c4153df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:52.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.806+0000 7fa6c8c4e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6c415bce0 con 0x7fa6c40cc8f0 2024-03-24T10:53:52.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.806+0000 7fa6c8c4e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fa6c40cd890 con 0x7fa6c40cf110 2024-03-24T10:53:52.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.806+0000 7fa6c8c4e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fa6c406cae0 con 0x7fa6c406c580 2024-03-24T10:53:52.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.806+0000 7fa6c2d9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa6c40cf110 0x7fa6c4153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:52.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.807+0000 7fa6c1d9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa6c406c580 0x7fa6c41538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:52.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.807+0000 7fa6c259c640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fa6c40cc8f0 0x7fa6c415b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59812/0 (socket says 172.21.15.5:59812) 2024-03-24T10:53:52.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.807+0000 7fa6c259c640 1 -- 172.21.15.5:0/387912962 learned_addr learned my addr 172.21.15.5:0/387912962 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:53:52.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.811+0000 7fa6c1d9b640 1 -- 172.21.15.5:0/387912962 >> v2:172.21.15.5:3300/0 conn(0x7fa6c40cf110 msgr2=0x7fa6c4153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:52.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.811+0000 7fa6c1d9b640 1 --2- 172.21.15.5:0/387912962 >> v2:172.21.15.5:3300/0 conn(0x7fa6c40cf110 0x7fa6c4153df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:52.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.811+0000 7fa6c1d9b640 1 -- 172.21.15.5:0/387912962 >> v1:172.21.15.5:6789/0 conn(0x7fa6c40cc8f0 legacy=0x7fa6c415b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:52.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.811+0000 7fa6c1d9b640 1 -- 172.21.15.5:0/387912962 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6c40cdb50 con 0x7fa6c406c580 2024-03-24T10:53:52.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.813+0000 7fa6c1d9b640 1 --2- 172.21.15.5:0/387912962 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa6c406c580 0x7fa6c41538b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa6b80097a0 tx=0x7fa6b802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fe543cc49f80374 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:52.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.818+0000 7fa6c159a640 1 -- 172.21.15.5:0/387912962 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8031830 con 0x7fa6c406c580 2024-03-24T10:53:52.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.818+0000 7fa6c159a640 1 -- 172.21.15.5:0/387912962 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6b8031990 con 0x7fa6c406c580 2024-03-24T10:53:52.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.818+0000 7fa6c159a640 1 -- 172.21.15.5:0/387912962 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8031d20 con 0x7fa6c406c580 2024-03-24T10:53:52.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.819+0000 7fa6c8c4e640 1 -- 172.21.15.5:0/387912962 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa6c406c580 msgr2=0x7fa6c41538b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:52.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.819+0000 7fa6c8c4e640 1 --2- 172.21.15.5:0/387912962 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa6c406c580 0x7fa6c41538b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa6b80097a0 tx=0x7fa6b802b8d0 comp rx=0 tx=0).stop 2024-03-24T10:53:52.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.819+0000 7fa6c8c4e640 1 -- 172.21.15.5:0/387912962 shutdown_connections 2024-03-24T10:53:52.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.819+0000 7fa6c8c4e640 1 --2- 172.21.15.5:0/387912962 >> v2:172.21.15.5:3300/0 conn(0x7fa6c40cf110 0x7fa6c4153df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:52.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.819+0000 7fa6c8c4e640 1 --2- 172.21.15.5:0/387912962 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa6c406c580 0x7fa6c41538b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:52.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.819+0000 7fa6c8c4e640 1 -- 172.21.15.5:0/387912962 >> 172.21.15.5:0/387912962 conn(0x7fa6c40b1820 msgr2=0x7fa6c40b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:52.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.819+0000 7fa6c8c4e640 1 -- 172.21.15.5:0/387912962 shutdown_connections 2024-03-24T10:53:52.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.819+0000 7fa6c8c4e640 1 -- 172.21.15.5:0/387912962 wait complete. 2024-03-24T10:53:52.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.821+0000 7fa6c8c4e640 1 Processor -- start 2024-03-24T10:53:52.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.821+0000 7fa6c8c4e640 1 -- start start 2024-03-24T10:53:52.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.821+0000 7fa6c8c4e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa6c406c580 0x7fa6c40c6db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:52.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.821+0000 7fa6c8c4e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa6c40cc8f0 0x7fa6c415a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:52.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.821+0000 7fa6c8c4e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6c40cf110 0x7fa6c4159960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:52.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.821+0000 7fa6c8c4e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c41f0120 con 0x7fa6c40cc8f0 2024-03-24T10:53:52.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.821+0000 7fa6c8c4e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c41f0400 con 0x7fa6c40cf110 2024-03-24T10:53:52.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.821+0000 7fa6c8c4e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fa6c41f06e0 con 0x7fa6c406c580 2024-03-24T10:53:52.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.822+0000 7fa6c1d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa6c40cc8f0 0x7fa6c415a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:52.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.822+0000 7fa6c2d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6c40cf110 0x7fa6c4159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:52.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.822+0000 7fa6c259c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa6c406c580 0x7fa6c40c6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:52.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.822+0000 7fa6c1d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa6c40cc8f0 0x7fa6c415a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:45918/0 (socket says 172.21.15.5:45918) 2024-03-24T10:53:52.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.822+0000 7fa6c1d9b640 1 -- 172.21.15.5:0/1793560703 learned_addr learned my addr 172.21.15.5:0/1793560703 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:52.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.826+0000 7fa6c1d9b640 1 -- 172.21.15.5:0/1793560703 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa6c406c580 msgr2=0x7fa6c40c6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:52.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.826+0000 7fa6c1d9b640 1 --2- 172.21.15.5:0/1793560703 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa6c406c580 0x7fa6c40c6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:52.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.826+0000 7fa6c1d9b640 1 -- 172.21.15.5:0/1793560703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6c40cf110 msgr2=0x7fa6c4159960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:52.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.826+0000 7fa6c1d9b640 1 --2- 172.21.15.5:0/1793560703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6c40cf110 0x7fa6c4159960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:52.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.826+0000 7fa6c1d9b640 1 -- 172.21.15.5:0/1793560703 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6b8009400 con 0x7fa6c40cc8f0 2024-03-24T10:53:52.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.828+0000 7fa6c1d9b640 1 --2- 172.21.15.5:0/1793560703 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa6c40cc8f0 0x7fa6c415a830 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8034040 tx=0x7fa6b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:52.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.832+0000 7fa6b2ffd640 1 -- 172.21.15.5:0/1793560703 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b803a8f0 con 0x7fa6c40cc8f0 2024-03-24T10:53:52.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.832+0000 7fa6b2ffd640 1 -- 172.21.15.5:0/1793560703 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6b803aa70 con 0x7fa6c40cc8f0 2024-03-24T10:53:52.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.832+0000 7fa6b2ffd640 1 -- 172.21.15.5:0/1793560703 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b803ae20 con 0x7fa6c40cc8f0 2024-03-24T10:53:52.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.833+0000 7fa6c8c4e640 1 -- 172.21.15.5:0/1793560703 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6c41f09c0 con 0x7fa6c40cc8f0 2024-03-24T10:53:52.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.833+0000 7fa6c8c4e640 1 -- 172.21.15.5:0/1793560703 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6c40c72f0 con 0x7fa6c40cc8f0 2024-03-24T10:53:52.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.834+0000 7fa6c8c4e640 1 -- 172.21.15.5:0/1793560703 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6c4159680 con 0x7fa6c40cc8f0 2024-03-24T10:53:52.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.857+0000 7fa6b2ffd640 1 -- 172.21.15.5:0/1793560703 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa6b804d440 con 0x7fa6c40cc8f0 2024-03-24T10:53:52.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.858+0000 7fa6b2ffd640 1 --2- 172.21.15.5:0/1793560703 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa6a406b810 0x7fa6a406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:52.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.858+0000 7fa6c259c640 1 --2- 172.21.15.5:0/1793560703 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa6a406b810 0x7fa6a406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:52.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:52.858+0000 7fa6c259c640 1 --2- 172.21.15.5:0/1793560703 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa6a406b810 0x7fa6a406dcd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4007fd0 tx=0x7fa6b4007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:53.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:53.137+0000 7fa6b2ffd640 1 -- 172.21.15.5:0/1793560703 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fa6b807ad90 con 0x7fa6c40cc8f0 2024-03-24T10:53:53.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:53.137+0000 7fa6b2ffd640 1 -- 172.21.15.5:0/1793560703 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa6b8077030 con 0x7fa6c40cc8f0 2024-03-24T10:53:53.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:53.357+0000 7fa6c8c4e640 1 -- 172.21.15.5:0/1793560703 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6c4159680 con 0x7fa6c40cc8f0 2024-03-24T10:53:53.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:53.372+0000 7fa6b2ffd640 1 -- 172.21.15.5:0/1793560703 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fa6c4159680 con 0x7fa6c40cc8f0 2024-03-24T10:53:53.373 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:53.373 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:53.373 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:53.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:53.375+0000 7fa6c8c4e640 1 -- 172.21.15.5:0/1793560703 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa6a406b810 msgr2=0x7fa6a406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:53.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:53.375+0000 7fa6c8c4e640 1 --2- 172.21.15.5:0/1793560703 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa6a406b810 0x7fa6a406dcd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4007fd0 tx=0x7fa6b4007e80 comp rx=0 tx=0).stop 2024-03-24T10:53:53.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:53.375+0000 7fa6c8c4e640 1 -- 172.21.15.5:0/1793560703 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa6c40cc8f0 msgr2=0x7fa6c415a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:53.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:53.375+0000 7fa6c8c4e640 1 --2- 172.21.15.5:0/1793560703 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa6c40cc8f0 0x7fa6c415a830 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8034040 tx=0x7fa6b80046c0 comp rx=0 tx=0).stop 2024-03-24T10:53:53.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:53.375+0000 7fa6c8c4e640 1 -- 172.21.15.5:0/1793560703 shutdown_connections 2024-03-24T10:53:53.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:53.375+0000 7fa6c8c4e640 1 --2- 172.21.15.5:0/1793560703 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa6a406b810 0x7fa6a406dcd0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:53.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:53.375+0000 7fa6c8c4e640 1 --2- 172.21.15.5:0/1793560703 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa6c40cc8f0 0x7fa6c415a830 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:53.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:53.375+0000 7fa6c8c4e640 1 --2- 172.21.15.5:0/1793560703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa6c40cf110 0x7fa6c4159960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:53.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:53.375+0000 7fa6c8c4e640 1 --2- 172.21.15.5:0/1793560703 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa6c406c580 0x7fa6c40c6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:53.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:53.375+0000 7fa6c8c4e640 1 -- 172.21.15.5:0/1793560703 >> 172.21.15.5:0/1793560703 conn(0x7fa6c40b1820 msgr2=0x7fa6c40b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:53.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:53.375+0000 7fa6c8c4e640 1 -- 172.21.15.5:0/1793560703 shutdown_connections 2024-03-24T10:53:53.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:53.375+0000 7fa6c8c4e640 1 -- 172.21.15.5:0/1793560703 wait complete. 2024-03-24T10:53:53.990 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:53.991 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:54.992 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:55.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.094+0000 7f205998a640 1 Processor -- start 2024-03-24T10:53:55.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.094+0000 7f205998a640 1 -- start start 2024-03-24T10:53:55.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.095+0000 7f205998a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f20541543a0 0x7f205415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:55.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.095+0000 7f205998a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f205406bf50 0x7f205406c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:55.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.095+0000 7f205998a640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20540cb2f0 con 0x7f205406d4f0 2024-03-24T10:53:55.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.095+0000 7f205998a640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f20540c6750 con 0x7f205406bf50 2024-03-24T10:53:55.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.095+0000 7f205998a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f20540cc380 con 0x7f20541543a0 2024-03-24T10:53:55.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.096+0000 7f20537fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f205406bf50 0x7f205406c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:55.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.096+0000 7f2052ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f20541543a0 0x7f205415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:55.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.097+0000 7f2052ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f20541543a0 0x7f205415ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:44530/0 (socket says 172.21.15.5:44530) 2024-03-24T10:53:55.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.097+0000 7f2052ffd640 1 -- 172.21.15.5:0/2039752241 learned_addr learned my addr 172.21.15.5:0/2039752241 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:55.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.099+0000 7f2052ffd640 1 -- 172.21.15.5:0/2039752241 >> v2:172.21.15.5:3300/0 conn(0x7f205406bf50 msgr2=0x7f205406c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:55.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.099+0000 7f2052ffd640 1 --2- 172.21.15.5:0/2039752241 >> v2:172.21.15.5:3300/0 conn(0x7f205406bf50 0x7f205406c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:55.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.099+0000 7f2052ffd640 1 -- 172.21.15.5:0/2039752241 >> v1:172.21.15.5:6789/0 conn(0x7f205406d4f0 legacy=0x7f20540cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:55.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.099+0000 7f2052ffd640 1 -- 172.21.15.5:0/2039752241 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20540ca7e0 con 0x7f20541543a0 2024-03-24T10:53:55.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.101+0000 7f2052ffd640 1 --2- 172.21.15.5:0/2039752241 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f20541543a0 0x7f205415ac00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2044009870 tx=0x7f204402ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db7ca1ff38f003fc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:55.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.106+0000 7f20527fc640 1 -- 172.21.15.5:0/2039752241 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204402f840 con 0x7f20541543a0 2024-03-24T10:53:55.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.106+0000 7f20527fc640 1 -- 172.21.15.5:0/2039752241 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f204402f9a0 con 0x7f20541543a0 2024-03-24T10:53:55.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.106+0000 7f20527fc640 1 -- 172.21.15.5:0/2039752241 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204402fd30 con 0x7f20541543a0 2024-03-24T10:53:55.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.107+0000 7f205998a640 1 -- 172.21.15.5:0/2039752241 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f20541543a0 msgr2=0x7f205415ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:55.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.107+0000 7f205998a640 1 --2- 172.21.15.5:0/2039752241 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f20541543a0 0x7f205415ac00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2044009870 tx=0x7f204402ba20 comp rx=0 tx=0).stop 2024-03-24T10:53:55.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.107+0000 7f205998a640 1 -- 172.21.15.5:0/2039752241 shutdown_connections 2024-03-24T10:53:55.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.107+0000 7f205998a640 1 --2- 172.21.15.5:0/2039752241 >> v2:172.21.15.5:3300/0 conn(0x7f205406bf50 0x7f205406c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:55.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.107+0000 7f205998a640 1 --2- 172.21.15.5:0/2039752241 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f20541543a0 0x7f205415ac00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:55.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.107+0000 7f205998a640 1 -- 172.21.15.5:0/2039752241 >> 172.21.15.5:0/2039752241 conn(0x7f2054072ef0 msgr2=0x7f2054061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:55.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.107+0000 7f205998a640 1 -- 172.21.15.5:0/2039752241 shutdown_connections 2024-03-24T10:53:55.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.107+0000 7f205998a640 1 -- 172.21.15.5:0/2039752241 wait complete. 2024-03-24T10:53:55.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.108+0000 7f205998a640 1 Processor -- start 2024-03-24T10:53:55.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.108+0000 7f205998a640 1 -- start start 2024-03-24T10:53:55.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.109+0000 7f205998a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f205406bf50 0x7f205414b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:55.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.109+0000 7f205998a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f205406d4f0 0x7f205414fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:55.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.109+0000 7f205998a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f20541543a0 0x7f205414a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:55.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.109+0000 7f205998a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f205414bf40 con 0x7f205406bf50 2024-03-24T10:53:55.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.109+0000 7f205998a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f205414c990 con 0x7f205406d4f0 2024-03-24T10:53:55.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.109+0000 7f205998a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f20540ce990 con 0x7f20541543a0 2024-03-24T10:53:55.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.109+0000 7f2052ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f205406bf50 0x7f205414b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:55.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.110+0000 7f204bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f205406d4f0 0x7f205414fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:55.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.110+0000 7f20537fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f20541543a0 0x7f205414a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:55.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.110+0000 7f2052ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f205406bf50 0x7f205414b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:45964/0 (socket says 172.21.15.5:45964) 2024-03-24T10:53:55.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.110+0000 7f2052ffd640 1 -- 172.21.15.5:0/1509236139 learned_addr learned my addr 172.21.15.5:0/1509236139 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:55.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.115+0000 7f2052ffd640 1 -- 172.21.15.5:0/1509236139 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f20541543a0 msgr2=0x7f205414a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:55.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.115+0000 7f2052ffd640 1 --2- 172.21.15.5:0/1509236139 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f20541543a0 0x7f205414a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:55.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.115+0000 7f2052ffd640 1 -- 172.21.15.5:0/1509236139 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f205406d4f0 msgr2=0x7f205414fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:55.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.115+0000 7f2052ffd640 1 --2- 172.21.15.5:0/1509236139 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f205406d4f0 0x7f205414fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:55.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.115+0000 7f2052ffd640 1 -- 172.21.15.5:0/1509236139 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20440094d0 con 0x7f205406bf50 2024-03-24T10:53:55.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.119+0000 7f2052ffd640 1 --2- 172.21.15.5:0/1509236139 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f205406bf50 0x7f205414b430 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2044000c00 tx=0x7f2044004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:55.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.124+0000 7f2050ff9640 1 -- 172.21.15.5:0/1509236139 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044037c40 con 0x7f205406bf50 2024-03-24T10:53:55.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.124+0000 7f2050ff9640 1 -- 172.21.15.5:0/1509236139 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2044037da0 con 0x7f205406bf50 2024-03-24T10:53:55.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.124+0000 7f2050ff9640 1 -- 172.21.15.5:0/1509236139 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044041640 con 0x7f205406bf50 2024-03-24T10:53:55.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.124+0000 7f205998a640 1 -- 172.21.15.5:0/1509236139 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2054157500 con 0x7f205406bf50 2024-03-24T10:53:55.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.124+0000 7f205998a640 1 -- 172.21.15.5:0/1509236139 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20540b47c0 con 0x7f205406bf50 2024-03-24T10:53:55.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.125+0000 7f205998a640 1 -- 172.21.15.5:0/1509236139 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20540b4620 con 0x7f205406bf50 2024-03-24T10:53:55.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.403+0000 7f2050ff9640 1 -- 172.21.15.5:0/1509236139 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f20440417a0 con 0x7f205406bf50 2024-03-24T10:53:55.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.403+0000 7f2050ff9640 1 --2- 172.21.15.5:0/1509236139 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f203406b810 0x7f203406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:55.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.403+0000 7f204bfff640 1 --2- 172.21.15.5:0/1509236139 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f203406b810 0x7f203406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:55.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.403+0000 7f204bfff640 1 --2- 172.21.15.5:0/1509236139 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f203406b810 0x7f203406dcd0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f20540ccfd0 tx=0x7f203c009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:55.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.429+0000 7f2050ff9640 1 -- 172.21.15.5:0/1509236139 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f204403c070 con 0x7f205406bf50 2024-03-24T10:53:55.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.429+0000 7f2050ff9640 1 -- 172.21.15.5:0/1509236139 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2044078190 con 0x7f205406bf50 2024-03-24T10:53:55.654 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.654+0000 7f205998a640 1 -- 172.21.15.5:0/1509236139 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20540b4620 con 0x7f205406bf50 2024-03-24T10:53:55.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.689+0000 7f2050ff9640 1 -- 172.21.15.5:0/1509236139 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f204407f890 con 0x7f205406bf50 2024-03-24T10:53:55.689 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:55.689 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:55.689 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:55.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.691+0000 7f205998a640 1 -- 172.21.15.5:0/1509236139 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f203406b810 msgr2=0x7f203406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:55.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.691+0000 7f205998a640 1 --2- 172.21.15.5:0/1509236139 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f203406b810 0x7f203406dcd0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f20540ccfd0 tx=0x7f203c009040 comp rx=0 tx=0).stop 2024-03-24T10:53:55.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.691+0000 7f205998a640 1 -- 172.21.15.5:0/1509236139 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f205406bf50 msgr2=0x7f205414b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:55.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.691+0000 7f205998a640 1 --2- 172.21.15.5:0/1509236139 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f205406bf50 0x7f205414b430 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2044000c00 tx=0x7f2044004730 comp rx=0 tx=0).stop 2024-03-24T10:53:55.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.691+0000 7f205998a640 1 -- 172.21.15.5:0/1509236139 shutdown_connections 2024-03-24T10:53:55.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.691+0000 7f205998a640 1 --2- 172.21.15.5:0/1509236139 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f203406b810 0x7f203406dcd0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:55.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.691+0000 7f205998a640 1 --2- 172.21.15.5:0/1509236139 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f20541543a0 0x7f205414a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:55.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.691+0000 7f205998a640 1 --2- 172.21.15.5:0/1509236139 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f205406d4f0 0x7f205414fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:55.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.691+0000 7f205998a640 1 --2- 172.21.15.5:0/1509236139 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f205406bf50 0x7f205414b430 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:55.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.691+0000 7f205998a640 1 -- 172.21.15.5:0/1509236139 >> 172.21.15.5:0/1509236139 conn(0x7f2054072ef0 msgr2=0x7f2054151b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:55.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.691+0000 7f205998a640 1 -- 172.21.15.5:0/1509236139 shutdown_connections 2024-03-24T10:53:55.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:55.691+0000 7f205998a640 1 -- 172.21.15.5:0/1509236139 wait complete. 2024-03-24T10:53:56.306 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:56.306 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:57.307 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:57.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.409+0000 7f3495125640 1 Processor -- start 2024-03-24T10:53:57.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.409+0000 7f3495125640 1 -- start start 2024-03-24T10:53:57.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.410+0000 7f3495125640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f349015ac00 0x7f34900cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:57.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.410+0000 7f3495125640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34900cb2f0 con 0x7f34900ccaa0 2024-03-24T10:53:57.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.410+0000 7f3495125640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f349015b070 con 0x7f34901543a0 2024-03-24T10:53:57.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.410+0000 7f3495125640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f34900c6750 con 0x7f349015ac00 2024-03-24T10:53:57.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.411+0000 7f348f59e640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f34900ccaa0 0x7f349006bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59820/0 (socket says 172.21.15.5:59820) 2024-03-24T10:53:57.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.411+0000 7f348f59e640 1 -- 172.21.15.5:0/859170167 learned_addr learned my addr 172.21.15.5:0/859170167 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:53:57.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.411+0000 7f348e59c640 1 --2- 172.21.15.5:0/859170167 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f349015ac00 0x7f34900cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:57.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.413+0000 7f348dd9b640 1 -- 172.21.15.5:0/859170167 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 946876947 0 0) 0x7f34900cb2f0 con 0x7f34900ccaa0 2024-03-24T10:53:57.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.413+0000 7f348dd9b640 1 -- 172.21.15.5:0/859170167 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3478003520 con 0x7f34900ccaa0 2024-03-24T10:53:57.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.414+0000 7f348dd9b640 1 -- 172.21.15.5:0/859170167 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1176507007 0 0) 0x7f349015b070 con 0x7f34901543a0 2024-03-24T10:53:57.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.414+0000 7f348dd9b640 1 -- 172.21.15.5:0/859170167 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34900cb2f0 con 0x7f34901543a0 2024-03-24T10:53:57.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.415+0000 7f348e59c640 1 -- 172.21.15.5:0/859170167 >> v1:172.21.15.57:6789/0 conn(0x7f34901543a0 legacy=0x7f349006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:57.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.415+0000 7f348e59c640 1 -- 172.21.15.5:0/859170167 >> v1:172.21.15.5:6789/0 conn(0x7f34900ccaa0 legacy=0x7f349006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:57.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.415+0000 7f348e59c640 1 -- 172.21.15.5:0/859170167 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34900ca7e0 con 0x7f349015ac00 2024-03-24T10:53:57.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.418+0000 7f348e59c640 1 --2- 172.21.15.5:0/859170167 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f349015ac00 0x7f34900cc560 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f34840097d0 tx=0x7f348402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=721bf7d64bea24cc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:57.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.423+0000 7f348dd9b640 1 -- 172.21.15.5:0/859170167 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3484031890 con 0x7f349015ac00 2024-03-24T10:53:57.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.423+0000 7f348dd9b640 1 -- 172.21.15.5:0/859170167 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34840319f0 con 0x7f349015ac00 2024-03-24T10:53:57.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.423+0000 7f348dd9b640 1 -- 172.21.15.5:0/859170167 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3484031d80 con 0x7f349015ac00 2024-03-24T10:53:57.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.424+0000 7f3495125640 1 -- 172.21.15.5:0/859170167 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f349015ac00 msgr2=0x7f34900cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:57.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.424+0000 7f3495125640 1 --2- 172.21.15.5:0/859170167 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f349015ac00 0x7f34900cc560 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f34840097d0 tx=0x7f348402b930 comp rx=0 tx=0).stop 2024-03-24T10:53:57.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.424+0000 7f3495125640 1 -- 172.21.15.5:0/859170167 shutdown_connections 2024-03-24T10:53:57.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.424+0000 7f3495125640 1 --2- 172.21.15.5:0/859170167 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f349015ac00 0x7f34900cc560 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:57.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.424+0000 7f3495125640 1 -- 172.21.15.5:0/859170167 >> 172.21.15.5:0/859170167 conn(0x7f3490072ef0 msgr2=0x7f3490061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:57.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.424+0000 7f3495125640 1 -- 172.21.15.5:0/859170167 shutdown_connections 2024-03-24T10:53:57.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.424+0000 7f3495125640 1 -- 172.21.15.5:0/859170167 wait complete. 2024-03-24T10:53:57.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.426+0000 7f3495125640 1 Processor -- start 2024-03-24T10:53:57.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.426+0000 7f3495125640 1 -- start start 2024-03-24T10:53:57.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.427+0000 7f3495125640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f34900ccaa0 0x7f34901596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:57.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.427+0000 7f3495125640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f34901543a0 0x7f3490159c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:57.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.427+0000 7f3495125640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f349015ac00 0x7f349015d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:57.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.427+0000 7f3495125640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f34901f01e0 con 0x7f34900ccaa0 2024-03-24T10:53:57.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.427+0000 7f3495125640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f34901f04c0 con 0x7f349015ac00 2024-03-24T10:53:57.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.427+0000 7f3495125640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f34901f07a0 con 0x7f34901543a0 2024-03-24T10:53:57.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.428+0000 7f348ed9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f34900ccaa0 0x7f34901596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:57.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.428+0000 7f348e59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f34901543a0 0x7f3490159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:57.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.428+0000 7f348f59e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f349015ac00 0x7f349015d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:57.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.428+0000 7f348ed9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f34900ccaa0 0x7f34901596c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:45970/0 (socket says 172.21.15.5:45970) 2024-03-24T10:53:57.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.428+0000 7f348ed9d640 1 -- 172.21.15.5:0/888130502 learned_addr learned my addr 172.21.15.5:0/888130502 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:57.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.432+0000 7f348ed9d640 1 -- 172.21.15.5:0/888130502 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f34901543a0 msgr2=0x7f3490159c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:57.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.432+0000 7f348ed9d640 1 --2- 172.21.15.5:0/888130502 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f34901543a0 0x7f3490159c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:57.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.432+0000 7f348ed9d640 1 -- 172.21.15.5:0/888130502 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f349015ac00 msgr2=0x7f349015d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:57.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.432+0000 7f348ed9d640 1 --2- 172.21.15.5:0/888130502 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f349015ac00 0x7f349015d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:57.433 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.432+0000 7f348ed9d640 1 -- 172.21.15.5:0/888130502 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3484009400 con 0x7f34900ccaa0 2024-03-24T10:53:57.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.435+0000 7f348ed9d640 1 --2- 172.21.15.5:0/888130502 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f34900ccaa0 0x7f34901596c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f348000c490 tx=0x7f348000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:57.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.439+0000 7f34777fe640 1 -- 172.21.15.5:0/888130502 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34800052d0 con 0x7f34900ccaa0 2024-03-24T10:53:57.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.439+0000 7f34777fe640 1 -- 172.21.15.5:0/888130502 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3480005430 con 0x7f34900ccaa0 2024-03-24T10:53:57.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.439+0000 7f34777fe640 1 -- 172.21.15.5:0/888130502 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3480012870 con 0x7f34900ccaa0 2024-03-24T10:53:57.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.440+0000 7f3495125640 1 -- 172.21.15.5:0/888130502 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34901f0a80 con 0x7f34900ccaa0 2024-03-24T10:53:57.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.440+0000 7f3495125640 1 -- 172.21.15.5:0/888130502 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34901591a0 con 0x7f34900ccaa0 2024-03-24T10:53:57.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.441+0000 7f3495125640 1 -- 172.21.15.5:0/888130502 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34900681d0 con 0x7f34900ccaa0 2024-03-24T10:53:57.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.738+0000 7f34777fe640 1 -- 172.21.15.5:0/888130502 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f34800129d0 con 0x7f34900ccaa0 2024-03-24T10:53:57.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.739+0000 7f34777fe640 1 --2- 172.21.15.5:0/888130502 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f347c06b810 0x7f347c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:57.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.739+0000 7f348e59c640 1 --2- 172.21.15.5:0/888130502 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f347c06b810 0x7f347c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:57.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.739+0000 7f348e59c640 1 --2- 172.21.15.5:0/888130502 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f347c06b810 0x7f347c06dcd0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3484036000 tx=0x7f348403c040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:57.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.762+0000 7f34777fe640 1 -- 172.21.15.5:0/888130502 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f3480058fc0 con 0x7f34900ccaa0 2024-03-24T10:53:57.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.762+0000 7f34777fe640 1 -- 172.21.15.5:0/888130502 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3480088e60 con 0x7f34900ccaa0 2024-03-24T10:53:57.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:57.990+0000 7f3495125640 1 -- 172.21.15.5:0/888130502 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f349015a470 con 0x7f34900ccaa0 2024-03-24T10:53:58.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:58.016+0000 7f34777fe640 1 -- 172.21.15.5:0/888130502 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f3480058b00 con 0x7f34900ccaa0 2024-03-24T10:53:58.016 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:53:58.016 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:53:58.016 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:53:58.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:58.018+0000 7f3495125640 1 -- 172.21.15.5:0/888130502 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f347c06b810 msgr2=0x7f347c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:58.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:58.018+0000 7f3495125640 1 --2- 172.21.15.5:0/888130502 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f347c06b810 0x7f347c06dcd0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3484036000 tx=0x7f348403c040 comp rx=0 tx=0).stop 2024-03-24T10:53:58.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:58.018+0000 7f3495125640 1 -- 172.21.15.5:0/888130502 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f34900ccaa0 msgr2=0x7f34901596c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:58.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:58.018+0000 7f3495125640 1 --2- 172.21.15.5:0/888130502 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f34900ccaa0 0x7f34901596c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f348000c490 tx=0x7f348000c950 comp rx=0 tx=0).stop 2024-03-24T10:53:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:58.018+0000 7f3495125640 1 -- 172.21.15.5:0/888130502 shutdown_connections 2024-03-24T10:53:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:58.018+0000 7f3495125640 1 --2- 172.21.15.5:0/888130502 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f347c06b810 0x7f347c06dcd0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:58.018+0000 7f3495125640 1 --2- 172.21.15.5:0/888130502 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f349015ac00 0x7f349015d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:58.018+0000 7f3495125640 1 --2- 172.21.15.5:0/888130502 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f34901543a0 0x7f3490159c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:58.018+0000 7f3495125640 1 --2- 172.21.15.5:0/888130502 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f34900ccaa0 0x7f34901596c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:58.018+0000 7f3495125640 1 -- 172.21.15.5:0/888130502 >> 172.21.15.5:0/888130502 conn(0x7f3490072ef0 msgr2=0x7f349006be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:58.018+0000 7f3495125640 1 -- 172.21.15.5:0/888130502 shutdown_connections 2024-03-24T10:53:58.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:58.018+0000 7f3495125640 1 -- 172.21.15.5:0/888130502 wait complete. 2024-03-24T10:53:58.635 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:53:58.635 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:53:59.154 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:09:50.655 36946== Warning: unimplemented fcntl command: 1036 2024-03-24T10:53:59.158 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:09:50.659 36946== Warning: unimplemented fcntl command: 1036 2024-03-24T10:53:59.176 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:09:50.640 36958== Warning: unimplemented fcntl command: 1036 2024-03-24T10:53:59.177 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:09:50.641 36957== Warning: unimplemented fcntl command: 1036 2024-03-24T10:53:59.179 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:09:50.643 36958== Warning: unimplemented fcntl command: 1036 2024-03-24T10:53:59.182 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:09:50.645 36957== Warning: unimplemented fcntl command: 1036 2024-03-24T10:53:59.218 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:09:50.719 36946== Warning: unimplemented fcntl command: 1036 2024-03-24T10:53:59.241 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:09:50.705 36958== Warning: unimplemented fcntl command: 1036 2024-03-24T10:53:59.256 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:09:50.720 36957== Warning: unimplemented fcntl command: 1036 2024-03-24T10:53:59.636 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:53:59.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.739+0000 7fd67f79a640 1 Processor -- start 2024-03-24T10:53:59.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.740+0000 7fd67f79a640 1 -- start start 2024-03-24T10:53:59.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.740+0000 7fd67f79a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd6780cae90 0x7fd6780cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:59.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.740+0000 7fd67f79a640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd6780cb270 0x7fd6780cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:59.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.740+0000 7fd67f79a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6780ccba0 con 0x7fd6780cd960 2024-03-24T10:53:59.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.740+0000 7fd67f79a640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fd6780cce40 con 0x7fd6780cae90 2024-03-24T10:53:59.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.740+0000 7fd67f79a640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fd67815a590 con 0x7fd6780cb270 2024-03-24T10:53:59.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.741+0000 7fd67d536640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd6780cae90 0x7fd6780cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:59.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.741+0000 7fd67d536640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd6780cae90 0x7fd6780cf530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:45976/0 (socket says 172.21.15.5:45976) 2024-03-24T10:53:59.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.741+0000 7fd67d536640 1 -- 172.21.15.5:0/2450292932 learned_addr learned my addr 172.21.15.5:0/2450292932 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:59.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.741+0000 7fd67cd35640 1 --2- 172.21.15.5:0/2450292932 >> v2:172.21.15.57:3300/0 conn(0x7fd6780cb270 0x7fd6780cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:59.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.744+0000 7fd67d536640 1 -- 172.21.15.5:0/2450292932 >> v2:172.21.15.57:3300/0 conn(0x7fd6780cb270 msgr2=0x7fd6780cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:53:59.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.744+0000 7fd67d536640 1 --2- 172.21.15.5:0/2450292932 >> v2:172.21.15.57:3300/0 conn(0x7fd6780cb270 0x7fd6780cc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:59.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.744+0000 7fd67d536640 1 -- 172.21.15.5:0/2450292932 >> v1:172.21.15.57:6789/0 conn(0x7fd6780cd960 legacy=0x7fd6780cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:59.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.744+0000 7fd67d536640 1 -- 172.21.15.5:0/2450292932 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd67815a870 con 0x7fd6780cae90 2024-03-24T10:53:59.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.746+0000 7fd67d536640 1 --2- 172.21.15.5:0/2450292932 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd6780cae90 0x7fd6780cf530 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd6780cd680 tx=0x7fd67402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48682904bb1a79ab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:59.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.752+0000 7fd66ffff640 1 -- 172.21.15.5:0/2450292932 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67402f850 con 0x7fd6780cae90 2024-03-24T10:53:59.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.752+0000 7fd66ffff640 1 -- 172.21.15.5:0/2450292932 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd67402f9b0 con 0x7fd6780cae90 2024-03-24T10:53:59.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.752+0000 7fd66ffff640 1 -- 172.21.15.5:0/2450292932 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67402fd40 con 0x7fd6780cae90 2024-03-24T10:53:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.752+0000 7fd67f79a640 1 -- 172.21.15.5:0/2450292932 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd6780cae90 msgr2=0x7fd6780cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.752+0000 7fd67f79a640 1 --2- 172.21.15.5:0/2450292932 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd6780cae90 0x7fd6780cf530 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd6780cd680 tx=0x7fd67402ba30 comp rx=0 tx=0).stop 2024-03-24T10:53:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.753+0000 7fd67f79a640 1 -- 172.21.15.5:0/2450292932 shutdown_connections 2024-03-24T10:53:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.753+0000 7fd67f79a640 1 --2- 172.21.15.5:0/2450292932 >> v2:172.21.15.57:3300/0 conn(0x7fd6780cb270 0x7fd6780cc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.753+0000 7fd67f79a640 1 --2- 172.21.15.5:0/2450292932 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd6780cae90 0x7fd6780cf530 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.753+0000 7fd67f79a640 1 -- 172.21.15.5:0/2450292932 >> 172.21.15.5:0/2450292932 conn(0x7fd6780cfb80 msgr2=0x7fd6780cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:53:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.753+0000 7fd67f79a640 1 -- 172.21.15.5:0/2450292932 shutdown_connections 2024-03-24T10:53:59.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.753+0000 7fd67f79a640 1 -- 172.21.15.5:0/2450292932 wait complete. 2024-03-24T10:53:59.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.754+0000 7fd67f79a640 1 Processor -- start 2024-03-24T10:53:59.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.754+0000 7fd67f79a640 1 -- start start 2024-03-24T10:53:59.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.755+0000 7fd67f79a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd6780cae90 0x7fd6781f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:59.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.755+0000 7fd67f79a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd6780cb270 0x7fd6781e0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:59.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.755+0000 7fd67f79a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6780cd960 0x7fd6780c3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:53:59.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.755+0000 7fd67f79a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd6780c53f0 con 0x7fd6780cb270 2024-03-24T10:53:59.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.755+0000 7fd67f79a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd6781f44c0 con 0x7fd6780cd960 2024-03-24T10:53:59.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.755+0000 7fd67f79a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd6780c3620 con 0x7fd6780cae90 2024-03-24T10:53:59.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.755+0000 7fd67d536640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd6780cae90 0x7fd6781f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:59.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.756+0000 7fd67d536640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd6780cae90 0x7fd6781f3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:44572/0 (socket says 172.21.15.5:44572) 2024-03-24T10:53:59.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.756+0000 7fd67d536640 1 -- 172.21.15.5:0/1766177979 learned_addr learned my addr 172.21.15.5:0/1766177979 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:53:59.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.756+0000 7fd67dd37640 1 --2- 172.21.15.5:0/1766177979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6780cd960 0x7fd6780c3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:59.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.758+0000 7fd67cd35640 1 --2- 172.21.15.5:0/1766177979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd6780cb270 0x7fd6781e0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:53:59.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.759+0000 7fd67d536640 1 -- 172.21.15.5:0/1766177979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6780cd960 msgr2=0x7fd6780c3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:59.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.759+0000 7fd67d536640 1 --2- 172.21.15.5:0/1766177979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6780cd960 0x7fd6780c3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:59.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.759+0000 7fd67d536640 1 -- 172.21.15.5:0/1766177979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd6780cb270 msgr2=0x7fd6781e0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:53:59.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.759+0000 7fd67d536640 1 --2- 172.21.15.5:0/1766177979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd6780cb270 0x7fd6781e0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:53:59.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.759+0000 7fd67d536640 1 -- 172.21.15.5:0/1766177979 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6740094d0 con 0x7fd6780cae90 2024-03-24T10:53:59.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.763+0000 7fd67d536640 1 --2- 172.21.15.5:0/1766177979 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd6780cae90 0x7fd6781f3f80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd674000c00 tx=0x7fd674004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:53:59.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.768+0000 7fd66dffb640 1 -- 172.21.15.5:0/1766177979 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd674037c40 con 0x7fd6780cae90 2024-03-24T10:53:59.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.768+0000 7fd66dffb640 1 -- 172.21.15.5:0/1766177979 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd674037dc0 con 0x7fd6780cae90 2024-03-24T10:53:59.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.768+0000 7fd67f79a640 1 -- 172.21.15.5:0/1766177979 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6781f0050 con 0x7fd6780cae90 2024-03-24T10:53:59.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.768+0000 7fd66dffb640 1 -- 172.21.15.5:0/1766177979 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6740426c0 con 0x7fd6780cae90 2024-03-24T10:53:59.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.769+0000 7fd67f79a640 1 -- 172.21.15.5:0/1766177979 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6781f05f0 con 0x7fd6780cae90 2024-03-24T10:53:59.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:53:59.769+0000 7fd67f79a640 1 -- 172.21.15.5:0/1766177979 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd67806d5a0 con 0x7fd6780cae90 2024-03-24T10:54:00.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:00.084+0000 7fd66dffb640 1 -- 172.21.15.5:0/1766177979 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd67404c440 con 0x7fd6780cae90 2024-03-24T10:54:00.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:00.084+0000 7fd66dffb640 1 --2- 172.21.15.5:0/1766177979 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd65c06b860 0x7fd65c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:00.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:00.084+0000 7fd67cd35640 1 --2- 172.21.15.5:0/1766177979 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd65c06b860 0x7fd65c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:00.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:00.085+0000 7fd67cd35640 1 --2- 172.21.15.5:0/1766177979 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd65c06b860 0x7fd65c06dd20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd6781e7510 tx=0x7fd668009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:00.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:00.090+0000 7fd66dffb640 1 -- 172.21.15.5:0/1766177979 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fd67403c070 con 0x7fd6780cae90 2024-03-24T10:54:00.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:00.109+0000 7fd66dffb640 1 -- 172.21.15.5:0/1766177979 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd674077730 con 0x7fd6780cae90 2024-03-24T10:54:00.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:00.350+0000 7fd67f79a640 1 -- 172.21.15.5:0/1766177979 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6780681d0 con 0x7fd6780cae90 2024-03-24T10:54:00.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:00.420+0000 7fd66dffb640 1 -- 172.21.15.5:0/1766177979 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fd674047030 con 0x7fd6780cae90 2024-03-24T10:54:00.422 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:00.422 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:00.422 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:00.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:00.422+0000 7fd67f79a640 1 -- 172.21.15.5:0/1766177979 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd65c06b860 msgr2=0x7fd65c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:00.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:00.422+0000 7fd67f79a640 1 --2- 172.21.15.5:0/1766177979 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd65c06b860 0x7fd65c06dd20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd6781e7510 tx=0x7fd668009190 comp rx=0 tx=0).stop 2024-03-24T10:54:00.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:00.423+0000 7fd67f79a640 1 -- 172.21.15.5:0/1766177979 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd6780cae90 msgr2=0x7fd6781f3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:00.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:00.423+0000 7fd67f79a640 1 --2- 172.21.15.5:0/1766177979 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd6780cae90 0x7fd6781f3f80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd674000c00 tx=0x7fd674004730 comp rx=0 tx=0).stop 2024-03-24T10:54:00.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:00.423+0000 7fd67f79a640 1 -- 172.21.15.5:0/1766177979 shutdown_connections 2024-03-24T10:54:00.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:00.423+0000 7fd67f79a640 1 --2- 172.21.15.5:0/1766177979 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd65c06b860 0x7fd65c06dd20 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:00.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:00.423+0000 7fd67f79a640 1 --2- 172.21.15.5:0/1766177979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd6780cb270 0x7fd6781e0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:00.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:00.423+0000 7fd67f79a640 1 --2- 172.21.15.5:0/1766177979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd6780cd960 0x7fd6780c3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:00.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:00.423+0000 7fd67f79a640 1 --2- 172.21.15.5:0/1766177979 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd6780cae90 0x7fd6781f3f80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:00.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:00.423+0000 7fd67f79a640 1 -- 172.21.15.5:0/1766177979 >> 172.21.15.5:0/1766177979 conn(0x7fd6780cfb80 msgr2=0x7fd6780c8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:00.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:00.423+0000 7fd67f79a640 1 -- 172.21.15.5:0/1766177979 shutdown_connections 2024-03-24T10:54:00.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:00.423+0000 7fd67f79a640 1 -- 172.21.15.5:0/1766177979 wait complete. 2024-03-24T10:54:01.038 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:01.038 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:02.039 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:02.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.141+0000 7fdab55a0640 1 Processor -- start 2024-03-24T10:54:02.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.141+0000 7fdab55a0640 1 -- start start 2024-03-24T10:54:02.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.142+0000 7fdab55a0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fdab0070120 0x7fdab0165ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:02.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.142+0000 7fdab55a0640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdab015a670 con 0x7fdab015a130 2024-03-24T10:54:02.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.142+0000 7fdab55a0640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdab015a930 con 0x7fdab01541b0 2024-03-24T10:54:02.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.142+0000 7fdab55a0640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fdab015ac10 con 0x7fdab0070120 2024-03-24T10:54:02.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.142+0000 7fdaaf7fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fdab015a130 0x7fdab00cda00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59828/0 (socket says 172.21.15.5:59828) 2024-03-24T10:54:02.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.142+0000 7fdaaf7fe640 1 -- 172.21.15.5:0/2786295129 learned_addr learned my addr 172.21.15.5:0/2786295129 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:54:02.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.142+0000 7fdab4d9f640 1 --2- 172.21.15.5:0/2786295129 >> v2:172.21.15.57:3300/0 conn(0x7fdab0070120 0x7fdab0165ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:02.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.145+0000 7fdaaeffd640 1 -- 172.21.15.5:0/2786295129 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1755204247 0 0) 0x7fdab015a670 con 0x7fdab015a130 2024-03-24T10:54:02.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.146+0000 7fdaaeffd640 1 -- 172.21.15.5:0/2786295129 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda98003520 con 0x7fdab015a130 2024-03-24T10:54:02.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.146+0000 7fdab4d9f640 1 -- 172.21.15.5:0/2786295129 >> v1:172.21.15.57:6789/0 conn(0x7fdab01541b0 legacy=0x7fdab015dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:02.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.146+0000 7fdab4d9f640 1 -- 172.21.15.5:0/2786295129 >> v1:172.21.15.5:6789/0 conn(0x7fdab015a130 legacy=0x7fdab00cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:02.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.146+0000 7fdab4d9f640 1 -- 172.21.15.5:0/2786295129 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab015aef0 con 0x7fdab0070120 2024-03-24T10:54:02.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.150+0000 7fdab4d9f640 1 --2- 172.21.15.5:0/2786295129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdab0070120 0x7fdab0165ac0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8009d90 tx=0x7fdaa802be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2750a78fa34bcd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:02.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.154+0000 7fdaaeffd640 1 -- 172.21.15.5:0/2786295129 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa802cbd0 con 0x7fdab0070120 2024-03-24T10:54:02.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.157+0000 7fdaaeffd640 1 -- 172.21.15.5:0/2786295129 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa802cd30 con 0x7fdab0070120 2024-03-24T10:54:02.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.157+0000 7fdaaeffd640 1 -- 172.21.15.5:0/2786295129 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa802cea0 con 0x7fdab0070120 2024-03-24T10:54:02.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.157+0000 7fdab55a0640 1 -- 172.21.15.5:0/2786295129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdab0070120 msgr2=0x7fdab0165ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:02.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.157+0000 7fdab55a0640 1 --2- 172.21.15.5:0/2786295129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdab0070120 0x7fdab0165ac0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8009d90 tx=0x7fdaa802be20 comp rx=0 tx=0).stop 2024-03-24T10:54:02.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.157+0000 7fdab55a0640 1 -- 172.21.15.5:0/2786295129 shutdown_connections 2024-03-24T10:54:02.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.157+0000 7fdab55a0640 1 --2- 172.21.15.5:0/2786295129 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdab0070120 0x7fdab0165ac0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:02.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.157+0000 7fdab55a0640 1 -- 172.21.15.5:0/2786295129 >> 172.21.15.5:0/2786295129 conn(0x7fdab0061d30 msgr2=0x7fdab0062140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:02.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.157+0000 7fdab55a0640 1 -- 172.21.15.5:0/2786295129 shutdown_connections 2024-03-24T10:54:02.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.157+0000 7fdab55a0640 1 -- 172.21.15.5:0/2786295129 wait complete. 2024-03-24T10:54:02.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.159+0000 7fdab55a0640 1 Processor -- start 2024-03-24T10:54:02.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.159+0000 7fdab55a0640 1 -- start start 2024-03-24T10:54:02.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.159+0000 7fdab55a0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdab0070120 0x7fdab0152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:02.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.159+0000 7fdab55a0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdab01541b0 0x7fdab00cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:02.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.159+0000 7fdab55a0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdab015a130 0x7fdab01f2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:02.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.159+0000 7fdab55a0640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fdab0152470 con 0x7fdab015a130 2024-03-24T10:54:02.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.159+0000 7fdab55a0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fdab00ca490 con 0x7fdab0070120 2024-03-24T10:54:02.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.159+0000 7fdab55a0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fdab0150970 con 0x7fdab01541b0 2024-03-24T10:54:02.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.160+0000 7fdab4d9f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdab015a130 0x7fdab01f2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:02.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.160+0000 7fdaaffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdab0070120 0x7fdab0152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:02.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.160+0000 7fdaaf7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdab01541b0 0x7fdab00cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:02.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.160+0000 7fdab4d9f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdab015a130 0x7fdab01f2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:45988/0 (socket says 172.21.15.5:45988) 2024-03-24T10:54:02.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.160+0000 7fdab4d9f640 1 -- 172.21.15.5:0/494298694 learned_addr learned my addr 172.21.15.5:0/494298694 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:02.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.164+0000 7fdab4d9f640 1 -- 172.21.15.5:0/494298694 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdab01541b0 msgr2=0x7fdab00cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:02.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.164+0000 7fdab4d9f640 1 --2- 172.21.15.5:0/494298694 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdab01541b0 0x7fdab00cc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:02.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.164+0000 7fdab4d9f640 1 -- 172.21.15.5:0/494298694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdab0070120 msgr2=0x7fdab0152a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:02.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.164+0000 7fdab4d9f640 1 --2- 172.21.15.5:0/494298694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdab0070120 0x7fdab0152a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:02.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.164+0000 7fdab4d9f640 1 -- 172.21.15.5:0/494298694 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa8009a70 con 0x7fdab015a130 2024-03-24T10:54:02.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.166+0000 7fdab4d9f640 1 --2- 172.21.15.5:0/494298694 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdab015a130 0x7fdab01f2cc0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8037040 tx=0x7fdaa8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:02.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.171+0000 7fdaacff9640 1 -- 172.21.15.5:0/494298694 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa802cea0 con 0x7fdab015a130 2024-03-24T10:54:02.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.171+0000 7fdaacff9640 1 -- 172.21.15.5:0/494298694 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa8004820 con 0x7fdab015a130 2024-03-24T10:54:02.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.171+0000 7fdab55a0640 1 -- 172.21.15.5:0/494298694 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdab0150c50 con 0x7fdab015a130 2024-03-24T10:54:02.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.171+0000 7fdaacff9640 1 -- 172.21.15.5:0/494298694 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8034680 con 0x7fdab015a130 2024-03-24T10:54:02.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.171+0000 7fdab55a0640 1 -- 172.21.15.5:0/494298694 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdab0151210 con 0x7fdab015a130 2024-03-24T10:54:02.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.172+0000 7fdab55a0640 1 -- 172.21.15.5:0/494298694 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdab0068d10 con 0x7fdab015a130 2024-03-24T10:54:02.341 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:54:02.344 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:54:02.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.465+0000 7fdaacff9640 1 -- 172.21.15.5:0/494298694 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdaa80347e0 con 0x7fdab015a130 2024-03-24T10:54:02.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.466+0000 7fdaacff9640 1 --2- 172.21.15.5:0/494298694 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda9c06b860 0x7fda9c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:02.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.466+0000 7fdaaffff640 1 --2- 172.21.15.5:0/494298694 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda9c06b860 0x7fda9c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:02.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.466+0000 7fdaaffff640 1 --2- 172.21.15.5:0/494298694 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda9c06b860 0x7fda9c06dd20 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0000f60 tx=0x7fdaa0009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:02.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.490+0000 7fdaacff9640 1 -- 172.21.15.5:0/494298694 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fdaa807b050 con 0x7fdab015a130 2024-03-24T10:54:02.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.490+0000 7fdaacff9640 1 -- 172.21.15.5:0/494298694 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdaa80aae60 con 0x7fdab015a130 2024-03-24T10:54:02.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.716+0000 7fdab55a0640 1 -- 172.21.15.5:0/494298694 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdab0151400 con 0x7fdab015a130 2024-03-24T10:54:02.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.733+0000 7fdaacff9640 1 -- 172.21.15.5:0/494298694 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fdaa8039090 con 0x7fdab015a130 2024-03-24T10:54:02.733 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:02.733 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:02.733 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:02.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.735+0000 7fdab55a0640 1 -- 172.21.15.5:0/494298694 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda9c06b860 msgr2=0x7fda9c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:02.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.735+0000 7fdab55a0640 1 --2- 172.21.15.5:0/494298694 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda9c06b860 0x7fda9c06dd20 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fdaa0000f60 tx=0x7fdaa0009040 comp rx=0 tx=0).stop 2024-03-24T10:54:02.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.735+0000 7fdab55a0640 1 -- 172.21.15.5:0/494298694 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdab015a130 msgr2=0x7fdab01f2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:02.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.735+0000 7fdab55a0640 1 --2- 172.21.15.5:0/494298694 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdab015a130 0x7fdab01f2cc0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8037040 tx=0x7fdaa8004a40 comp rx=0 tx=0).stop 2024-03-24T10:54:02.735 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.735+0000 7fdab55a0640 1 -- 172.21.15.5:0/494298694 shutdown_connections 2024-03-24T10:54:02.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.735+0000 7fdab55a0640 1 --2- 172.21.15.5:0/494298694 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda9c06b860 0x7fda9c06dd20 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:02.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.735+0000 7fdab55a0640 1 --2- 172.21.15.5:0/494298694 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdab015a130 0x7fdab01f2cc0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:02.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.735+0000 7fdab55a0640 1 --2- 172.21.15.5:0/494298694 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdab01541b0 0x7fdab00cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:02.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.735+0000 7fdab55a0640 1 --2- 172.21.15.5:0/494298694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdab0070120 0x7fdab0152a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:02.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.735+0000 7fdab55a0640 1 -- 172.21.15.5:0/494298694 >> 172.21.15.5:0/494298694 conn(0x7fdab0061d30 msgr2=0x7fdab0062110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:02.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.735+0000 7fdab55a0640 1 -- 172.21.15.5:0/494298694 shutdown_connections 2024-03-24T10:54:02.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:02.735+0000 7fdab55a0640 1 -- 172.21.15.5:0/494298694 wait complete. 2024-03-24T10:54:03.350 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:03.350 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:04.351 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:04.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.464+0000 7f1ac94eb640 1 Processor -- start 2024-03-24T10:54:04.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.464+0000 7f1ac94eb640 1 -- start start 2024-03-24T10:54:04.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.465+0000 7f1ac94eb640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1ac406bf50 0x7f1ac406c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:04.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.465+0000 7f1ac94eb640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ac40cb2f0 con 0x7f1ac415ac00 2024-03-24T10:54:04.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.465+0000 7f1ac94eb640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ac415b070 con 0x7f1ac41543a0 2024-03-24T10:54:04.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.465+0000 7f1ac94eb640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f1ac40c6750 con 0x7f1ac406bf50 2024-03-24T10:54:04.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.465+0000 7f1ac37fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1ac406bf50 0x7f1ac406c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:04.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.466+0000 7f1ac2ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1ac41543a0 0x7f1ac406d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:33336/0 (socket says 172.21.15.5:33336) 2024-03-24T10:54:04.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.466+0000 7f1ac2ffd640 1 -- 172.21.15.5:0/265144328 learned_addr learned my addr 172.21.15.5:0/265144328 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:54:04.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.469+0000 7f1ac1ffb640 1 -- 172.21.15.5:0/265144328 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1100264575 0 0) 0x7f1ac415b070 con 0x7f1ac41543a0 2024-03-24T10:54:04.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.469+0000 7f1ac1ffb640 1 -- 172.21.15.5:0/265144328 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1aac003520 con 0x7f1ac41543a0 2024-03-24T10:54:04.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.469+0000 7f1ac37fe640 1 -- 172.21.15.5:0/265144328 >> v1:172.21.15.57:6789/0 conn(0x7f1ac41543a0 legacy=0x7f1ac406d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:04.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.469+0000 7f1ac37fe640 1 -- 172.21.15.5:0/265144328 >> v1:172.21.15.5:6789/0 conn(0x7f1ac415ac00 legacy=0x7f1ac40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:04.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.469+0000 7f1ac37fe640 1 -- 172.21.15.5:0/265144328 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ac40ca7e0 con 0x7f1ac406bf50 2024-03-24T10:54:04.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.474+0000 7f1ac37fe640 1 --2- 172.21.15.5:0/265144328 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ac406bf50 0x7f1ac406c330 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1abc009d90 tx=0x7f1abc02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff5cd400517c5c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:04.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.494+0000 7f1ac1ffb640 1 -- 172.21.15.5:0/265144328 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc02cd00 con 0x7f1ac406bf50 2024-03-24T10:54:04.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.494+0000 7f1ac1ffb640 1 -- 172.21.15.5:0/265144328 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1abc02ce60 con 0x7f1ac406bf50 2024-03-24T10:54:04.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.494+0000 7f1ac1ffb640 1 -- 172.21.15.5:0/265144328 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc034660 con 0x7f1ac406bf50 2024-03-24T10:54:04.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.494+0000 7f1ac94eb640 1 -- 172.21.15.5:0/265144328 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ac406bf50 msgr2=0x7f1ac406c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:04.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.494+0000 7f1ac94eb640 1 --2- 172.21.15.5:0/265144328 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ac406bf50 0x7f1ac406c330 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f1abc009d90 tx=0x7f1abc02bf50 comp rx=0 tx=0).stop 2024-03-24T10:54:04.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.495+0000 7f1ac94eb640 1 -- 172.21.15.5:0/265144328 shutdown_connections 2024-03-24T10:54:04.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.495+0000 7f1ac94eb640 1 --2- 172.21.15.5:0/265144328 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ac406bf50 0x7f1ac406c330 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:04.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.495+0000 7f1ac94eb640 1 -- 172.21.15.5:0/265144328 >> 172.21.15.5:0/265144328 conn(0x7f1ac4072ef0 msgr2=0x7f1ac4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:04.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.495+0000 7f1ac94eb640 1 -- 172.21.15.5:0/265144328 shutdown_connections 2024-03-24T10:54:04.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.495+0000 7f1ac94eb640 1 -- 172.21.15.5:0/265144328 wait complete. 2024-03-24T10:54:04.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.497+0000 7f1ac94eb640 1 Processor -- start 2024-03-24T10:54:04.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.497+0000 7f1ac94eb640 1 -- start start 2024-03-24T10:54:04.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.498+0000 7f1ac94eb640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1ac406bf50 0x7f1ac414cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:04.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.498+0000 7f1ac94eb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ac41543a0 0x7f1ac40c9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:04.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.498+0000 7f1ac94eb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ac415ac00 0x7f1ac40c9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:04.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.498+0000 7f1ac94eb640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac40c7e00 con 0x7f1ac415ac00 2024-03-24T10:54:04.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.498+0000 7f1ac94eb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac414c1d0 con 0x7f1ac41543a0 2024-03-24T10:54:04.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.498+0000 7f1ac94eb640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1ac40c7500 con 0x7f1ac406bf50 2024-03-24T10:54:04.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.498+0000 7f1ac37fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ac415ac00 0x7f1ac40c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:04.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.499+0000 7f1ac2ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1ac406bf50 0x7f1ac414cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:04.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.499+0000 7f1ac37fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ac415ac00 0x7f1ac40c9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:42398/0 (socket says 172.21.15.5:42398) 2024-03-24T10:54:04.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.499+0000 7f1ac37fe640 1 -- 172.21.15.5:0/1753192120 learned_addr learned my addr 172.21.15.5:0/1753192120 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:04.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.499+0000 7f1ac27fc640 1 --2- 172.21.15.5:0/1753192120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ac41543a0 0x7f1ac40c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:04.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.502+0000 7f1ac37fe640 1 -- 172.21.15.5:0/1753192120 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1ac406bf50 msgr2=0x7f1ac414cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:04.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.502+0000 7f1ac37fe640 1 --2- 172.21.15.5:0/1753192120 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1ac406bf50 0x7f1ac414cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:04.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.502+0000 7f1ac37fe640 1 -- 172.21.15.5:0/1753192120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ac41543a0 msgr2=0x7f1ac40c9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:04.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.502+0000 7f1ac37fe640 1 --2- 172.21.15.5:0/1753192120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ac41543a0 0x7f1ac40c9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:04.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.502+0000 7f1ac37fe640 1 -- 172.21.15.5:0/1753192120 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1abc009a70 con 0x7f1ac415ac00 2024-03-24T10:54:04.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.505+0000 7f1ac37fe640 1 --2- 172.21.15.5:0/1753192120 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ac415ac00 0x7f1ac40c9760 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1abc009eb0 tx=0x7f1abc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:04.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.509+0000 7f1aab7fe640 1 -- 172.21.15.5:0/1753192120 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc03e610 con 0x7f1ac415ac00 2024-03-24T10:54:04.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.509+0000 7f1aab7fe640 1 -- 172.21.15.5:0/1753192120 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1abc03e770 con 0x7f1ac415ac00 2024-03-24T10:54:04.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.509+0000 7f1ac94eb640 1 -- 172.21.15.5:0/1753192120 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ac40c8010 con 0x7f1ac415ac00 2024-03-24T10:54:04.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.509+0000 7f1aab7fe640 1 -- 172.21.15.5:0/1753192120 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1abc03eb20 con 0x7f1ac415ac00 2024-03-24T10:54:04.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.509+0000 7f1ac94eb640 1 -- 172.21.15.5:0/1753192120 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ac40c85d0 con 0x7f1ac415ac00 2024-03-24T10:54:04.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.510+0000 7f1ac94eb640 1 -- 172.21.15.5:0/1753192120 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ac40c87f0 con 0x7f1ac415ac00 2024-03-24T10:54:04.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.538+0000 7f1aab7fe640 1 -- 172.21.15.5:0/1753192120 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1abc03b050 con 0x7f1ac415ac00 2024-03-24T10:54:04.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.539+0000 7f1aab7fe640 1 --2- 172.21.15.5:0/1753192120 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1ab006b860 0x7f1ab006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:04.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.539+0000 7f1ac2ffd640 1 --2- 172.21.15.5:0/1753192120 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1ab006b860 0x7f1ab006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:04.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.539+0000 7f1ac2ffd640 1 --2- 172.21.15.5:0/1753192120 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1ab006b860 0x7f1ab006dd20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40011e0 tx=0x7f1ab4007d30 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:04.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.828+0000 7f1aab7fe640 1 -- 172.21.15.5:0/1753192120 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f1abc07b940 con 0x7f1ac415ac00 2024-03-24T10:54:04.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:04.828+0000 7f1aab7fe640 1 -- 172.21.15.5:0/1753192120 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1abc077be0 con 0x7f1ac415ac00 2024-03-24T10:54:05.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:05.053+0000 7f1ac94eb640 1 -- 172.21.15.5:0/1753192120 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ac40cac70 con 0x7f1ac415ac00 2024-03-24T10:54:05.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:05.069+0000 7f1aab7fe640 1 -- 172.21.15.5:0/1753192120 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f1abc039070 con 0x7f1ac415ac00 2024-03-24T10:54:05.070 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:05.070 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:05.070 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:05.072+0000 7f1ac94eb640 1 -- 172.21.15.5:0/1753192120 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1ab006b860 msgr2=0x7f1ab006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:05.072+0000 7f1ac94eb640 1 --2- 172.21.15.5:0/1753192120 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1ab006b860 0x7f1ab006dd20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40011e0 tx=0x7f1ab4007d30 comp rx=0 tx=0).stop 2024-03-24T10:54:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:05.072+0000 7f1ac94eb640 1 -- 172.21.15.5:0/1753192120 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ac415ac00 msgr2=0x7f1ac40c9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:05.072+0000 7f1ac94eb640 1 --2- 172.21.15.5:0/1753192120 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ac415ac00 0x7f1ac40c9760 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1abc009eb0 tx=0x7f1abc004a40 comp rx=0 tx=0).stop 2024-03-24T10:54:05.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:05.072+0000 7f1ac94eb640 1 -- 172.21.15.5:0/1753192120 shutdown_connections 2024-03-24T10:54:05.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:05.072+0000 7f1ac94eb640 1 --2- 172.21.15.5:0/1753192120 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1ab006b860 0x7f1ab006dd20 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:05.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:05.072+0000 7f1ac94eb640 1 --2- 172.21.15.5:0/1753192120 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ac415ac00 0x7f1ac40c9760 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:05.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:05.072+0000 7f1ac94eb640 1 --2- 172.21.15.5:0/1753192120 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ac41543a0 0x7f1ac40c9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:05.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:05.072+0000 7f1ac94eb640 1 --2- 172.21.15.5:0/1753192120 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1ac406bf50 0x7f1ac414cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:05.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:05.072+0000 7f1ac94eb640 1 -- 172.21.15.5:0/1753192120 >> 172.21.15.5:0/1753192120 conn(0x7f1ac4072ef0 msgr2=0x7f1ac41508a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:05.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:05.072+0000 7f1ac94eb640 1 -- 172.21.15.5:0/1753192120 shutdown_connections 2024-03-24T10:54:05.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:05.072+0000 7f1ac94eb640 1 -- 172.21.15.5:0/1753192120 wait complete. 2024-03-24T10:54:05.692 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:05.692 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:06.693 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:06.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.794+0000 7fbd9bbe9640 1 Processor -- start 2024-03-24T10:54:06.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.794+0000 7fbd9bbe9640 1 -- start start 2024-03-24T10:54:06.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.795+0000 7fbd9bbe9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbd940cf110 0x7fbd94153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:06.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.795+0000 7fbd9bbe9640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd9415bce0 con 0x7fbd9406c580 2024-03-24T10:54:06.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.795+0000 7fbd9bbe9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd940cd890 con 0x7fbd940cc8f0 2024-03-24T10:54:06.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.795+0000 7fbd9bbe9640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fbd9406cae0 con 0x7fbd940cf110 2024-03-24T10:54:06.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.796+0000 7fbd99184640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fbd9406c580 0x7fbd941538b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:43722/0 (socket says 172.21.15.5:43722) 2024-03-24T10:54:06.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.796+0000 7fbd99184640 1 -- 172.21.15.5:0/2256036491 learned_addr learned my addr 172.21.15.5:0/2256036491 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:54:06.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.796+0000 7fbd9a186640 1 --2- 172.21.15.5:0/2256036491 >> v2:172.21.15.57:3300/0 conn(0x7fbd940cf110 0x7fbd94153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:06.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.799+0000 7fbd98983640 1 -- 172.21.15.5:0/2256036491 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2787723485 0 0) 0x7fbd9415bce0 con 0x7fbd9406c580 2024-03-24T10:54:06.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.799+0000 7fbd98983640 1 -- 172.21.15.5:0/2256036491 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd80003520 con 0x7fbd9406c580 2024-03-24T10:54:06.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.801+0000 7fbd9a186640 1 -- 172.21.15.5:0/2256036491 >> v1:172.21.15.57:6789/0 conn(0x7fbd940cc8f0 legacy=0x7fbd9415b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:06.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.801+0000 7fbd9a186640 1 -- 172.21.15.5:0/2256036491 >> v1:172.21.15.5:6789/0 conn(0x7fbd9406c580 legacy=0x7fbd941538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:06.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.801+0000 7fbd9a186640 1 -- 172.21.15.5:0/2256036491 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd940cdb50 con 0x7fbd940cf110 2024-03-24T10:54:06.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.804+0000 7fbd9a186640 1 --2- 172.21.15.5:0/2256036491 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd940cf110 0x7fbd94153fc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c009d90 tx=0x7fbd8c02bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3253f2d00185570c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:06.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.809+0000 7fbd98983640 1 -- 172.21.15.5:0/2256036491 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c02cd80 con 0x7fbd940cf110 2024-03-24T10:54:06.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.809+0000 7fbd98983640 1 -- 172.21.15.5:0/2256036491 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd8c034440 con 0x7fbd940cf110 2024-03-24T10:54:06.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.809+0000 7fbd98983640 1 -- 172.21.15.5:0/2256036491 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c0347d0 con 0x7fbd940cf110 2024-03-24T10:54:06.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.810+0000 7fbd9bbe9640 1 -- 172.21.15.5:0/2256036491 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd940cf110 msgr2=0x7fbd94153fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:06.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.810+0000 7fbd9bbe9640 1 --2- 172.21.15.5:0/2256036491 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd940cf110 0x7fbd94153fc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c009d90 tx=0x7fbd8c02bfd0 comp rx=0 tx=0).stop 2024-03-24T10:54:06.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.810+0000 7fbd9bbe9640 1 -- 172.21.15.5:0/2256036491 shutdown_connections 2024-03-24T10:54:06.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.810+0000 7fbd9bbe9640 1 --2- 172.21.15.5:0/2256036491 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd940cf110 0x7fbd94153fc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:06.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.810+0000 7fbd9bbe9640 1 -- 172.21.15.5:0/2256036491 >> 172.21.15.5:0/2256036491 conn(0x7fbd940b1820 msgr2=0x7fbd940b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:06.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.810+0000 7fbd9bbe9640 1 -- 172.21.15.5:0/2256036491 shutdown_connections 2024-03-24T10:54:06.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.810+0000 7fbd9bbe9640 1 -- 172.21.15.5:0/2256036491 wait complete. 2024-03-24T10:54:06.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.811+0000 7fbd9bbe9640 1 Processor -- start 2024-03-24T10:54:06.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.811+0000 7fbd9bbe9640 1 -- start start 2024-03-24T10:54:06.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.812+0000 7fbd9bbe9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd9406c580 0x7fbd941f07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:06.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.812+0000 7fbd9bbe9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd940cc8f0 0x7fbd94158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:06.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.812+0000 7fbd9bbe9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbd940cf110 0x7fbd941e4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:06.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.812+0000 7fbd9bbe9640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fbd94158a80 con 0x7fbd9406c580 2024-03-24T10:54:06.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.812+0000 7fbd9bbe9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbd941f0d30 con 0x7fbd940cc8f0 2024-03-24T10:54:06.813 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.812+0000 7fbd9bbe9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fbd9415a280 con 0x7fbd940cf110 2024-03-24T10:54:06.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.813+0000 7fbd99184640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd940cc8f0 0x7fbd94158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:06.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.813+0000 7fbd99985640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd9406c580 0x7fbd941f07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:06.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.813+0000 7fbd99184640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd940cc8f0 0x7fbd94158030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:38932/0 (socket says 172.21.15.5:38932) 2024-03-24T10:54:06.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.813+0000 7fbd99184640 1 -- 172.21.15.5:0/2431327829 learned_addr learned my addr 172.21.15.5:0/2431327829 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:06.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.813+0000 7fbd9a186640 1 --2- 172.21.15.5:0/2431327829 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbd940cf110 0x7fbd941e4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:06.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.816+0000 7fbd99985640 1 -- 172.21.15.5:0/2431327829 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbd940cf110 msgr2=0x7fbd941e4b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:06.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.816+0000 7fbd99985640 1 --2- 172.21.15.5:0/2431327829 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbd940cf110 0x7fbd941e4b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:06.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.816+0000 7fbd99985640 1 -- 172.21.15.5:0/2431327829 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd940cc8f0 msgr2=0x7fbd94158030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:06.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.816+0000 7fbd99985640 1 --2- 172.21.15.5:0/2431327829 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd940cc8f0 0x7fbd94158030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:06.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.817+0000 7fbd99985640 1 -- 172.21.15.5:0/2431327829 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd8c009a70 con 0x7fbd9406c580 2024-03-24T10:54:06.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.819+0000 7fbd99985640 1 --2- 172.21.15.5:0/2431327829 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd9406c580 0x7fbd941f07f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fbd9000c340 tx=0x7fbd9000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:06.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.823+0000 7fbd8a7fc640 1 -- 172.21.15.5:0/2431327829 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd900052d0 con 0x7fbd9406c580 2024-03-24T10:54:06.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.823+0000 7fbd8a7fc640 1 -- 172.21.15.5:0/2431327829 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd90005450 con 0x7fbd9406c580 2024-03-24T10:54:06.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.823+0000 7fbd9bbe9640 1 -- 172.21.15.5:0/2431327829 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd9415a560 con 0x7fbd9406c580 2024-03-24T10:54:06.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.823+0000 7fbd8a7fc640 1 -- 172.21.15.5:0/2431327829 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd90007260 con 0x7fbd9406c580 2024-03-24T10:54:06.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.823+0000 7fbd9bbe9640 1 -- 172.21.15.5:0/2431327829 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd9415ab20 con 0x7fbd9406c580 2024-03-24T10:54:06.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.824+0000 7fbd9bbe9640 1 -- 172.21.15.5:0/2431327829 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd940681d0 con 0x7fbd9406c580 2024-03-24T10:54:06.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.843+0000 7fbd8a7fc640 1 -- 172.21.15.5:0/2431327829 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbd900073c0 con 0x7fbd9406c580 2024-03-24T10:54:06.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.844+0000 7fbd8a7fc640 1 --2- 172.21.15.5:0/2431327829 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbd7806b810 0x7fbd7806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:06.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.844+0000 7fbd99184640 1 --2- 172.21.15.5:0/2431327829 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbd7806b810 0x7fbd7806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:06.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:06.844+0000 7fbd99184640 1 --2- 172.21.15.5:0/2431327829 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbd7806b810 0x7fbd7806dcd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbd94157d50 tx=0x7fbd84009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:07.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:07.107+0000 7fbd8a7fc640 1 -- 172.21.15.5:0/2431327829 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fbd900582a0 con 0x7fbd9406c580 2024-03-24T10:54:07.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:07.108+0000 7fbd8a7fc640 1 -- 172.21.15.5:0/2431327829 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbd90054540 con 0x7fbd9406c580 2024-03-24T10:54:07.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:07.329+0000 7fbd9bbe9640 1 -- 172.21.15.5:0/2431327829 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd940cf720 con 0x7fbd9406c580 2024-03-24T10:54:07.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:07.342+0000 7fbd8a7fc640 1 -- 172.21.15.5:0/2431327829 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fbd90020020 con 0x7fbd9406c580 2024-03-24T10:54:07.344 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:07.344 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:07.344 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:07.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:07.345+0000 7fbd9bbe9640 1 -- 172.21.15.5:0/2431327829 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbd7806b810 msgr2=0x7fbd7806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:07.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:07.345+0000 7fbd9bbe9640 1 --2- 172.21.15.5:0/2431327829 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbd7806b810 0x7fbd7806dcd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbd94157d50 tx=0x7fbd84009040 comp rx=0 tx=0).stop 2024-03-24T10:54:07.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:07.345+0000 7fbd9bbe9640 1 -- 172.21.15.5:0/2431327829 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd9406c580 msgr2=0x7fbd941f07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:07.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:07.345+0000 7fbd9bbe9640 1 --2- 172.21.15.5:0/2431327829 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd9406c580 0x7fbd941f07f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fbd9000c340 tx=0x7fbd9000c800 comp rx=0 tx=0).stop 2024-03-24T10:54:07.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:07.345+0000 7fbd9bbe9640 1 -- 172.21.15.5:0/2431327829 shutdown_connections 2024-03-24T10:54:07.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:07.345+0000 7fbd9bbe9640 1 --2- 172.21.15.5:0/2431327829 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbd7806b810 0x7fbd7806dcd0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:07.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:07.345+0000 7fbd9bbe9640 1 --2- 172.21.15.5:0/2431327829 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbd940cf110 0x7fbd941e4b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:07.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:07.345+0000 7fbd9bbe9640 1 --2- 172.21.15.5:0/2431327829 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbd940cc8f0 0x7fbd94158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:07.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:07.345+0000 7fbd9bbe9640 1 --2- 172.21.15.5:0/2431327829 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbd9406c580 0x7fbd941f07f0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:07.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:07.345+0000 7fbd9bbe9640 1 -- 172.21.15.5:0/2431327829 >> 172.21.15.5:0/2431327829 conn(0x7fbd940b1820 msgr2=0x7fbd940b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:07.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:07.345+0000 7fbd9bbe9640 1 -- 172.21.15.5:0/2431327829 shutdown_connections 2024-03-24T10:54:07.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:07.345+0000 7fbd9bbe9640 1 -- 172.21.15.5:0/2431327829 wait complete. 2024-03-24T10:54:07.961 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:07.961 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:08.962 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:09.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.063+0000 7f4970c7c640 1 Processor -- start 2024-03-24T10:54:09.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.064+0000 7f4970c7c640 1 -- start start 2024-03-24T10:54:09.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.064+0000 7f4970c7c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f496c06bf50 0x7f496c06c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:09.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.064+0000 7f4970c7c640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f496c0cb2f0 con 0x7f496c15ac00 2024-03-24T10:54:09.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.064+0000 7f4970c7c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f496c15b070 con 0x7f496c1543a0 2024-03-24T10:54:09.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.064+0000 7f4970c7c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f496c0c6750 con 0x7f496c06bf50 2024-03-24T10:54:09.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.065+0000 7f4963fff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f496c15ac00 0x7f496c0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:43738/0 (socket says 172.21.15.5:43738) 2024-03-24T10:54:09.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.065+0000 7f4963fff640 1 -- 172.21.15.5:0/1946933863 learned_addr learned my addr 172.21.15.5:0/1946933863 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:54:09.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.065+0000 7f496ad9d640 1 --2- 172.21.15.5:0/1946933863 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f496c06bf50 0x7f496c06c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:09.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.068+0000 7f4969d9b640 1 -- 172.21.15.5:0/1946933863 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1020076693 0 0) 0x7f496c0cb2f0 con 0x7f496c15ac00 2024-03-24T10:54:09.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.068+0000 7f4969d9b640 1 -- 172.21.15.5:0/1946933863 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4958003520 con 0x7f496c15ac00 2024-03-24T10:54:09.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.068+0000 7f496ad9d640 1 -- 172.21.15.5:0/1946933863 >> v1:172.21.15.57:6789/0 conn(0x7f496c1543a0 legacy=0x7f496c06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:09.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.068+0000 7f496ad9d640 1 -- 172.21.15.5:0/1946933863 >> v1:172.21.15.5:6789/0 conn(0x7f496c15ac00 legacy=0x7f496c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:09.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.068+0000 7f496ad9d640 1 -- 172.21.15.5:0/1946933863 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f496c0ca7e0 con 0x7f496c06bf50 2024-03-24T10:54:09.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.070+0000 7f496ad9d640 1 --2- 172.21.15.5:0/1946933863 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f496c06bf50 0x7f496c06c360 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4964009dc0 tx=0x7f496402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=966858e2dfa84071 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:09.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.075+0000 7f4969d9b640 1 -- 172.21.15.5:0/1946933863 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496402ccd0 con 0x7f496c06bf50 2024-03-24T10:54:09.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.075+0000 7f4969d9b640 1 -- 172.21.15.5:0/1946933863 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f496402ce30 con 0x7f496c06bf50 2024-03-24T10:54:09.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.075+0000 7f4969d9b640 1 -- 172.21.15.5:0/1946933863 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4964034660 con 0x7f496c06bf50 2024-03-24T10:54:09.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.076+0000 7f4970c7c640 1 -- 172.21.15.5:0/1946933863 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f496c06bf50 msgr2=0x7f496c06c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:09.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.076+0000 7f4970c7c640 1 --2- 172.21.15.5:0/1946933863 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f496c06bf50 0x7f496c06c360 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4964009dc0 tx=0x7f496402bf20 comp rx=0 tx=0).stop 2024-03-24T10:54:09.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.076+0000 7f4970c7c640 1 -- 172.21.15.5:0/1946933863 shutdown_connections 2024-03-24T10:54:09.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.076+0000 7f4970c7c640 1 --2- 172.21.15.5:0/1946933863 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f496c06bf50 0x7f496c06c360 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:09.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.076+0000 7f4970c7c640 1 -- 172.21.15.5:0/1946933863 >> 172.21.15.5:0/1946933863 conn(0x7f496c072ef0 msgr2=0x7f496c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:09.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.076+0000 7f4970c7c640 1 -- 172.21.15.5:0/1946933863 shutdown_connections 2024-03-24T10:54:09.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.076+0000 7f4970c7c640 1 -- 172.21.15.5:0/1946933863 wait complete. 2024-03-24T10:54:09.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.078+0000 7f4970c7c640 1 Processor -- start 2024-03-24T10:54:09.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.078+0000 7f4970c7c640 1 -- start start 2024-03-24T10:54:09.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.079+0000 7f4970c7c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f496c06bf50 0x7f496c14e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:09.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.079+0000 7f4970c7c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f496c1543a0 0x7f496c150520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:09.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.079+0000 7f4970c7c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f496c15ac00 0x7f496c150a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:09.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.079+0000 7f4970c7c640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f496c15e140 con 0x7f496c1543a0 2024-03-24T10:54:09.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.079+0000 7f4970c7c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f496c1510d0 con 0x7f496c06bf50 2024-03-24T10:54:09.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.079+0000 7f4970c7c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f496c15d4c0 con 0x7f496c15ac00 2024-03-24T10:54:09.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.080+0000 7f4963fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f496c1543a0 0x7f496c150520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:09.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.080+0000 7f496ad9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f496c15ac00 0x7f496c150a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:09.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.080+0000 7f496a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f496c06bf50 0x7f496c14e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:09.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.080+0000 7f496ad9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f496c15ac00 0x7f496c150a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:32890/0 (socket says 172.21.15.5:32890) 2024-03-24T10:54:09.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.080+0000 7f496ad9d640 1 -- 172.21.15.5:0/715810075 learned_addr learned my addr 172.21.15.5:0/715810075 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:09.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.083+0000 7f496ad9d640 1 -- 172.21.15.5:0/715810075 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f496c06bf50 msgr2=0x7f496c14e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:09.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.083+0000 7f496ad9d640 1 --2- 172.21.15.5:0/715810075 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f496c06bf50 0x7f496c14e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:09.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.083+0000 7f496ad9d640 1 -- 172.21.15.5:0/715810075 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f496c1543a0 msgr2=0x7f496c150520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:09.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.083+0000 7f496ad9d640 1 --2- 172.21.15.5:0/715810075 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f496c1543a0 0x7f496c150520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:09.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.084+0000 7f496ad9d640 1 -- 172.21.15.5:0/715810075 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4964009a70 con 0x7f496c15ac00 2024-03-24T10:54:09.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.086+0000 7f496ad9d640 1 --2- 172.21.15.5:0/715810075 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f496c15ac00 0x7f496c150a60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f496402dfd0 tx=0x7f4964004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:09.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.090+0000 7f4962ffd640 1 -- 172.21.15.5:0/715810075 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496403e610 con 0x7f496c15ac00 2024-03-24T10:54:09.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.091+0000 7f4962ffd640 1 -- 172.21.15.5:0/715810075 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f496403e790 con 0x7f496c15ac00 2024-03-24T10:54:09.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.091+0000 7f4962ffd640 1 -- 172.21.15.5:0/715810075 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496403eb40 con 0x7f496c15ac00 2024-03-24T10:54:09.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.091+0000 7f4970c7c640 1 -- 172.21.15.5:0/715810075 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f496c15d7a0 con 0x7f496c15ac00 2024-03-24T10:54:09.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.091+0000 7f4970c7c640 1 -- 172.21.15.5:0/715810075 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f496c15dd60 con 0x7f496c15ac00 2024-03-24T10:54:09.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.092+0000 7f4970c7c640 1 -- 172.21.15.5:0/715810075 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f496c15e280 con 0x7f496c15ac00 2024-03-24T10:54:09.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.117+0000 7f4962ffd640 1 -- 172.21.15.5:0/715810075 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f496403b050 con 0x7f496c15ac00 2024-03-24T10:54:09.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.117+0000 7f4962ffd640 1 --2- 172.21.15.5:0/715810075 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f494c06b810 0x7f494c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:09.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.117+0000 7f496a59c640 1 --2- 172.21.15.5:0/715810075 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f494c06b810 0x7f494c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:09.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.118+0000 7f496a59c640 1 --2- 172.21.15.5:0/715810075 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f494c06b810 0x7f494c06dcd0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f495c000f60 tx=0x7f495c009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:09.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.123+0000 7f4962ffd640 1 -- 172.21.15.5:0/715810075 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f496407b450 con 0x7f496c15ac00 2024-03-24T10:54:09.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.417+0000 7f4962ffd640 1 -- 172.21.15.5:0/715810075 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f49640776f0 con 0x7f496c15ac00 2024-03-24T10:54:09.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.661+0000 7f4970c7c640 1 -- 172.21.15.5:0/715810075 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f496c15e280 con 0x7f496c15ac00 2024-03-24T10:54:09.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.760+0000 7f4962ffd640 1 -- 172.21.15.5:0/715810075 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f496c15e280 con 0x7f496c15ac00 2024-03-24T10:54:09.761 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:09.761 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:09.762 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:09.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.762+0000 7f4970c7c640 1 -- 172.21.15.5:0/715810075 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f494c06b810 msgr2=0x7f494c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:09.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.762+0000 7f4970c7c640 1 --2- 172.21.15.5:0/715810075 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f494c06b810 0x7f494c06dcd0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f495c000f60 tx=0x7f495c009040 comp rx=0 tx=0).stop 2024-03-24T10:54:09.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.762+0000 7f4970c7c640 1 -- 172.21.15.5:0/715810075 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f496c15ac00 msgr2=0x7f496c150a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:09.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.762+0000 7f4970c7c640 1 --2- 172.21.15.5:0/715810075 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f496c15ac00 0x7f496c150a60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f496402dfd0 tx=0x7f4964004a40 comp rx=0 tx=0).stop 2024-03-24T10:54:09.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.762+0000 7f4970c7c640 1 -- 172.21.15.5:0/715810075 shutdown_connections 2024-03-24T10:54:09.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.762+0000 7f4970c7c640 1 --2- 172.21.15.5:0/715810075 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f494c06b810 0x7f494c06dcd0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:09.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.762+0000 7f4970c7c640 1 --2- 172.21.15.5:0/715810075 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f496c1543a0 0x7f496c150520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:09.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.762+0000 7f4970c7c640 1 --2- 172.21.15.5:0/715810075 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f496c15ac00 0x7f496c150a60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:09.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.762+0000 7f4970c7c640 1 --2- 172.21.15.5:0/715810075 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f496c06bf50 0x7f496c14e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:09.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.762+0000 7f4970c7c640 1 -- 172.21.15.5:0/715810075 >> 172.21.15.5:0/715810075 conn(0x7f496c072ef0 msgr2=0x7f496c159f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:09.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.762+0000 7f4970c7c640 1 -- 172.21.15.5:0/715810075 shutdown_connections 2024-03-24T10:54:09.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:09.762+0000 7f4970c7c640 1 -- 172.21.15.5:0/715810075 wait complete. 2024-03-24T10:54:10.378 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:10.378 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:11.379 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:11.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.481+0000 7f7f1ebf7640 1 Processor -- start 2024-03-24T10:54:11.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.481+0000 7f7f1ebf7640 1 -- start start 2024-03-24T10:54:11.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.481+0000 7f7f1ebf7640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7f181543a0 0x7f7f1815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:11.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.481+0000 7f7f1ebf7640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f180cb2f0 con 0x7f7f1806d4f0 2024-03-24T10:54:11.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.481+0000 7f7f1ebf7640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f180ca7e0 con 0x7f7f1806bf50 2024-03-24T10:54:11.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.481+0000 7f7f1ebf7640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f7f180c6750 con 0x7f7f181543a0 2024-03-24T10:54:11.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.482+0000 7f7f17fff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f7f1806d4f0 0x7f7f180cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:43748/0 (socket says 172.21.15.5:43748) 2024-03-24T10:54:11.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.482+0000 7f7f17fff640 1 -- 172.21.15.5:0/3084264283 learned_addr learned my addr 172.21.15.5:0/3084264283 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:54:11.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.482+0000 7f7f1c993640 1 --2- 172.21.15.5:0/3084264283 >> v2:172.21.15.57:3300/0 conn(0x7f7f181543a0 0x7f7f1815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:11.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.485+0000 7f7f177fe640 1 -- 172.21.15.5:0/3084264283 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1567173344 0 0) 0x7f7f180cb2f0 con 0x7f7f1806d4f0 2024-03-24T10:54:11.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.485+0000 7f7f177fe640 1 -- 172.21.15.5:0/3084264283 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f08003520 con 0x7f7f1806d4f0 2024-03-24T10:54:11.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.488+0000 7f7f1c993640 1 -- 172.21.15.5:0/3084264283 >> v1:172.21.15.57:6789/0 conn(0x7f7f1806bf50 legacy=0x7f7f1806c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:11.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.488+0000 7f7f1c993640 1 -- 172.21.15.5:0/3084264283 >> v1:172.21.15.5:6789/0 conn(0x7f7f1806d4f0 legacy=0x7f7f180cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:11.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.488+0000 7f7f1c993640 1 -- 172.21.15.5:0/3084264283 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f180cb520 con 0x7f7f181543a0 2024-03-24T10:54:11.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.492+0000 7f7f1c993640 1 --2- 172.21.15.5:0/3084264283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f181543a0 0x7f7f1815ac00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0097a0 tx=0x7f7f0c02b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85b31059c722db0e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:11.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.496+0000 7f7f177fe640 1 -- 172.21.15.5:0/3084264283 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c02f810 con 0x7f7f181543a0 2024-03-24T10:54:11.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.496+0000 7f7f177fe640 1 -- 172.21.15.5:0/3084264283 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f0c02f970 con 0x7f7f181543a0 2024-03-24T10:54:11.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.496+0000 7f7f177fe640 1 -- 172.21.15.5:0/3084264283 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0c02fd00 con 0x7f7f181543a0 2024-03-24T10:54:11.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.497+0000 7f7f1ebf7640 1 -- 172.21.15.5:0/3084264283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f181543a0 msgr2=0x7f7f1815ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:11.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.497+0000 7f7f1ebf7640 1 --2- 172.21.15.5:0/3084264283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f181543a0 0x7f7f1815ac00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0097a0 tx=0x7f7f0c02b9f0 comp rx=0 tx=0).stop 2024-03-24T10:54:11.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.497+0000 7f7f1ebf7640 1 -- 172.21.15.5:0/3084264283 shutdown_connections 2024-03-24T10:54:11.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.497+0000 7f7f1ebf7640 1 --2- 172.21.15.5:0/3084264283 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f181543a0 0x7f7f1815ac00 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:11.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.497+0000 7f7f1ebf7640 1 -- 172.21.15.5:0/3084264283 >> 172.21.15.5:0/3084264283 conn(0x7f7f18072ef0 msgr2=0x7f7f18061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:11.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.497+0000 7f7f1ebf7640 1 -- 172.21.15.5:0/3084264283 shutdown_connections 2024-03-24T10:54:11.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.497+0000 7f7f1ebf7640 1 -- 172.21.15.5:0/3084264283 wait complete. 2024-03-24T10:54:11.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.498+0000 7f7f1ebf7640 1 Processor -- start 2024-03-24T10:54:11.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.498+0000 7f7f1ebf7640 1 -- start start 2024-03-24T10:54:11.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.499+0000 7f7f1ebf7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f1806bf50 0x7f7f180ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:11.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.499+0000 7f7f1ebf7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f1806d4f0 0x7f7f180cea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:11.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.499+0000 7f7f1ebf7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f181543a0 0x7f7f18146290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:11.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.499+0000 7f7f1ebf7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f7f180b4110 con 0x7f7f181543a0 2024-03-24T10:54:11.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.499+0000 7f7f1ebf7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7f180b43f0 con 0x7f7f1806d4f0 2024-03-24T10:54:11.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.499+0000 7f7f1ebf7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7f180cda00 con 0x7f7f1806bf50 2024-03-24T10:54:11.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.500+0000 7f7f17fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f1806d4f0 0x7f7f180cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:11.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.500+0000 7f7f1c993640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f1806bf50 0x7f7f180ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:11.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.500+0000 7f7f17fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f1806d4f0 0x7f7f180cea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:38968/0 (socket says 172.21.15.5:38968) 2024-03-24T10:54:11.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.500+0000 7f7f17fff640 1 -- 172.21.15.5:0/235639146 learned_addr learned my addr 172.21.15.5:0/235639146 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:11.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.505+0000 7f7f17fff640 1 -- 172.21.15.5:0/235639146 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f1806bf50 msgr2=0x7f7f180ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:11.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.505+0000 7f7f17fff640 1 --2- 172.21.15.5:0/235639146 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f1806bf50 0x7f7f180ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:11.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.505+0000 7f7f17fff640 1 -- 172.21.15.5:0/235639146 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f181543a0 msgr2=0x7f7f18146290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:11.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.505+0000 7f7f17fff640 1 --2- 172.21.15.5:0/235639146 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f181543a0 0x7f7f18146290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:11.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.505+0000 7f7f17fff640 1 -- 172.21.15.5:0/235639146 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f0c009400 con 0x7f7f1806d4f0 2024-03-24T10:54:11.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.508+0000 7f7f17fff640 1 --2- 172.21.15.5:0/235639146 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f1806d4f0 0x7f7f180cea20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7f0400c500 tx=0x7f7f0400c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:11.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.512+0000 7f7f157fa640 1 -- 172.21.15.5:0/235639146 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f04005ce0 con 0x7f7f1806d4f0 2024-03-24T10:54:11.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.512+0000 7f7f157fa640 1 -- 172.21.15.5:0/235639146 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f04005e40 con 0x7f7f1806d4f0 2024-03-24T10:54:11.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.512+0000 7f7f1ebf7640 1 -- 172.21.15.5:0/235639146 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f180cdce0 con 0x7f7f1806d4f0 2024-03-24T10:54:11.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.513+0000 7f7f157fa640 1 -- 172.21.15.5:0/235639146 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f04007330 con 0x7f7f1806d4f0 2024-03-24T10:54:11.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.513+0000 7f7f1ebf7640 1 -- 172.21.15.5:0/235639146 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f180b46d0 con 0x7f7f1806d4f0 2024-03-24T10:54:11.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.513+0000 7f7f1ebf7640 1 -- 172.21.15.5:0/235639146 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f180ce200 con 0x7f7f1806d4f0 2024-03-24T10:54:11.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.782+0000 7f7f157fa640 1 -- 172.21.15.5:0/235639146 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7f04007550 con 0x7f7f1806d4f0 2024-03-24T10:54:11.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.782+0000 7f7f157fa640 1 --2- 172.21.15.5:0/235639146 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7efc06b860 0x7f7efc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:11.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.783+0000 7f7f1c993640 1 --2- 172.21.15.5:0/235639146 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7efc06b860 0x7f7efc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:11.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.783+0000 7f7f1c993640 1 --2- 172.21.15.5:0/235639146 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7efc06b860 0x7f7efc06dd20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c036000 tx=0x7f7f0c008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:11.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.786+0000 7f7f157fa640 1 -- 172.21.15.5:0/235639146 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f7f040583d0 con 0x7f7f1806d4f0 2024-03-24T10:54:11.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:11.798+0000 7f7f157fa640 1 -- 172.21.15.5:0/235639146 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7f04054670 con 0x7f7f1806d4f0 2024-03-24T10:54:12.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:12.020+0000 7f7f1ebf7640 1 -- 172.21.15.5:0/235639146 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f180ce200 con 0x7f7f1806d4f0 2024-03-24T10:54:12.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:12.034+0000 7f7f157fa640 1 -- 172.21.15.5:0/235639146 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f7f180ce200 con 0x7f7f1806d4f0 2024-03-24T10:54:12.035 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:12.036 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:12.036 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:12.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:12.037+0000 7f7f1ebf7640 1 -- 172.21.15.5:0/235639146 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7efc06b860 msgr2=0x7f7efc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:12.038 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:12.037+0000 7f7f1ebf7640 1 --2- 172.21.15.5:0/235639146 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7efc06b860 0x7f7efc06dd20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c036000 tx=0x7f7f0c008030 comp rx=0 tx=0).stop 2024-03-24T10:54:12.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:12.037+0000 7f7f1ebf7640 1 -- 172.21.15.5:0/235639146 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f1806d4f0 msgr2=0x7f7f180cea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:12.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:12.037+0000 7f7f1ebf7640 1 --2- 172.21.15.5:0/235639146 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f1806d4f0 0x7f7f180cea20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7f0400c500 tx=0x7f7f0400c9c0 comp rx=0 tx=0).stop 2024-03-24T10:54:12.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:12.037+0000 7f7f1ebf7640 1 -- 172.21.15.5:0/235639146 shutdown_connections 2024-03-24T10:54:12.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:12.037+0000 7f7f1ebf7640 1 --2- 172.21.15.5:0/235639146 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7efc06b860 0x7f7efc06dd20 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:12.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:12.037+0000 7f7f1ebf7640 1 --2- 172.21.15.5:0/235639146 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7f181543a0 0x7f7f18146290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:12.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:12.037+0000 7f7f1ebf7640 1 --2- 172.21.15.5:0/235639146 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7f1806d4f0 0x7f7f180cea20 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:12.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:12.037+0000 7f7f1ebf7640 1 --2- 172.21.15.5:0/235639146 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7f1806bf50 0x7f7f180ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:12.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:12.037+0000 7f7f1ebf7640 1 -- 172.21.15.5:0/235639146 >> 172.21.15.5:0/235639146 conn(0x7f7f18072ef0 msgr2=0x7f7f180cc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:12.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:12.037+0000 7f7f1ebf7640 1 -- 172.21.15.5:0/235639146 shutdown_connections 2024-03-24T10:54:12.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:12.037+0000 7f7f1ebf7640 1 -- 172.21.15.5:0/235639146 wait complete. 2024-03-24T10:54:12.657 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:12.657 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:13.658 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:13.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.761+0000 7f0484dc7640 1 Processor -- start 2024-03-24T10:54:13.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.761+0000 7f0484dc7640 1 -- start start 2024-03-24T10:54:13.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.762+0000 7f0484dc7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f04800cf530 0x7f04800cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:13.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.762+0000 7f0484dc7640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04800cf9e0 con 0x7f04800cae90 2024-03-24T10:54:13.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.762+0000 7f0484dc7640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04800ccba0 con 0x7f04800cb270 2024-03-24T10:54:13.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.762+0000 7f0484dc7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f04800cc9b0 con 0x7f04800cf530 2024-03-24T10:54:13.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.763+0000 7f047e59c640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f04800cae90 0x7f04800cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:42214/0 (socket says 172.21.15.5:42214) 2024-03-24T10:54:13.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.763+0000 7f047e59c640 1 -- 172.21.15.5:0/2379256210 learned_addr learned my addr 172.21.15.5:0/2379256210 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:54:13.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.763+0000 7f047ed9d640 1 --2- 172.21.15.5:0/2379256210 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f04800cf530 0x7f04800cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:13.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.766+0000 7f047d59a640 1 -- 172.21.15.5:0/2379256210 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3220776513 0 0) 0x7f0470001110 con 0x7f04800cae90 2024-03-24T10:54:13.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.766+0000 7f047d59a640 1 -- 172.21.15.5:0/2379256210 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0468003520 con 0x7f04800cae90 2024-03-24T10:54:13.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.766+0000 7f047ed9d640 1 -- 172.21.15.5:0/2379256210 >> v1:172.21.15.57:6789/0 conn(0x7f04800cb270 legacy=0x7f04800cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:13.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.766+0000 7f047ed9d640 1 -- 172.21.15.5:0/2379256210 >> v1:172.21.15.5:6789/0 conn(0x7f04800cae90 legacy=0x7f04800cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:13.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.766+0000 7f047ed9d640 1 -- 172.21.15.5:0/2379256210 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04800cce40 con 0x7f04800cf530 2024-03-24T10:54:13.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.768+0000 7f047ed9d640 1 --2- 172.21.15.5:0/2379256210 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f04800cf530 0x7f04800cb6f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0478009dc0 tx=0x7f047802bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=749041b103524ff9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:13.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.773+0000 7f047d59a640 1 -- 172.21.15.5:0/2379256210 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047802cd50 con 0x7f04800cf530 2024-03-24T10:54:13.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.773+0000 7f047d59a640 1 -- 172.21.15.5:0/2379256210 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0478034440 con 0x7f04800cf530 2024-03-24T10:54:13.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.773+0000 7f047d59a640 1 -- 172.21.15.5:0/2379256210 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04780347d0 con 0x7f04800cf530 2024-03-24T10:54:13.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.773+0000 7f0484dc7640 1 -- 172.21.15.5:0/2379256210 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f04800cf530 msgr2=0x7f04800cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:13.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.773+0000 7f0484dc7640 1 --2- 172.21.15.5:0/2379256210 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f04800cf530 0x7f04800cb6f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0478009dc0 tx=0x7f047802bfa0 comp rx=0 tx=0).stop 2024-03-24T10:54:13.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.773+0000 7f0484dc7640 1 -- 172.21.15.5:0/2379256210 shutdown_connections 2024-03-24T10:54:13.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.773+0000 7f0484dc7640 1 --2- 172.21.15.5:0/2379256210 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f04800cf530 0x7f04800cb6f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:13.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.773+0000 7f0484dc7640 1 -- 172.21.15.5:0/2379256210 >> 172.21.15.5:0/2379256210 conn(0x7f04800cfb80 msgr2=0x7f04800cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:13.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.773+0000 7f0484dc7640 1 -- 172.21.15.5:0/2379256210 shutdown_connections 2024-03-24T10:54:13.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.773+0000 7f0484dc7640 1 -- 172.21.15.5:0/2379256210 wait complete. 2024-03-24T10:54:13.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.775+0000 7f0484dc7640 1 Processor -- start 2024-03-24T10:54:13.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.775+0000 7f0484dc7640 1 -- start start 2024-03-24T10:54:13.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.775+0000 7f0484dc7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f04800cae90 0x7f0480156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:13.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.775+0000 7f0484dc7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f04800cb270 0x7f04801f3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:13.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.775+0000 7f0484dc7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04800cf530 0x7f0480155440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:13.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.775+0000 7f0484dc7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f048015b880 con 0x7f04800cb270 2024-03-24T10:54:13.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.775+0000 7f0484dc7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0480157950 con 0x7f04800cf530 2024-03-24T10:54:13.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.775+0000 7f0484dc7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0480157e80 con 0x7f04800cae90 2024-03-24T10:54:13.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.776+0000 7f047e59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f04800cae90 0x7f0480156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:13.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.776+0000 7f047dd9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f04800cb270 0x7f04801f3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:13.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.776+0000 7f047ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04800cf530 0x7f0480155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:13.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.776+0000 7f047e59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f04800cae90 0x7f0480156d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:53452/0 (socket says 172.21.15.5:53452) 2024-03-24T10:54:13.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.776+0000 7f047e59c640 1 -- 172.21.15.5:0/1830399926 learned_addr learned my addr 172.21.15.5:0/1830399926 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:13.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.782+0000 7f047dd9b640 1 -- 172.21.15.5:0/1830399926 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f04800cae90 msgr2=0x7f0480156d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:13.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.782+0000 7f047dd9b640 1 --2- 172.21.15.5:0/1830399926 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f04800cae90 0x7f0480156d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:13.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.782+0000 7f047dd9b640 1 -- 172.21.15.5:0/1830399926 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04800cf530 msgr2=0x7f0480155440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:13.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.782+0000 7f047dd9b640 1 --2- 172.21.15.5:0/1830399926 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04800cf530 0x7f0480155440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:13.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.782+0000 7f047dd9b640 1 -- 172.21.15.5:0/1830399926 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0478009a70 con 0x7f04800cb270 2024-03-24T10:54:13.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.785+0000 7f047dd9b640 1 --2- 172.21.15.5:0/1830399926 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f04800cb270 0x7f04801f3100 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f047400c370 tx=0x7f047400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:13.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.790+0000 7f046effd640 1 -- 172.21.15.5:0/1830399926 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04740052d0 con 0x7f04800cb270 2024-03-24T10:54:13.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.790+0000 7f046effd640 1 -- 172.21.15.5:0/1830399926 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0474005450 con 0x7f04800cb270 2024-03-24T10:54:13.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.790+0000 7f046effd640 1 -- 172.21.15.5:0/1830399926 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0474007260 con 0x7f04800cb270 2024-03-24T10:54:13.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.790+0000 7f0484dc7640 1 -- 172.21.15.5:0/1830399926 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04801f39c0 con 0x7f04800cb270 2024-03-24T10:54:13.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.790+0000 7f0484dc7640 1 -- 172.21.15.5:0/1830399926 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0480157ba0 con 0x7f04800cb270 2024-03-24T10:54:13.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:13.791+0000 7f0484dc7640 1 -- 172.21.15.5:0/1830399926 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04800681d0 con 0x7f04800cb270 2024-03-24T10:54:14.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:14.075+0000 7f046effd640 1 -- 172.21.15.5:0/1830399926 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f04740073c0 con 0x7f04800cb270 2024-03-24T10:54:14.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:14.075+0000 7f046effd640 1 --2- 172.21.15.5:0/1830399926 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f046006b860 0x7f046006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:14.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:14.076+0000 7f047e59c640 1 --2- 172.21.15.5:0/1830399926 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f046006b860 0x7f046006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:14.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:14.076+0000 7f047e59c640 1 --2- 172.21.15.5:0/1830399926 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f046006b860 0x7f046006dd20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0470005880 tx=0x7f0470007c30 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:14.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:14.100+0000 7f046effd640 1 -- 172.21.15.5:0/1830399926 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f04740581b0 con 0x7f04800cb270 2024-03-24T10:54:14.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:14.100+0000 7f046effd640 1 -- 172.21.15.5:0/1830399926 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0474054450 con 0x7f04800cb270 2024-03-24T10:54:14.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:14.355+0000 7f0484dc7640 1 -- 172.21.15.5:0/1830399926 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04800cff60 con 0x7f04800cb270 2024-03-24T10:54:14.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:14.367+0000 7f046effd640 1 -- 172.21.15.5:0/1830399926 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f0474020020 con 0x7f04800cb270 2024-03-24T10:54:14.367 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:14.367 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:14.367 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:14.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:14.370+0000 7f0484dc7640 1 -- 172.21.15.5:0/1830399926 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f046006b860 msgr2=0x7f046006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:14.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:14.370+0000 7f0484dc7640 1 --2- 172.21.15.5:0/1830399926 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f046006b860 0x7f046006dd20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0470005880 tx=0x7f0470007c30 comp rx=0 tx=0).stop 2024-03-24T10:54:14.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:14.370+0000 7f0484dc7640 1 -- 172.21.15.5:0/1830399926 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f04800cb270 msgr2=0x7f04801f3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:14.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:14.370+0000 7f0484dc7640 1 --2- 172.21.15.5:0/1830399926 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f04800cb270 0x7f04801f3100 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f047400c370 tx=0x7f047400c830 comp rx=0 tx=0).stop 2024-03-24T10:54:14.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:14.370+0000 7f0484dc7640 1 -- 172.21.15.5:0/1830399926 shutdown_connections 2024-03-24T10:54:14.370 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:14.370+0000 7f0484dc7640 1 --2- 172.21.15.5:0/1830399926 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f046006b860 0x7f046006dd20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:14.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:14.370+0000 7f0484dc7640 1 --2- 172.21.15.5:0/1830399926 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f04800cb270 0x7f04801f3100 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:14.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:14.370+0000 7f0484dc7640 1 --2- 172.21.15.5:0/1830399926 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f04800cf530 0x7f0480155440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:14.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:14.370+0000 7f0484dc7640 1 --2- 172.21.15.5:0/1830399926 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f04800cae90 0x7f0480156d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:14.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:14.370+0000 7f0484dc7640 1 -- 172.21.15.5:0/1830399926 >> 172.21.15.5:0/1830399926 conn(0x7f04800cfb80 msgr2=0x7f04800c7970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:14.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:14.370+0000 7f0484dc7640 1 -- 172.21.15.5:0/1830399926 shutdown_connections 2024-03-24T10:54:14.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:14.370+0000 7f0484dc7640 1 -- 172.21.15.5:0/1830399926 wait complete. 2024-03-24T10:54:14.985 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:14.985 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:15.986 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:16.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.087+0000 7f57901dd640 1 Processor -- start 2024-03-24T10:54:16.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.088+0000 7f57901dd640 1 -- start start 2024-03-24T10:54:16.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.088+0000 7f57901dd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f57880c59c0 0x7f578806b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:16.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.088+0000 7f57901dd640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57880c4d70 con 0x7f57881548e0 2024-03-24T10:54:16.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.088+0000 7f57901dd640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57880ca160 con 0x7f5788154500 2024-03-24T10:54:16.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.088+0000 7f57901dd640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f57880c9550 con 0x7f57880c59c0 2024-03-24T10:54:16.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.089+0000 7f578d778640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f57881548e0 0x7f57880c52b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:42228/0 (socket says 172.21.15.5:42228) 2024-03-24T10:54:16.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.089+0000 7f578d778640 1 -- 172.21.15.5:0/5730704 learned_addr learned my addr 172.21.15.5:0/5730704 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:54:16.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.090+0000 7f578e77a640 1 --2- 172.21.15.5:0/5730704 >> v2:172.21.15.57:3300/0 conn(0x7f57880c59c0 0x7f578806b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:16.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.092+0000 7f578cf77640 1 -- 172.21.15.5:0/5730704 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2075108321 0 0) 0x7f57880c4d70 con 0x7f57881548e0 2024-03-24T10:54:16.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.092+0000 7f578cf77640 1 -- 172.21.15.5:0/5730704 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5774003520 con 0x7f57881548e0 2024-03-24T10:54:16.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.095+0000 7f578e77a640 1 -- 172.21.15.5:0/5730704 >> v1:172.21.15.57:6789/0 conn(0x7f5788154500 legacy=0x7f57880cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:16.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.095+0000 7f578e77a640 1 -- 172.21.15.5:0/5730704 >> v1:172.21.15.5:6789/0 conn(0x7f57881548e0 legacy=0x7f57880c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:16.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.095+0000 7f578e77a640 1 -- 172.21.15.5:0/5730704 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57880c9830 con 0x7f57880c59c0 2024-03-24T10:54:16.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.101+0000 7f578e77a640 1 --2- 172.21.15.5:0/5730704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57880c59c0 0x7f578806b8c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5780009d90 tx=0x7f578002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d18d0f25c18a3a6c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.106+0000 7f578cf77640 1 -- 172.21.15.5:0/5730704 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f578002cd00 con 0x7f57880c59c0 2024-03-24T10:54:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.106+0000 7f578cf77640 1 -- 172.21.15.5:0/5730704 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f578002ce60 con 0x7f57880c59c0 2024-03-24T10:54:16.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.106+0000 7f578cf77640 1 -- 172.21.15.5:0/5730704 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5780034660 con 0x7f57880c59c0 2024-03-24T10:54:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.106+0000 7f57901dd640 1 -- 172.21.15.5:0/5730704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57880c59c0 msgr2=0x7f578806b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.106+0000 7f57901dd640 1 --2- 172.21.15.5:0/5730704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57880c59c0 0x7f578806b8c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5780009d90 tx=0x7f578002bf50 comp rx=0 tx=0).stop 2024-03-24T10:54:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.106+0000 7f57901dd640 1 -- 172.21.15.5:0/5730704 shutdown_connections 2024-03-24T10:54:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.106+0000 7f57901dd640 1 --2- 172.21.15.5:0/5730704 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f57880c59c0 0x7f578806b8c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.106+0000 7f57901dd640 1 -- 172.21.15.5:0/5730704 >> 172.21.15.5:0/5730704 conn(0x7f578806c240 msgr2=0x7f578806c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.107+0000 7f57901dd640 1 -- 172.21.15.5:0/5730704 shutdown_connections 2024-03-24T10:54:16.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.107+0000 7f57901dd640 1 -- 172.21.15.5:0/5730704 wait complete. 2024-03-24T10:54:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.108+0000 7f57901dd640 1 Processor -- start 2024-03-24T10:54:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.108+0000 7f57901dd640 1 -- start start 2024-03-24T10:54:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.108+0000 7f57901dd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f57880c59c0 0x7f5788157010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:16.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.108+0000 7f57901dd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5788154500 0x7f5788148140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:16.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.108+0000 7f57901dd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57881548e0 0x7f57881491e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:16.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.109+0000 7f57901dd640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f5788149720 con 0x7f57881548e0 2024-03-24T10:54:16.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.109+0000 7f57901dd640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5788157bf0 con 0x7f5788154500 2024-03-24T10:54:16.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.109+0000 7f57901dd640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f5788156060 con 0x7f57880c59c0 2024-03-24T10:54:16.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.109+0000 7f578d778640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5788154500 0x7f5788148140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:16.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.109+0000 7f578df79640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f57880c59c0 0x7f5788157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:16.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.110+0000 7f578d778640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5788154500 0x7f5788148140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:50478/0 (socket says 172.21.15.5:50478) 2024-03-24T10:54:16.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.110+0000 7f578d778640 1 -- 172.21.15.5:0/1330269870 learned_addr learned my addr 172.21.15.5:0/1330269870 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:16.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.110+0000 7f578e77a640 1 --2- 172.21.15.5:0/1330269870 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57881548e0 0x7f57881491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:16.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.115+0000 7f578df79640 1 -- 172.21.15.5:0/1330269870 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5788154500 msgr2=0x7f5788148140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:16.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.115+0000 7f578df79640 1 --2- 172.21.15.5:0/1330269870 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5788154500 0x7f5788148140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:16.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.115+0000 7f578df79640 1 -- 172.21.15.5:0/1330269870 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57881548e0 msgr2=0x7f57881491e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:16.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.115+0000 7f578df79640 1 --2- 172.21.15.5:0/1330269870 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57881548e0 0x7f57881491e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:16.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.115+0000 7f578df79640 1 -- 172.21.15.5:0/1330269870 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5780009a70 con 0x7f57880c59c0 2024-03-24T10:54:16.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.118+0000 7f578df79640 1 --2- 172.21.15.5:0/1330269870 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f57880c59c0 0x7f5788157010 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f578400c340 tx=0x7f578400c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:16.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.126+0000 7f577e7fc640 1 -- 172.21.15.5:0/1330269870 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57840052d0 con 0x7f57880c59c0 2024-03-24T10:54:16.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.126+0000 7f577e7fc640 1 -- 172.21.15.5:0/1330269870 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5784005450 con 0x7f57880c59c0 2024-03-24T10:54:16.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.126+0000 7f57901dd640 1 -- 172.21.15.5:0/1330269870 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57881564f0 con 0x7f57880c59c0 2024-03-24T10:54:16.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.126+0000 7f577e7fc640 1 -- 172.21.15.5:0/1330269870 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5784007260 con 0x7f57880c59c0 2024-03-24T10:54:16.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.126+0000 7f57901dd640 1 -- 172.21.15.5:0/1330269870 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5788157910 con 0x7f57880c59c0 2024-03-24T10:54:16.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.127+0000 7f57901dd640 1 -- 172.21.15.5:0/1330269870 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5788152ad0 con 0x7f57880c59c0 2024-03-24T10:54:16.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.435+0000 7f577e7fc640 1 -- 172.21.15.5:0/1330269870 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f57840073c0 con 0x7f57880c59c0 2024-03-24T10:54:16.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.435+0000 7f577e7fc640 1 --2- 172.21.15.5:0/1330269870 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f576c06b860 0x7f576c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:16.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.435+0000 7f578d778640 1 --2- 172.21.15.5:0/1330269870 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f576c06b860 0x7f576c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:16.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.436+0000 7f578d778640 1 --2- 172.21.15.5:0/1330269870 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f576c06b860 0x7f576c06dd20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f57880c3590 tx=0x7f5778009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:16.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.442+0000 7f577e7fc640 1 -- 172.21.15.5:0/1330269870 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f57840584a0 con 0x7f57880c59c0 2024-03-24T10:54:16.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.459+0000 7f577e7fc640 1 -- 172.21.15.5:0/1330269870 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5784054740 con 0x7f57880c59c0 2024-03-24T10:54:16.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.701+0000 7f57901dd640 1 -- 172.21.15.5:0/1330269870 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57880c91e0 con 0x7f57880c59c0 2024-03-24T10:54:16.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.787+0000 7f577e7fc640 1 -- 172.21.15.5:0/1330269870 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f5784057e00 con 0x7f57880c59c0 2024-03-24T10:54:16.787 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:16.788 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:16.788 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:16.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.790+0000 7f57901dd640 1 -- 172.21.15.5:0/1330269870 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f576c06b860 msgr2=0x7f576c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:16.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.790+0000 7f57901dd640 1 --2- 172.21.15.5:0/1330269870 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f576c06b860 0x7f576c06dd20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f57880c3590 tx=0x7f5778009040 comp rx=0 tx=0).stop 2024-03-24T10:54:16.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.790+0000 7f57901dd640 1 -- 172.21.15.5:0/1330269870 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f57880c59c0 msgr2=0x7f5788157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:16.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.790+0000 7f57901dd640 1 --2- 172.21.15.5:0/1330269870 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f57880c59c0 0x7f5788157010 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f578400c340 tx=0x7f578400c800 comp rx=0 tx=0).stop 2024-03-24T10:54:16.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.790+0000 7f57901dd640 1 -- 172.21.15.5:0/1330269870 shutdown_connections 2024-03-24T10:54:16.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.790+0000 7f57901dd640 1 --2- 172.21.15.5:0/1330269870 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f576c06b860 0x7f576c06dd20 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:16.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.790+0000 7f57901dd640 1 --2- 172.21.15.5:0/1330269870 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f57881548e0 0x7f57881491e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:16.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.790+0000 7f57901dd640 1 --2- 172.21.15.5:0/1330269870 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5788154500 0x7f5788148140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:16.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.790+0000 7f57901dd640 1 --2- 172.21.15.5:0/1330269870 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f57880c59c0 0x7f5788157010 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:16.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.790+0000 7f57901dd640 1 -- 172.21.15.5:0/1330269870 >> 172.21.15.5:0/1330269870 conn(0x7f578806c240 msgr2=0x7f5788153120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:16.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.790+0000 7f57901dd640 1 -- 172.21.15.5:0/1330269870 shutdown_connections 2024-03-24T10:54:16.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:16.790+0000 7f57901dd640 1 -- 172.21.15.5:0/1330269870 wait complete. 2024-03-24T10:54:17.405 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:17.405 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:18.406 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:18.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.509+0000 7f3832c2a640 1 Processor -- start 2024-03-24T10:54:18.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.509+0000 7f3832c2a640 1 -- start start 2024-03-24T10:54:18.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.510+0000 7f3832c2a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f382c1543a0 0x7f382c15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:18.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.510+0000 7f3832c2a640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f382c06d4f0 0x7f382c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:18.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.510+0000 7f3832c2a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f382c0cb2f0 con 0x7f382c0ccaa0 2024-03-24T10:54:18.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.510+0000 7f3832c2a640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f382c0c6750 con 0x7f382c1543a0 2024-03-24T10:54:18.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.510+0000 7f3832c2a640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f382c0cc380 con 0x7f382c06d4f0 2024-03-24T10:54:18.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.511+0000 7f38309c6640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f382c1543a0 0x7f382c15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:18.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.511+0000 7f38309c6640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f382c1543a0 0x7f382c15ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50282/0 (socket says 172.21.15.5:50282) 2024-03-24T10:54:18.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.511+0000 7f38309c6640 1 -- 172.21.15.5:0/2431301296 learned_addr learned my addr 172.21.15.5:0/2431301296 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:18.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.511+0000 7f382bfff640 1 --2- 172.21.15.5:0/2431301296 >> v2:172.21.15.57:3300/0 conn(0x7f382c06d4f0 0x7f382c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:18.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.513+0000 7f38309c6640 1 -- 172.21.15.5:0/2431301296 >> v2:172.21.15.57:3300/0 conn(0x7f382c06d4f0 msgr2=0x7f382c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:54:18.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.513+0000 7f38309c6640 1 --2- 172.21.15.5:0/2431301296 >> v2:172.21.15.57:3300/0 conn(0x7f382c06d4f0 0x7f382c0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:18.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.513+0000 7f38309c6640 1 -- 172.21.15.5:0/2431301296 >> v1:172.21.15.57:6789/0 conn(0x7f382c0ccaa0 legacy=0x7f382c06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:18.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.513+0000 7f38309c6640 1 -- 172.21.15.5:0/2431301296 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f382c0ca7e0 con 0x7f382c1543a0 2024-03-24T10:54:18.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.515+0000 7f38309c6640 1 --2- 172.21.15.5:0/2431301296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f382c1543a0 0x7f382c15ac00 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f382c0c6990 tx=0x7f382002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b7ceb817ea007d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:18.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.519+0000 7f382b7fe640 1 -- 172.21.15.5:0/2431301296 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382002f850 con 0x7f382c1543a0 2024-03-24T10:54:18.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.519+0000 7f382b7fe640 1 -- 172.21.15.5:0/2431301296 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f382002f9b0 con 0x7f382c1543a0 2024-03-24T10:54:18.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.519+0000 7f382b7fe640 1 -- 172.21.15.5:0/2431301296 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382002fd40 con 0x7f382c1543a0 2024-03-24T10:54:18.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.520+0000 7f3832c2a640 1 -- 172.21.15.5:0/2431301296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f382c1543a0 msgr2=0x7f382c15ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:18.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.520+0000 7f3832c2a640 1 --2- 172.21.15.5:0/2431301296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f382c1543a0 0x7f382c15ac00 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f382c0c6990 tx=0x7f382002ba30 comp rx=0 tx=0).stop 2024-03-24T10:54:18.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.520+0000 7f3832c2a640 1 -- 172.21.15.5:0/2431301296 shutdown_connections 2024-03-24T10:54:18.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.520+0000 7f3832c2a640 1 --2- 172.21.15.5:0/2431301296 >> v2:172.21.15.57:3300/0 conn(0x7f382c06d4f0 0x7f382c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:18.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.520+0000 7f3832c2a640 1 --2- 172.21.15.5:0/2431301296 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f382c1543a0 0x7f382c15ac00 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:18.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.520+0000 7f3832c2a640 1 -- 172.21.15.5:0/2431301296 >> 172.21.15.5:0/2431301296 conn(0x7f382c072ef0 msgr2=0x7f382c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:18.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.520+0000 7f3832c2a640 1 -- 172.21.15.5:0/2431301296 shutdown_connections 2024-03-24T10:54:18.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.520+0000 7f3832c2a640 1 -- 172.21.15.5:0/2431301296 wait complete. 2024-03-24T10:54:18.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.522+0000 7f3832c2a640 1 Processor -- start 2024-03-24T10:54:18.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.522+0000 7f3832c2a640 1 -- start start 2024-03-24T10:54:18.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.523+0000 7f3832c2a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f382c06d4f0 0x7f382c0b4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:18.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.523+0000 7f3832c2a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f382c0ccaa0 0x7f382c15c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:18.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.523+0000 7f3832c2a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f382c1543a0 0x7f382c157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:18.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.523+0000 7f3832c2a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f382c1f3dd0 con 0x7f382c0ccaa0 2024-03-24T10:54:18.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.523+0000 7f3832c2a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f382c154a60 con 0x7f382c1543a0 2024-03-24T10:54:18.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.523+0000 7f3832c2a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f382c0b47b0 con 0x7f382c06d4f0 2024-03-24T10:54:18.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.524+0000 7f38311c7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f382c1543a0 0x7f382c157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:18.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.524+0000 7f382bfff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f382c0ccaa0 0x7f382c15c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:18.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.524+0000 7f38309c6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f382c06d4f0 0x7f382c0b4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:18.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.524+0000 7f38311c7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f382c1543a0 0x7f382c157480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:50500/0 (socket says 172.21.15.5:50500) 2024-03-24T10:54:18.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.524+0000 7f38311c7640 1 -- 172.21.15.5:0/1301528008 learned_addr learned my addr 172.21.15.5:0/1301528008 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:18.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.528+0000 7f38311c7640 1 -- 172.21.15.5:0/1301528008 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f382c06d4f0 msgr2=0x7f382c0b4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:18.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.528+0000 7f38311c7640 1 --2- 172.21.15.5:0/1301528008 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f382c06d4f0 0x7f382c0b4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:18.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.528+0000 7f38311c7640 1 -- 172.21.15.5:0/1301528008 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f382c0ccaa0 msgr2=0x7f382c15c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:18.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.528+0000 7f38311c7640 1 --2- 172.21.15.5:0/1301528008 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f382c0ccaa0 0x7f382c15c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:18.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.528+0000 7f38311c7640 1 -- 172.21.15.5:0/1301528008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38200094d0 con 0x7f382c1543a0 2024-03-24T10:54:18.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.531+0000 7f38311c7640 1 --2- 172.21.15.5:0/1301528008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f382c1543a0 0x7f382c157480 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f382400c7d0 tx=0x7f382400cc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:18.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.536+0000 7f38297fa640 1 -- 172.21.15.5:0/1301528008 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3824003b70 con 0x7f382c1543a0 2024-03-24T10:54:18.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.537+0000 7f38297fa640 1 -- 172.21.15.5:0/1301528008 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3824003cf0 con 0x7f382c1543a0 2024-03-24T10:54:18.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.537+0000 7f3832c2a640 1 -- 172.21.15.5:0/1301528008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f382c0b30f0 con 0x7f382c1543a0 2024-03-24T10:54:18.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.537+0000 7f38297fa640 1 -- 172.21.15.5:0/1301528008 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382400aa90 con 0x7f382c1543a0 2024-03-24T10:54:18.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.537+0000 7f3832c2a640 1 -- 172.21.15.5:0/1301528008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f382c1f3af0 con 0x7f382c1543a0 2024-03-24T10:54:18.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.538+0000 7f3832c2a640 1 -- 172.21.15.5:0/1301528008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f382c0681d0 con 0x7f382c1543a0 2024-03-24T10:54:18.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.563+0000 7f38297fa640 1 -- 172.21.15.5:0/1301528008 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f382400abf0 con 0x7f382c1543a0 2024-03-24T10:54:18.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.563+0000 7f38297fa640 1 --2- 172.21.15.5:0/1301528008 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f381006b890 0x7f381006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:18.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.563+0000 7f38309c6640 1 --2- 172.21.15.5:0/1301528008 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f381006b890 0x7f381006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:18.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.564+0000 7f38309c6640 1 --2- 172.21.15.5:0/1301528008 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f381006b890 0x7f381006dd50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3820036000 tx=0x7f3820008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:18.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.566+0000 7f38297fa640 1 -- 172.21.15.5:0/1301528008 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f3824059170 con 0x7f382c1543a0 2024-03-24T10:54:18.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:18.852+0000 7f38297fa640 1 -- 172.21.15.5:0/1301528008 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3824055410 con 0x7f382c1543a0 2024-03-24T10:54:19.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:19.079+0000 7f3832c2a640 1 -- 172.21.15.5:0/1301528008 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f382c0b48f0 con 0x7f382c1543a0 2024-03-24T10:54:19.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:19.094+0000 7f38297fa640 1 -- 172.21.15.5:0/1301528008 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f3824003e60 con 0x7f382c1543a0 2024-03-24T10:54:19.094 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:19.094 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:19.094 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:19.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:19.096+0000 7f3832c2a640 1 -- 172.21.15.5:0/1301528008 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f381006b890 msgr2=0x7f381006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:19.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:19.096+0000 7f3832c2a640 1 --2- 172.21.15.5:0/1301528008 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f381006b890 0x7f381006dd50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f3820036000 tx=0x7f3820008030 comp rx=0 tx=0).stop 2024-03-24T10:54:19.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:19.096+0000 7f3832c2a640 1 -- 172.21.15.5:0/1301528008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f382c1543a0 msgr2=0x7f382c157480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:19.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:19.096+0000 7f3832c2a640 1 --2- 172.21.15.5:0/1301528008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f382c1543a0 0x7f382c157480 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f382400c7d0 tx=0x7f382400cc90 comp rx=0 tx=0).stop 2024-03-24T10:54:19.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:19.096+0000 7f3832c2a640 1 -- 172.21.15.5:0/1301528008 shutdown_connections 2024-03-24T10:54:19.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:19.096+0000 7f3832c2a640 1 --2- 172.21.15.5:0/1301528008 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f381006b890 0x7f381006dd50 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:19.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:19.096+0000 7f3832c2a640 1 --2- 172.21.15.5:0/1301528008 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f382c0ccaa0 0x7f382c15c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:19.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:19.096+0000 7f3832c2a640 1 --2- 172.21.15.5:0/1301528008 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f382c1543a0 0x7f382c157480 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:19.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:19.096+0000 7f3832c2a640 1 --2- 172.21.15.5:0/1301528008 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f382c06d4f0 0x7f382c0b4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:19.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:19.096+0000 7f3832c2a640 1 -- 172.21.15.5:0/1301528008 >> 172.21.15.5:0/1301528008 conn(0x7f382c072ef0 msgr2=0x7f382c153480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:19.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:19.096+0000 7f3832c2a640 1 -- 172.21.15.5:0/1301528008 shutdown_connections 2024-03-24T10:54:19.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:19.096+0000 7f3832c2a640 1 -- 172.21.15.5:0/1301528008 wait complete. 2024-03-24T10:54:19.712 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:19.713 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:20.714 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:20.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.817+0000 7f5209bd6640 1 Processor -- start 2024-03-24T10:54:20.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.817+0000 7f5209bd6640 1 -- start start 2024-03-24T10:54:20.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.817+0000 7f5209bd6640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f520415ac00 0x7f52040cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:20.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.817+0000 7f5209bd6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f52040ccaa0 0x7f520406bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:20.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.817+0000 7f5209bd6640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52040cb2f0 con 0x7f52041543a0 2024-03-24T10:54:20.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.817+0000 7f5209bd6640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f52040c6750 con 0x7f520415ac00 2024-03-24T10:54:20.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.817+0000 7f5209bd6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f52040cc380 con 0x7f52040ccaa0 2024-03-24T10:54:20.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.818+0000 7f5203fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f52040ccaa0 0x7f520406bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:20.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.818+0000 7f51faffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f520415ac00 0x7f52040cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:20.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.818+0000 7f52037fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f52041543a0 0x7f520406d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:42240/0 (socket says 172.21.15.5:42240) 2024-03-24T10:54:20.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.818+0000 7f52037fe640 1 -- 172.21.15.5:0/2003181720 learned_addr learned my addr 172.21.15.5:0/2003181720 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:54:20.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.821+0000 7f5203fff640 1 -- 172.21.15.5:0/2003181720 >> v2:172.21.15.5:3300/0 conn(0x7f520415ac00 msgr2=0x7f52040cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:54:20.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.821+0000 7f5203fff640 1 --2- 172.21.15.5:0/2003181720 >> v2:172.21.15.5:3300/0 conn(0x7f520415ac00 0x7f52040cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:20.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.821+0000 7f5203fff640 1 -- 172.21.15.5:0/2003181720 >> v1:172.21.15.5:6789/0 conn(0x7f52041543a0 legacy=0x7f520406d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:20.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.821+0000 7f5203fff640 1 -- 172.21.15.5:0/2003181720 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f520415b070 con 0x7f52040ccaa0 2024-03-24T10:54:20.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.823+0000 7f5203fff640 1 --2- 172.21.15.5:0/2003181720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52040ccaa0 0x7f520406bf50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f52040b51c0 tx=0x7f51fc02bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76983ea1b010c2b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:20.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.828+0000 7f5202ffd640 1 -- 172.21.15.5:0/2003181720 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc02cc80 con 0x7f52040ccaa0 2024-03-24T10:54:20.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.828+0000 7f5202ffd640 1 -- 172.21.15.5:0/2003181720 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51fc02cde0 con 0x7f52040ccaa0 2024-03-24T10:54:20.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.828+0000 7f5202ffd640 1 -- 172.21.15.5:0/2003181720 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc034660 con 0x7f52040ccaa0 2024-03-24T10:54:20.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.829+0000 7f5209bd6640 1 -- 172.21.15.5:0/2003181720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52040ccaa0 msgr2=0x7f520406bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:20.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.829+0000 7f5209bd6640 1 --2- 172.21.15.5:0/2003181720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52040ccaa0 0x7f520406bf50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f52040b51c0 tx=0x7f51fc02bed0 comp rx=0 tx=0).stop 2024-03-24T10:54:20.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.829+0000 7f5209bd6640 1 -- 172.21.15.5:0/2003181720 shutdown_connections 2024-03-24T10:54:20.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.829+0000 7f5209bd6640 1 --2- 172.21.15.5:0/2003181720 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52040ccaa0 0x7f520406bf50 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:20.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.829+0000 7f5209bd6640 1 --2- 172.21.15.5:0/2003181720 >> v2:172.21.15.5:3300/0 conn(0x7f520415ac00 0x7f52040cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:20.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.829+0000 7f5209bd6640 1 -- 172.21.15.5:0/2003181720 >> 172.21.15.5:0/2003181720 conn(0x7f5204072ef0 msgr2=0x7f5204061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:20.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.829+0000 7f5209bd6640 1 -- 172.21.15.5:0/2003181720 shutdown_connections 2024-03-24T10:54:20.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.829+0000 7f5209bd6640 1 -- 172.21.15.5:0/2003181720 wait complete. 2024-03-24T10:54:20.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.830+0000 7f5209bd6640 1 Processor -- start 2024-03-24T10:54:20.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.830+0000 7f5209bd6640 1 -- start start 2024-03-24T10:54:20.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.831+0000 7f5209bd6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f52040ccaa0 0x7f52041f3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:20.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.831+0000 7f5209bd6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52041543a0 0x7f52041f03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:20.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.831+0000 7f5209bd6640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520415ac00 0x7f520414f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:20.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.831+0000 7f5209bd6640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f5204147b20 con 0x7f520415ac00 2024-03-24T10:54:20.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.831+0000 7f5209bd6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5204145660 con 0x7f52041543a0 2024-03-24T10:54:20.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.831+0000 7f5209bd6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f52041ef9b0 con 0x7f52040ccaa0 2024-03-24T10:54:20.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.831+0000 7f5203fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520415ac00 0x7f520414f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:20.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.831+0000 7f52037fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f52040ccaa0 0x7f52041f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:20.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.831+0000 7f51faffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52041543a0 0x7f52041f03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:20.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.832+0000 7f5203fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520415ac00 0x7f520414f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50318/0 (socket says 172.21.15.5:50318) 2024-03-24T10:54:20.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.832+0000 7f5203fff640 1 -- 172.21.15.5:0/2957785763 learned_addr learned my addr 172.21.15.5:0/2957785763 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:20.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.835+0000 7f52037fe640 1 -- 172.21.15.5:0/2957785763 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52041543a0 msgr2=0x7f52041f03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:20.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.835+0000 7f52037fe640 1 --2- 172.21.15.5:0/2957785763 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52041543a0 0x7f52041f03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:20.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.835+0000 7f52037fe640 1 -- 172.21.15.5:0/2957785763 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520415ac00 msgr2=0x7f520414f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:20.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.835+0000 7f52037fe640 1 --2- 172.21.15.5:0/2957785763 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520415ac00 0x7f520414f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:20.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.835+0000 7f52037fe640 1 -- 172.21.15.5:0/2957785763 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51fc009a70 con 0x7f52040ccaa0 2024-03-24T10:54:20.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.838+0000 7f52037fe640 1 --2- 172.21.15.5:0/2957785763 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f52040ccaa0 0x7f52041f3930 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f51f400c370 tx=0x7f51f400c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:20.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.840+0000 7f5200ff9640 1 -- 172.21.15.5:0/2957785763 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f4005ce0 con 0x7f52040ccaa0 2024-03-24T10:54:20.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.840+0000 7f5209bd6640 1 -- 172.21.15.5:0/2957785763 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52041eff30 con 0x7f52040ccaa0 2024-03-24T10:54:20.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.840+0000 7f5209bd6640 1 -- 172.21.15.5:0/2957785763 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5204144dc0 con 0x7f52040ccaa0 2024-03-24T10:54:20.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:20.841+0000 7f5209bd6640 1 -- 172.21.15.5:0/2957785763 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5204145100 con 0x7f52040ccaa0 2024-03-24T10:54:21.051 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.051+0000 7f5200ff9640 1 -- 172.21.15.5:0/2957785763 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51f4005e40 con 0x7f52040ccaa0 2024-03-24T10:54:21.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.178+0000 7f5200ff9640 1 -- 172.21.15.5:0/2957785763 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f4005230 con 0x7f52040ccaa0 2024-03-24T10:54:21.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.178+0000 7f5200ff9640 1 -- 172.21.15.5:0/2957785763 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f51f4005450 con 0x7f52040ccaa0 2024-03-24T10:54:21.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.179+0000 7f5200ff9640 1 --2- 172.21.15.5:0/2957785763 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f51e406b840 0x7f51e406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:21.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.179+0000 7f51faffd640 1 --2- 172.21.15.5:0/2957785763 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f51e406b840 0x7f51e406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:21.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.179+0000 7f51faffd640 1 --2- 172.21.15.5:0/2957785763 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f51e406b840 0x7f51e406dd00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5204150730 tx=0x7f51ec009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:21.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.186+0000 7f5200ff9640 1 -- 172.21.15.5:0/2957785763 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f51f4058330 con 0x7f52040ccaa0 2024-03-24T10:54:21.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.207+0000 7f5200ff9640 1 -- 172.21.15.5:0/2957785763 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f51f40545d0 con 0x7f52040ccaa0 2024-03-24T10:54:21.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.430+0000 7f5209bd6640 1 -- 172.21.15.5:0/2957785763 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52040681d0 con 0x7f52040ccaa0 2024-03-24T10:54:21.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.454+0000 7f5200ff9640 1 -- 172.21.15.5:0/2957785763 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f51f4017090 con 0x7f52040ccaa0 2024-03-24T10:54:21.454 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:21.455 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:21.455 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:21.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.456+0000 7f5209bd6640 1 -- 172.21.15.5:0/2957785763 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f51e406b840 msgr2=0x7f51e406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:21.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.456+0000 7f5209bd6640 1 --2- 172.21.15.5:0/2957785763 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f51e406b840 0x7f51e406dd00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5204150730 tx=0x7f51ec009190 comp rx=0 tx=0).stop 2024-03-24T10:54:21.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.456+0000 7f5209bd6640 1 -- 172.21.15.5:0/2957785763 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f52040ccaa0 msgr2=0x7f52041f3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:21.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.456+0000 7f5209bd6640 1 --2- 172.21.15.5:0/2957785763 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f52040ccaa0 0x7f52041f3930 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f51f400c370 tx=0x7f51f400c830 comp rx=0 tx=0).stop 2024-03-24T10:54:21.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.456+0000 7f5209bd6640 1 -- 172.21.15.5:0/2957785763 shutdown_connections 2024-03-24T10:54:21.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.456+0000 7f5209bd6640 1 --2- 172.21.15.5:0/2957785763 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f51e406b840 0x7f51e406dd00 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:21.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.456+0000 7f5209bd6640 1 --2- 172.21.15.5:0/2957785763 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520415ac00 0x7f520414f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:21.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.456+0000 7f5209bd6640 1 --2- 172.21.15.5:0/2957785763 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f52041543a0 0x7f52041f03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:21.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.456+0000 7f5209bd6640 1 --2- 172.21.15.5:0/2957785763 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f52040ccaa0 0x7f52041f3930 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:21.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.456+0000 7f5209bd6640 1 -- 172.21.15.5:0/2957785763 >> 172.21.15.5:0/2957785763 conn(0x7f5204072ef0 msgr2=0x7f5204159580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:21.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.456+0000 7f5209bd6640 1 -- 172.21.15.5:0/2957785763 shutdown_connections 2024-03-24T10:54:21.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:21.456+0000 7f5209bd6640 1 -- 172.21.15.5:0/2957785763 wait complete. 2024-03-24T10:54:22.071 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:22.071 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:23.072 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:23.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.176+0000 7f93ff447640 1 Processor -- start 2024-03-24T10:54:23.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.176+0000 7f93ff447640 1 -- start start 2024-03-24T10:54:23.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.177+0000 7f93ff447640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f93f81543a0 0x7f93f815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:23.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.177+0000 7f93ff447640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f93f806bf50 0x7f93f806c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:23.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.177+0000 7f93ff447640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93f80cb2f0 con 0x7f93f806d4f0 2024-03-24T10:54:23.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.177+0000 7f93ff447640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f93f80c6750 con 0x7f93f806bf50 2024-03-24T10:54:23.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.177+0000 7f93ff447640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f93f80cc380 con 0x7f93f81543a0 2024-03-24T10:54:23.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.178+0000 7f93fc9e2640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f93f806d4f0 0x7f93f80cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:50424/0 (socket says 172.21.15.5:50424) 2024-03-24T10:54:23.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.178+0000 7f93fc9e2640 1 -- 172.21.15.5:0/1155204685 learned_addr learned my addr 172.21.15.5:0/1155204685 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:54:23.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.178+0000 7f93fd1e3640 1 --2- 172.21.15.5:0/1155204685 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f93f81543a0 0x7f93f815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:23.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.178+0000 7f93fd9e4640 1 --2- 172.21.15.5:0/1155204685 >> v2:172.21.15.57:3300/0 conn(0x7f93f806bf50 0x7f93f806c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:23.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.181+0000 7f93effff640 1 -- 172.21.15.5:0/1155204685 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 248637212 0 0) 0x7f93f80cb2f0 con 0x7f93f806d4f0 2024-03-24T10:54:23.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.181+0000 7f93effff640 1 -- 172.21.15.5:0/1155204685 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93e4003520 con 0x7f93f806d4f0 2024-03-24T10:54:23.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.182+0000 7f93fd9e4640 1 -- 172.21.15.5:0/1155204685 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f93f81543a0 msgr2=0x7f93f815ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:54:23.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.182+0000 7f93fd9e4640 1 --2- 172.21.15.5:0/1155204685 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f93f81543a0 0x7f93f815ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:23.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.182+0000 7f93fd9e4640 1 -- 172.21.15.5:0/1155204685 >> v1:172.21.15.5:6789/0 conn(0x7f93f806d4f0 legacy=0x7f93f80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:23.182 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.182+0000 7f93fd9e4640 1 -- 172.21.15.5:0/1155204685 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93f80ca7e0 con 0x7f93f806bf50 2024-03-24T10:54:23.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.185+0000 7f93fd9e4640 1 --2- 172.21.15.5:0/1155204685 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93f806bf50 0x7f93f806c380 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f93f0009d90 tx=0x7f93f002bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8a7c21578d28970 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:23.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.191+0000 7f93effff640 1 -- 172.21.15.5:0/1155204685 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f002cce0 con 0x7f93f806bf50 2024-03-24T10:54:23.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.191+0000 7f93effff640 1 -- 172.21.15.5:0/1155204685 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93f002ce40 con 0x7f93f806bf50 2024-03-24T10:54:23.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.191+0000 7f93effff640 1 -- 172.21.15.5:0/1155204685 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f0034660 con 0x7f93f806bf50 2024-03-24T10:54:23.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.191+0000 7f93ff447640 1 -- 172.21.15.5:0/1155204685 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93f806bf50 msgr2=0x7f93f806c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:23.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.191+0000 7f93ff447640 1 --2- 172.21.15.5:0/1155204685 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93f806bf50 0x7f93f806c380 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f93f0009d90 tx=0x7f93f002bf30 comp rx=0 tx=0).stop 2024-03-24T10:54:23.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.191+0000 7f93ff447640 1 -- 172.21.15.5:0/1155204685 shutdown_connections 2024-03-24T10:54:23.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.191+0000 7f93ff447640 1 --2- 172.21.15.5:0/1155204685 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93f806bf50 0x7f93f806c380 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:23.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.191+0000 7f93ff447640 1 --2- 172.21.15.5:0/1155204685 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f93f81543a0 0x7f93f815ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:23.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.191+0000 7f93ff447640 1 -- 172.21.15.5:0/1155204685 >> 172.21.15.5:0/1155204685 conn(0x7f93f8072ef0 msgr2=0x7f93f8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:23.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.191+0000 7f93ff447640 1 -- 172.21.15.5:0/1155204685 shutdown_connections 2024-03-24T10:54:23.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.192+0000 7f93ff447640 1 -- 172.21.15.5:0/1155204685 wait complete. 2024-03-24T10:54:23.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.193+0000 7f93ff447640 1 Processor -- start 2024-03-24T10:54:23.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.193+0000 7f93ff447640 1 -- start start 2024-03-24T10:54:23.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.193+0000 7f93ff447640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f93f806bf50 0x7f93f80b4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:23.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.193+0000 7f93ff447640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93f806d4f0 0x7f93f81451a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:23.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.193+0000 7f93ff447640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93f81543a0 0x7f93f81f3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:23.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.193+0000 7f93ff447640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f93f81599c0 con 0x7f93f81543a0 2024-03-24T10:54:23.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.193+0000 7f93ff447640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f93f8146b10 con 0x7f93f806d4f0 2024-03-24T10:54:23.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.193+0000 7f93ff447640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f93f8147690 con 0x7f93f806bf50 2024-03-24T10:54:23.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.194+0000 7f93fd9e4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93f81543a0 0x7f93f81f3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:23.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.194+0000 7f93fd1e3640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f93f806bf50 0x7f93f80b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:23.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.194+0000 7f93fd9e4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93f81543a0 0x7f93f81f3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57906/0 (socket says 172.21.15.5:57906) 2024-03-24T10:54:23.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.194+0000 7f93fd9e4640 1 -- 172.21.15.5:0/3452102923 learned_addr learned my addr 172.21.15.5:0/3452102923 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:23.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.194+0000 7f93fc9e2640 1 --2- 172.21.15.5:0/3452102923 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93f806d4f0 0x7f93f81451a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:23.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.198+0000 7f93fd9e4640 1 -- 172.21.15.5:0/3452102923 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f93f806bf50 msgr2=0x7f93f80b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:23.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.198+0000 7f93fd9e4640 1 --2- 172.21.15.5:0/3452102923 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f93f806bf50 0x7f93f80b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:23.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.198+0000 7f93fd9e4640 1 -- 172.21.15.5:0/3452102923 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93f806d4f0 msgr2=0x7f93f81451a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:23.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.198+0000 7f93fd9e4640 1 --2- 172.21.15.5:0/3452102923 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93f806d4f0 0x7f93f81451a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:23.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.198+0000 7f93fd9e4640 1 -- 172.21.15.5:0/3452102923 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93f0009a70 con 0x7f93f81543a0 2024-03-24T10:54:23.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.201+0000 7f93fd9e4640 1 --2- 172.21.15.5:0/3452102923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93f81543a0 0x7f93f81f3020 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f93f0009d90 tx=0x7f93f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:23.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.203+0000 7f93edffb640 1 -- 172.21.15.5:0/3452102923 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f003e610 con 0x7f93f81543a0 2024-03-24T10:54:23.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.203+0000 7f93ff447640 1 -- 172.21.15.5:0/3452102923 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93f8146dd0 con 0x7f93f81543a0 2024-03-24T10:54:23.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.203+0000 7f93ff447640 1 -- 172.21.15.5:0/3452102923 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93f8159ed0 con 0x7f93f81543a0 2024-03-24T10:54:23.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.204+0000 7f93ff447640 1 -- 172.21.15.5:0/3452102923 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93f8147820 con 0x7f93f81543a0 2024-03-24T10:54:23.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.204+0000 7f93edffb640 1 -- 172.21.15.5:0/3452102923 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93f0034e40 con 0x7f93f81543a0 2024-03-24T10:54:23.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.204+0000 7f93edffb640 1 -- 172.21.15.5:0/3452102923 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f003e990 con 0x7f93f81543a0 2024-03-24T10:54:23.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.476+0000 7f93edffb640 1 -- 172.21.15.5:0/3452102923 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f93f003b050 con 0x7f93f81543a0 2024-03-24T10:54:23.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.476+0000 7f93edffb640 1 --2- 172.21.15.5:0/3452102923 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f93dc06b810 0x7f93dc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:23.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.477+0000 7f93fd1e3640 1 --2- 172.21.15.5:0/3452102923 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f93dc06b810 0x7f93dc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:23.477 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.477+0000 7f93fd1e3640 1 --2- 172.21.15.5:0/3452102923 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f93dc06b810 0x7f93dc06dcd0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f93f4009b40 tx=0x7f93f4009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:23.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.490+0000 7f93edffb640 1 -- 172.21.15.5:0/3452102923 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f93f007b530 con 0x7f93f81543a0 2024-03-24T10:54:23.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.490+0000 7f93edffb640 1 -- 172.21.15.5:0/3452102923 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f93f00777d0 con 0x7f93f81543a0 2024-03-24T10:54:23.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.740+0000 7f93ff447640 1 -- 172.21.15.5:0/3452102923 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93f80681d0 con 0x7f93f81543a0 2024-03-24T10:54:23.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.761+0000 7f93edffb640 1 -- 172.21.15.5:0/3452102923 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f93f004e090 con 0x7f93f81543a0 2024-03-24T10:54:23.762 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:23.762 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:23.762 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:23.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.764+0000 7f93ff447640 1 -- 172.21.15.5:0/3452102923 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f93dc06b810 msgr2=0x7f93dc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:23.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.764+0000 7f93ff447640 1 --2- 172.21.15.5:0/3452102923 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f93dc06b810 0x7f93dc06dcd0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f93f4009b40 tx=0x7f93f4009190 comp rx=0 tx=0).stop 2024-03-24T10:54:23.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.764+0000 7f93ff447640 1 -- 172.21.15.5:0/3452102923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93f81543a0 msgr2=0x7f93f81f3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:23.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.764+0000 7f93ff447640 1 --2- 172.21.15.5:0/3452102923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93f81543a0 0x7f93f81f3020 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f93f0009d90 tx=0x7f93f0004a40 comp rx=0 tx=0).stop 2024-03-24T10:54:23.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.764+0000 7f93ff447640 1 -- 172.21.15.5:0/3452102923 shutdown_connections 2024-03-24T10:54:23.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.764+0000 7f93ff447640 1 --2- 172.21.15.5:0/3452102923 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f93dc06b810 0x7f93dc06dcd0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:23.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.764+0000 7f93ff447640 1 --2- 172.21.15.5:0/3452102923 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93f81543a0 0x7f93f81f3020 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:23.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.764+0000 7f93ff447640 1 --2- 172.21.15.5:0/3452102923 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93f806d4f0 0x7f93f81451a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:23.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.764+0000 7f93ff447640 1 --2- 172.21.15.5:0/3452102923 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f93f806bf50 0x7f93f80b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:23.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.764+0000 7f93ff447640 1 -- 172.21.15.5:0/3452102923 >> 172.21.15.5:0/3452102923 conn(0x7f93f8072ef0 msgr2=0x7f93f81535a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:23.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.764+0000 7f93ff447640 1 -- 172.21.15.5:0/3452102923 shutdown_connections 2024-03-24T10:54:23.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:23.764+0000 7f93ff447640 1 -- 172.21.15.5:0/3452102923 wait complete. 2024-03-24T10:54:24.378 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:24.379 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:25.379 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:25.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.482+0000 7f1af5d80640 1 Processor -- start 2024-03-24T10:54:25.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.482+0000 7f1af5d80640 1 -- start start 2024-03-24T10:54:25.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.483+0000 7f1af5d80640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1af0150fd0 0x7f1af00c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:25.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.483+0000 7f1af5d80640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1af015e590 0x7f1af006bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:25.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.483+0000 7f1af5d80640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1af006ca40 con 0x7f1af01513b0 2024-03-24T10:54:25.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.483+0000 7f1af5d80640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f1af015ea00 con 0x7f1af015e590 2024-03-24T10:54:25.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.483+0000 7f1af5d80640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1af015e100 con 0x7f1af0150fd0 2024-03-24T10:54:25.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.484+0000 7f1aef7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1af0150fd0 0x7f1af00c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:25.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.484+0000 7f1aef7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1af0150fd0 0x7f1af00c70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:34298/0 (socket says 172.21.15.5:34298) 2024-03-24T10:54:25.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.484+0000 7f1aef7fe640 1 -- 172.21.15.5:0/4097163384 learned_addr learned my addr 172.21.15.5:0/4097163384 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:25.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.486+0000 7f1aeffff640 1 --2- 172.21.15.5:0/4097163384 >> v2:172.21.15.5:3300/0 conn(0x7f1af015e590 0x7f1af006bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:25.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.488+0000 7f1aee7fc640 1 -- 172.21.15.5:0/4097163384 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 597476631 0 0) 0x7f1af006ca40 con 0x7f1af01513b0 2024-03-24T10:54:25.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.488+0000 7f1aee7fc640 1 -- 172.21.15.5:0/4097163384 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ad8003520 con 0x7f1af01513b0 2024-03-24T10:54:25.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.489+0000 7f1aeffff640 1 -- 172.21.15.5:0/4097163384 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1af0150fd0 msgr2=0x7f1af00c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:54:25.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.489+0000 7f1aeffff640 1 --2- 172.21.15.5:0/4097163384 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1af0150fd0 0x7f1af00c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:25.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.489+0000 7f1aeffff640 1 -- 172.21.15.5:0/4097163384 >> v1:172.21.15.57:6789/0 conn(0x7f1af01513b0 legacy=0x7f1af015aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:25.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.489+0000 7f1aeffff640 1 -- 172.21.15.5:0/4097163384 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af006c230 con 0x7f1af015e590 2024-03-24T10:54:25.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.491+0000 7f1aeffff640 1 --2- 172.21.15.5:0/4097163384 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1af015e590 0x7f1af006bcf0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8009d90 tx=0x7f1ae802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35ab054107208223 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:25.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.496+0000 7f1aee7fc640 1 -- 172.21.15.5:0/4097163384 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae802cc50 con 0x7f1af015e590 2024-03-24T10:54:25.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.496+0000 7f1aee7fc640 1 -- 172.21.15.5:0/4097163384 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ae802cdb0 con 0x7f1af015e590 2024-03-24T10:54:25.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.496+0000 7f1aee7fc640 1 -- 172.21.15.5:0/4097163384 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8034660 con 0x7f1af015e590 2024-03-24T10:54:25.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.497+0000 7f1af5d80640 1 -- 172.21.15.5:0/4097163384 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1af015e590 msgr2=0x7f1af006bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:25.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.497+0000 7f1af5d80640 1 --2- 172.21.15.5:0/4097163384 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1af015e590 0x7f1af006bcf0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8009d90 tx=0x7f1ae802bea0 comp rx=0 tx=0).stop 2024-03-24T10:54:25.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.497+0000 7f1af5d80640 1 -- 172.21.15.5:0/4097163384 shutdown_connections 2024-03-24T10:54:25.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.497+0000 7f1af5d80640 1 --2- 172.21.15.5:0/4097163384 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1af015e590 0x7f1af006bcf0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:25.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.497+0000 7f1af5d80640 1 --2- 172.21.15.5:0/4097163384 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1af0150fd0 0x7f1af00c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:25.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.497+0000 7f1af5d80640 1 -- 172.21.15.5:0/4097163384 >> 172.21.15.5:0/4097163384 conn(0x7f1af006f8b0 msgr2=0x7f1af006fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:25.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.497+0000 7f1af5d80640 1 -- 172.21.15.5:0/4097163384 shutdown_connections 2024-03-24T10:54:25.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.497+0000 7f1af5d80640 1 -- 172.21.15.5:0/4097163384 wait complete. 2024-03-24T10:54:25.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.498+0000 7f1af5d80640 1 Processor -- start 2024-03-24T10:54:25.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.498+0000 7f1af5d80640 1 -- start start 2024-03-24T10:54:25.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.499+0000 7f1af5d80640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1af0150fd0 0x7f1af01f0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:25.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.499+0000 7f1af5d80640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1af01513b0 0x7f1af01555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:25.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.499+0000 7f1af5d80640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1af015e590 0x7f1af014bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:25.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.499+0000 7f1af5d80640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f1af01f0b40 con 0x7f1af01513b0 2024-03-24T10:54:25.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.499+0000 7f1af5d80640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1af014d890 con 0x7f1af0150fd0 2024-03-24T10:54:25.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.499+0000 7f1af5d80640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1af014d5b0 con 0x7f1af015e590 2024-03-24T10:54:25.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.500+0000 7f1aef7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1af0150fd0 0x7f1af01f0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:25.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.500+0000 7f1aeffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1af015e590 0x7f1af014bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:25.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.500+0000 7f1aef7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1af0150fd0 0x7f1af01f0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:52020/0 (socket says 172.21.15.5:52020) 2024-03-24T10:54:25.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.500+0000 7f1aef7fe640 1 -- 172.21.15.5:0/3261883390 learned_addr learned my addr 172.21.15.5:0/3261883390 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:25.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.505+0000 7f1aef7fe640 1 -- 172.21.15.5:0/3261883390 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1af015e590 msgr2=0x7f1af014bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:25.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.505+0000 7f1aef7fe640 1 --2- 172.21.15.5:0/3261883390 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1af015e590 0x7f1af014bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:25.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.505+0000 7f1aef7fe640 1 -- 172.21.15.5:0/3261883390 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1af01513b0 msgr2=0x7f1af01555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:25.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.505+0000 7f1aef7fe640 1 --2- 172.21.15.5:0/3261883390 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1af01513b0 0x7f1af01555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:25.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.505+0000 7f1aef7fe640 1 -- 172.21.15.5:0/3261883390 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae8009a70 con 0x7f1af0150fd0 2024-03-24T10:54:25.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.508+0000 7f1aef7fe640 1 --2- 172.21.15.5:0/3261883390 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1af0150fd0 0x7f1af01f0600 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009fd0 tx=0x7f1ae000ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:25.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.513+0000 7f1ad7fff640 1 -- 172.21.15.5:0/3261883390 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae000c940 con 0x7f1af0150fd0 2024-03-24T10:54:25.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.513+0000 7f1ad7fff640 1 -- 172.21.15.5:0/3261883390 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ae000caa0 con 0x7f1af0150fd0 2024-03-24T10:54:25.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.513+0000 7f1af5d80640 1 -- 172.21.15.5:0/3261883390 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1af0147bb0 con 0x7f1af0150fd0 2024-03-24T10:54:25.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.513+0000 7f1ad7fff640 1 -- 172.21.15.5:0/3261883390 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae000ce30 con 0x7f1af0150fd0 2024-03-24T10:54:25.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.513+0000 7f1af5d80640 1 -- 172.21.15.5:0/3261883390 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1af0148170 con 0x7f1af0150fd0 2024-03-24T10:54:25.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.514+0000 7f1af5d80640 1 -- 172.21.15.5:0/3261883390 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1af00681d0 con 0x7f1af0150fd0 2024-03-24T10:54:25.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.781+0000 7f1ad7fff640 1 -- 172.21.15.5:0/3261883390 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1ae0013470 con 0x7f1af0150fd0 2024-03-24T10:54:25.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.782+0000 7f1ad7fff640 1 --2- 172.21.15.5:0/3261883390 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1adc06b810 0x7f1adc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:25.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.782+0000 7f1aeeffd640 1 --2- 172.21.15.5:0/3261883390 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1adc06b810 0x7f1adc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:25.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.782+0000 7f1aeeffd640 1 --2- 172.21.15.5:0/3261883390 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1adc06b810 0x7f1adc06dcd0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1ae40010e0 tx=0x7f1ae4007950 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:25.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.786+0000 7f1ad7fff640 1 -- 172.21.15.5:0/3261883390 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f1ae0059300 con 0x7f1af0150fd0 2024-03-24T10:54:25.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:25.798+0000 7f1ad7fff640 1 -- 172.21.15.5:0/3261883390 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1ae00555a0 con 0x7f1af0150fd0 2024-03-24T10:54:26.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:26.064+0000 7f1af5d80640 1 -- 172.21.15.5:0/3261883390 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1af006bad0 con 0x7f1af0150fd0 2024-03-24T10:54:26.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:26.077+0000 7f1ad7fff640 1 -- 172.21.15.5:0/3261883390 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f1ae0017090 con 0x7f1af0150fd0 2024-03-24T10:54:26.078 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:26.078 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:26.078 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:26.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:26.079+0000 7f1af5d80640 1 -- 172.21.15.5:0/3261883390 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1adc06b810 msgr2=0x7f1adc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:26.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:26.079+0000 7f1af5d80640 1 --2- 172.21.15.5:0/3261883390 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1adc06b810 0x7f1adc06dcd0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1ae40010e0 tx=0x7f1ae4007950 comp rx=0 tx=0).stop 2024-03-24T10:54:26.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:26.079+0000 7f1af5d80640 1 -- 172.21.15.5:0/3261883390 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1af0150fd0 msgr2=0x7f1af01f0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:26.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:26.079+0000 7f1af5d80640 1 --2- 172.21.15.5:0/3261883390 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1af0150fd0 0x7f1af01f0600 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009fd0 tx=0x7f1ae000ed40 comp rx=0 tx=0).stop 2024-03-24T10:54:26.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:26.079+0000 7f1af5d80640 1 -- 172.21.15.5:0/3261883390 shutdown_connections 2024-03-24T10:54:26.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:26.079+0000 7f1af5d80640 1 --2- 172.21.15.5:0/3261883390 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1adc06b810 0x7f1adc06dcd0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:26.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:26.079+0000 7f1af5d80640 1 --2- 172.21.15.5:0/3261883390 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1af01513b0 0x7f1af01555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:26.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:26.079+0000 7f1af5d80640 1 --2- 172.21.15.5:0/3261883390 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1af015e590 0x7f1af014bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:26.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:26.079+0000 7f1af5d80640 1 --2- 172.21.15.5:0/3261883390 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1af0150fd0 0x7f1af01f0600 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:26.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:26.079+0000 7f1af5d80640 1 -- 172.21.15.5:0/3261883390 >> 172.21.15.5:0/3261883390 conn(0x7f1af006f8b0 msgr2=0x7f1af014e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:26.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:26.080+0000 7f1af5d80640 1 -- 172.21.15.5:0/3261883390 shutdown_connections 2024-03-24T10:54:26.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:26.080+0000 7f1af5d80640 1 -- 172.21.15.5:0/3261883390 wait complete. 2024-03-24T10:54:26.695 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:26.696 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:27.697 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:27.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.800+0000 7f39d4947640 1 Processor -- start 2024-03-24T10:54:27.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.800+0000 7f39d4947640 1 -- start start 2024-03-24T10:54:27.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.801+0000 7f39d4947640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f39d01543a0 0x7f39d015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:27.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.801+0000 7f39d4947640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f39d006d4f0 0x7f39d00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:27.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.801+0000 7f39d4947640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f39d00ccaa0 0x7f39d006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:27.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.801+0000 7f39d4947640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f39d00c6750 con 0x7f39d00ccaa0 2024-03-24T10:54:27.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.801+0000 7f39d4947640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f39d00cc380 con 0x7f39d01543a0 2024-03-24T10:54:27.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.801+0000 7f39d4947640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f39d00cb2f0 con 0x7f39d006d4f0 2024-03-24T10:54:27.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.801+0000 7f39ced9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f39d00ccaa0 0x7f39d006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:27.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.801+0000 7f39ce59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f39d01543a0 0x7f39d015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:27.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.801+0000 7f39cdd9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f39d006d4f0 0x7f39d00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:27.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.802+0000 7f39ced9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f39d00ccaa0 0x7f39d006bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57944/0 (socket says 172.21.15.5:57944) 2024-03-24T10:54:27.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.802+0000 7f39ced9d640 1 -- 172.21.15.5:0/4045853185 learned_addr learned my addr 172.21.15.5:0/4045853185 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:27.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.804+0000 7f39ced9d640 1 -- 172.21.15.5:0/4045853185 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f39d006d4f0 msgr2=0x7f39d00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:54:27.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.804+0000 7f39ced9d640 1 --2- 172.21.15.5:0/4045853185 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f39d006d4f0 0x7f39d00cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:27.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.804+0000 7f39ced9d640 1 -- 172.21.15.5:0/4045853185 >> v2:172.21.15.57:3300/0 conn(0x7f39d01543a0 msgr2=0x7f39d015ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:54:27.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.804+0000 7f39ced9d640 1 --2- 172.21.15.5:0/4045853185 >> v2:172.21.15.57:3300/0 conn(0x7f39d01543a0 0x7f39d015ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:27.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.804+0000 7f39ced9d640 1 -- 172.21.15.5:0/4045853185 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39d00ca7e0 con 0x7f39d00ccaa0 2024-03-24T10:54:27.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.806+0000 7f39ced9d640 1 --2- 172.21.15.5:0/4045853185 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39d00ccaa0 0x7f39d006bf50 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f39d00cd600 tx=0x7f39c802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f56d42a9dacce3c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:27.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.808+0000 7f39cd59a640 1 -- 172.21.15.5:0/4045853185 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c802cb00 con 0x7f39d00ccaa0 2024-03-24T10:54:27.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.810+0000 7f39cd59a640 1 -- 172.21.15.5:0/4045853185 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39c802cc60 con 0x7f39d00ccaa0 2024-03-24T10:54:27.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.810+0000 7f39cd59a640 1 -- 172.21.15.5:0/4045853185 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c80343f0 con 0x7f39d00ccaa0 2024-03-24T10:54:27.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.811+0000 7f39d4947640 1 -- 172.21.15.5:0/4045853185 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39d00ccaa0 msgr2=0x7f39d006bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:27.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.811+0000 7f39d4947640 1 --2- 172.21.15.5:0/4045853185 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39d00ccaa0 0x7f39d006bf50 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f39d00cd600 tx=0x7f39c802bf60 comp rx=0 tx=0).stop 2024-03-24T10:54:27.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.811+0000 7f39d4947640 1 -- 172.21.15.5:0/4045853185 shutdown_connections 2024-03-24T10:54:27.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.811+0000 7f39d4947640 1 --2- 172.21.15.5:0/4045853185 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f39d006d4f0 0x7f39d00cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:27.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.811+0000 7f39d4947640 1 --2- 172.21.15.5:0/4045853185 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39d00ccaa0 0x7f39d006bf50 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:27.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.811+0000 7f39d4947640 1 --2- 172.21.15.5:0/4045853185 >> v2:172.21.15.57:3300/0 conn(0x7f39d01543a0 0x7f39d015ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:27.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.811+0000 7f39d4947640 1 -- 172.21.15.5:0/4045853185 >> 172.21.15.5:0/4045853185 conn(0x7f39d0072ef0 msgr2=0x7f39d0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:27.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.811+0000 7f39d4947640 1 -- 172.21.15.5:0/4045853185 shutdown_connections 2024-03-24T10:54:27.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.811+0000 7f39d4947640 1 -- 172.21.15.5:0/4045853185 wait complete. 2024-03-24T10:54:27.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.812+0000 7f39d4947640 1 Processor -- start 2024-03-24T10:54:27.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.813+0000 7f39d4947640 1 -- start start 2024-03-24T10:54:27.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.813+0000 7f39d4947640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39d006d4f0 0x7f39d00cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:27.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.813+0000 7f39d4947640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f39d00ccaa0 0x7f39d01efbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:27.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.813+0000 7f39d4947640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39d01543a0 0x7f39d01f0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:27.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.813+0000 7f39d4947640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f39d01f0640 con 0x7f39d006d4f0 2024-03-24T10:54:27.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.813+0000 7f39d4947640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f39d00ca220 con 0x7f39d01543a0 2024-03-24T10:54:27.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.813+0000 7f39d4947640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f39d01f3d40 con 0x7f39d00ccaa0 2024-03-24T10:54:27.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.814+0000 7f39ce59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39d006d4f0 0x7f39d00cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:27.816 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.814+0000 7f39ced9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39d01543a0 0x7f39d01f0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:27.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.814+0000 7f39cdd9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f39d00ccaa0 0x7f39d01efbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:27.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.814+0000 7f39ce59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39d006d4f0 0x7f39d00cecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57950/0 (socket says 172.21.15.5:57950) 2024-03-24T10:54:27.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.814+0000 7f39ce59c640 1 -- 172.21.15.5:0/202009135 learned_addr learned my addr 172.21.15.5:0/202009135 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:27.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.817+0000 7f39ce59c640 1 -- 172.21.15.5:0/202009135 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f39d00ccaa0 msgr2=0x7f39d01efbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:27.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.818+0000 7f39ce59c640 1 --2- 172.21.15.5:0/202009135 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f39d00ccaa0 0x7f39d01efbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:27.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.818+0000 7f39ce59c640 1 -- 172.21.15.5:0/202009135 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39d01543a0 msgr2=0x7f39d01f0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:27.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.818+0000 7f39ce59c640 1 --2- 172.21.15.5:0/202009135 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39d01543a0 0x7f39d01f0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:27.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.818+0000 7f39ce59c640 1 -- 172.21.15.5:0/202009135 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c8009b30 con 0x7f39d006d4f0 2024-03-24T10:54:27.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.822+0000 7f39ce59c640 1 --2- 172.21.15.5:0/202009135 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39d006d4f0 0x7f39d00cecc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f39c000e8a0 tx=0x7f39c000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:27.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.830+0000 7f39beffd640 1 -- 172.21.15.5:0/202009135 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c00094f0 con 0x7f39d006d4f0 2024-03-24T10:54:27.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.830+0000 7f39beffd640 1 -- 172.21.15.5:0/202009135 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39c000c940 con 0x7f39d006d4f0 2024-03-24T10:54:27.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.830+0000 7f39d4947640 1 -- 172.21.15.5:0/202009135 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39d0155840 con 0x7f39d006d4f0 2024-03-24T10:54:27.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.830+0000 7f39beffd640 1 -- 172.21.15.5:0/202009135 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c000ccf0 con 0x7f39d006d4f0 2024-03-24T10:54:27.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.830+0000 7f39d4947640 1 -- 172.21.15.5:0/202009135 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39d01f3410 con 0x7f39d006d4f0 2024-03-24T10:54:27.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:27.831+0000 7f39d4947640 1 -- 172.21.15.5:0/202009135 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39d00681d0 con 0x7f39d006d4f0 2024-03-24T10:54:28.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:28.161+0000 7f39beffd640 1 -- 172.21.15.5:0/202009135 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f39c0013470 con 0x7f39d006d4f0 2024-03-24T10:54:28.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:28.162+0000 7f39beffd640 1 --2- 172.21.15.5:0/202009135 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f39b006b810 0x7f39b006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:28.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:28.162+0000 7f39cdd9b640 1 --2- 172.21.15.5:0/202009135 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f39b006b810 0x7f39b006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:28.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:28.162+0000 7f39cdd9b640 1 --2- 172.21.15.5:0/202009135 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f39b006b810 0x7f39b006dcd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f39d00cf980 tx=0x7f39c4009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:28.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:28.181+0000 7f39beffd640 1 -- 172.21.15.5:0/202009135 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f39c0059190 con 0x7f39d006d4f0 2024-03-24T10:54:28.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:28.181+0000 7f39beffd640 1 -- 172.21.15.5:0/202009135 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f39c0055430 con 0x7f39d006d4f0 2024-03-24T10:54:28.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:28.435+0000 7f39d4947640 1 -- 172.21.15.5:0/202009135 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39d00caec0 con 0x7f39d006d4f0 2024-03-24T10:54:28.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:28.449+0000 7f39beffd640 1 -- 172.21.15.5:0/202009135 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f39c0021090 con 0x7f39d006d4f0 2024-03-24T10:54:28.450 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:28.450 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:28.450 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:28.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:28.453+0000 7f39d4947640 1 -- 172.21.15.5:0/202009135 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f39b006b810 msgr2=0x7f39b006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:28.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:28.453+0000 7f39d4947640 1 --2- 172.21.15.5:0/202009135 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f39b006b810 0x7f39b006dcd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f39d00cf980 tx=0x7f39c4009190 comp rx=0 tx=0).stop 2024-03-24T10:54:28.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:28.453+0000 7f39d4947640 1 -- 172.21.15.5:0/202009135 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39d006d4f0 msgr2=0x7f39d00cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:28.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:28.453+0000 7f39d4947640 1 --2- 172.21.15.5:0/202009135 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39d006d4f0 0x7f39d00cecc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f39c000e8a0 tx=0x7f39c000ed60 comp rx=0 tx=0).stop 2024-03-24T10:54:28.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:28.453+0000 7f39d4947640 1 -- 172.21.15.5:0/202009135 shutdown_connections 2024-03-24T10:54:28.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:28.453+0000 7f39d4947640 1 --2- 172.21.15.5:0/202009135 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f39b006b810 0x7f39b006dcd0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:28.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:28.453+0000 7f39d4947640 1 --2- 172.21.15.5:0/202009135 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39d01543a0 0x7f39d01f0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:28.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:28.453+0000 7f39d4947640 1 --2- 172.21.15.5:0/202009135 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f39d00ccaa0 0x7f39d01efbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:28.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:28.453+0000 7f39d4947640 1 --2- 172.21.15.5:0/202009135 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39d006d4f0 0x7f39d00cecc0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:28.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:28.453+0000 7f39d4947640 1 -- 172.21.15.5:0/202009135 >> 172.21.15.5:0/202009135 conn(0x7f39d0072ef0 msgr2=0x7f39d00c5490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:28.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:28.453+0000 7f39d4947640 1 -- 172.21.15.5:0/202009135 shutdown_connections 2024-03-24T10:54:28.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:28.453+0000 7f39d4947640 1 -- 172.21.15.5:0/202009135 wait complete. 2024-03-24T10:54:29.067 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:29.067 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:30.069 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:30.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.173+0000 7fa27c5c7640 1 Processor -- start 2024-03-24T10:54:30.173 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.173+0000 7fa27c5c7640 1 -- start start 2024-03-24T10:54:30.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.174+0000 7fa27c5c7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa27415aea0 0x7fa2740cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:30.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.174+0000 7fa27c5c7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa2740cf100 0x7fa2740ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:30.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.174+0000 7fa27c5c7640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa27406cd50 con 0x7fa27415aac0 2024-03-24T10:54:30.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.174+0000 7fa27c5c7640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fa2740ce790 con 0x7fa2740cf100 2024-03-24T10:54:30.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.174+0000 7fa27c5c7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fa2740cdb10 con 0x7fa27415aea0 2024-03-24T10:54:30.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.175+0000 7fa279b62640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa27415aea0 0x7fa2740cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:30.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.175+0000 7fa27a363640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fa27415aac0 0x7fa27406c230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:50440/0 (socket says 172.21.15.5:50440) 2024-03-24T10:54:30.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.175+0000 7fa27a363640 1 -- 172.21.15.5:0/491540104 learned_addr learned my addr 172.21.15.5:0/491540104 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:54:30.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.175+0000 7fa27ab64640 1 --2- 172.21.15.5:0/491540104 >> v2:172.21.15.5:3300/0 conn(0x7fa2740cf100 0x7fa2740ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:30.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.177+0000 7fa279b62640 1 -- 172.21.15.5:0/491540104 >> v2:172.21.15.5:3300/0 conn(0x7fa2740cf100 msgr2=0x7fa2740ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:54:30.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.177+0000 7fa279b62640 1 --2- 172.21.15.5:0/491540104 >> v2:172.21.15.5:3300/0 conn(0x7fa2740cf100 0x7fa2740ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:30.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.177+0000 7fa279b62640 1 -- 172.21.15.5:0/491540104 >> v1:172.21.15.5:6789/0 conn(0x7fa27415aac0 legacy=0x7fa27406c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:30.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.177+0000 7fa279b62640 1 -- 172.21.15.5:0/491540104 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2740cddf0 con 0x7fa27415aea0 2024-03-24T10:54:30.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.179+0000 7fa279b62640 1 --2- 172.21.15.5:0/491540104 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa27415aea0 0x7fa2740cebc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa2640097a0 tx=0x7fa26402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ec4e3d4c5819470 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:30.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.184+0000 7fa279361640 1 -- 172.21.15.5:0/491540104 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa264030610 con 0x7fa27415aea0 2024-03-24T10:54:30.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.184+0000 7fa279361640 1 -- 172.21.15.5:0/491540104 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa264030770 con 0x7fa27415aea0 2024-03-24T10:54:30.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.184+0000 7fa279361640 1 -- 172.21.15.5:0/491540104 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa264030b00 con 0x7fa27415aea0 2024-03-24T10:54:30.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.184+0000 7fa27c5c7640 1 -- 172.21.15.5:0/491540104 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa27415aea0 msgr2=0x7fa2740cebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:30.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.184+0000 7fa27c5c7640 1 --2- 172.21.15.5:0/491540104 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa27415aea0 0x7fa2740cebc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa2640097a0 tx=0x7fa26402b960 comp rx=0 tx=0).stop 2024-03-24T10:54:30.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.184+0000 7fa27c5c7640 1 -- 172.21.15.5:0/491540104 shutdown_connections 2024-03-24T10:54:30.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.184+0000 7fa27c5c7640 1 --2- 172.21.15.5:0/491540104 >> v2:172.21.15.5:3300/0 conn(0x7fa2740cf100 0x7fa2740ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:30.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.184+0000 7fa27c5c7640 1 --2- 172.21.15.5:0/491540104 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa27415aea0 0x7fa2740cebc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:30.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.184+0000 7fa27c5c7640 1 -- 172.21.15.5:0/491540104 >> 172.21.15.5:0/491540104 conn(0x7fa2740cfb90 msgr2=0x7fa2740cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:30.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.184+0000 7fa27c5c7640 1 -- 172.21.15.5:0/491540104 shutdown_connections 2024-03-24T10:54:30.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.184+0000 7fa27c5c7640 1 -- 172.21.15.5:0/491540104 wait complete. 2024-03-24T10:54:30.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.186+0000 7fa27c5c7640 1 Processor -- start 2024-03-24T10:54:30.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.186+0000 7fa27c5c7640 1 -- start start 2024-03-24T10:54:30.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.186+0000 7fa27c5c7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2740cf100 0x7fa274157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:30.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.186+0000 7fa27c5c7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa27415aac0 0x7fa2741f0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:30.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.186+0000 7fa27c5c7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa27415aea0 0x7fa2740b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:30.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.186+0000 7fa27c5c7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa27415d240 con 0x7fa27415aea0 2024-03-24T10:54:30.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.186+0000 7fa27c5c7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa274156400 con 0x7fa2740cf100 2024-03-24T10:54:30.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.186+0000 7fa27c5c7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fa2741566e0 con 0x7fa27415aac0 2024-03-24T10:54:30.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.187+0000 7fa27ab64640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa27415aea0 0x7fa2740b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:30.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.187+0000 7fa279b62640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa27415aac0 0x7fa2741f0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:30.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.187+0000 7fa27ab64640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa27415aea0 0x7fa2740b07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57968/0 (socket says 172.21.15.5:57968) 2024-03-24T10:54:30.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.187+0000 7fa27ab64640 1 -- 172.21.15.5:0/688526869 learned_addr learned my addr 172.21.15.5:0/688526869 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:30.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.187+0000 7fa27a363640 1 --2- 172.21.15.5:0/688526869 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2740cf100 0x7fa274157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:30.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.191+0000 7fa27ab64640 1 -- 172.21.15.5:0/688526869 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa27415aac0 msgr2=0x7fa2741f0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:30.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.191+0000 7fa27ab64640 1 --2- 172.21.15.5:0/688526869 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa27415aac0 0x7fa2741f0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:30.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.191+0000 7fa27ab64640 1 -- 172.21.15.5:0/688526869 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2740cf100 msgr2=0x7fa274157540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:30.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.191+0000 7fa27ab64640 1 --2- 172.21.15.5:0/688526869 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2740cf100 0x7fa274157540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:30.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.191+0000 7fa27ab64640 1 -- 172.21.15.5:0/688526869 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa264009400 con 0x7fa27415aea0 2024-03-24T10:54:30.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.193+0000 7fa27ab64640 1 --2- 172.21.15.5:0/688526869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa27415aea0 0x7fa2740b07b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa26c007e40 tx=0x7fa26c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:30.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.200+0000 7fa26affd640 1 -- 172.21.15.5:0/688526869 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26c014620 con 0x7fa27415aea0 2024-03-24T10:54:30.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.200+0000 7fa26affd640 1 -- 172.21.15.5:0/688526869 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa26c00ee90 con 0x7fa27415aea0 2024-03-24T10:54:30.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.200+0000 7fa27c5c7640 1 -- 172.21.15.5:0/688526869 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2741569c0 con 0x7fa27415aea0 2024-03-24T10:54:30.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.200+0000 7fa26affd640 1 -- 172.21.15.5:0/688526869 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa26c0149e0 con 0x7fa27415aea0 2024-03-24T10:54:30.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.200+0000 7fa27c5c7640 1 -- 172.21.15.5:0/688526869 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa274156f80 con 0x7fa27415aea0 2024-03-24T10:54:30.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.201+0000 7fa27c5c7640 1 -- 172.21.15.5:0/688526869 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2740c8340 con 0x7fa27415aea0 2024-03-24T10:54:30.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.490+0000 7fa26affd640 1 -- 172.21.15.5:0/688526869 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa26c014b40 con 0x7fa27415aea0 2024-03-24T10:54:30.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.490+0000 7fa26affd640 1 --2- 172.21.15.5:0/688526869 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa25806b860 0x7fa25806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:30.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.491+0000 7fa27a363640 1 --2- 172.21.15.5:0/688526869 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa25806b860 0x7fa25806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:30.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.491+0000 7fa27a363640 1 --2- 172.21.15.5:0/688526869 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa25806b860 0x7fa25806dd20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa270007fd0 tx=0x7fa270007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:30.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.506+0000 7fa26affd640 1 -- 172.21.15.5:0/688526869 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fa26c05a110 con 0x7fa27415aea0 2024-03-24T10:54:30.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.506+0000 7fa26affd640 1 -- 172.21.15.5:0/688526869 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa26c0563b0 con 0x7fa27415aea0 2024-03-24T10:54:30.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.754+0000 7fa27c5c7640 1 -- 172.21.15.5:0/688526869 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2740681d0 con 0x7fa27415aea0 2024-03-24T10:54:30.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.796+0000 7fa26affd640 1 -- 172.21.15.5:0/688526869 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fa26c059a70 con 0x7fa27415aea0 2024-03-24T10:54:30.796 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:30.796 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:30.797 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:30.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.798+0000 7fa27c5c7640 1 -- 172.21.15.5:0/688526869 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa25806b860 msgr2=0x7fa25806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:30.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.798+0000 7fa27c5c7640 1 --2- 172.21.15.5:0/688526869 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa25806b860 0x7fa25806dd20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa270007fd0 tx=0x7fa270007e80 comp rx=0 tx=0).stop 2024-03-24T10:54:30.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.798+0000 7fa27c5c7640 1 -- 172.21.15.5:0/688526869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa27415aea0 msgr2=0x7fa2740b07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:30.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.798+0000 7fa27c5c7640 1 --2- 172.21.15.5:0/688526869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa27415aea0 0x7fa2740b07b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa26c007e40 tx=0x7fa26c00e4b0 comp rx=0 tx=0).stop 2024-03-24T10:54:30.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.798+0000 7fa27c5c7640 1 -- 172.21.15.5:0/688526869 shutdown_connections 2024-03-24T10:54:30.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.798+0000 7fa27c5c7640 1 --2- 172.21.15.5:0/688526869 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa25806b860 0x7fa25806dd20 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:30.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.798+0000 7fa27c5c7640 1 --2- 172.21.15.5:0/688526869 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa27415aea0 0x7fa2740b07b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:30.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.798+0000 7fa27c5c7640 1 --2- 172.21.15.5:0/688526869 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa27415aac0 0x7fa2741f0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:30.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.798+0000 7fa27c5c7640 1 --2- 172.21.15.5:0/688526869 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2740cf100 0x7fa274157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:30.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.798+0000 7fa27c5c7640 1 -- 172.21.15.5:0/688526869 >> 172.21.15.5:0/688526869 conn(0x7fa2740cfb90 msgr2=0x7fa274153040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:30.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.798+0000 7fa27c5c7640 1 -- 172.21.15.5:0/688526869 shutdown_connections 2024-03-24T10:54:30.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:30.798+0000 7fa27c5c7640 1 -- 172.21.15.5:0/688526869 wait complete. 2024-03-24T10:54:31.413 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:31.414 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:32.414 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:32.421 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:54:32.460 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:54:32.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.516+0000 7fb81fa0e640 1 Processor -- start 2024-03-24T10:54:32.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.517+0000 7fb81fa0e640 1 -- start start 2024-03-24T10:54:32.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.517+0000 7fb81fa0e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb8181543a0 0x7fb81815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:32.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.517+0000 7fb81fa0e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8180cb2f0 con 0x7fb81806d4f0 2024-03-24T10:54:32.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.517+0000 7fb81fa0e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8180ca7e0 con 0x7fb81806bf50 2024-03-24T10:54:32.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.517+0000 7fb81fa0e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fb8180c6750 con 0x7fb8181543a0 2024-03-24T10:54:32.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.518+0000 7fb81cfa9640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fb81806d4f0 0x7fb8180cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:57590/0 (socket says 172.21.15.5:57590) 2024-03-24T10:54:32.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.518+0000 7fb81cfa9640 1 -- 172.21.15.5:0/1036726235 learned_addr learned my addr 172.21.15.5:0/1036726235 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:54:32.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.518+0000 7fb81d7aa640 1 --2- 172.21.15.5:0/1036726235 >> v2:172.21.15.57:3300/0 conn(0x7fb8181543a0 0x7fb81815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:32.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.521+0000 7fb80ffff640 1 -- 172.21.15.5:0/1036726235 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3737738709 0 0) 0x7fb8180cb2f0 con 0x7fb81806d4f0 2024-03-24T10:54:32.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.521+0000 7fb80ffff640 1 -- 172.21.15.5:0/1036726235 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb804003520 con 0x7fb81806d4f0 2024-03-24T10:54:32.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.522+0000 7fb81d7aa640 1 -- 172.21.15.5:0/1036726235 >> v1:172.21.15.57:6789/0 conn(0x7fb81806bf50 legacy=0x7fb81806c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:32.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.522+0000 7fb81d7aa640 1 -- 172.21.15.5:0/1036726235 >> v1:172.21.15.5:6789/0 conn(0x7fb81806d4f0 legacy=0x7fb8180cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:32.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.522+0000 7fb81d7aa640 1 -- 172.21.15.5:0/1036726235 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8180cb520 con 0x7fb8181543a0 2024-03-24T10:54:32.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.526+0000 7fb81d7aa640 1 --2- 172.21.15.5:0/1036726235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8181543a0 0x7fb81815ac00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb8140097a0 tx=0x7fb81402b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f123b742fb3e9d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:32.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.530+0000 7fb80ffff640 1 -- 172.21.15.5:0/1036726235 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81402f810 con 0x7fb8181543a0 2024-03-24T10:54:32.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.531+0000 7fb80ffff640 1 -- 172.21.15.5:0/1036726235 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb81402f970 con 0x7fb8181543a0 2024-03-24T10:54:32.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.531+0000 7fb80ffff640 1 -- 172.21.15.5:0/1036726235 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81402fd00 con 0x7fb8181543a0 2024-03-24T10:54:32.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.531+0000 7fb81fa0e640 1 -- 172.21.15.5:0/1036726235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8181543a0 msgr2=0x7fb81815ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:32.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.531+0000 7fb81fa0e640 1 --2- 172.21.15.5:0/1036726235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8181543a0 0x7fb81815ac00 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb8140097a0 tx=0x7fb81402b9f0 comp rx=0 tx=0).stop 2024-03-24T10:54:32.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.532+0000 7fb81fa0e640 1 -- 172.21.15.5:0/1036726235 shutdown_connections 2024-03-24T10:54:32.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.532+0000 7fb81fa0e640 1 --2- 172.21.15.5:0/1036726235 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb8181543a0 0x7fb81815ac00 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:32.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.532+0000 7fb81fa0e640 1 -- 172.21.15.5:0/1036726235 >> 172.21.15.5:0/1036726235 conn(0x7fb818072ef0 msgr2=0x7fb818061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:32.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.532+0000 7fb81fa0e640 1 -- 172.21.15.5:0/1036726235 shutdown_connections 2024-03-24T10:54:32.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.532+0000 7fb81fa0e640 1 -- 172.21.15.5:0/1036726235 wait complete. 2024-03-24T10:54:32.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.533+0000 7fb81fa0e640 1 Processor -- start 2024-03-24T10:54:32.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.533+0000 7fb81fa0e640 1 -- start start 2024-03-24T10:54:32.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.534+0000 7fb81fa0e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb81806bf50 0x7fb8180ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:32.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.534+0000 7fb81fa0e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb81806d4f0 0x7fb8180cea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:32.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.534+0000 7fb81fa0e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb8181543a0 0x7fb818146290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:32.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.534+0000 7fb81fa0e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fb8180b4110 con 0x7fb81806d4f0 2024-03-24T10:54:32.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.534+0000 7fb81fa0e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb8180b43f0 con 0x7fb81806bf50 2024-03-24T10:54:32.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.534+0000 7fb81fa0e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fb8180cda00 con 0x7fb8181543a0 2024-03-24T10:54:32.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.534+0000 7fb81cfa9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb81806d4f0 0x7fb8180cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:32.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.534+0000 7fb81d7aa640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb81806bf50 0x7fb8180ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:32.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.534+0000 7fb81dfab640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb8181543a0 0x7fb818146290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:32.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.535+0000 7fb81cfa9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb81806d4f0 0x7fb8180cea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50874/0 (socket says 172.21.15.5:50874) 2024-03-24T10:54:32.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.535+0000 7fb81cfa9640 1 -- 172.21.15.5:0/5045618 learned_addr learned my addr 172.21.15.5:0/5045618 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:32.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.539+0000 7fb81dfab640 1 -- 172.21.15.5:0/5045618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb81806bf50 msgr2=0x7fb8180ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:32.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.539+0000 7fb81dfab640 1 --2- 172.21.15.5:0/5045618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb81806bf50 0x7fb8180ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:32.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.539+0000 7fb81dfab640 1 -- 172.21.15.5:0/5045618 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb81806d4f0 msgr2=0x7fb8180cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:32.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.539+0000 7fb81dfab640 1 --2- 172.21.15.5:0/5045618 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb81806d4f0 0x7fb8180cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:32.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.539+0000 7fb81dfab640 1 -- 172.21.15.5:0/5045618 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb814009400 con 0x7fb8181543a0 2024-03-24T10:54:32.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.558+0000 7fb81dfab640 1 --2- 172.21.15.5:0/5045618 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb8181543a0 0x7fb818146290 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb81000c690 tx=0x7fb81000cb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:32.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.569+0000 7fb80dffb640 1 -- 172.21.15.5:0/5045618 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb810003b70 con 0x7fb8181543a0 2024-03-24T10:54:32.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.569+0000 7fb80dffb640 1 -- 172.21.15.5:0/5045618 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb810003cd0 con 0x7fb8181543a0 2024-03-24T10:54:32.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.569+0000 7fb80dffb640 1 -- 172.21.15.5:0/5045618 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb810005310 con 0x7fb8181543a0 2024-03-24T10:54:32.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.569+0000 7fb81fa0e640 1 -- 172.21.15.5:0/5045618 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8180cdce0 con 0x7fb8181543a0 2024-03-24T10:54:32.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.569+0000 7fb81fa0e640 1 -- 172.21.15.5:0/5045618 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8180b46d0 con 0x7fb8181543a0 2024-03-24T10:54:32.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.570+0000 7fb81fa0e640 1 -- 172.21.15.5:0/5045618 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8180ce200 con 0x7fb8181543a0 2024-03-24T10:54:32.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.851+0000 7fb80dffb640 1 -- 172.21.15.5:0/5045618 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb810005470 con 0x7fb8181543a0 2024-03-24T10:54:32.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.851+0000 7fb80dffb640 1 --2- 172.21.15.5:0/5045618 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb7fc06b860 0x7fb7fc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:32.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.851+0000 7fb81d7aa640 1 --2- 172.21.15.5:0/5045618 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb7fc06b860 0x7fb7fc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:32.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.852+0000 7fb81d7aa640 1 --2- 172.21.15.5:0/5045618 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb7fc06b860 0x7fb7fc06dd20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb814036000 tx=0x7fb814008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:32.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.857+0000 7fb80dffb640 1 -- 172.21.15.5:0/5045618 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fb810058f40 con 0x7fb8181543a0 2024-03-24T10:54:32.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:32.872+0000 7fb80dffb640 1 -- 172.21.15.5:0/5045618 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb8100551e0 con 0x7fb8181543a0 2024-03-24T10:54:33.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:33.127+0000 7fb81fa0e640 1 -- 172.21.15.5:0/5045618 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8180ce200 con 0x7fb8181543a0 2024-03-24T10:54:33.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:33.148+0000 7fb80dffb640 1 -- 172.21.15.5:0/5045618 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fb8180ce200 con 0x7fb8181543a0 2024-03-24T10:54:33.149 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:33.149 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:33.149 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:33.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:33.150+0000 7fb81fa0e640 1 -- 172.21.15.5:0/5045618 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb7fc06b860 msgr2=0x7fb7fc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:33.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:33.150+0000 7fb81fa0e640 1 --2- 172.21.15.5:0/5045618 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb7fc06b860 0x7fb7fc06dd20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb814036000 tx=0x7fb814008030 comp rx=0 tx=0).stop 2024-03-24T10:54:33.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:33.150+0000 7fb81fa0e640 1 -- 172.21.15.5:0/5045618 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb8181543a0 msgr2=0x7fb818146290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:33.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:33.150+0000 7fb81fa0e640 1 --2- 172.21.15.5:0/5045618 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb8181543a0 0x7fb818146290 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb81000c690 tx=0x7fb81000cb50 comp rx=0 tx=0).stop 2024-03-24T10:54:33.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:33.150+0000 7fb81fa0e640 1 -- 172.21.15.5:0/5045618 shutdown_connections 2024-03-24T10:54:33.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:33.150+0000 7fb81fa0e640 1 --2- 172.21.15.5:0/5045618 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb7fc06b860 0x7fb7fc06dd20 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:33.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:33.150+0000 7fb81fa0e640 1 --2- 172.21.15.5:0/5045618 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb81806d4f0 0x7fb8180cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:33.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:33.150+0000 7fb81fa0e640 1 --2- 172.21.15.5:0/5045618 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb8181543a0 0x7fb818146290 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:33.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:33.150+0000 7fb81fa0e640 1 --2- 172.21.15.5:0/5045618 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb81806bf50 0x7fb8180ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:33.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:33.150+0000 7fb81fa0e640 1 -- 172.21.15.5:0/5045618 >> 172.21.15.5:0/5045618 conn(0x7fb818072ef0 msgr2=0x7fb8180cc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:33.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:33.151+0000 7fb81fa0e640 1 -- 172.21.15.5:0/5045618 shutdown_connections 2024-03-24T10:54:33.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:33.151+0000 7fb81fa0e640 1 -- 172.21.15.5:0/5045618 wait complete. 2024-03-24T10:54:33.767 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:33.767 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:34.768 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:34.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.870+0000 7fad07bf4640 1 Processor -- start 2024-03-24T10:54:34.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.871+0000 7fad07bf4640 1 -- start start 2024-03-24T10:54:34.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.871+0000 7fad07bf4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fad001543a0 0x7fad0015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:34.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.871+0000 7fad07bf4640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fad0006bf50 0x7fad0006c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:34.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.871+0000 7fad07bf4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad000cb2f0 con 0x7fad0006d4f0 2024-03-24T10:54:34.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.871+0000 7fad07bf4640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fad000c6750 con 0x7fad0006bf50 2024-03-24T10:54:34.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.871+0000 7fad07bf4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fad000cc380 con 0x7fad001543a0 2024-03-24T10:54:34.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.872+0000 7fad05990640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fad001543a0 0x7fad0015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:34.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.872+0000 7fad0518f640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fad0006d4f0 0x7fad000cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:57600/0 (socket says 172.21.15.5:57600) 2024-03-24T10:54:34.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.872+0000 7fad0518f640 1 -- 172.21.15.5:0/2009242116 learned_addr learned my addr 172.21.15.5:0/2009242116 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:54:34.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.872+0000 7fad06191640 1 --2- 172.21.15.5:0/2009242116 >> v2:172.21.15.5:3300/0 conn(0x7fad0006bf50 0x7fad0006c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:34.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.876+0000 7fad05990640 1 -- 172.21.15.5:0/2009242116 >> v2:172.21.15.5:3300/0 conn(0x7fad0006bf50 msgr2=0x7fad0006c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:54:34.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.876+0000 7fad05990640 1 --2- 172.21.15.5:0/2009242116 >> v2:172.21.15.5:3300/0 conn(0x7fad0006bf50 0x7fad0006c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:34.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.876+0000 7fad05990640 1 -- 172.21.15.5:0/2009242116 >> v1:172.21.15.5:6789/0 conn(0x7fad0006d4f0 legacy=0x7fad000cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:34.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.876+0000 7fad05990640 1 -- 172.21.15.5:0/2009242116 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad000ca7e0 con 0x7fad001543a0 2024-03-24T10:54:34.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.879+0000 7fad05990640 1 --2- 172.21.15.5:0/2009242116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad001543a0 0x7fad0015ac00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fad000cd600 tx=0x7facfc02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdd9bad64b8f9c07 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:34.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.882+0000 7fad0498e640 1 -- 172.21.15.5:0/2009242116 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc030610 con 0x7fad001543a0 2024-03-24T10:54:34.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.884+0000 7fad0498e640 1 -- 172.21.15.5:0/2009242116 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facfc030770 con 0x7fad001543a0 2024-03-24T10:54:34.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.884+0000 7fad0498e640 1 -- 172.21.15.5:0/2009242116 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc030b00 con 0x7fad001543a0 2024-03-24T10:54:34.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.885+0000 7fad07bf4640 1 -- 172.21.15.5:0/2009242116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad001543a0 msgr2=0x7fad0015ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:34.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.885+0000 7fad07bf4640 1 --2- 172.21.15.5:0/2009242116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad001543a0 0x7fad0015ac00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fad000cd600 tx=0x7facfc02b960 comp rx=0 tx=0).stop 2024-03-24T10:54:34.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.885+0000 7fad07bf4640 1 -- 172.21.15.5:0/2009242116 shutdown_connections 2024-03-24T10:54:34.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.885+0000 7fad07bf4640 1 --2- 172.21.15.5:0/2009242116 >> v2:172.21.15.5:3300/0 conn(0x7fad0006bf50 0x7fad0006c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:34.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.885+0000 7fad07bf4640 1 --2- 172.21.15.5:0/2009242116 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad001543a0 0x7fad0015ac00 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:34.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.885+0000 7fad07bf4640 1 -- 172.21.15.5:0/2009242116 >> 172.21.15.5:0/2009242116 conn(0x7fad00072ef0 msgr2=0x7fad00061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:34.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.885+0000 7fad07bf4640 1 -- 172.21.15.5:0/2009242116 shutdown_connections 2024-03-24T10:54:34.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.885+0000 7fad07bf4640 1 -- 172.21.15.5:0/2009242116 wait complete. 2024-03-24T10:54:34.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.886+0000 7fad07bf4640 1 Processor -- start 2024-03-24T10:54:34.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.886+0000 7fad07bf4640 1 -- start start 2024-03-24T10:54:34.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.887+0000 7fad07bf4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fad0006bf50 0x7fad0014fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:34.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.887+0000 7fad07bf4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad0006d4f0 0x7fad001f12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:34.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.887+0000 7fad07bf4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad001543a0 0x7fad001f1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:34.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.887+0000 7fad07bf4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fad001f0e50 con 0x7fad001543a0 2024-03-24T10:54:34.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.887+0000 7fad07bf4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fad00154ce0 con 0x7fad0006d4f0 2024-03-24T10:54:34.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.887+0000 7fad07bf4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fad0014f940 con 0x7fad0006bf50 2024-03-24T10:54:34.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.887+0000 7fad06191640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad001543a0 0x7fad001f1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:34.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.888+0000 7fad0518f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad0006d4f0 0x7fad001f12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:34.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.888+0000 7fad06191640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad001543a0 0x7fad001f1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50896/0 (socket says 172.21.15.5:50896) 2024-03-24T10:54:34.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.888+0000 7fad06191640 1 -- 172.21.15.5:0/2397347199 learned_addr learned my addr 172.21.15.5:0/2397347199 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:34.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.888+0000 7fad05990640 1 --2- 172.21.15.5:0/2397347199 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fad0006bf50 0x7fad0014fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:34.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.891+0000 7fad06191640 1 -- 172.21.15.5:0/2397347199 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fad0006bf50 msgr2=0x7fad0014fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:34.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.891+0000 7fad06191640 1 --2- 172.21.15.5:0/2397347199 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fad0006bf50 0x7fad0014fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:34.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.891+0000 7fad06191640 1 -- 172.21.15.5:0/2397347199 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad0006d4f0 msgr2=0x7fad001f12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:34.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.891+0000 7fad06191640 1 --2- 172.21.15.5:0/2397347199 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad0006d4f0 0x7fad001f12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:34.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.891+0000 7fad06191640 1 -- 172.21.15.5:0/2397347199 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facfc009400 con 0x7fad001543a0 2024-03-24T10:54:34.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.893+0000 7fad06191640 1 --2- 172.21.15.5:0/2397347199 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad001543a0 0x7fad001f1820 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7facf800efc0 tx=0x7facf800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:34.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.898+0000 7facf67fc640 1 -- 172.21.15.5:0/2397347199 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf8014480 con 0x7fad001543a0 2024-03-24T10:54:34.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.898+0000 7facf67fc640 1 -- 172.21.15.5:0/2397347199 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facf800f040 con 0x7fad001543a0 2024-03-24T10:54:34.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.898+0000 7facf67fc640 1 -- 172.21.15.5:0/2397347199 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf8014820 con 0x7fad001543a0 2024-03-24T10:54:34.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.898+0000 7fad07bf4640 1 -- 172.21.15.5:0/2397347199 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad0014f0a0 con 0x7fad001543a0 2024-03-24T10:54:34.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.898+0000 7fad07bf4640 1 -- 172.21.15.5:0/2397347199 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad0014f660 con 0x7fad001543a0 2024-03-24T10:54:34.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.899+0000 7fad07bf4640 1 -- 172.21.15.5:0/2397347199 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad001f1000 con 0x7fad001543a0 2024-03-24T10:54:34.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.923+0000 7facf67fc640 1 -- 172.21.15.5:0/2397347199 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7facf8014980 con 0x7fad001543a0 2024-03-24T10:54:34.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.924+0000 7facf67fc640 1 --2- 172.21.15.5:0/2397347199 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7face406b810 0x7face406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:34.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.924+0000 7fad05990640 1 --2- 172.21.15.5:0/2397347199 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7face406b810 0x7face406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:34.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.924+0000 7fad05990640 1 --2- 172.21.15.5:0/2397347199 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7face406b810 0x7face406dcd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7facfc036000 tx=0x7facfc03b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:34.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:34.926+0000 7facf67fc640 1 -- 172.21.15.5:0/2397347199 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7facf8059c20 con 0x7fad001543a0 2024-03-24T10:54:35.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:35.202+0000 7facf67fc640 1 -- 172.21.15.5:0/2397347199 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7facf8055ec0 con 0x7fad001543a0 2024-03-24T10:54:35.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:35.422+0000 7fad07bf4640 1 -- 172.21.15.5:0/2397347199 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad000681d0 con 0x7fad001543a0 2024-03-24T10:54:35.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:35.437+0000 7facf67fc640 1 -- 172.21.15.5:0/2397347199 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7facf8018070 con 0x7fad001543a0 2024-03-24T10:54:35.538 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:35.538 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:35.539 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:35.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:35.439+0000 7fad07bf4640 1 -- 172.21.15.5:0/2397347199 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7face406b810 msgr2=0x7face406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:35.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:35.439+0000 7fad07bf4640 1 --2- 172.21.15.5:0/2397347199 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7face406b810 0x7face406dcd0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7facfc036000 tx=0x7facfc03b040 comp rx=0 tx=0).stop 2024-03-24T10:54:35.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:35.439+0000 7fad07bf4640 1 -- 172.21.15.5:0/2397347199 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad001543a0 msgr2=0x7fad001f1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:35.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:35.439+0000 7fad07bf4640 1 --2- 172.21.15.5:0/2397347199 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad001543a0 0x7fad001f1820 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7facf800efc0 tx=0x7facf800c490 comp rx=0 tx=0).stop 2024-03-24T10:54:35.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:35.439+0000 7fad07bf4640 1 -- 172.21.15.5:0/2397347199 shutdown_connections 2024-03-24T10:54:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:35.439+0000 7fad07bf4640 1 --2- 172.21.15.5:0/2397347199 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7face406b810 0x7face406dcd0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:35.439+0000 7fad07bf4640 1 --2- 172.21.15.5:0/2397347199 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fad001543a0 0x7fad001f1820 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:35.439+0000 7fad07bf4640 1 --2- 172.21.15.5:0/2397347199 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fad0006d4f0 0x7fad001f12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:35.439+0000 7fad07bf4640 1 --2- 172.21.15.5:0/2397347199 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fad0006bf50 0x7fad0014fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:35.440+0000 7fad07bf4640 1 -- 172.21.15.5:0/2397347199 >> 172.21.15.5:0/2397347199 conn(0x7fad00072ef0 msgr2=0x7fad001512e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:35.440+0000 7fad07bf4640 1 -- 172.21.15.5:0/2397347199 shutdown_connections 2024-03-24T10:54:35.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:35.440+0000 7fad07bf4640 1 -- 172.21.15.5:0/2397347199 wait complete. 2024-03-24T10:54:36.145 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:36.145 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:37.146 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:37.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.248+0000 7f0a58415640 1 Processor -- start 2024-03-24T10:54:37.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.248+0000 7f0a58415640 1 -- start start 2024-03-24T10:54:37.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.249+0000 7f0a58415640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0a50154500 0x7f0a5006b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:37.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.249+0000 7f0a58415640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a500c4d70 con 0x7f0a500732e0 2024-03-24T10:54:37.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.249+0000 7f0a58415640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a500ca160 con 0x7f0a501548e0 2024-03-24T10:54:37.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.249+0000 7f0a58415640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f0a500c9550 con 0x7f0a50154500 2024-03-24T10:54:37.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.250+0000 7f0a561b1640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0a50154500 0x7f0a5006b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:37.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.250+0000 7f0a559b0640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f0a501548e0 0x7f0a500cc8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:57438/0 (socket says 172.21.15.5:57438) 2024-03-24T10:54:37.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.250+0000 7f0a559b0640 1 -- 172.21.15.5:0/1561222327 learned_addr learned my addr 172.21.15.5:0/1561222327 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:54:37.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.253+0000 7f0a561b1640 1 -- 172.21.15.5:0/1561222327 >> v1:172.21.15.57:6789/0 conn(0x7f0a501548e0 legacy=0x7f0a500cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:37.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.253+0000 7f0a561b1640 1 -- 172.21.15.5:0/1561222327 >> v1:172.21.15.5:6789/0 conn(0x7f0a500732e0 legacy=0x7f0a500c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:37.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.253+0000 7f0a551af640 1 -- 172.21.15.5:0/1561222327 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2080325741 0 0) 0x7f0a500ca160 con 0x7f0a501548e0 2024-03-24T10:54:37.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.253+0000 7f0a561b1640 1 -- 172.21.15.5:0/1561222327 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a500c9830 con 0x7f0a50154500 2024-03-24T10:54:37.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.256+0000 7f0a561b1640 1 --2- 172.21.15.5:0/1561222327 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a50154500 0x7f0a5006b8c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c0097a0 tx=0x7f0a4c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b215c9b8a0887cb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:37.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.263+0000 7f0a551af640 1 -- 172.21.15.5:0/1561222327 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c030610 con 0x7f0a50154500 2024-03-24T10:54:37.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.263+0000 7f0a551af640 1 -- 172.21.15.5:0/1561222327 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a4c030770 con 0x7f0a50154500 2024-03-24T10:54:37.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.263+0000 7f0a551af640 1 -- 172.21.15.5:0/1561222327 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c030b00 con 0x7f0a50154500 2024-03-24T10:54:37.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.264+0000 7f0a58415640 1 -- 172.21.15.5:0/1561222327 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a50154500 msgr2=0x7f0a5006b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:37.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.264+0000 7f0a58415640 1 --2- 172.21.15.5:0/1561222327 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a50154500 0x7f0a5006b8c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c0097a0 tx=0x7f0a4c02b960 comp rx=0 tx=0).stop 2024-03-24T10:54:37.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.264+0000 7f0a58415640 1 -- 172.21.15.5:0/1561222327 shutdown_connections 2024-03-24T10:54:37.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.264+0000 7f0a58415640 1 --2- 172.21.15.5:0/1561222327 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a50154500 0x7f0a5006b8c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:37.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.264+0000 7f0a58415640 1 -- 172.21.15.5:0/1561222327 >> 172.21.15.5:0/1561222327 conn(0x7f0a5006c240 msgr2=0x7f0a5006c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:37.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.264+0000 7f0a58415640 1 -- 172.21.15.5:0/1561222327 shutdown_connections 2024-03-24T10:54:37.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.264+0000 7f0a58415640 1 -- 172.21.15.5:0/1561222327 wait complete. 2024-03-24T10:54:37.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.265+0000 7f0a58415640 1 Processor -- start 2024-03-24T10:54:37.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.266+0000 7f0a58415640 1 -- start start 2024-03-24T10:54:37.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.266+0000 7f0a58415640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0a500732e0 0x7f0a501e6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:37.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.266+0000 7f0a58415640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a50154500 0x7f0a501e5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:37.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.266+0000 7f0a58415640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a501548e0 0x7f0a501e5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:37.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.266+0000 7f0a58415640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f0a501efea0 con 0x7f0a501548e0 2024-03-24T10:54:37.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.266+0000 7f0a58415640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0a500c39f0 con 0x7f0a50154500 2024-03-24T10:54:37.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.266+0000 7f0a58415640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0a501f35b0 con 0x7f0a500732e0 2024-03-24T10:54:37.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.267+0000 7f0a569b2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a501548e0 0x7f0a501e5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:37.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.267+0000 7f0a559b0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a50154500 0x7f0a501e5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:37.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.267+0000 7f0a561b1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0a500732e0 0x7f0a501e6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:37.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.267+0000 7f0a569b2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a501548e0 0x7f0a501e5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50916/0 (socket says 172.21.15.5:50916) 2024-03-24T10:54:37.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.267+0000 7f0a569b2640 1 -- 172.21.15.5:0/2408783603 learned_addr learned my addr 172.21.15.5:0/2408783603 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:37.270 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.271+0000 7f0a569b2640 1 -- 172.21.15.5:0/2408783603 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0a500732e0 msgr2=0x7f0a501e6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:37.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.271+0000 7f0a569b2640 1 --2- 172.21.15.5:0/2408783603 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0a500732e0 0x7f0a501e6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:37.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.271+0000 7f0a569b2640 1 -- 172.21.15.5:0/2408783603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a50154500 msgr2=0x7f0a501e5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:37.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.271+0000 7f0a569b2640 1 --2- 172.21.15.5:0/2408783603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a50154500 0x7f0a501e5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:37.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.271+0000 7f0a569b2640 1 -- 172.21.15.5:0/2408783603 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a4c009400 con 0x7f0a501548e0 2024-03-24T10:54:37.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.273+0000 7f0a569b2640 1 --2- 172.21.15.5:0/2408783603 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a501548e0 0x7f0a501e5b50 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0a4800c680 tx=0x7f0a4800cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:37.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.277+0000 7f0a46ffd640 1 -- 172.21.15.5:0/2408783603 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a48003b70 con 0x7f0a501548e0 2024-03-24T10:54:37.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.277+0000 7f0a46ffd640 1 -- 172.21.15.5:0/2408783603 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a48003cd0 con 0x7f0a501548e0 2024-03-24T10:54:37.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.277+0000 7f0a58415640 1 -- 172.21.15.5:0/2408783603 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a501f0160 con 0x7f0a501548e0 2024-03-24T10:54:37.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.277+0000 7f0a46ffd640 1 -- 172.21.15.5:0/2408783603 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a48005310 con 0x7f0a501548e0 2024-03-24T10:54:37.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.277+0000 7f0a58415640 1 -- 172.21.15.5:0/2408783603 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a501f0720 con 0x7f0a501548e0 2024-03-24T10:54:37.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.279+0000 7f0a58415640 1 -- 172.21.15.5:0/2408783603 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a5006b130 con 0x7f0a501548e0 2024-03-24T10:54:37.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.303+0000 7f0a46ffd640 1 -- 172.21.15.5:0/2408783603 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0a48006b50 con 0x7f0a501548e0 2024-03-24T10:54:37.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.303+0000 7f0a46ffd640 1 --2- 172.21.15.5:0/2408783603 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0a3406b840 0x7f0a3406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:37.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.304+0000 7f0a561b1640 1 --2- 172.21.15.5:0/2408783603 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0a3406b840 0x7f0a3406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:37.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.304+0000 7f0a561b1640 1 --2- 172.21.15.5:0/2408783603 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0a3406b840 0x7f0a3406dd00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c036000 tx=0x7f0a4c008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:37.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.308+0000 7f0a46ffd640 1 -- 172.21.15.5:0/2408783603 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f0a48059120 con 0x7f0a501548e0 2024-03-24T10:54:37.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.580+0000 7f0a46ffd640 1 -- 172.21.15.5:0/2408783603 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0a480553c0 con 0x7f0a501548e0 2024-03-24T10:54:37.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.808+0000 7f0a58415640 1 -- 172.21.15.5:0/2408783603 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a500c91e0 con 0x7f0a501548e0 2024-03-24T10:54:37.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.822+0000 7f0a46ffd640 1 -- 172.21.15.5:0/2408783603 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f0a48058a80 con 0x7f0a501548e0 2024-03-24T10:54:37.822 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:37.822 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:37.822 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:37.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.824+0000 7f0a58415640 1 -- 172.21.15.5:0/2408783603 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0a3406b840 msgr2=0x7f0a3406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:37.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.824+0000 7f0a58415640 1 --2- 172.21.15.5:0/2408783603 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0a3406b840 0x7f0a3406dd00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c036000 tx=0x7f0a4c008030 comp rx=0 tx=0).stop 2024-03-24T10:54:37.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.824+0000 7f0a58415640 1 -- 172.21.15.5:0/2408783603 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a501548e0 msgr2=0x7f0a501e5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:37.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.824+0000 7f0a58415640 1 --2- 172.21.15.5:0/2408783603 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a501548e0 0x7f0a501e5b50 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0a4800c680 tx=0x7f0a4800cb40 comp rx=0 tx=0).stop 2024-03-24T10:54:37.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.824+0000 7f0a58415640 1 -- 172.21.15.5:0/2408783603 shutdown_connections 2024-03-24T10:54:37.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.824+0000 7f0a58415640 1 --2- 172.21.15.5:0/2408783603 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0a3406b840 0x7f0a3406dd00 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:37.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.824+0000 7f0a58415640 1 --2- 172.21.15.5:0/2408783603 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0a501548e0 0x7f0a501e5b50 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:37.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.824+0000 7f0a58415640 1 --2- 172.21.15.5:0/2408783603 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0a50154500 0x7f0a501e5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:37.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.824+0000 7f0a58415640 1 --2- 172.21.15.5:0/2408783603 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0a500732e0 0x7f0a501e6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:37.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.824+0000 7f0a58415640 1 -- 172.21.15.5:0/2408783603 >> 172.21.15.5:0/2408783603 conn(0x7f0a5006c240 msgr2=0x7f0a500c78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:37.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.825+0000 7f0a58415640 1 -- 172.21.15.5:0/2408783603 shutdown_connections 2024-03-24T10:54:37.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:37.825+0000 7f0a58415640 1 -- 172.21.15.5:0/2408783603 wait complete. 2024-03-24T10:54:38.441 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:38.441 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:39.442 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:39.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.555+0000 7fbb92842640 1 Processor -- start 2024-03-24T10:54:39.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.555+0000 7fbb92842640 1 -- start start 2024-03-24T10:54:39.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.556+0000 7fbb92842640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbb8c15ac00 0x7fbb8c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:39.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.556+0000 7fbb92842640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbb8c0ccaa0 0x7fbb8c06bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:39.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.556+0000 7fbb92842640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb8c0cb2f0 con 0x7fbb8c1543a0 2024-03-24T10:54:39.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.556+0000 7fbb92842640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fbb8c0c6750 con 0x7fbb8c0ccaa0 2024-03-24T10:54:39.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.556+0000 7fbb92842640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fbb8c0cc380 con 0x7fbb8c15ac00 2024-03-24T10:54:39.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.557+0000 7fbb8b7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbb8c15ac00 0x7fbb8c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:39.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.557+0000 7fbb90ddf640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbb8c0ccaa0 0x7fbb8c06bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:39.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.557+0000 7fbb90ddf640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbb8c0ccaa0 0x7fbb8c06bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50920/0 (socket says 172.21.15.5:50920) 2024-03-24T10:54:39.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.557+0000 7fbb90ddf640 1 -- 172.21.15.5:0/2101888503 learned_addr learned my addr 172.21.15.5:0/2101888503 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:39.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.560+0000 7fbb90ddf640 1 -- 172.21.15.5:0/2101888503 >> v2:172.21.15.57:3300/0 conn(0x7fbb8c15ac00 msgr2=0x7fbb8c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:54:39.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.560+0000 7fbb90ddf640 1 --2- 172.21.15.5:0/2101888503 >> v2:172.21.15.57:3300/0 conn(0x7fbb8c15ac00 0x7fbb8c0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:39.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.560+0000 7fbb90ddf640 1 -- 172.21.15.5:0/2101888503 >> v1:172.21.15.57:6789/0 conn(0x7fbb8c1543a0 legacy=0x7fbb8c06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:39.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.560+0000 7fbb90ddf640 1 -- 172.21.15.5:0/2101888503 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb8c15b070 con 0x7fbb8c0ccaa0 2024-03-24T10:54:39.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.563+0000 7fbb90ddf640 1 --2- 172.21.15.5:0/2101888503 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb8c0ccaa0 0x7fbb8c06bf50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c0c6990 tx=0x7fbb8402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc19a569bf1280e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:39.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.567+0000 7fbb8affd640 1 -- 172.21.15.5:0/2101888503 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8402cb20 con 0x7fbb8c0ccaa0 2024-03-24T10:54:39.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.567+0000 7fbb8affd640 1 -- 172.21.15.5:0/2101888503 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb8402cc80 con 0x7fbb8c0ccaa0 2024-03-24T10:54:39.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.567+0000 7fbb8affd640 1 -- 172.21.15.5:0/2101888503 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb840343f0 con 0x7fbb8c0ccaa0 2024-03-24T10:54:39.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.567+0000 7fbb92842640 1 -- 172.21.15.5:0/2101888503 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb8c0ccaa0 msgr2=0x7fbb8c06bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:39.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.567+0000 7fbb92842640 1 --2- 172.21.15.5:0/2101888503 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb8c0ccaa0 0x7fbb8c06bf50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c0c6990 tx=0x7fbb8402bf80 comp rx=0 tx=0).stop 2024-03-24T10:54:39.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.568+0000 7fbb92842640 1 -- 172.21.15.5:0/2101888503 shutdown_connections 2024-03-24T10:54:39.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.568+0000 7fbb92842640 1 --2- 172.21.15.5:0/2101888503 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb8c0ccaa0 0x7fbb8c06bf50 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:39.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.568+0000 7fbb92842640 1 --2- 172.21.15.5:0/2101888503 >> v2:172.21.15.57:3300/0 conn(0x7fbb8c15ac00 0x7fbb8c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:39.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.568+0000 7fbb92842640 1 -- 172.21.15.5:0/2101888503 >> 172.21.15.5:0/2101888503 conn(0x7fbb8c072ef0 msgr2=0x7fbb8c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:39.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.568+0000 7fbb92842640 1 -- 172.21.15.5:0/2101888503 shutdown_connections 2024-03-24T10:54:39.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.568+0000 7fbb92842640 1 -- 172.21.15.5:0/2101888503 wait complete. 2024-03-24T10:54:39.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.569+0000 7fbb92842640 1 Processor -- start 2024-03-24T10:54:39.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.569+0000 7fbb92842640 1 -- start start 2024-03-24T10:54:39.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.570+0000 7fbb92842640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb8c0ccaa0 0x7fbb8c0ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:39.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.570+0000 7fbb92842640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb8c1543a0 0x7fbb8c1597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:39.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.570+0000 7fbb92842640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbb8c15ac00 0x7fbb8c1f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:39.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.570+0000 7fbb92842640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8c1efca0 con 0x7fbb8c1543a0 2024-03-24T10:54:39.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.570+0000 7fbb92842640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8c15c0b0 con 0x7fbb8c0ccaa0 2024-03-24T10:54:39.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.570+0000 7fbb92842640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fbb8c1f33d0 con 0x7fbb8c15ac00 2024-03-24T10:54:39.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.570+0000 7fbb8b7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb8c1543a0 0x7fbb8c1597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:39.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.570+0000 7fbb8bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb8c0ccaa0 0x7fbb8c0ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:39.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.571+0000 7fbb8b7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb8c1543a0 0x7fbb8c1597e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50936/0 (socket says 172.21.15.5:50936) 2024-03-24T10:54:39.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.571+0000 7fbb8b7fe640 1 -- 172.21.15.5:0/2658829159 learned_addr learned my addr 172.21.15.5:0/2658829159 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:39.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.571+0000 7fbb90ddf640 1 --2- 172.21.15.5:0/2658829159 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbb8c15ac00 0x7fbb8c1f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:39.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.575+0000 7fbb8b7fe640 1 -- 172.21.15.5:0/2658829159 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbb8c15ac00 msgr2=0x7fbb8c1f3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:39.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.575+0000 7fbb8b7fe640 1 --2- 172.21.15.5:0/2658829159 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbb8c15ac00 0x7fbb8c1f3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:39.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.575+0000 7fbb8b7fe640 1 -- 172.21.15.5:0/2658829159 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb8c0ccaa0 msgr2=0x7fbb8c0ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:39.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.575+0000 7fbb8b7fe640 1 --2- 172.21.15.5:0/2658829159 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb8c0ccaa0 0x7fbb8c0ceb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:39.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.575+0000 7fbb8b7fe640 1 -- 172.21.15.5:0/2658829159 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb84009b30 con 0x7fbb8c1543a0 2024-03-24T10:54:39.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.577+0000 7fbb8b7fe640 1 --2- 172.21.15.5:0/2658829159 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb8c1543a0 0x7fbb8c1597e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000e8a0 tx=0x7fbb8000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:39.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.581+0000 7fbb88ff9640 1 -- 172.21.15.5:0/2658829159 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb800094f0 con 0x7fbb8c1543a0 2024-03-24T10:54:39.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.582+0000 7fbb88ff9640 1 -- 172.21.15.5:0/2658829159 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb8000c940 con 0x7fbb8c1543a0 2024-03-24T10:54:39.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.582+0000 7fbb92842640 1 -- 172.21.15.5:0/2658829159 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb8c15b2b0 con 0x7fbb8c1543a0 2024-03-24T10:54:39.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.582+0000 7fbb88ff9640 1 -- 172.21.15.5:0/2658829159 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8000ccf0 con 0x7fbb8c1543a0 2024-03-24T10:54:39.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.582+0000 7fbb92842640 1 -- 172.21.15.5:0/2658829159 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb8c15b810 con 0x7fbb8c1543a0 2024-03-24T10:54:39.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.583+0000 7fbb92842640 1 -- 172.21.15.5:0/2658829159 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb8c0681d0 con 0x7fbb8c1543a0 2024-03-24T10:54:39.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.883+0000 7fbb88ff9640 1 -- 172.21.15.5:0/2658829159 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbb80013470 con 0x7fbb8c1543a0 2024-03-24T10:54:39.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.883+0000 7fbb88ff9640 1 --2- 172.21.15.5:0/2658829159 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbb7806b860 0x7fbb7806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:39.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.883+0000 7fbb8bfff640 1 --2- 172.21.15.5:0/2658829159 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbb7806b860 0x7fbb7806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:39.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.884+0000 7fbb8bfff640 1 --2- 172.21.15.5:0/2658829159 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbb7806b860 0x7fbb7806dd20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c000f60 tx=0x7fbb7c009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:39.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.905+0000 7fbb88ff9640 1 -- 172.21.15.5:0/2658829159 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fbb80059260 con 0x7fbb8c1543a0 2024-03-24T10:54:39.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:39.905+0000 7fbb88ff9640 1 -- 172.21.15.5:0/2658829159 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbb80055500 con 0x7fbb8c1543a0 2024-03-24T10:54:40.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:40.135+0000 7fbb92842640 1 -- 172.21.15.5:0/2658829159 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb8c14f610 con 0x7fbb8c1543a0 2024-03-24T10:54:40.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:40.158+0000 7fbb88ff9640 1 -- 172.21.15.5:0/2658829159 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fbb80021090 con 0x7fbb8c1543a0 2024-03-24T10:54:40.158 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:40.159 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:40.159 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:40.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:40.160+0000 7fbb92842640 1 -- 172.21.15.5:0/2658829159 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbb7806b860 msgr2=0x7fbb7806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:40.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:40.160+0000 7fbb92842640 1 --2- 172.21.15.5:0/2658829159 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbb7806b860 0x7fbb7806dd20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c000f60 tx=0x7fbb7c009040 comp rx=0 tx=0).stop 2024-03-24T10:54:40.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:40.160+0000 7fbb92842640 1 -- 172.21.15.5:0/2658829159 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb8c1543a0 msgr2=0x7fbb8c1597e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:40.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:40.160+0000 7fbb92842640 1 --2- 172.21.15.5:0/2658829159 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb8c1543a0 0x7fbb8c1597e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000e8a0 tx=0x7fbb8000ed60 comp rx=0 tx=0).stop 2024-03-24T10:54:40.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:40.160+0000 7fbb92842640 1 -- 172.21.15.5:0/2658829159 shutdown_connections 2024-03-24T10:54:40.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:40.160+0000 7fbb92842640 1 --2- 172.21.15.5:0/2658829159 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbb7806b860 0x7fbb7806dd20 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:40.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:40.160+0000 7fbb92842640 1 --2- 172.21.15.5:0/2658829159 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbb8c1543a0 0x7fbb8c1597e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:40.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:40.160+0000 7fbb92842640 1 --2- 172.21.15.5:0/2658829159 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbb8c15ac00 0x7fbb8c1f3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:40.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:40.160+0000 7fbb92842640 1 --2- 172.21.15.5:0/2658829159 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbb8c0ccaa0 0x7fbb8c0ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:40.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:40.161+0000 7fbb92842640 1 -- 172.21.15.5:0/2658829159 >> 172.21.15.5:0/2658829159 conn(0x7fbb8c072ef0 msgr2=0x7fbb8c15a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:40.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:40.161+0000 7fbb92842640 1 -- 172.21.15.5:0/2658829159 shutdown_connections 2024-03-24T10:54:40.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:40.161+0000 7fbb92842640 1 -- 172.21.15.5:0/2658829159 wait complete. 2024-03-24T10:54:40.776 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:40.776 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:41.777 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:41.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.879+0000 7f38371d6640 1 Processor -- start 2024-03-24T10:54:41.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.879+0000 7f38371d6640 1 -- start start 2024-03-24T10:54:41.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.880+0000 7f38371d6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f38300cb270 0x7f38300cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:41.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.880+0000 7f38371d6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38300cd960 0x7f38300cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:41.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.880+0000 7f38371d6640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38300ccba0 con 0x7f38300cae90 2024-03-24T10:54:41.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.880+0000 7f38371d6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f38300cc9b0 con 0x7f38300cb270 2024-03-24T10:54:41.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.880+0000 7f38371d6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f38300cce40 con 0x7f38300cd960 2024-03-24T10:54:41.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.881+0000 7f3834f72640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f38300cae90 0x7f38300cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:57632/0 (socket says 172.21.15.5:57632) 2024-03-24T10:54:41.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.881+0000 7f3834f72640 1 -- 172.21.15.5:0/2925437894 learned_addr learned my addr 172.21.15.5:0/2925437894 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:54:41.881 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.881+0000 7f3835773640 1 --2- 172.21.15.5:0/2925437894 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38300cd960 0x7f38300cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:41.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.881+0000 7f382ffff640 1 --2- 172.21.15.5:0/2925437894 >> v2:172.21.15.57:3300/0 conn(0x7f38300cb270 0x7f38300cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:41.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.884+0000 7f382ffff640 1 -- 172.21.15.5:0/2925437894 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38300cd960 msgr2=0x7f38300cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:54:41.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.884+0000 7f382ffff640 1 --2- 172.21.15.5:0/2925437894 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38300cd960 0x7f38300cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:41.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.884+0000 7f382ffff640 1 -- 172.21.15.5:0/2925437894 >> v1:172.21.15.5:6789/0 conn(0x7f38300cae90 legacy=0x7f38300cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:41.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.884+0000 7f382ffff640 1 -- 172.21.15.5:0/2925437894 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f383015a590 con 0x7f38300cb270 2024-03-24T10:54:41.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.886+0000 7f382ffff640 1 --2- 172.21.15.5:0/2925437894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38300cb270 0x7f38300cf530 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f381c0097a0 tx=0x7f381c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86fbac70cdffc005 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:41.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.888+0000 7f382f7fe640 1 -- 172.21.15.5:0/2925437894 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f381c030610 con 0x7f38300cb270 2024-03-24T10:54:41.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.890+0000 7f382f7fe640 1 -- 172.21.15.5:0/2925437894 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f381c030770 con 0x7f38300cb270 2024-03-24T10:54:41.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.890+0000 7f38371d6640 1 -- 172.21.15.5:0/2925437894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38300cb270 msgr2=0x7f38300cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:41.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.890+0000 7f38371d6640 1 --2- 172.21.15.5:0/2925437894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38300cb270 0x7f38300cf530 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f381c0097a0 tx=0x7f381c02b960 comp rx=0 tx=0).stop 2024-03-24T10:54:41.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.890+0000 7f38371d6640 1 -- 172.21.15.5:0/2925437894 shutdown_connections 2024-03-24T10:54:41.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.890+0000 7f38371d6640 1 --2- 172.21.15.5:0/2925437894 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38300cd960 0x7f38300cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:41.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.890+0000 7f38371d6640 1 --2- 172.21.15.5:0/2925437894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38300cb270 0x7f38300cf530 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:41.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.890+0000 7f38371d6640 1 -- 172.21.15.5:0/2925437894 >> 172.21.15.5:0/2925437894 conn(0x7f38300cfb80 msgr2=0x7f38300cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:41.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.890+0000 7f38371d6640 1 -- 172.21.15.5:0/2925437894 shutdown_connections 2024-03-24T10:54:41.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.891+0000 7f38371d6640 1 -- 172.21.15.5:0/2925437894 wait complete. 2024-03-24T10:54:41.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.892+0000 7f38371d6640 1 Processor -- start 2024-03-24T10:54:41.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.892+0000 7f38371d6640 1 -- start start 2024-03-24T10:54:41.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.893+0000 7f38371d6640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38300cae90 0x7f38301f2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:41.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.893+0000 7f38371d6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38300cb270 0x7f38300b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:41.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.893+0000 7f38371d6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38300cd960 0x7f38300b1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:41.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.893+0000 7f38371d6640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f383015e970 con 0x7f38300cae90 2024-03-24T10:54:41.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.893+0000 7f38371d6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f38300af240 con 0x7f38300cd960 2024-03-24T10:54:41.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.893+0000 7f38371d6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f38300b30c0 con 0x7f38300cb270 2024-03-24T10:54:41.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.893+0000 7f382ffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38300cb270 0x7f38300b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:41.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.893+0000 7f3834f72640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38300cae90 0x7f38301f2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:41.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.894+0000 7f3835773640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38300cd960 0x7f38300b1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:41.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.894+0000 7f3834f72640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38300cae90 0x7f38301f2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50950/0 (socket says 172.21.15.5:50950) 2024-03-24T10:54:41.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.894+0000 7f3834f72640 1 -- 172.21.15.5:0/4169811649 learned_addr learned my addr 172.21.15.5:0/4169811649 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:41.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.897+0000 7f3834f72640 1 -- 172.21.15.5:0/4169811649 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38300cb270 msgr2=0x7f38300b0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:41.897 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.897+0000 7f3834f72640 1 --2- 172.21.15.5:0/4169811649 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38300cb270 0x7f38300b0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:41.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.897+0000 7f3834f72640 1 -- 172.21.15.5:0/4169811649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38300cd960 msgr2=0x7f38300b1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:41.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.897+0000 7f3834f72640 1 --2- 172.21.15.5:0/4169811649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38300cd960 0x7f38300b1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:41.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.897+0000 7f3834f72640 1 -- 172.21.15.5:0/4169811649 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f381c009400 con 0x7f38300cae90 2024-03-24T10:54:41.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.900+0000 7f3834f72640 1 --2- 172.21.15.5:0/4169811649 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38300cae90 0x7f38301f2bc0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f382400c400 tx=0x7f382400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:41.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.904+0000 7f382d7fa640 1 -- 172.21.15.5:0/4169811649 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3824005ce0 con 0x7f38300cae90 2024-03-24T10:54:41.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.904+0000 7f382d7fa640 1 -- 172.21.15.5:0/4169811649 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3824005e40 con 0x7f38300cae90 2024-03-24T10:54:41.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.904+0000 7f382d7fa640 1 -- 172.21.15.5:0/4169811649 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3824007310 con 0x7f38300cae90 2024-03-24T10:54:41.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.904+0000 7f38371d6640 1 -- 172.21.15.5:0/4169811649 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38301f0500 con 0x7f38300cae90 2024-03-24T10:54:41.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.904+0000 7f38371d6640 1 -- 172.21.15.5:0/4169811649 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38301efa20 con 0x7f38300cae90 2024-03-24T10:54:41.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.905+0000 7f38371d6640 1 -- 172.21.15.5:0/4169811649 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38300681d0 con 0x7f38300cae90 2024-03-24T10:54:41.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.924+0000 7f382d7fa640 1 -- 172.21.15.5:0/4169811649 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f38240074f0 con 0x7f38300cae90 2024-03-24T10:54:41.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.925+0000 7f382d7fa640 1 --2- 172.21.15.5:0/4169811649 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f381406b860 0x7f381406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:41.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.925+0000 7f382ffff640 1 --2- 172.21.15.5:0/4169811649 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f381406b860 0x7f381406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:41.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.926+0000 7f382ffff640 1 --2- 172.21.15.5:0/4169811649 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f381406b860 0x7f381406dd20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f381c036000 tx=0x7f381c008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:41.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:41.929+0000 7f382d7fa640 1 -- 172.21.15.5:0/4169811649 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f3824058420 con 0x7f38300cae90 2024-03-24T10:54:42.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:42.205+0000 7f382d7fa640 1 -- 172.21.15.5:0/4169811649 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f38240546c0 con 0x7f38300cae90 2024-03-24T10:54:42.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:42.450+0000 7f38371d6640 1 -- 172.21.15.5:0/4169811649 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38300d0330 con 0x7f38300cae90 2024-03-24T10:54:42.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:42.492+0000 7f382d7fa640 1 -- 172.21.15.5:0/4169811649 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f3824017020 con 0x7f38300cae90 2024-03-24T10:54:42.492 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:42.492 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:42.492 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:42.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:42.494+0000 7f38371d6640 1 -- 172.21.15.5:0/4169811649 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f381406b860 msgr2=0x7f381406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:42.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:42.494+0000 7f38371d6640 1 --2- 172.21.15.5:0/4169811649 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f381406b860 0x7f381406dd20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f381c036000 tx=0x7f381c008030 comp rx=0 tx=0).stop 2024-03-24T10:54:42.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:42.494+0000 7f38371d6640 1 -- 172.21.15.5:0/4169811649 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38300cae90 msgr2=0x7f38301f2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:42.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:42.494+0000 7f38371d6640 1 --2- 172.21.15.5:0/4169811649 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38300cae90 0x7f38301f2bc0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f382400c400 tx=0x7f382400c8c0 comp rx=0 tx=0).stop 2024-03-24T10:54:42.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:42.494+0000 7f38371d6640 1 -- 172.21.15.5:0/4169811649 shutdown_connections 2024-03-24T10:54:42.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:42.494+0000 7f38371d6640 1 --2- 172.21.15.5:0/4169811649 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f381406b860 0x7f381406dd20 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:42.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:42.494+0000 7f38371d6640 1 --2- 172.21.15.5:0/4169811649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f38300cd960 0x7f38300b1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:42.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:42.494+0000 7f38371d6640 1 --2- 172.21.15.5:0/4169811649 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f38300cb270 0x7f38300b0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:42.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:42.494+0000 7f38371d6640 1 --2- 172.21.15.5:0/4169811649 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f38300cae90 0x7f38301f2bc0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:42.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:42.494+0000 7f38371d6640 1 -- 172.21.15.5:0/4169811649 >> 172.21.15.5:0/4169811649 conn(0x7f38300cfb80 msgr2=0x7f38300ce410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:42.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:42.494+0000 7f38371d6640 1 -- 172.21.15.5:0/4169811649 shutdown_connections 2024-03-24T10:54:42.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:42.494+0000 7f38371d6640 1 -- 172.21.15.5:0/4169811649 wait complete. 2024-03-24T10:54:43.109 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:43.110 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:44.111 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:44.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.214+0000 7ff7adc83640 1 Processor -- start 2024-03-24T10:54:44.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.214+0000 7ff7adc83640 1 -- start start 2024-03-24T10:54:44.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.215+0000 7ff7adc83640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff7a81543a0 0x7ff7a815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:44.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.215+0000 7ff7adc83640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7a80cb2f0 con 0x7ff7a806d4f0 2024-03-24T10:54:44.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.215+0000 7ff7adc83640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7a80ca7e0 con 0x7ff7a806bf50 2024-03-24T10:54:44.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.215+0000 7ff7adc83640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff7a80c6750 con 0x7ff7a81543a0 2024-03-24T10:54:44.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.216+0000 7ff7a6ffd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7ff7a806d4f0 0x7ff7a80cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:38952/0 (socket says 172.21.15.5:38952) 2024-03-24T10:54:44.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.216+0000 7ff7a6ffd640 1 -- 172.21.15.5:0/2503567064 learned_addr learned my addr 172.21.15.5:0/2503567064 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:54:44.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.216+0000 7ff7a77fe640 1 --2- 172.21.15.5:0/2503567064 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff7a81543a0 0x7ff7a815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:44.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.219+0000 7ff7a67fc640 1 -- 172.21.15.5:0/2503567064 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2727281747 0 0) 0x7ff7a80ca7e0 con 0x7ff7a806bf50 2024-03-24T10:54:44.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.219+0000 7ff7a67fc640 1 -- 172.21.15.5:0/2503567064 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff790003520 con 0x7ff7a806bf50 2024-03-24T10:54:44.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.219+0000 7ff7a67fc640 1 -- 172.21.15.5:0/2503567064 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1762852660 0 0) 0x7ff7a80cb2f0 con 0x7ff7a806d4f0 2024-03-24T10:54:44.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.219+0000 7ff7a67fc640 1 -- 172.21.15.5:0/2503567064 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7a80ca7e0 con 0x7ff7a806d4f0 2024-03-24T10:54:44.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.220+0000 7ff7a77fe640 1 -- 172.21.15.5:0/2503567064 >> v1:172.21.15.57:6789/0 conn(0x7ff7a806bf50 legacy=0x7ff7a806c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:44.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.220+0000 7ff7a77fe640 1 -- 172.21.15.5:0/2503567064 >> v1:172.21.15.5:6789/0 conn(0x7ff7a806d4f0 legacy=0x7ff7a80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:44.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.220+0000 7ff7a77fe640 1 -- 172.21.15.5:0/2503567064 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7a80cb520 con 0x7ff7a81543a0 2024-03-24T10:54:44.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.222+0000 7ff7a77fe640 1 --2- 172.21.15.5:0/2503567064 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff7a81543a0 0x7ff7a815ac00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff7980097d0 tx=0x7ff79802b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41d8ce1f7a26fdbf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:44.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.228+0000 7ff7a67fc640 1 -- 172.21.15.5:0/2503567064 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff798030880 con 0x7ff7a81543a0 2024-03-24T10:54:44.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.228+0000 7ff7a67fc640 1 -- 172.21.15.5:0/2503567064 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7980309e0 con 0x7ff7a81543a0 2024-03-24T10:54:44.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.228+0000 7ff7a67fc640 1 -- 172.21.15.5:0/2503567064 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff798030d70 con 0x7ff7a81543a0 2024-03-24T10:54:44.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.229+0000 7ff7adc83640 1 -- 172.21.15.5:0/2503567064 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff7a81543a0 msgr2=0x7ff7a815ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:44.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.229+0000 7ff7adc83640 1 --2- 172.21.15.5:0/2503567064 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff7a81543a0 0x7ff7a815ac00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff7980097d0 tx=0x7ff79802b9c0 comp rx=0 tx=0).stop 2024-03-24T10:54:44.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.229+0000 7ff7adc83640 1 -- 172.21.15.5:0/2503567064 shutdown_connections 2024-03-24T10:54:44.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.229+0000 7ff7adc83640 1 --2- 172.21.15.5:0/2503567064 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff7a81543a0 0x7ff7a815ac00 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:44.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.229+0000 7ff7adc83640 1 -- 172.21.15.5:0/2503567064 >> 172.21.15.5:0/2503567064 conn(0x7ff7a8072ef0 msgr2=0x7ff7a8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:44.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.229+0000 7ff7adc83640 1 -- 172.21.15.5:0/2503567064 shutdown_connections 2024-03-24T10:54:44.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.229+0000 7ff7adc83640 1 -- 172.21.15.5:0/2503567064 wait complete. 2024-03-24T10:54:44.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.230+0000 7ff7adc83640 1 Processor -- start 2024-03-24T10:54:44.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.231+0000 7ff7adc83640 1 -- start start 2024-03-24T10:54:44.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.231+0000 7ff7adc83640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7a806bf50 0x7ff7a8159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:44.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.231+0000 7ff7adc83640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff7a806d4f0 0x7ff7a81593e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:44.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.231+0000 7ff7adc83640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7a81543a0 0x7ff7a80b4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:44.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.231+0000 7ff7adc83640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a815a170 con 0x7ff7a806bf50 2024-03-24T10:54:44.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.231+0000 7ff7adc83640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff7a81f0ce0 con 0x7ff7a81543a0 2024-03-24T10:54:44.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.231+0000 7ff7adc83640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff7a8159920 con 0x7ff7a806d4f0 2024-03-24T10:54:44.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.232+0000 7ff7a7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7a81543a0 0x7ff7a80b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:44.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.232+0000 7ff7a6ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff7a806d4f0 0x7ff7a81593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:44.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.232+0000 7ff7a77fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7a806bf50 0x7ff7a8159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:44.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.232+0000 7ff7a77fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7a806bf50 0x7ff7a8159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50608/0 (socket says 172.21.15.5:50608) 2024-03-24T10:54:44.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.232+0000 7ff7a77fe640 1 -- 172.21.15.5:0/1640199499 learned_addr learned my addr 172.21.15.5:0/1640199499 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:44.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.237+0000 7ff7a77fe640 1 -- 172.21.15.5:0/1640199499 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff7a806d4f0 msgr2=0x7ff7a81593e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:44.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.237+0000 7ff7a77fe640 1 --2- 172.21.15.5:0/1640199499 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff7a806d4f0 0x7ff7a81593e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:44.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.237+0000 7ff7a77fe640 1 -- 172.21.15.5:0/1640199499 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7a81543a0 msgr2=0x7ff7a80b4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:44.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.237+0000 7ff7a77fe640 1 --2- 172.21.15.5:0/1640199499 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7a81543a0 0x7ff7a80b4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:44.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.237+0000 7ff7a77fe640 1 -- 172.21.15.5:0/1640199499 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff798009400 con 0x7ff7a806bf50 2024-03-24T10:54:44.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.240+0000 7ff7a77fe640 1 --2- 172.21.15.5:0/1640199499 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7a806bf50 0x7ff7a8159c30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff7980098f0 tx=0x7ff798004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:44.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.244+0000 7ff78ffff640 1 -- 172.21.15.5:0/1640199499 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79803e040 con 0x7ff7a806bf50 2024-03-24T10:54:44.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.244+0000 7ff7adc83640 1 -- 172.21.15.5:0/1640199499 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7a81efe10 con 0x7ff7a806bf50 2024-03-24T10:54:44.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.244+0000 7ff7adc83640 1 -- 172.21.15.5:0/1640199499 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7a81f03b0 con 0x7ff7a806bf50 2024-03-24T10:54:44.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.245+0000 7ff7adc83640 1 -- 172.21.15.5:0/1640199499 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7a80681d0 con 0x7ff7a806bf50 2024-03-24T10:54:44.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.246+0000 7ff78ffff640 1 -- 172.21.15.5:0/1640199499 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff798039c90 con 0x7ff7a806bf50 2024-03-24T10:54:44.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.246+0000 7ff78ffff640 1 -- 172.21.15.5:0/1640199499 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79804e440 con 0x7ff7a806bf50 2024-03-24T10:54:44.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.529+0000 7ff78ffff640 1 -- 172.21.15.5:0/1640199499 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff79802f3d0 con 0x7ff7a806bf50 2024-03-24T10:54:44.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.530+0000 7ff78ffff640 1 --2- 172.21.15.5:0/1640199499 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff79406b810 0x7ff79406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:44.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.530+0000 7ff7a6ffd640 1 --2- 172.21.15.5:0/1640199499 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff79406b810 0x7ff79406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:44.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.530+0000 7ff7a6ffd640 1 --2- 172.21.15.5:0/1640199499 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff79406b810 0x7ff79406dcd0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8159370 tx=0x7ff79c007d30 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:44.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.555+0000 7ff78ffff640 1 -- 172.21.15.5:0/1640199499 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7ff79807b560 con 0x7ff7a806bf50 2024-03-24T10:54:44.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.555+0000 7ff78ffff640 1 -- 172.21.15.5:0/1640199499 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff798077800 con 0x7ff7a806bf50 2024-03-24T10:54:44.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.786+0000 7ff7adc83640 1 -- 172.21.15.5:0/1640199499 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7a814ed90 con 0x7ff7a806bf50 2024-03-24T10:54:44.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.807+0000 7ff78ffff640 1 -- 172.21.15.5:0/1640199499 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7ff79807aec0 con 0x7ff7a806bf50 2024-03-24T10:54:44.808 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:44.808 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:44.808 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:44.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.810+0000 7ff7adc83640 1 -- 172.21.15.5:0/1640199499 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff79406b810 msgr2=0x7ff79406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:44.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.810+0000 7ff7adc83640 1 --2- 172.21.15.5:0/1640199499 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff79406b810 0x7ff79406dcd0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8159370 tx=0x7ff79c007d30 comp rx=0 tx=0).stop 2024-03-24T10:54:44.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.810+0000 7ff7adc83640 1 -- 172.21.15.5:0/1640199499 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7a806bf50 msgr2=0x7ff7a8159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:44.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.810+0000 7ff7adc83640 1 --2- 172.21.15.5:0/1640199499 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7a806bf50 0x7ff7a8159c30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff7980098f0 tx=0x7ff798004730 comp rx=0 tx=0).stop 2024-03-24T10:54:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.810+0000 7ff7adc83640 1 -- 172.21.15.5:0/1640199499 shutdown_connections 2024-03-24T10:54:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.810+0000 7ff7adc83640 1 --2- 172.21.15.5:0/1640199499 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff79406b810 0x7ff79406dcd0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.810+0000 7ff7adc83640 1 --2- 172.21.15.5:0/1640199499 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff7a81543a0 0x7ff7a80b4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.810+0000 7ff7adc83640 1 --2- 172.21.15.5:0/1640199499 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff7a806d4f0 0x7ff7a81593e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.810+0000 7ff7adc83640 1 --2- 172.21.15.5:0/1640199499 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff7a806bf50 0x7ff7a8159c30 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.810+0000 7ff7adc83640 1 -- 172.21.15.5:0/1640199499 >> 172.21.15.5:0/1640199499 conn(0x7ff7a8072ef0 msgr2=0x7ff7a80c5550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.810+0000 7ff7adc83640 1 -- 172.21.15.5:0/1640199499 shutdown_connections 2024-03-24T10:54:44.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:44.810+0000 7ff7adc83640 1 -- 172.21.15.5:0/1640199499 wait complete. 2024-03-24T10:54:45.429 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:45.429 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:46.430 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:46.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.535+0000 7f7742c9d640 1 Processor -- start 2024-03-24T10:54:46.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.535+0000 7f7742c9d640 1 -- start start 2024-03-24T10:54:46.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.536+0000 7f7742c9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f773c15ac00 0x7f773c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:46.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.536+0000 7f7742c9d640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f773c0cb2f0 con 0x7f773c0ccaa0 2024-03-24T10:54:46.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.536+0000 7f7742c9d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f773c15b070 con 0x7f773c1543a0 2024-03-24T10:54:46.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.536+0000 7f7742c9d640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f773c0c6750 con 0x7f773c15ac00 2024-03-24T10:54:46.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.537+0000 7f774123a640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f773c0ccaa0 0x7f773c06bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:38968/0 (socket says 172.21.15.5:38968) 2024-03-24T10:54:46.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.537+0000 7f774123a640 1 -- 172.21.15.5:0/302896553 learned_addr learned my addr 172.21.15.5:0/302896553 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:54:46.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.537+0000 7f773bfff640 1 --2- 172.21.15.5:0/302896553 >> v2:172.21.15.5:3300/0 conn(0x7f773c15ac00 0x7f773c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:46.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.541+0000 7f773b7fe640 1 -- 172.21.15.5:0/302896553 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2476569087 0 0) 0x7f773c0cb2f0 con 0x7f773c0ccaa0 2024-03-24T10:54:46.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.541+0000 7f773b7fe640 1 -- 172.21.15.5:0/302896553 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f772c003520 con 0x7f773c0ccaa0 2024-03-24T10:54:46.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.541+0000 7f773b7fe640 1 -- 172.21.15.5:0/302896553 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3960842556 0 0) 0x7f773c15b070 con 0x7f773c1543a0 2024-03-24T10:54:46.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.541+0000 7f773b7fe640 1 -- 172.21.15.5:0/302896553 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f773c0cb2f0 con 0x7f773c1543a0 2024-03-24T10:54:46.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.543+0000 7f773bfff640 1 -- 172.21.15.5:0/302896553 >> v1:172.21.15.57:6789/0 conn(0x7f773c1543a0 legacy=0x7f773c06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:46.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.543+0000 7f773bfff640 1 -- 172.21.15.5:0/302896553 >> v1:172.21.15.5:6789/0 conn(0x7f773c0ccaa0 legacy=0x7f773c06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:46.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.543+0000 7f773bfff640 1 -- 172.21.15.5:0/302896553 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f773c0ca7e0 con 0x7f773c15ac00 2024-03-24T10:54:46.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.547+0000 7f773bfff640 1 --2- 172.21.15.5:0/302896553 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f773c15ac00 0x7f773c0cc560 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f77280097a0 tx=0x7f772802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d41a9e82eed244ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:46.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.552+0000 7f773b7fe640 1 -- 172.21.15.5:0/302896553 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7728030610 con 0x7f773c15ac00 2024-03-24T10:54:46.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.552+0000 7f773b7fe640 1 -- 172.21.15.5:0/302896553 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7728030770 con 0x7f773c15ac00 2024-03-24T10:54:46.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.552+0000 7f773b7fe640 1 -- 172.21.15.5:0/302896553 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7728030b00 con 0x7f773c15ac00 2024-03-24T10:54:46.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.552+0000 7f7742c9d640 1 -- 172.21.15.5:0/302896553 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f773c15ac00 msgr2=0x7f773c0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:46.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.552+0000 7f7742c9d640 1 --2- 172.21.15.5:0/302896553 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f773c15ac00 0x7f773c0cc560 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f77280097a0 tx=0x7f772802b960 comp rx=0 tx=0).stop 2024-03-24T10:54:46.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.552+0000 7f7742c9d640 1 -- 172.21.15.5:0/302896553 shutdown_connections 2024-03-24T10:54:46.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.552+0000 7f7742c9d640 1 --2- 172.21.15.5:0/302896553 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f773c15ac00 0x7f773c0cc560 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:46.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.552+0000 7f7742c9d640 1 -- 172.21.15.5:0/302896553 >> 172.21.15.5:0/302896553 conn(0x7f773c072ef0 msgr2=0x7f773c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:46.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.552+0000 7f7742c9d640 1 -- 172.21.15.5:0/302896553 shutdown_connections 2024-03-24T10:54:46.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.553+0000 7f7742c9d640 1 -- 172.21.15.5:0/302896553 wait complete. 2024-03-24T10:54:46.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.554+0000 7f7742c9d640 1 Processor -- start 2024-03-24T10:54:46.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.554+0000 7f7742c9d640 1 -- start start 2024-03-24T10:54:46.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.555+0000 7f7742c9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f773c0ccaa0 0x7f773c144930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:46.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.555+0000 7f7742c9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f773c1543a0 0x7f773c154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:46.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.555+0000 7f7742c9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f773c15ac00 0x7f773c159830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:46.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.555+0000 7f7742c9d640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f773c0cf690 con 0x7f773c0ccaa0 2024-03-24T10:54:46.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.555+0000 7f7742c9d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f773c157810 con 0x7f773c15ac00 2024-03-24T10:54:46.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.555+0000 7f7742c9d640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f773c0cf420 con 0x7f773c1543a0 2024-03-24T10:54:46.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.555+0000 7f7740a39640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f773c0ccaa0 0x7f773c144930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:46.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.555+0000 7f774123a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f773c15ac00 0x7f773c159830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:46.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.555+0000 7f7740a39640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f773c0ccaa0 0x7f773c144930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50624/0 (socket says 172.21.15.5:50624) 2024-03-24T10:54:46.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.555+0000 7f7740a39640 1 -- 172.21.15.5:0/21731075 learned_addr learned my addr 172.21.15.5:0/21731075 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:46.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.556+0000 7f773bfff640 1 --2- 172.21.15.5:0/21731075 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f773c1543a0 0x7f773c154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:46.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.559+0000 7f7740a39640 1 -- 172.21.15.5:0/21731075 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f773c1543a0 msgr2=0x7f773c154b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:46.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.559+0000 7f7740a39640 1 --2- 172.21.15.5:0/21731075 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f773c1543a0 0x7f773c154b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:46.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.559+0000 7f7740a39640 1 -- 172.21.15.5:0/21731075 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f773c15ac00 msgr2=0x7f773c159830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:46.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.559+0000 7f7740a39640 1 --2- 172.21.15.5:0/21731075 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f773c15ac00 0x7f773c159830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:46.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.559+0000 7f7740a39640 1 -- 172.21.15.5:0/21731075 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7728009400 con 0x7f773c0ccaa0 2024-03-24T10:54:46.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.561+0000 7f7740a39640 1 --2- 172.21.15.5:0/21731075 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f773c0ccaa0 0x7f773c144930 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f773000c490 tx=0x7f773000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:46.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.564+0000 7f77397fa640 1 -- 172.21.15.5:0/21731075 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77300052d0 con 0x7f773c0ccaa0 2024-03-24T10:54:46.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.564+0000 7f7742c9d640 1 -- 172.21.15.5:0/21731075 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f773c157510 con 0x7f773c0ccaa0 2024-03-24T10:54:46.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.564+0000 7f7742c9d640 1 -- 172.21.15.5:0/21731075 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f773c144e70 con 0x7f773c0ccaa0 2024-03-24T10:54:46.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.565+0000 7f7742c9d640 1 -- 172.21.15.5:0/21731075 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f773c0681d0 con 0x7f773c0ccaa0 2024-03-24T10:54:46.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.567+0000 7f77397fa640 1 -- 172.21.15.5:0/21731075 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7730005430 con 0x7f773c0ccaa0 2024-03-24T10:54:46.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.567+0000 7f77397fa640 1 -- 172.21.15.5:0/21731075 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7730012830 con 0x7f773c0ccaa0 2024-03-24T10:54:46.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.850+0000 7f77397fa640 1 -- 172.21.15.5:0/21731075 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7730012990 con 0x7f773c0ccaa0 2024-03-24T10:54:46.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.850+0000 7f77397fa640 1 --2- 172.21.15.5:0/21731075 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f772006b810 0x7f772006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:46.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.851+0000 7f773bfff640 1 --2- 172.21.15.5:0/21731075 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f772006b810 0x7f772006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:46.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.851+0000 7f773bfff640 1 --2- 172.21.15.5:0/21731075 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f772006b810 0x7f772006dcd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7728036000 tx=0x7f772803b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:46.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.864+0000 7f77397fa640 1 -- 172.21.15.5:0/21731075 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f7730058fb0 con 0x7f773c0ccaa0 2024-03-24T10:54:46.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:46.864+0000 7f77397fa640 1 -- 172.21.15.5:0/21731075 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7730055250 con 0x7f773c0ccaa0 2024-03-24T10:54:47.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:47.090+0000 7f7742c9d640 1 -- 172.21.15.5:0/21731075 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f773c1587d0 con 0x7f773c0ccaa0 2024-03-24T10:54:47.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:47.111+0000 7f77397fa640 1 -- 172.21.15.5:0/21731075 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f7730058910 con 0x7f773c0ccaa0 2024-03-24T10:54:47.111 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:47.111 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:47.111 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:47.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:47.113+0000 7f7742c9d640 1 -- 172.21.15.5:0/21731075 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f772006b810 msgr2=0x7f772006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:47.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:47.113+0000 7f7742c9d640 1 --2- 172.21.15.5:0/21731075 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f772006b810 0x7f772006dcd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7728036000 tx=0x7f772803b040 comp rx=0 tx=0).stop 2024-03-24T10:54:47.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:47.113+0000 7f7742c9d640 1 -- 172.21.15.5:0/21731075 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f773c0ccaa0 msgr2=0x7f773c144930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:47.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:47.113+0000 7f7742c9d640 1 --2- 172.21.15.5:0/21731075 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f773c0ccaa0 0x7f773c144930 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f773000c490 tx=0x7f773000c950 comp rx=0 tx=0).stop 2024-03-24T10:54:47.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:47.113+0000 7f7742c9d640 1 -- 172.21.15.5:0/21731075 shutdown_connections 2024-03-24T10:54:47.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:47.113+0000 7f7742c9d640 1 --2- 172.21.15.5:0/21731075 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f772006b810 0x7f772006dcd0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:47.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:47.113+0000 7f7742c9d640 1 --2- 172.21.15.5:0/21731075 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f773c15ac00 0x7f773c159830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:47.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:47.113+0000 7f7742c9d640 1 --2- 172.21.15.5:0/21731075 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f773c1543a0 0x7f773c154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:47.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:47.113+0000 7f7742c9d640 1 --2- 172.21.15.5:0/21731075 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f773c0ccaa0 0x7f773c144930 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:47.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:47.113+0000 7f7742c9d640 1 -- 172.21.15.5:0/21731075 >> 172.21.15.5:0/21731075 conn(0x7f773c072ef0 msgr2=0x7f773c06be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:47.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:47.113+0000 7f7742c9d640 1 -- 172.21.15.5:0/21731075 shutdown_connections 2024-03-24T10:54:47.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:47.113+0000 7f7742c9d640 1 -- 172.21.15.5:0/21731075 wait complete. 2024-03-24T10:54:47.728 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:47.728 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:48.729 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:48.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.831+0000 7f0c086df640 1 Processor -- start 2024-03-24T10:54:48.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.831+0000 7f0c086df640 1 -- start start 2024-03-24T10:54:48.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.832+0000 7f0c086df640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0c000cb270 0x7f0c000cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:48.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.832+0000 7f0c086df640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c000ccba0 con 0x7f0c000cd960 2024-03-24T10:54:48.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.832+0000 7f0c086df640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c000cce40 con 0x7f0c000cae90 2024-03-24T10:54:48.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.832+0000 7f0c086df640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f0c000cc9b0 con 0x7f0c000cb270 2024-03-24T10:54:48.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.833+0000 7f0c0647b640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f0c000cae90 0x7f0c000cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:44604/0 (socket says 172.21.15.5:44604) 2024-03-24T10:54:48.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.833+0000 7f0c0647b640 1 -- 172.21.15.5:0/1010577996 learned_addr learned my addr 172.21.15.5:0/1010577996 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:54:48.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.833+0000 7f0c05c7a640 1 --2- 172.21.15.5:0/1010577996 >> v2:172.21.15.5:3300/0 conn(0x7f0c000cb270 0x7f0c000cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:48.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.836+0000 7f0c05479640 1 -- 172.21.15.5:0/1010577996 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4065981673 0 0) 0x7f0c000cce40 con 0x7f0c000cae90 2024-03-24T10:54:48.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.836+0000 7f0c05479640 1 -- 172.21.15.5:0/1010577996 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bec003520 con 0x7f0c000cae90 2024-03-24T10:54:48.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.838+0000 7f0c05c7a640 1 -- 172.21.15.5:0/1010577996 >> v1:172.21.15.57:6789/0 conn(0x7f0c000cae90 legacy=0x7f0c000cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:48.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.838+0000 7f0c05c7a640 1 -- 172.21.15.5:0/1010577996 >> v1:172.21.15.5:6789/0 conn(0x7f0c000cd960 legacy=0x7f0c000cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:48.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.838+0000 7f0c05c7a640 1 -- 172.21.15.5:0/1010577996 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0015a590 con 0x7f0c000cb270 2024-03-24T10:54:48.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.842+0000 7f0c05c7a640 1 --2- 172.21.15.5:0/1010577996 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c000cb270 0x7f0c000cf530 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00097a0 tx=0x7f0bf002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=276fbfb79568fd4b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:48.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.847+0000 7f0c05479640 1 -- 172.21.15.5:0/1010577996 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf002f810 con 0x7f0c000cb270 2024-03-24T10:54:48.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.847+0000 7f0c05479640 1 -- 172.21.15.5:0/1010577996 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bf002f970 con 0x7f0c000cb270 2024-03-24T10:54:48.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.847+0000 7f0c05479640 1 -- 172.21.15.5:0/1010577996 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf002fd00 con 0x7f0c000cb270 2024-03-24T10:54:48.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.848+0000 7f0c086df640 1 -- 172.21.15.5:0/1010577996 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c000cb270 msgr2=0x7f0c000cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:48.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.848+0000 7f0c086df640 1 --2- 172.21.15.5:0/1010577996 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c000cb270 0x7f0c000cf530 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00097a0 tx=0x7f0bf002b9f0 comp rx=0 tx=0).stop 2024-03-24T10:54:48.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.848+0000 7f0c086df640 1 -- 172.21.15.5:0/1010577996 shutdown_connections 2024-03-24T10:54:48.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.848+0000 7f0c086df640 1 --2- 172.21.15.5:0/1010577996 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c000cb270 0x7f0c000cf530 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:48.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.848+0000 7f0c086df640 1 -- 172.21.15.5:0/1010577996 >> 172.21.15.5:0/1010577996 conn(0x7f0c000cfb80 msgr2=0x7f0c000cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:48.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.848+0000 7f0c086df640 1 -- 172.21.15.5:0/1010577996 shutdown_connections 2024-03-24T10:54:48.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.848+0000 7f0c086df640 1 -- 172.21.15.5:0/1010577996 wait complete. 2024-03-24T10:54:48.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.849+0000 7f0c086df640 1 Processor -- start 2024-03-24T10:54:48.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.849+0000 7f0c086df640 1 -- start start 2024-03-24T10:54:48.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.850+0000 7f0c086df640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c000cae90 0x7f0c000c48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:48.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.850+0000 7f0c086df640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c000cb270 0x7f0c000c6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:48.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.850+0000 7f0c086df640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c000cd960 0x7f0c001e6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:48.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.850+0000 7f0c086df640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f0c001f3b70 con 0x7f0c000cd960 2024-03-24T10:54:48.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.850+0000 7f0c086df640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0c001f40a0 con 0x7f0c000cae90 2024-03-24T10:54:48.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.850+0000 7f0c086df640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0c000c3e40 con 0x7f0c000cb270 2024-03-24T10:54:48.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.851+0000 7f0c06c7c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c000cd960 0x7f0c001e6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:48.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.851+0000 7f0c0647b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c000cae90 0x7f0c000c48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:48.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.851+0000 7f0c06c7c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c000cd960 0x7f0c001e6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50644/0 (socket says 172.21.15.5:50644) 2024-03-24T10:54:48.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.851+0000 7f0c06c7c640 1 -- 172.21.15.5:0/4255354269 learned_addr learned my addr 172.21.15.5:0/4255354269 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:48.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.851+0000 7f0c05c7a640 1 --2- 172.21.15.5:0/4255354269 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c000cb270 0x7f0c000c6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:48.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.854+0000 7f0c06c7c640 1 -- 172.21.15.5:0/4255354269 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c000cb270 msgr2=0x7f0c000c6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:48.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.854+0000 7f0c06c7c640 1 --2- 172.21.15.5:0/4255354269 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c000cb270 0x7f0c000c6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:48.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.854+0000 7f0c06c7c640 1 -- 172.21.15.5:0/4255354269 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c000cae90 msgr2=0x7f0c000c48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:48.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.854+0000 7f0c06c7c640 1 --2- 172.21.15.5:0/4255354269 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c000cae90 0x7f0c000c48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:48.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.854+0000 7f0c06c7c640 1 -- 172.21.15.5:0/4255354269 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf0009400 con 0x7f0c000cd960 2024-03-24T10:54:48.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.860+0000 7f0c06c7c640 1 --2- 172.21.15.5:0/4255354269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c000cd960 0x7f0c001e6cb0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f0bf800c6b0 tx=0x7f0bf800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:48.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.862+0000 7f0bf6ffd640 1 -- 172.21.15.5:0/4255354269 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8003b70 con 0x7f0c000cd960 2024-03-24T10:54:48.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.862+0000 7f0c086df640 1 -- 172.21.15.5:0/4255354269 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c001f3dc0 con 0x7f0c000cd960 2024-03-24T10:54:48.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.862+0000 7f0c086df640 1 -- 172.21.15.5:0/4255354269 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c000c43c0 con 0x7f0c000cd960 2024-03-24T10:54:48.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.863+0000 7f0c086df640 1 -- 172.21.15.5:0/4255354269 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c0006dad0 con 0x7f0c000cd960 2024-03-24T10:54:48.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.864+0000 7f0bf6ffd640 1 -- 172.21.15.5:0/4255354269 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bf8001790 con 0x7f0c000cd960 2024-03-24T10:54:48.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:48.864+0000 7f0bf6ffd640 1 -- 172.21.15.5:0/4255354269 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf80050d0 con 0x7f0c000cd960 2024-03-24T10:54:49.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:49.132+0000 7f0bf6ffd640 1 -- 172.21.15.5:0/4255354269 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0bf8005230 con 0x7f0c000cd960 2024-03-24T10:54:49.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:49.133+0000 7f0bf6ffd640 1 --2- 172.21.15.5:0/4255354269 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0be406b810 0x7f0be406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:49.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:49.133+0000 7f0c0647b640 1 --2- 172.21.15.5:0/4255354269 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0be406b810 0x7f0be406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:49.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:49.133+0000 7f0c0647b640 1 --2- 172.21.15.5:0/4255354269 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0be406b810 0x7f0be406dcd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc0011e0 tx=0x7f0bfc007d30 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:49.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:49.149+0000 7f0bf6ffd640 1 -- 172.21.15.5:0/4255354269 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f0bf8059190 con 0x7f0c000cd960 2024-03-24T10:54:49.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:49.149+0000 7f0bf6ffd640 1 -- 172.21.15.5:0/4255354269 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0bf8058f60 con 0x7f0c000cd960 2024-03-24T10:54:49.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:49.371+0000 7f0c086df640 1 -- 172.21.15.5:0/4255354269 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c000c8190 con 0x7f0c000cd960 2024-03-24T10:54:49.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:49.392+0000 7f0bf6ffd640 1 -- 172.21.15.5:0/4255354269 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f0bf8058af0 con 0x7f0c000cd960 2024-03-24T10:54:49.392 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:49.392 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:49.392 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:49.394+0000 7f0c086df640 1 -- 172.21.15.5:0/4255354269 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0be406b810 msgr2=0x7f0be406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:49.394+0000 7f0c086df640 1 --2- 172.21.15.5:0/4255354269 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0be406b810 0x7f0be406dcd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc0011e0 tx=0x7f0bfc007d30 comp rx=0 tx=0).stop 2024-03-24T10:54:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:49.394+0000 7f0c086df640 1 -- 172.21.15.5:0/4255354269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c000cd960 msgr2=0x7f0c001e6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:49.394+0000 7f0c086df640 1 --2- 172.21.15.5:0/4255354269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c000cd960 0x7f0c001e6cb0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f0bf800c6b0 tx=0x7f0bf800cb70 comp rx=0 tx=0).stop 2024-03-24T10:54:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:49.394+0000 7f0c086df640 1 -- 172.21.15.5:0/4255354269 shutdown_connections 2024-03-24T10:54:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:49.394+0000 7f0c086df640 1 --2- 172.21.15.5:0/4255354269 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0be406b810 0x7f0be406dcd0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:49.394+0000 7f0c086df640 1 --2- 172.21.15.5:0/4255354269 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c000cd960 0x7f0c001e6cb0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:49.394+0000 7f0c086df640 1 --2- 172.21.15.5:0/4255354269 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c000cb270 0x7f0c000c6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:49.394+0000 7f0c086df640 1 --2- 172.21.15.5:0/4255354269 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c000cae90 0x7f0c000c48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:49.394+0000 7f0c086df640 1 -- 172.21.15.5:0/4255354269 >> 172.21.15.5:0/4255354269 conn(0x7f0c000cfb80 msgr2=0x7f0c000c7960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:49.394+0000 7f0c086df640 1 -- 172.21.15.5:0/4255354269 shutdown_connections 2024-03-24T10:54:49.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:49.394+0000 7f0c086df640 1 -- 172.21.15.5:0/4255354269 wait complete. 2024-03-24T10:54:50.009 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:50.009 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:51.010 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:51.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.113+0000 7fb051f43640 1 Processor -- start 2024-03-24T10:54:51.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.113+0000 7fb051f43640 1 -- start start 2024-03-24T10:54:51.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.113+0000 7fb051f43640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb04c06bf50 0x7fb04c06c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:51.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.113+0000 7fb051f43640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb04c0cb2f0 con 0x7fb04c1543a0 2024-03-24T10:54:51.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.113+0000 7fb051f43640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb04c15b070 con 0x7fb04c15ac00 2024-03-24T10:54:51.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.113+0000 7fb051f43640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fb04c0c6750 con 0x7fb04c06bf50 2024-03-24T10:54:51.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.114+0000 7fb04affd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fb04c15ac00 0x7fb04c0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:44610/0 (socket says 172.21.15.5:44610) 2024-03-24T10:54:51.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.114+0000 7fb04affd640 1 -- 172.21.15.5:0/2824424517 learned_addr learned my addr 172.21.15.5:0/2824424517 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:54:51.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.114+0000 7fb04bfff640 1 --2- 172.21.15.5:0/2824424517 >> v2:172.21.15.5:3300/0 conn(0x7fb04c06bf50 0x7fb04c06c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:51.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.118+0000 7fb04a7fc640 1 -- 172.21.15.5:0/2824424517 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3263595711 0 0) 0x7fb04c15b070 con 0x7fb04c15ac00 2024-03-24T10:54:51.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.118+0000 7fb04a7fc640 1 -- 172.21.15.5:0/2824424517 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb034003520 con 0x7fb04c15ac00 2024-03-24T10:54:51.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.118+0000 7fb04bfff640 1 -- 172.21.15.5:0/2824424517 >> v1:172.21.15.57:6789/0 conn(0x7fb04c15ac00 legacy=0x7fb04c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:51.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.118+0000 7fb04bfff640 1 -- 172.21.15.5:0/2824424517 >> v1:172.21.15.5:6789/0 conn(0x7fb04c1543a0 legacy=0x7fb04c06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:51.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.118+0000 7fb04bfff640 1 -- 172.21.15.5:0/2824424517 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb04c0ca7e0 con 0x7fb04c06bf50 2024-03-24T10:54:51.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.123+0000 7fb04bfff640 1 --2- 172.21.15.5:0/2824424517 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb04c06bf50 0x7fb04c06c330 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb044009d90 tx=0x7fb04402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dd38e5f4104217b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:51.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.127+0000 7fb04a7fc640 1 -- 172.21.15.5:0/2824424517 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04402cd00 con 0x7fb04c06bf50 2024-03-24T10:54:51.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.128+0000 7fb04a7fc640 1 -- 172.21.15.5:0/2824424517 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb04402ce60 con 0x7fb04c06bf50 2024-03-24T10:54:51.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.128+0000 7fb04a7fc640 1 -- 172.21.15.5:0/2824424517 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb044034660 con 0x7fb04c06bf50 2024-03-24T10:54:51.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.128+0000 7fb051f43640 1 -- 172.21.15.5:0/2824424517 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb04c06bf50 msgr2=0x7fb04c06c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:51.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.128+0000 7fb051f43640 1 --2- 172.21.15.5:0/2824424517 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb04c06bf50 0x7fb04c06c330 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb044009d90 tx=0x7fb04402bf50 comp rx=0 tx=0).stop 2024-03-24T10:54:51.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.128+0000 7fb051f43640 1 -- 172.21.15.5:0/2824424517 shutdown_connections 2024-03-24T10:54:51.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.128+0000 7fb051f43640 1 --2- 172.21.15.5:0/2824424517 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb04c06bf50 0x7fb04c06c330 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:51.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.128+0000 7fb051f43640 1 -- 172.21.15.5:0/2824424517 >> 172.21.15.5:0/2824424517 conn(0x7fb04c072ef0 msgr2=0x7fb04c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:51.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.128+0000 7fb051f43640 1 -- 172.21.15.5:0/2824424517 shutdown_connections 2024-03-24T10:54:51.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.128+0000 7fb051f43640 1 -- 172.21.15.5:0/2824424517 wait complete. 2024-03-24T10:54:51.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.130+0000 7fb051f43640 1 Processor -- start 2024-03-24T10:54:51.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.130+0000 7fb051f43640 1 -- start start 2024-03-24T10:54:51.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.131+0000 7fb051f43640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb04c06bf50 0x7fb04c148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:51.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.131+0000 7fb051f43640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb04c1543a0 0x7fb04c1472f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:51.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.131+0000 7fb051f43640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb04c15ac00 0x7fb04c0b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:51.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.131+0000 7fb051f43640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fb04c148170 con 0x7fb04c1543a0 2024-03-24T10:54:51.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.131+0000 7fb051f43640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb04c0b3530 con 0x7fb04c06bf50 2024-03-24T10:54:51.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.131+0000 7fb051f43640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fb04c1579e0 con 0x7fb04c15ac00 2024-03-24T10:54:51.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.131+0000 7fb04affd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb04c1543a0 0x7fb04c1472f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:51.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.132+0000 7fb04b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb04c06bf50 0x7fb04c148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:51.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.132+0000 7fb04bfff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb04c15ac00 0x7fb04c0b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:51.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.132+0000 7fb04affd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb04c1543a0 0x7fb04c1472f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50654/0 (socket says 172.21.15.5:50654) 2024-03-24T10:54:51.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.132+0000 7fb04affd640 1 -- 172.21.15.5:0/1968212014 learned_addr learned my addr 172.21.15.5:0/1968212014 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:51.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.135+0000 7fb04affd640 1 -- 172.21.15.5:0/1968212014 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb04c15ac00 msgr2=0x7fb04c0b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:51.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.135+0000 7fb04affd640 1 --2- 172.21.15.5:0/1968212014 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb04c15ac00 0x7fb04c0b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:51.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.135+0000 7fb04affd640 1 -- 172.21.15.5:0/1968212014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb04c06bf50 msgr2=0x7fb04c148340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:51.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.135+0000 7fb04affd640 1 --2- 172.21.15.5:0/1968212014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb04c06bf50 0x7fb04c148340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:51.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.135+0000 7fb04affd640 1 -- 172.21.15.5:0/1968212014 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb044009a70 con 0x7fb04c1543a0 2024-03-24T10:54:51.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.138+0000 7fb04affd640 1 --2- 172.21.15.5:0/1968212014 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb04c1543a0 0x7fb04c1472f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb04000c410 tx=0x7fb04000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:51.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.143+0000 7fb033fff640 1 -- 172.21.15.5:0/1968212014 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb040005ce0 con 0x7fb04c1543a0 2024-03-24T10:54:51.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.143+0000 7fb033fff640 1 -- 172.21.15.5:0/1968212014 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb040005e40 con 0x7fb04c1543a0 2024-03-24T10:54:51.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.143+0000 7fb033fff640 1 -- 172.21.15.5:0/1968212014 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0400072f0 con 0x7fb04c1543a0 2024-03-24T10:54:51.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.143+0000 7fb051f43640 1 -- 172.21.15.5:0/1968212014 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb04c1f02e0 con 0x7fb04c1543a0 2024-03-24T10:54:51.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.143+0000 7fb051f43640 1 -- 172.21.15.5:0/1968212014 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb04c147e90 con 0x7fb04c1543a0 2024-03-24T10:54:51.144 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.144+0000 7fb051f43640 1 -- 172.21.15.5:0/1968212014 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb04c0681d0 con 0x7fb04c1543a0 2024-03-24T10:54:51.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.437+0000 7fb033fff640 1 -- 172.21.15.5:0/1968212014 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb040007510 con 0x7fb04c1543a0 2024-03-24T10:54:51.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.437+0000 7fb033fff640 1 --2- 172.21.15.5:0/1968212014 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb03806b860 0x7fb03806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:51.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.438+0000 7fb04b7fe640 1 --2- 172.21.15.5:0/1968212014 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb03806b860 0x7fb03806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:51.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.438+0000 7fb04b7fe640 1 --2- 172.21.15.5:0/1968212014 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb03806b860 0x7fb03806dd20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb03c001780 tx=0x7fb03c009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:51.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.452+0000 7fb033fff640 1 -- 172.21.15.5:0/1968212014 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fb0400584e0 con 0x7fb04c1543a0 2024-03-24T10:54:51.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.452+0000 7fb033fff640 1 -- 172.21.15.5:0/1968212014 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb040054780 con 0x7fb04c1543a0 2024-03-24T10:54:51.689 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.689+0000 7fb051f43640 1 -- 172.21.15.5:0/1968212014 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb04c0b3670 con 0x7fb04c1543a0 2024-03-24T10:54:51.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.715+0000 7fb033fff640 1 -- 172.21.15.5:0/1968212014 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fb040017090 con 0x7fb04c1543a0 2024-03-24T10:54:51.715 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:51.716 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:51.716 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:51.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.717+0000 7fb051f43640 1 -- 172.21.15.5:0/1968212014 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb03806b860 msgr2=0x7fb03806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:51.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.717+0000 7fb051f43640 1 --2- 172.21.15.5:0/1968212014 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb03806b860 0x7fb03806dd20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb03c001780 tx=0x7fb03c009040 comp rx=0 tx=0).stop 2024-03-24T10:54:51.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.717+0000 7fb051f43640 1 -- 172.21.15.5:0/1968212014 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb04c1543a0 msgr2=0x7fb04c1472f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:51.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.717+0000 7fb051f43640 1 --2- 172.21.15.5:0/1968212014 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb04c1543a0 0x7fb04c1472f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb04000c410 tx=0x7fb04000c8d0 comp rx=0 tx=0).stop 2024-03-24T10:54:51.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.718+0000 7fb051f43640 1 -- 172.21.15.5:0/1968212014 shutdown_connections 2024-03-24T10:54:51.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.718+0000 7fb051f43640 1 --2- 172.21.15.5:0/1968212014 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb03806b860 0x7fb03806dd20 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:51.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.718+0000 7fb051f43640 1 --2- 172.21.15.5:0/1968212014 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb04c1543a0 0x7fb04c1472f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:51.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.718+0000 7fb051f43640 1 --2- 172.21.15.5:0/1968212014 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb04c15ac00 0x7fb04c0b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:51.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.718+0000 7fb051f43640 1 --2- 172.21.15.5:0/1968212014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb04c06bf50 0x7fb04c148340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:51.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.718+0000 7fb051f43640 1 -- 172.21.15.5:0/1968212014 >> 172.21.15.5:0/1968212014 conn(0x7fb04c072ef0 msgr2=0x7fb04c150580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:51.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.718+0000 7fb051f43640 1 -- 172.21.15.5:0/1968212014 shutdown_connections 2024-03-24T10:54:51.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:51.718+0000 7fb051f43640 1 -- 172.21.15.5:0/1968212014 wait complete. 2024-03-24T10:54:52.333 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:52.333 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:53.334 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:53.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.436+0000 7f1cb2563640 1 Processor -- start 2024-03-24T10:54:53.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.437+0000 7f1cb2563640 1 -- start start 2024-03-24T10:54:53.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.437+0000 7f1cb2563640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1cac0cace0 0x7f1cac1432d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:53.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.437+0000 7f1cb2563640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1cac15a310 0x7f1cac143810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:53.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.437+0000 7f1cb2563640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cac0cb0c0 con 0x7f1cac06c240 2024-03-24T10:54:53.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.437+0000 7f1cb2563640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f1cac06f110 con 0x7f1cac15a310 2024-03-24T10:54:53.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.437+0000 7f1cb2563640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1cac15b320 con 0x7f1cac0cace0 2024-03-24T10:54:53.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.438+0000 7f1cab7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1cac15a310 0x7f1cac143810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:53.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.438+0000 7f1cb0b00640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f1cac06c240 0x7f1cac06f530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:32972/0 (socket says 172.21.15.5:32972) 2024-03-24T10:54:53.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.438+0000 7f1cb0b00640 1 -- 172.21.15.5:0/1616750614 learned_addr learned my addr 172.21.15.5:0/1616750614 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:54:53.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.439+0000 7f1cabfff640 1 --2- 172.21.15.5:0/1616750614 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1cac0cace0 0x7f1cac1432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:53.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.441+0000 7f1caaffd640 1 -- 172.21.15.5:0/1616750614 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3893799901 0 0) 0x7f1cac0cb0c0 con 0x7f1cac06c240 2024-03-24T10:54:53.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.441+0000 7f1caaffd640 1 -- 172.21.15.5:0/1616750614 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c94003520 con 0x7f1cac06c240 2024-03-24T10:54:53.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.442+0000 7f1cab7fe640 1 -- 172.21.15.5:0/1616750614 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1cac0cace0 msgr2=0x7f1cac1432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:54:53.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.442+0000 7f1cab7fe640 1 --2- 172.21.15.5:0/1616750614 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1cac0cace0 0x7f1cac1432d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:53.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.442+0000 7f1cab7fe640 1 -- 172.21.15.5:0/1616750614 >> v1:172.21.15.5:6789/0 conn(0x7f1cac06c240 legacy=0x7f1cac06f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:53.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.442+0000 7f1cab7fe640 1 -- 172.21.15.5:0/1616750614 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cac06cf70 con 0x7f1cac15a310 2024-03-24T10:54:53.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.444+0000 7f1cab7fe640 1 --2- 172.21.15.5:0/1616750614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cac15a310 0x7f1cac143810 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1ca00097a0 tx=0x7f1ca002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fd9e32214376655 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:53.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.449+0000 7f1caaffd640 1 -- 172.21.15.5:0/1616750614 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca0030610 con 0x7f1cac15a310 2024-03-24T10:54:53.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.450+0000 7f1caaffd640 1 -- 172.21.15.5:0/1616750614 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ca0030770 con 0x7f1cac15a310 2024-03-24T10:54:53.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.450+0000 7f1caaffd640 1 -- 172.21.15.5:0/1616750614 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ca0030b00 con 0x7f1cac15a310 2024-03-24T10:54:53.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.450+0000 7f1cb2563640 1 -- 172.21.15.5:0/1616750614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cac15a310 msgr2=0x7f1cac143810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:53.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.450+0000 7f1cb2563640 1 --2- 172.21.15.5:0/1616750614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cac15a310 0x7f1cac143810 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1ca00097a0 tx=0x7f1ca002b960 comp rx=0 tx=0).stop 2024-03-24T10:54:53.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.450+0000 7f1cb2563640 1 -- 172.21.15.5:0/1616750614 shutdown_connections 2024-03-24T10:54:53.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.450+0000 7f1cb2563640 1 --2- 172.21.15.5:0/1616750614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cac15a310 0x7f1cac143810 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:53.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.450+0000 7f1cb2563640 1 --2- 172.21.15.5:0/1616750614 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1cac0cace0 0x7f1cac1432d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:53.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.450+0000 7f1cb2563640 1 -- 172.21.15.5:0/1616750614 >> 172.21.15.5:0/1616750614 conn(0x7f1cac071240 msgr2=0x7f1cac071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:53.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.450+0000 7f1cb2563640 1 -- 172.21.15.5:0/1616750614 shutdown_connections 2024-03-24T10:54:53.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.450+0000 7f1cb2563640 1 -- 172.21.15.5:0/1616750614 wait complete. 2024-03-24T10:54:53.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.452+0000 7f1cb2563640 1 Processor -- start 2024-03-24T10:54:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.452+0000 7f1cb2563640 1 -- start start 2024-03-24T10:54:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.452+0000 7f1cb2563640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cac06c240 0x7f1cac0cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.452+0000 7f1cb2563640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cac0cace0 0x7f1cac13f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.452+0000 7f1cb2563640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1cac0ce1b0 0x7f1cac13f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.452+0000 7f1cb2563640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f1cac1efc90 con 0x7f1cac0cace0 2024-03-24T10:54:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.452+0000 7f1cb2563640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1cac1efe80 con 0x7f1cac06c240 2024-03-24T10:54:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.452+0000 7f1cb2563640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1cac1f0160 con 0x7f1cac0ce1b0 2024-03-24T10:54:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.453+0000 7f1cab7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cac0cace0 0x7f1cac13f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.453+0000 7f1cabfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cac06c240 0x7f1cac0cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.453+0000 7f1cb0b00640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1cac0ce1b0 0x7f1cac13f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.454+0000 7f1cab7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cac0cace0 0x7f1cac13f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36212/0 (socket says 172.21.15.5:36212) 2024-03-24T10:54:53.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.454+0000 7f1cab7fe640 1 -- 172.21.15.5:0/716554803 learned_addr learned my addr 172.21.15.5:0/716554803 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:53.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.457+0000 7f1cabfff640 1 -- 172.21.15.5:0/716554803 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1cac0ce1b0 msgr2=0x7f1cac13f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:53.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.457+0000 7f1cabfff640 1 --2- 172.21.15.5:0/716554803 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1cac0ce1b0 0x7f1cac13f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:53.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.457+0000 7f1cabfff640 1 -- 172.21.15.5:0/716554803 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cac0cace0 msgr2=0x7f1cac13f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:53.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.457+0000 7f1cabfff640 1 --2- 172.21.15.5:0/716554803 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cac0cace0 0x7f1cac13f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:53.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.457+0000 7f1cabfff640 1 -- 172.21.15.5:0/716554803 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca0009400 con 0x7f1cac06c240 2024-03-24T10:54:53.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.460+0000 7f1cabfff640 1 --2- 172.21.15.5:0/716554803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cac06c240 0x7f1cac0cdc70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00e880 tx=0x7f1c9c00ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:53.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.464+0000 7f1ca8ff9640 1 -- 172.21.15.5:0/716554803 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c00c940 con 0x7f1cac06c240 2024-03-24T10:54:53.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.464+0000 7f1ca8ff9640 1 -- 172.21.15.5:0/716554803 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c9c00caa0 con 0x7f1cac06c240 2024-03-24T10:54:53.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.464+0000 7f1ca8ff9640 1 -- 172.21.15.5:0/716554803 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c00ce50 con 0x7f1cac06c240 2024-03-24T10:54:53.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.464+0000 7f1cb2563640 1 -- 172.21.15.5:0/716554803 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cac1f0440 con 0x7f1cac06c240 2024-03-24T10:54:53.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.464+0000 7f1cb2563640 1 -- 172.21.15.5:0/716554803 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cac1f0a00 con 0x7f1cac06c240 2024-03-24T10:54:53.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.465+0000 7f1cb2563640 1 -- 172.21.15.5:0/716554803 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cac1f0bf0 con 0x7f1cac06c240 2024-03-24T10:54:53.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.485+0000 7f1ca8ff9640 1 -- 172.21.15.5:0/716554803 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1c9c013470 con 0x7f1cac06c240 2024-03-24T10:54:53.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.486+0000 7f1ca8ff9640 1 --2- 172.21.15.5:0/716554803 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1c9806b840 0x7f1c9806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:53.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.486+0000 7f1cab7fe640 1 --2- 172.21.15.5:0/716554803 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1c9806b840 0x7f1c9806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:53.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.486+0000 7f1cab7fe640 1 --2- 172.21.15.5:0/716554803 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1c9806b840 0x7f1c9806dd00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1ca002b9a0 tx=0x7f1ca0008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:53.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.740+0000 7f1ca8ff9640 1 -- 172.21.15.5:0/716554803 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f1c9c059320 con 0x7f1cac06c240 2024-03-24T10:54:53.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.754+0000 7f1ca8ff9640 1 -- 172.21.15.5:0/716554803 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1c9c0555c0 con 0x7f1cac06c240 2024-03-24T10:54:53.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:53.978+0000 7f1cb2563640 1 -- 172.21.15.5:0/716554803 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1cac0681d0 con 0x7f1cac06c240 2024-03-24T10:54:54.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:54.000+0000 7f1ca8ff9640 1 -- 172.21.15.5:0/716554803 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f1c9c017090 con 0x7f1cac06c240 2024-03-24T10:54:54.000 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:54.001 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:54.001 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:54.002+0000 7f1cb2563640 1 -- 172.21.15.5:0/716554803 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1c9806b840 msgr2=0x7f1c9806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:54.002+0000 7f1cb2563640 1 --2- 172.21.15.5:0/716554803 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1c9806b840 0x7f1c9806dd00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1ca002b9a0 tx=0x7f1ca0008030 comp rx=0 tx=0).stop 2024-03-24T10:54:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:54.002+0000 7f1cb2563640 1 -- 172.21.15.5:0/716554803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cac06c240 msgr2=0x7f1cac0cdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:54.002+0000 7f1cb2563640 1 --2- 172.21.15.5:0/716554803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cac06c240 0x7f1cac0cdc70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00e880 tx=0x7f1c9c00ed40 comp rx=0 tx=0).stop 2024-03-24T10:54:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:54.002+0000 7f1cb2563640 1 -- 172.21.15.5:0/716554803 shutdown_connections 2024-03-24T10:54:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:54.002+0000 7f1cb2563640 1 --2- 172.21.15.5:0/716554803 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1c9806b840 0x7f1c9806dd00 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:54.002+0000 7f1cb2563640 1 --2- 172.21.15.5:0/716554803 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1cac0cace0 0x7f1cac13f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:54.002+0000 7f1cb2563640 1 --2- 172.21.15.5:0/716554803 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1cac0ce1b0 0x7f1cac13f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:54.002+0000 7f1cb2563640 1 --2- 172.21.15.5:0/716554803 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1cac06c240 0x7f1cac0cdc70 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:54.002+0000 7f1cb2563640 1 -- 172.21.15.5:0/716554803 >> 172.21.15.5:0/716554803 conn(0x7f1cac071240 msgr2=0x7f1cac0b2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:54.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:54.002+0000 7f1cb2563640 1 -- 172.21.15.5:0/716554803 shutdown_connections 2024-03-24T10:54:54.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:54.002+0000 7f1cb2563640 1 -- 172.21.15.5:0/716554803 wait complete. 2024-03-24T10:54:54.619 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:54.619 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:55.620 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:55.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.724+0000 7f8bf1e34640 1 Processor -- start 2024-03-24T10:54:55.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.725+0000 7f8bf1e34640 1 -- start start 2024-03-24T10:54:55.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.726+0000 7f8bf1e34640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8bec0cb270 0x7f8bec0cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:55.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.726+0000 7f8bf1e34640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8bec0cd960 0x7f8bec0cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:55.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.726+0000 7f8bf1e34640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bec0ccba0 con 0x7f8bec0cae90 2024-03-24T10:54:55.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.726+0000 7f8bf1e34640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f8bec0cc9b0 con 0x7f8bec0cb270 2024-03-24T10:54:55.726 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.726+0000 7f8bf1e34640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f8bec0cce40 con 0x7f8bec0cd960 2024-03-24T10:54:55.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.726+0000 7f8beaffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8bec0cb270 0x7f8bec0cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:55.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.727+0000 7f8bebfff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8bec0cd960 0x7f8bec0cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:55.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.727+0000 7f8beb7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f8bec0cae90 0x7f8bec0cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:46556/0 (socket says 172.21.15.5:46556) 2024-03-24T10:54:55.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.727+0000 7f8beb7fe640 1 -- 172.21.15.5:0/3103570350 learned_addr learned my addr 172.21.15.5:0/3103570350 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:54:55.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.729+0000 7f8beaffd640 1 -- 172.21.15.5:0/3103570350 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8bec0cd960 msgr2=0x7f8bec0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:54:55.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.729+0000 7f8beaffd640 1 --2- 172.21.15.5:0/3103570350 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8bec0cd960 0x7f8bec0cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.729+0000 7f8beaffd640 1 -- 172.21.15.5:0/3103570350 >> v1:172.21.15.57:6789/0 conn(0x7f8bec0cae90 legacy=0x7f8bec0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:55.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.729+0000 7f8beaffd640 1 -- 172.21.15.5:0/3103570350 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bec15a590 con 0x7f8bec0cb270 2024-03-24T10:54:55.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.732+0000 7f8beaffd640 1 --2- 172.21.15.5:0/3103570350 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bec0cb270 0x7f8bec0cf530 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f8be00097a0 tx=0x7f8be002b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d4a43a7f818f372 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:55.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.734+0000 7f8bea7fc640 1 -- 172.21.15.5:0/3103570350 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be0030800 con 0x7f8bec0cb270 2024-03-24T10:54:55.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.736+0000 7f8bea7fc640 1 -- 172.21.15.5:0/3103570350 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8be0030960 con 0x7f8bec0cb270 2024-03-24T10:54:55.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.736+0000 7f8bea7fc640 1 -- 172.21.15.5:0/3103570350 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be0030cf0 con 0x7f8bec0cb270 2024-03-24T10:54:55.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.737+0000 7f8bf1e34640 1 -- 172.21.15.5:0/3103570350 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bec0cb270 msgr2=0x7f8bec0cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:55.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.737+0000 7f8bf1e34640 1 --2- 172.21.15.5:0/3103570350 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bec0cb270 0x7f8bec0cf530 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f8be00097a0 tx=0x7f8be002b940 comp rx=0 tx=0).stop 2024-03-24T10:54:55.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.737+0000 7f8bf1e34640 1 -- 172.21.15.5:0/3103570350 shutdown_connections 2024-03-24T10:54:55.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.737+0000 7f8bf1e34640 1 --2- 172.21.15.5:0/3103570350 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8bec0cd960 0x7f8bec0cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:55.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.737+0000 7f8bf1e34640 1 --2- 172.21.15.5:0/3103570350 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bec0cb270 0x7f8bec0cf530 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:55.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.737+0000 7f8bf1e34640 1 -- 172.21.15.5:0/3103570350 >> 172.21.15.5:0/3103570350 conn(0x7f8bec0cfb80 msgr2=0x7f8bec0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:55.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.737+0000 7f8bf1e34640 1 -- 172.21.15.5:0/3103570350 shutdown_connections 2024-03-24T10:54:55.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.737+0000 7f8bf1e34640 1 -- 172.21.15.5:0/3103570350 wait complete. 2024-03-24T10:54:55.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.739+0000 7f8bf1e34640 1 Processor -- start 2024-03-24T10:54:55.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.739+0000 7f8bf1e34640 1 -- start start 2024-03-24T10:54:55.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.740+0000 7f8bf1e34640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8bec0cae90 0x7f8bec15c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:55.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.740+0000 7f8bf1e34640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bec0cb270 0x7f8bec0b5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:55.741 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.740+0000 7f8bf1e34640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8bec0cd960 0x7f8bec0abf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:55.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.740+0000 7f8bf1e34640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f8bec1599c0 con 0x7f8bec0cb270 2024-03-24T10:54:55.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.740+0000 7f8bf1e34640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8bec15e210 con 0x7f8bec0cae90 2024-03-24T10:54:55.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.740+0000 7f8bf1e34640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f8bec15e490 con 0x7f8bec0cd960 2024-03-24T10:54:55.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.741+0000 7f8beaffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bec0cb270 0x7f8bec0b5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:55.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.741+0000 7f8bebfff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8bec0cd960 0x7f8bec0abf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:55.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.741+0000 7f8beaffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bec0cb270 0x7f8bec0b5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36224/0 (socket says 172.21.15.5:36224) 2024-03-24T10:54:55.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.741+0000 7f8beaffd640 1 -- 172.21.15.5:0/4150274599 learned_addr learned my addr 172.21.15.5:0/4150274599 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:55.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.745+0000 7f8beaffd640 1 -- 172.21.15.5:0/4150274599 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8bec0cd960 msgr2=0x7f8bec0abf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:55.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.745+0000 7f8beaffd640 1 --2- 172.21.15.5:0/4150274599 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8bec0cd960 0x7f8bec0abf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:55.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.745+0000 7f8beaffd640 1 -- 172.21.15.5:0/4150274599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8bec0cae90 msgr2=0x7f8bec15c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:55.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.745+0000 7f8beaffd640 1 --2- 172.21.15.5:0/4150274599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8bec0cae90 0x7f8bec15c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:55.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.745+0000 7f8beaffd640 1 -- 172.21.15.5:0/4150274599 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8be0009400 con 0x7f8bec0cb270 2024-03-24T10:54:55.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.747+0000 7f8beaffd640 1 --2- 172.21.15.5:0/4150274599 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bec0cb270 0x7f8bec0b5200 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8be002bf50 tx=0x7f8be0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:55.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.751+0000 7f8bd3fff640 1 -- 172.21.15.5:0/4150274599 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be003e040 con 0x7f8bec0cb270 2024-03-24T10:54:55.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.751+0000 7f8bd3fff640 1 -- 172.21.15.5:0/4150274599 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8be0039cb0 con 0x7f8bec0cb270 2024-03-24T10:54:55.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.751+0000 7f8bf1e34640 1 -- 172.21.15.5:0/4150274599 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bec159400 con 0x7f8bec0cb270 2024-03-24T10:54:55.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.751+0000 7f8bd3fff640 1 -- 172.21.15.5:0/4150274599 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be0044630 con 0x7f8bec0cb270 2024-03-24T10:54:55.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.751+0000 7f8bf1e34640 1 -- 172.21.15.5:0/4150274599 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bec0b4a00 con 0x7f8bec0cb270 2024-03-24T10:54:55.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.752+0000 7f8bf1e34640 1 -- 172.21.15.5:0/4150274599 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bec0b4860 con 0x7f8bec0cb270 2024-03-24T10:54:55.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.772+0000 7f8bd3fff640 1 -- 172.21.15.5:0/4150274599 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8be004d3b0 con 0x7f8bec0cb270 2024-03-24T10:54:55.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.772+0000 7f8bd3fff640 1 --2- 172.21.15.5:0/4150274599 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8bd806b860 0x7f8bd806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:55.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.773+0000 7f8beb7fe640 1 --2- 172.21.15.5:0/4150274599 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8bd806b860 0x7f8bd806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:55.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:55.773+0000 7f8beb7fe640 1 --2- 172.21.15.5:0/4150274599 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8bd806b860 0x7f8bd806dd20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc005880 tx=0x7f8bdc007a00 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:56.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:56.032+0000 7f8bd3fff640 1 -- 172.21.15.5:0/4150274599 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f8be0039e20 con 0x7f8bec0cb270 2024-03-24T10:54:56.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:56.033+0000 7f8bd3fff640 1 -- 172.21.15.5:0/4150274599 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8be0077560 con 0x7f8bec0cb270 2024-03-24T10:54:56.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:56.276+0000 7f8bf1e34640 1 -- 172.21.15.5:0/4150274599 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8bec0b4860 con 0x7f8bec0cb270 2024-03-24T10:54:56.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:56.300+0000 7f8bd3fff640 1 -- 172.21.15.5:0/4150274599 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f8bec0b4860 con 0x7f8bec0cb270 2024-03-24T10:54:56.301 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:56.302 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:56.302 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:56.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:56.303+0000 7f8bf1e34640 1 -- 172.21.15.5:0/4150274599 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8bd806b860 msgr2=0x7f8bd806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:56.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:56.303+0000 7f8bf1e34640 1 --2- 172.21.15.5:0/4150274599 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8bd806b860 0x7f8bd806dd20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc005880 tx=0x7f8bdc007a00 comp rx=0 tx=0).stop 2024-03-24T10:54:56.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:56.303+0000 7f8bf1e34640 1 -- 172.21.15.5:0/4150274599 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bec0cb270 msgr2=0x7f8bec0b5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:56.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:56.303+0000 7f8bf1e34640 1 --2- 172.21.15.5:0/4150274599 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bec0cb270 0x7f8bec0b5200 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8be002bf50 tx=0x7f8be0004730 comp rx=0 tx=0).stop 2024-03-24T10:54:56.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:56.303+0000 7f8bf1e34640 1 -- 172.21.15.5:0/4150274599 shutdown_connections 2024-03-24T10:54:56.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:56.303+0000 7f8bf1e34640 1 --2- 172.21.15.5:0/4150274599 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8bd806b860 0x7f8bd806dd20 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:56.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:56.303+0000 7f8bf1e34640 1 --2- 172.21.15.5:0/4150274599 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8bec0cb270 0x7f8bec0b5200 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:56.304 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:56.303+0000 7f8bf1e34640 1 --2- 172.21.15.5:0/4150274599 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8bec0cd960 0x7f8bec0abf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:56.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:56.303+0000 7f8bf1e34640 1 --2- 172.21.15.5:0/4150274599 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8bec0cae90 0x7f8bec15c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:56.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:56.303+0000 7f8bf1e34640 1 -- 172.21.15.5:0/4150274599 >> 172.21.15.5:0/4150274599 conn(0x7f8bec0cfb80 msgr2=0x7f8bec159e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:56.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:56.303+0000 7f8bf1e34640 1 -- 172.21.15.5:0/4150274599 shutdown_connections 2024-03-24T10:54:56.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:56.303+0000 7f8bf1e34640 1 -- 172.21.15.5:0/4150274599 wait complete. 2024-03-24T10:54:56.918 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:56.918 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:54:57.919 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:54:58.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.019+0000 7fc52c93a640 1 Processor -- start 2024-03-24T10:54:58.021 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.019+0000 7fc52c93a640 1 -- start start 2024-03-24T10:54:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.021+0000 7fc52c93a640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc5281543a0 0x7fc52815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.021+0000 7fc52c93a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc52806d4f0 0x7fc5280cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.021+0000 7fc52c93a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5280ccaa0 0x7fc52806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.021+0000 7fc52c93a640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fc5280c6750 con 0x7fc52806d4f0 2024-03-24T10:54:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.021+0000 7fc52c93a640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fc5280cceb0 con 0x7fc5281543a0 2024-03-24T10:54:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.021+0000 7fc52c93a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc5280cc380 con 0x7fc5280ccaa0 2024-03-24T10:54:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.021+0000 7fc525d9b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc52806d4f0 0x7fc5280cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.022+0000 7fc526d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5280ccaa0 0x7fc52806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.022+0000 7fc52659c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc5281543a0 0x7fc52815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.022+0000 7fc525d9b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc52806d4f0 0x7fc5280cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36240/0 (socket says 172.21.15.5:36240) 2024-03-24T10:54:58.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.022+0000 7fc525d9b640 1 -- 172.21.15.5:0/1975070451 learned_addr learned my addr 172.21.15.5:0/1975070451 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:58.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.024+0000 7fc525d9b640 1 -- 172.21.15.5:0/1975070451 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5280ccaa0 msgr2=0x7fc52806bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:54:58.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.024+0000 7fc525d9b640 1 --2- 172.21.15.5:0/1975070451 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5280ccaa0 0x7fc52806bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:58.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.024+0000 7fc525d9b640 1 -- 172.21.15.5:0/1975070451 >> v2:172.21.15.57:3300/0 conn(0x7fc5281543a0 msgr2=0x7fc52815ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:54:58.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.024+0000 7fc525d9b640 1 --2- 172.21.15.5:0/1975070451 >> v2:172.21.15.57:3300/0 conn(0x7fc5281543a0 0x7fc52815ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:58.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.024+0000 7fc525d9b640 1 -- 172.21.15.5:0/1975070451 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5280cb2f0 con 0x7fc52806d4f0 2024-03-24T10:54:58.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.026+0000 7fc525d9b640 1 --2- 172.21.15.5:0/1975070451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc52806d4f0 0x7fc5280cc560 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fc5280cd600 tx=0x7fc51c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54f631048a40d68c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:58.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.031+0000 7fc52559a640 1 -- 172.21.15.5:0/1975070451 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51c030860 con 0x7fc52806d4f0 2024-03-24T10:54:58.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.031+0000 7fc52559a640 1 -- 172.21.15.5:0/1975070451 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc51c0309c0 con 0x7fc52806d4f0 2024-03-24T10:54:58.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.031+0000 7fc52559a640 1 -- 172.21.15.5:0/1975070451 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51c030d50 con 0x7fc52806d4f0 2024-03-24T10:54:58.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.031+0000 7fc52c93a640 1 -- 172.21.15.5:0/1975070451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc52806d4f0 msgr2=0x7fc5280cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:58.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.031+0000 7fc52c93a640 1 --2- 172.21.15.5:0/1975070451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc52806d4f0 0x7fc5280cc560 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fc5280cd600 tx=0x7fc51c02b9a0 comp rx=0 tx=0).stop 2024-03-24T10:54:58.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.031+0000 7fc52c93a640 1 -- 172.21.15.5:0/1975070451 shutdown_connections 2024-03-24T10:54:58.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.031+0000 7fc52c93a640 1 --2- 172.21.15.5:0/1975070451 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5280ccaa0 0x7fc52806bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:58.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.031+0000 7fc52c93a640 1 --2- 172.21.15.5:0/1975070451 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc52806d4f0 0x7fc5280cc560 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:58.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.031+0000 7fc52c93a640 1 --2- 172.21.15.5:0/1975070451 >> v2:172.21.15.57:3300/0 conn(0x7fc5281543a0 0x7fc52815ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:58.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.031+0000 7fc52c93a640 1 -- 172.21.15.5:0/1975070451 >> 172.21.15.5:0/1975070451 conn(0x7fc528072ef0 msgr2=0x7fc528061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:58.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.031+0000 7fc52c93a640 1 -- 172.21.15.5:0/1975070451 shutdown_connections 2024-03-24T10:54:58.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.031+0000 7fc52c93a640 1 -- 172.21.15.5:0/1975070451 wait complete. 2024-03-24T10:54:58.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.033+0000 7fc52c93a640 1 Processor -- start 2024-03-24T10:54:58.034 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.033+0000 7fc52c93a640 1 -- start start 2024-03-24T10:54:58.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.034+0000 7fc52c93a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc52806d4f0 0x7fc5280ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:58.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.034+0000 7fc52c93a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5280ccaa0 0x7fc5281f3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:58.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.034+0000 7fc52c93a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5281543a0 0x7fc5281f37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:58.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.034+0000 7fc52c93a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fc528155d30 con 0x7fc52806d4f0 2024-03-24T10:54:58.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.034+0000 7fc52c93a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc5280cea10 con 0x7fc5280ccaa0 2024-03-24T10:54:58.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.034+0000 7fc52c93a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc528157760 con 0x7fc5281543a0 2024-03-24T10:54:58.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.034+0000 7fc52659c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc52806d4f0 0x7fc5280ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:58.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.034+0000 7fc526d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5281543a0 0x7fc5281f37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:58.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.034+0000 7fc525d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5280ccaa0 0x7fc5281f3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:58.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.034+0000 7fc52659c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc52806d4f0 0x7fc5280ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36250/0 (socket says 172.21.15.5:36250) 2024-03-24T10:54:58.036 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.034+0000 7fc52659c640 1 -- 172.21.15.5:0/4095831309 learned_addr learned my addr 172.21.15.5:0/4095831309 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:54:58.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.038+0000 7fc52659c640 1 -- 172.21.15.5:0/4095831309 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5281543a0 msgr2=0x7fc5281f37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:58.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.038+0000 7fc52659c640 1 --2- 172.21.15.5:0/4095831309 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5281543a0 0x7fc5281f37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:58.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.038+0000 7fc52659c640 1 -- 172.21.15.5:0/4095831309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5280ccaa0 msgr2=0x7fc5281f3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:58.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.038+0000 7fc52659c640 1 --2- 172.21.15.5:0/4095831309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5280ccaa0 0x7fc5281f3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:58.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.038+0000 7fc52659c640 1 -- 172.21.15.5:0/4095831309 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc51c0094d0 con 0x7fc52806d4f0 2024-03-24T10:54:58.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.041+0000 7fc52659c640 1 --2- 172.21.15.5:0/4095831309 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc52806d4f0 0x7fc5280ceea0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc51800e870 tx=0x7fc51800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:58.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.046+0000 7fc516ffd640 1 -- 172.21.15.5:0/4095831309 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5180094f0 con 0x7fc52806d4f0 2024-03-24T10:54:58.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.046+0000 7fc516ffd640 1 -- 172.21.15.5:0/4095831309 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc51800c940 con 0x7fc52806d4f0 2024-03-24T10:54:58.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.046+0000 7fc52c93a640 1 -- 172.21.15.5:0/4095831309 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5281590f0 con 0x7fc52806d4f0 2024-03-24T10:54:58.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.046+0000 7fc516ffd640 1 -- 172.21.15.5:0/4095831309 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc51800ccf0 con 0x7fc52806d4f0 2024-03-24T10:54:58.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.046+0000 7fc52c93a640 1 -- 172.21.15.5:0/4095831309 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc528157480 con 0x7fc52806d4f0 2024-03-24T10:54:58.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.047+0000 7fc52c93a640 1 -- 172.21.15.5:0/4095831309 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5280681d0 con 0x7fc52806d4f0 2024-03-24T10:54:58.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.320+0000 7fc516ffd640 1 -- 172.21.15.5:0/4095831309 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc518013470 con 0x7fc52806d4f0 2024-03-24T10:54:58.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.320+0000 7fc516ffd640 1 --2- 172.21.15.5:0/4095831309 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc50806b890 0x7fc50806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:54:58.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.320+0000 7fc525d9b640 1 --2- 172.21.15.5:0/4095831309 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc50806b890 0x7fc50806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:54:58.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.321+0000 7fc525d9b640 1 --2- 172.21.15.5:0/4095831309 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc50806b890 0x7fc50806dd50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc51c036000 tx=0x7fc51c008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:54:58.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.335+0000 7fc516ffd640 1 -- 172.21.15.5:0/4095831309 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fc518059170 con 0x7fc52806d4f0 2024-03-24T10:54:58.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.335+0000 7fc516ffd640 1 -- 172.21.15.5:0/4095831309 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc518055410 con 0x7fc52806d4f0 2024-03-24T10:54:58.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.580+0000 7fc52c93a640 1 -- 172.21.15.5:0/4095831309 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5280ce760 con 0x7fc52806d4f0 2024-03-24T10:54:58.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.593+0000 7fc516ffd640 1 -- 172.21.15.5:0/4095831309 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fc518021090 con 0x7fc52806d4f0 2024-03-24T10:54:58.594 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:54:58.594 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:54:58.594 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:54:58.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.595+0000 7fc52c93a640 1 -- 172.21.15.5:0/4095831309 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc50806b890 msgr2=0x7fc50806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:58.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.595+0000 7fc52c93a640 1 --2- 172.21.15.5:0/4095831309 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc50806b890 0x7fc50806dd50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc51c036000 tx=0x7fc51c008030 comp rx=0 tx=0).stop 2024-03-24T10:54:58.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.596+0000 7fc52c93a640 1 -- 172.21.15.5:0/4095831309 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc52806d4f0 msgr2=0x7fc5280ceea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:54:58.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.596+0000 7fc52c93a640 1 --2- 172.21.15.5:0/4095831309 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc52806d4f0 0x7fc5280ceea0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fc51800e870 tx=0x7fc51800ed30 comp rx=0 tx=0).stop 2024-03-24T10:54:58.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.596+0000 7fc52c93a640 1 -- 172.21.15.5:0/4095831309 shutdown_connections 2024-03-24T10:54:58.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.596+0000 7fc52c93a640 1 --2- 172.21.15.5:0/4095831309 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc50806b890 0x7fc50806dd50 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:58.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.596+0000 7fc52c93a640 1 --2- 172.21.15.5:0/4095831309 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc5281543a0 0x7fc5281f37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:58.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.596+0000 7fc52c93a640 1 --2- 172.21.15.5:0/4095831309 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5280ccaa0 0x7fc5281f3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:58.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.596+0000 7fc52c93a640 1 --2- 172.21.15.5:0/4095831309 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc52806d4f0 0x7fc5280ceea0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:54:58.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.596+0000 7fc52c93a640 1 -- 172.21.15.5:0/4095831309 >> 172.21.15.5:0/4095831309 conn(0x7fc528072ef0 msgr2=0x7fc5281525d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:54:58.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.596+0000 7fc52c93a640 1 -- 172.21.15.5:0/4095831309 shutdown_connections 2024-03-24T10:54:58.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:54:58.596+0000 7fc52c93a640 1 -- 172.21.15.5:0/4095831309 wait complete. 2024-03-24T10:54:59.215 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:54:59.215 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:00.216 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:00.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.319+0000 7faacd901640 1 Processor -- start 2024-03-24T10:55:00.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.319+0000 7faacd901640 1 -- start start 2024-03-24T10:55:00.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.320+0000 7faacd901640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faac815ac00 0x7faac80cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:00.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.320+0000 7faacd901640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faac80ccaa0 0x7faac806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:00.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.320+0000 7faacd901640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faac80cb2f0 con 0x7faac81543a0 2024-03-24T10:55:00.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.320+0000 7faacd901640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7faac80c6750 con 0x7faac80ccaa0 2024-03-24T10:55:00.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.320+0000 7faacd901640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7faac80cc380 con 0x7faac815ac00 2024-03-24T10:55:00.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.321+0000 7faac77fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faac80ccaa0 0x7faac806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:00.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.321+0000 7faabffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faac815ac00 0x7faac80cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:00.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.321+0000 7faac77fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faac80ccaa0 0x7faac806bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36258/0 (socket says 172.21.15.5:36258) 2024-03-24T10:55:00.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.321+0000 7faac77fe640 1 -- 172.21.15.5:0/1239579782 learned_addr learned my addr 172.21.15.5:0/1239579782 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:00.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.324+0000 7faac77fe640 1 -- 172.21.15.5:0/1239579782 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faac815ac00 msgr2=0x7faac80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:00.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.324+0000 7faac77fe640 1 --2- 172.21.15.5:0/1239579782 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faac815ac00 0x7faac80cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:00.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.324+0000 7faac77fe640 1 -- 172.21.15.5:0/1239579782 >> v1:172.21.15.5:6789/0 conn(0x7faac81543a0 legacy=0x7faac806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:00.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.324+0000 7faac77fe640 1 -- 172.21.15.5:0/1239579782 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac815b070 con 0x7faac80ccaa0 2024-03-24T10:55:00.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.328+0000 7faac77fe640 1 --2- 172.21.15.5:0/1239579782 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faac80ccaa0 0x7faac806bf50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7faac0009e50 tx=0x7faac002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a73d3582aa326ef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:00.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.332+0000 7faac67fc640 1 -- 172.21.15.5:0/1239579782 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac002cb00 con 0x7faac80ccaa0 2024-03-24T10:55:00.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.332+0000 7faac67fc640 1 -- 172.21.15.5:0/1239579782 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faac002cc60 con 0x7faac80ccaa0 2024-03-24T10:55:00.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.332+0000 7faac67fc640 1 -- 172.21.15.5:0/1239579782 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac00343f0 con 0x7faac80ccaa0 2024-03-24T10:55:00.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.332+0000 7faacd901640 1 -- 172.21.15.5:0/1239579782 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faac80ccaa0 msgr2=0x7faac806bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:00.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.332+0000 7faacd901640 1 --2- 172.21.15.5:0/1239579782 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faac80ccaa0 0x7faac806bf50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7faac0009e50 tx=0x7faac002bf60 comp rx=0 tx=0).stop 2024-03-24T10:55:00.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.332+0000 7faacd901640 1 -- 172.21.15.5:0/1239579782 shutdown_connections 2024-03-24T10:55:00.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.332+0000 7faacd901640 1 --2- 172.21.15.5:0/1239579782 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faac80ccaa0 0x7faac806bf50 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:00.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.332+0000 7faacd901640 1 --2- 172.21.15.5:0/1239579782 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faac815ac00 0x7faac80cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:00.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.332+0000 7faacd901640 1 -- 172.21.15.5:0/1239579782 >> 172.21.15.5:0/1239579782 conn(0x7faac8072ef0 msgr2=0x7faac8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:00.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.332+0000 7faacd901640 1 -- 172.21.15.5:0/1239579782 shutdown_connections 2024-03-24T10:55:00.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.332+0000 7faacd901640 1 -- 172.21.15.5:0/1239579782 wait complete. 2024-03-24T10:55:00.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.334+0000 7faacd901640 1 Processor -- start 2024-03-24T10:55:00.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.334+0000 7faacd901640 1 -- start start 2024-03-24T10:55:00.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.335+0000 7faacd901640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faac80ccaa0 0x7faac8156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:00.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.335+0000 7faacd901640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faac81543a0 0x7faac81f0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:00.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.335+0000 7faacd901640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faac815ac00 0x7faac81f3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:00.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.335+0000 7faacd901640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7faac814f6d0 con 0x7faac80ccaa0 2024-03-24T10:55:00.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.335+0000 7faacd901640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7faac81ef840 con 0x7faac815ac00 2024-03-24T10:55:00.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.335+0000 7faacd901640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7faac815b2b0 con 0x7faac81543a0 2024-03-24T10:55:00.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.335+0000 7faac6ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faac80ccaa0 0x7faac8156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:00.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.335+0000 7faabffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faac81543a0 0x7faac81f0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:00.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.336+0000 7faac77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faac815ac00 0x7faac81f3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:00.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.336+0000 7faac6ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faac80ccaa0 0x7faac8156190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36268/0 (socket says 172.21.15.5:36268) 2024-03-24T10:55:00.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.336+0000 7faac6ffd640 1 -- 172.21.15.5:0/2262048485 learned_addr learned my addr 172.21.15.5:0/2262048485 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:00.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.339+0000 7faac6ffd640 1 -- 172.21.15.5:0/2262048485 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faac81543a0 msgr2=0x7faac81f0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:00.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.339+0000 7faac6ffd640 1 --2- 172.21.15.5:0/2262048485 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faac81543a0 0x7faac81f0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:00.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.339+0000 7faac6ffd640 1 -- 172.21.15.5:0/2262048485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faac815ac00 msgr2=0x7faac81f3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:00.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.339+0000 7faac6ffd640 1 --2- 172.21.15.5:0/2262048485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faac815ac00 0x7faac81f3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:00.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.339+0000 7faac6ffd640 1 -- 172.21.15.5:0/2262048485 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac0009b30 con 0x7faac80ccaa0 2024-03-24T10:55:00.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.342+0000 7faac6ffd640 1 --2- 172.21.15.5:0/2262048485 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faac80ccaa0 0x7faac8156190 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7faab800c400 tx=0x7faab800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:00.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.344+0000 7faac4ff9640 1 -- 172.21.15.5:0/2262048485 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab80052d0 con 0x7faac80ccaa0 2024-03-24T10:55:00.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.344+0000 7faacd901640 1 -- 172.21.15.5:0/2262048485 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faac815b590 con 0x7faac80ccaa0 2024-03-24T10:55:00.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.344+0000 7faacd901640 1 -- 172.21.15.5:0/2262048485 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faac81efb00 con 0x7faac80ccaa0 2024-03-24T10:55:00.346 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.345+0000 7faacd901640 1 -- 172.21.15.5:0/2262048485 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faac80681d0 con 0x7faac80ccaa0 2024-03-24T10:55:00.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.366+0000 7faac4ff9640 1 -- 172.21.15.5:0/2262048485 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faab8005430 con 0x7faac80ccaa0 2024-03-24T10:55:00.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.366+0000 7faac4ff9640 1 -- 172.21.15.5:0/2262048485 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faab8007240 con 0x7faac80ccaa0 2024-03-24T10:55:00.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.367+0000 7faac4ff9640 1 -- 172.21.15.5:0/2262048485 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7faab8007460 con 0x7faac80ccaa0 2024-03-24T10:55:00.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.368+0000 7faac4ff9640 1 --2- 172.21.15.5:0/2262048485 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faaa806b840 0x7faaa806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:00.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.368+0000 7faabffff640 1 --2- 172.21.15.5:0/2262048485 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faaa806b840 0x7faaa806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:00.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.368+0000 7faabffff640 1 --2- 172.21.15.5:0/2262048485 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faaa806b840 0x7faaa806dd00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7faac80ccfc0 tx=0x7faab0009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:00.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.370+0000 7faac4ff9640 1 -- 172.21.15.5:0/2262048485 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7faab8058290 con 0x7faac80ccaa0 2024-03-24T10:55:00.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.628+0000 7faac4ff9640 1 -- 172.21.15.5:0/2262048485 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7faab8054530 con 0x7faac80ccaa0 2024-03-24T10:55:00.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.872+0000 7faacd901640 1 -- 172.21.15.5:0/2262048485 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faac815e510 con 0x7faac80ccaa0 2024-03-24T10:55:00.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.885+0000 7faac4ff9640 1 -- 172.21.15.5:0/2262048485 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7faab8020020 con 0x7faac80ccaa0 2024-03-24T10:55:00.887 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:00.887 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:55:00.887 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:55:00.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.887+0000 7faacd901640 1 -- 172.21.15.5:0/2262048485 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faaa806b840 msgr2=0x7faaa806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:00.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.887+0000 7faacd901640 1 --2- 172.21.15.5:0/2262048485 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faaa806b840 0x7faaa806dd00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7faac80ccfc0 tx=0x7faab0009190 comp rx=0 tx=0).stop 2024-03-24T10:55:00.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.888+0000 7faacd901640 1 -- 172.21.15.5:0/2262048485 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faac80ccaa0 msgr2=0x7faac8156190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:00.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.888+0000 7faacd901640 1 --2- 172.21.15.5:0/2262048485 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faac80ccaa0 0x7faac8156190 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7faab800c400 tx=0x7faab800c8c0 comp rx=0 tx=0).stop 2024-03-24T10:55:00.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.888+0000 7faacd901640 1 -- 172.21.15.5:0/2262048485 shutdown_connections 2024-03-24T10:55:00.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.888+0000 7faacd901640 1 --2- 172.21.15.5:0/2262048485 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faaa806b840 0x7faaa806dd00 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:00.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.888+0000 7faacd901640 1 --2- 172.21.15.5:0/2262048485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faac815ac00 0x7faac81f3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:00.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.888+0000 7faacd901640 1 --2- 172.21.15.5:0/2262048485 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faac81543a0 0x7faac81f0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:00.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.888+0000 7faacd901640 1 --2- 172.21.15.5:0/2262048485 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faac80ccaa0 0x7faac8156190 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:00.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.888+0000 7faacd901640 1 -- 172.21.15.5:0/2262048485 >> 172.21.15.5:0/2262048485 conn(0x7faac8072ef0 msgr2=0x7faac8158090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:00.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.888+0000 7faacd901640 1 -- 172.21.15.5:0/2262048485 shutdown_connections 2024-03-24T10:55:00.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:00.888+0000 7faacd901640 1 -- 172.21.15.5:0/2262048485 wait complete. 2024-03-24T10:55:01.504 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:55:01.505 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:02.506 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:02.535 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:55:02.551 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:55:02.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.608+0000 7fb0a4fb5640 1 Processor -- start 2024-03-24T10:55:02.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.608+0000 7fb0a4fb5640 1 -- start start 2024-03-24T10:55:02.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.608+0000 7fb0a4fb5640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0a00c92f0 0x7fb0a00cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:02.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.609+0000 7fb0a4fb5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb0a006c750 0x7fb0a0164c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:02.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.609+0000 7fb0a4fb5640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0a00ce350 con 0x7fb0a00c96d0 2024-03-24T10:55:02.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.609+0000 7fb0a4fb5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fb0a015e320 con 0x7fb0a006c750 2024-03-24T10:55:02.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.609+0000 7fb0a4fb5640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fb0a015e7b0 con 0x7fb0a00c92f0 2024-03-24T10:55:02.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.609+0000 7fb09dd9b640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fb0a00c96d0 0x7fb0a015ad50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:53134/0 (socket says 172.21.15.5:53134) 2024-03-24T10:55:02.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.609+0000 7fb09dd9b640 1 -- 172.21.15.5:0/1843279137 learned_addr learned my addr 172.21.15.5:0/1843279137 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:55:02.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.609+0000 7fb09e59c640 1 --2- 172.21.15.5:0/1843279137 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0a00c92f0 0x7fb0a00cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:02.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.609+0000 7fb09ed9d640 1 --2- 172.21.15.5:0/1843279137 >> v2:172.21.15.57:3300/0 conn(0x7fb0a006c750 0x7fb0a0164c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:02.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.612+0000 7fb09d59a640 1 -- 172.21.15.5:0/1843279137 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1780413662 0 0) 0x7fb0a00ce350 con 0x7fb0a00c96d0 2024-03-24T10:55:02.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.612+0000 7fb09d59a640 1 -- 172.21.15.5:0/1843279137 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb088003520 con 0x7fb0a00c96d0 2024-03-24T10:55:02.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.612+0000 7fb09e59c640 1 -- 172.21.15.5:0/1843279137 >> v2:172.21.15.57:3300/0 conn(0x7fb0a006c750 msgr2=0x7fb0a0164c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:02.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.612+0000 7fb09e59c640 1 --2- 172.21.15.5:0/1843279137 >> v2:172.21.15.57:3300/0 conn(0x7fb0a006c750 0x7fb0a0164c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:02.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.612+0000 7fb09e59c640 1 -- 172.21.15.5:0/1843279137 >> v1:172.21.15.5:6789/0 conn(0x7fb0a00c96d0 legacy=0x7fb0a015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:02.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.612+0000 7fb09e59c640 1 -- 172.21.15.5:0/1843279137 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0a00ce070 con 0x7fb0a00c92f0 2024-03-24T10:55:02.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.615+0000 7fb09e59c640 1 --2- 172.21.15.5:0/1843279137 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0a00c92f0 0x7fb0a00cd9c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb0900097a0 tx=0x7fb09002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b799b9963c6a7286 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:02.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.619+0000 7fb09d59a640 1 -- 172.21.15.5:0/1843279137 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb090031830 con 0x7fb0a00c92f0 2024-03-24T10:55:02.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.619+0000 7fb09d59a640 1 -- 172.21.15.5:0/1843279137 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb090031990 con 0x7fb0a00c92f0 2024-03-24T10:55:02.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.619+0000 7fb09d59a640 1 -- 172.21.15.5:0/1843279137 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb090031d20 con 0x7fb0a00c92f0 2024-03-24T10:55:02.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.620+0000 7fb0a4fb5640 1 -- 172.21.15.5:0/1843279137 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0a00c92f0 msgr2=0x7fb0a00cd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:02.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.620+0000 7fb0a4fb5640 1 --2- 172.21.15.5:0/1843279137 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0a00c92f0 0x7fb0a00cd9c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb0900097a0 tx=0x7fb09002b8d0 comp rx=0 tx=0).stop 2024-03-24T10:55:02.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.620+0000 7fb0a4fb5640 1 -- 172.21.15.5:0/1843279137 shutdown_connections 2024-03-24T10:55:02.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.620+0000 7fb0a4fb5640 1 --2- 172.21.15.5:0/1843279137 >> v2:172.21.15.57:3300/0 conn(0x7fb0a006c750 0x7fb0a0164c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:02.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.620+0000 7fb0a4fb5640 1 --2- 172.21.15.5:0/1843279137 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0a00c92f0 0x7fb0a00cd9c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:02.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.620+0000 7fb0a4fb5640 1 -- 172.21.15.5:0/1843279137 >> 172.21.15.5:0/1843279137 conn(0x7fb0a006f820 msgr2=0x7fb0a0061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:02.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.620+0000 7fb0a4fb5640 1 -- 172.21.15.5:0/1843279137 shutdown_connections 2024-03-24T10:55:02.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.620+0000 7fb0a4fb5640 1 -- 172.21.15.5:0/1843279137 wait complete. 2024-03-24T10:55:02.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.621+0000 7fb0a4fb5640 1 Processor -- start 2024-03-24T10:55:02.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.622+0000 7fb0a4fb5640 1 -- start start 2024-03-24T10:55:02.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.622+0000 7fb0a4fb5640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0a006c750 0x7fb0a01e3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:02.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.622+0000 7fb0a4fb5640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0a00c92f0 0x7fb0a00cc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:02.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.622+0000 7fb0a4fb5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0a00c96d0 0x7fb0a00c36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:02.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.622+0000 7fb0a4fb5640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a00cc9d0 con 0x7fb0a006c750 2024-03-24T10:55:02.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.622+0000 7fb0a4fb5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb0a00cb9f0 con 0x7fb0a00c96d0 2024-03-24T10:55:02.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.622+0000 7fb0a4fb5640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fb0a01f0910 con 0x7fb0a00c92f0 2024-03-24T10:55:02.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.623+0000 7fb09e59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0a006c750 0x7fb0a01e3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:02.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.623+0000 7fb09dd9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0a00c92f0 0x7fb0a00cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:02.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.623+0000 7fb09ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0a00c96d0 0x7fb0a00c36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:02.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.623+0000 7fb09e59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0a006c750 0x7fb0a01e3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:53026/0 (socket says 172.21.15.5:53026) 2024-03-24T10:55:02.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.623+0000 7fb09e59c640 1 -- 172.21.15.5:0/972085255 learned_addr learned my addr 172.21.15.5:0/972085255 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:02.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.627+0000 7fb09e59c640 1 -- 172.21.15.5:0/972085255 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0a00c92f0 msgr2=0x7fb0a00cc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:02.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.627+0000 7fb09e59c640 1 --2- 172.21.15.5:0/972085255 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0a00c92f0 0x7fb0a00cc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:02.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.627+0000 7fb09e59c640 1 -- 172.21.15.5:0/972085255 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0a00c96d0 msgr2=0x7fb0a00c36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:02.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.627+0000 7fb09e59c640 1 --2- 172.21.15.5:0/972085255 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0a00c96d0 0x7fb0a00c36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:02.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.627+0000 7fb09e59c640 1 -- 172.21.15.5:0/972085255 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb090009400 con 0x7fb0a006c750 2024-03-24T10:55:02.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.629+0000 7fb09e59c640 1 --2- 172.21.15.5:0/972085255 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0a006c750 0x7fb0a01e3f00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb0900098c0 tx=0x7fb0900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:02.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.631+0000 7fb08effd640 1 -- 172.21.15.5:0/972085255 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb09003a630 con 0x7fb0a006c750 2024-03-24T10:55:02.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.631+0000 7fb0a4fb5640 1 -- 172.21.15.5:0/972085255 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0a00cbc90 con 0x7fb0a006c750 2024-03-24T10:55:02.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.631+0000 7fb0a4fb5640 1 -- 172.21.15.5:0/972085255 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0a01f0bf0 con 0x7fb0a006c750 2024-03-24T10:55:02.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.632+0000 7fb0a4fb5640 1 -- 172.21.15.5:0/972085255 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0a015ce70 con 0x7fb0a006c750 2024-03-24T10:55:02.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.658+0000 7fb08effd640 1 -- 172.21.15.5:0/972085255 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb09003a790 con 0x7fb0a006c750 2024-03-24T10:55:02.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.658+0000 7fb08effd640 1 -- 172.21.15.5:0/972085255 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb09003ab40 con 0x7fb0a006c750 2024-03-24T10:55:02.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.659+0000 7fb08effd640 1 -- 172.21.15.5:0/972085255 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb09003ace0 con 0x7fb0a006c750 2024-03-24T10:55:02.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.659+0000 7fb08effd640 1 --2- 172.21.15.5:0/972085255 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb08006b810 0x7fb08006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:02.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.659+0000 7fb09dd9b640 1 --2- 172.21.15.5:0/972085255 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb08006b810 0x7fb08006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:02.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.659+0000 7fb09dd9b640 1 --2- 172.21.15.5:0/972085255 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb08006b810 0x7fb08006dcd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb0a00ccd60 tx=0x7fb094009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:02.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.662+0000 7fb08effd640 1 -- 172.21.15.5:0/972085255 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fb09007a920 con 0x7fb0a006c750 2024-03-24T10:55:02.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:02.955+0000 7fb08effd640 1 -- 172.21.15.5:0/972085255 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb090076bc0 con 0x7fb0a006c750 2024-03-24T10:55:03.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:03.184+0000 7fb0a4fb5640 1 -- 172.21.15.5:0/972085255 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0a00681d0 con 0x7fb0a006c750 2024-03-24T10:55:03.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:03.198+0000 7fb08effd640 1 -- 172.21.15.5:0/972085255 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fb090052090 con 0x7fb0a006c750 2024-03-24T10:55:03.199 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:03.199 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:55:03.200 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:55:03.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:03.200+0000 7fb0a4fb5640 1 -- 172.21.15.5:0/972085255 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb08006b810 msgr2=0x7fb08006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:03.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:03.200+0000 7fb0a4fb5640 1 --2- 172.21.15.5:0/972085255 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb08006b810 0x7fb08006dcd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb0a00ccd60 tx=0x7fb094009040 comp rx=0 tx=0).stop 2024-03-24T10:55:03.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:03.200+0000 7fb0a4fb5640 1 -- 172.21.15.5:0/972085255 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0a006c750 msgr2=0x7fb0a01e3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:03.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:03.200+0000 7fb0a4fb5640 1 --2- 172.21.15.5:0/972085255 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0a006c750 0x7fb0a01e3f00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb0900098c0 tx=0x7fb0900046c0 comp rx=0 tx=0).stop 2024-03-24T10:55:03.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:03.200+0000 7fb0a4fb5640 1 -- 172.21.15.5:0/972085255 shutdown_connections 2024-03-24T10:55:03.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:03.200+0000 7fb0a4fb5640 1 --2- 172.21.15.5:0/972085255 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb08006b810 0x7fb08006dcd0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:03.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:03.200+0000 7fb0a4fb5640 1 --2- 172.21.15.5:0/972085255 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0a00c96d0 0x7fb0a00c36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:03.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:03.200+0000 7fb0a4fb5640 1 --2- 172.21.15.5:0/972085255 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0a00c92f0 0x7fb0a00cc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:03.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:03.200+0000 7fb0a4fb5640 1 --2- 172.21.15.5:0/972085255 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0a006c750 0x7fb0a01e3f00 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:03.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:03.200+0000 7fb0a4fb5640 1 -- 172.21.15.5:0/972085255 >> 172.21.15.5:0/972085255 conn(0x7fb0a006f820 msgr2=0x7fb0a006dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:03.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:03.201+0000 7fb0a4fb5640 1 -- 172.21.15.5:0/972085255 shutdown_connections 2024-03-24T10:55:03.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:03.201+0000 7fb0a4fb5640 1 -- 172.21.15.5:0/972085255 wait complete. 2024-03-24T10:55:03.816 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:55:03.816 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:04.817 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:04.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.921+0000 7fa09413e640 1 Processor -- start 2024-03-24T10:55:04.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.921+0000 7fa09413e640 1 -- start start 2024-03-24T10:55:04.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.922+0000 7fa09413e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa08c0c95b0 0x7fa08c0c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:04.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.922+0000 7fa09413e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa08c0c6d40 con 0x7fa08c156fe0 2024-03-24T10:55:04.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.922+0000 7fa09413e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa08c0c4530 con 0x7fa08c0c9990 2024-03-24T10:55:04.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.922+0000 7fa09413e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fa08c0c5800 con 0x7fa08c0c95b0 2024-03-24T10:55:04.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.923+0000 7fa091eda640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa08c0c95b0 0x7fa08c0c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:04.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.923+0000 7fa0916d9640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa08c0c9990 0x7fa08c06e980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:43774/0 (socket says 172.21.15.5:43774) 2024-03-24T10:55:04.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.923+0000 7fa0916d9640 1 -- 172.21.15.5:0/3312425020 learned_addr learned my addr 172.21.15.5:0/3312425020 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:55:04.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.926+0000 7fa090ed8640 1 -- 172.21.15.5:0/3312425020 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 714096669 0 0) 0x7fa08c0c4530 con 0x7fa08c0c9990 2024-03-24T10:55:04.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.926+0000 7fa090ed8640 1 -- 172.21.15.5:0/3312425020 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa078003520 con 0x7fa08c0c9990 2024-03-24T10:55:04.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.927+0000 7fa091eda640 1 -- 172.21.15.5:0/3312425020 >> v1:172.21.15.57:6789/0 conn(0x7fa08c0c9990 legacy=0x7fa08c06e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:04.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.927+0000 7fa091eda640 1 -- 172.21.15.5:0/3312425020 >> v1:172.21.15.5:6789/0 conn(0x7fa08c156fe0 legacy=0x7fa08c150cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:04.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.927+0000 7fa091eda640 1 -- 172.21.15.5:0/3312425020 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa08c06f090 con 0x7fa08c0c95b0 2024-03-24T10:55:04.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.931+0000 7fa091eda640 1 --2- 172.21.15.5:0/3312425020 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa08c0c95b0 0x7fa08c0c60b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa0880097a0 tx=0x7fa08802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f91c61dc884bf0e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:04.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.935+0000 7fa090ed8640 1 -- 172.21.15.5:0/3312425020 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa088031830 con 0x7fa08c0c95b0 2024-03-24T10:55:04.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.935+0000 7fa090ed8640 1 -- 172.21.15.5:0/3312425020 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa088031990 con 0x7fa08c0c95b0 2024-03-24T10:55:04.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.935+0000 7fa090ed8640 1 -- 172.21.15.5:0/3312425020 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa088031d20 con 0x7fa08c0c95b0 2024-03-24T10:55:04.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.936+0000 7fa09413e640 1 -- 172.21.15.5:0/3312425020 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa08c0c95b0 msgr2=0x7fa08c0c60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:04.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.936+0000 7fa09413e640 1 --2- 172.21.15.5:0/3312425020 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa08c0c95b0 0x7fa08c0c60b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa0880097a0 tx=0x7fa08802b8d0 comp rx=0 tx=0).stop 2024-03-24T10:55:04.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.936+0000 7fa09413e640 1 -- 172.21.15.5:0/3312425020 shutdown_connections 2024-03-24T10:55:04.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.936+0000 7fa09413e640 1 --2- 172.21.15.5:0/3312425020 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa08c0c95b0 0x7fa08c0c60b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:04.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.936+0000 7fa09413e640 1 -- 172.21.15.5:0/3312425020 >> 172.21.15.5:0/3312425020 conn(0x7fa08c06c240 msgr2=0x7fa08c06c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:04.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.936+0000 7fa09413e640 1 -- 172.21.15.5:0/3312425020 shutdown_connections 2024-03-24T10:55:04.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.936+0000 7fa09413e640 1 -- 172.21.15.5:0/3312425020 wait complete. 2024-03-24T10:55:04.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.938+0000 7fa09413e640 1 Processor -- start 2024-03-24T10:55:04.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.938+0000 7fa09413e640 1 -- start start 2024-03-24T10:55:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.939+0000 7fa09413e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa08c0c95b0 0x7fa08c14d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.939+0000 7fa09413e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa08c0c9990 0x7fa08c151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.939+0000 7fa09413e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa08c156fe0 0x7fa08c152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.939+0000 7fa09413e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa08c14af10 con 0x7fa08c156fe0 2024-03-24T10:55:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.939+0000 7fa09413e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa08c14b530 con 0x7fa08c0c9990 2024-03-24T10:55:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.939+0000 7fa09413e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fa08c14b810 con 0x7fa08c0c95b0 2024-03-24T10:55:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.939+0000 7fa0926db640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa08c156fe0 0x7fa08c152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.940+0000 7fa091eda640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa08c0c95b0 0x7fa08c14d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.940+0000 7fa0916d9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa08c0c9990 0x7fa08c151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.940+0000 7fa0926db640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa08c156fe0 0x7fa08c152150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:53046/0 (socket says 172.21.15.5:53046) 2024-03-24T10:55:04.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.940+0000 7fa0926db640 1 -- 172.21.15.5:0/2613860014 learned_addr learned my addr 172.21.15.5:0/2613860014 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:04.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.943+0000 7fa0926db640 1 -- 172.21.15.5:0/2613860014 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa08c0c95b0 msgr2=0x7fa08c14d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:04.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.943+0000 7fa0926db640 1 --2- 172.21.15.5:0/2613860014 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa08c0c95b0 0x7fa08c14d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:04.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.943+0000 7fa0926db640 1 -- 172.21.15.5:0/2613860014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa08c0c9990 msgr2=0x7fa08c151c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:04.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.943+0000 7fa0926db640 1 --2- 172.21.15.5:0/2613860014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa08c0c9990 0x7fa08c151c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:04.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.943+0000 7fa0926db640 1 -- 172.21.15.5:0/2613860014 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa088009400 con 0x7fa08c156fe0 2024-03-24T10:55:04.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.946+0000 7fa0926db640 1 --2- 172.21.15.5:0/2613860014 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa08c156fe0 0x7fa08c152150 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa08400c730 tx=0x7fa08400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:04.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.950+0000 7fa0827fc640 1 -- 172.21.15.5:0/2613860014 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa084003b70 con 0x7fa08c156fe0 2024-03-24T10:55:04.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.950+0000 7fa0827fc640 1 -- 172.21.15.5:0/2613860014 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa084003cf0 con 0x7fa08c156fe0 2024-03-24T10:55:04.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.950+0000 7fa0827fc640 1 -- 172.21.15.5:0/2613860014 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa084003ea0 con 0x7fa08c156fe0 2024-03-24T10:55:04.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.950+0000 7fa09413e640 1 -- 172.21.15.5:0/2613860014 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa08c14baf0 con 0x7fa08c156fe0 2024-03-24T10:55:04.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.951+0000 7fa09413e640 1 -- 172.21.15.5:0/2613860014 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa08c14c0b0 con 0x7fa08c156fe0 2024-03-24T10:55:04.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:04.952+0000 7fa09413e640 1 -- 172.21.15.5:0/2613860014 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa08c151930 con 0x7fa08c156fe0 2024-03-24T10:55:05.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:05.220+0000 7fa0827fc640 1 -- 172.21.15.5:0/2613860014 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa084005310 con 0x7fa08c156fe0 2024-03-24T10:55:05.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:05.221+0000 7fa0827fc640 1 --2- 172.21.15.5:0/2613860014 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa07006b840 0x7fa07006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:05.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:05.221+0000 7fa091eda640 1 --2- 172.21.15.5:0/2613860014 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa07006b840 0x7fa07006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:05.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:05.221+0000 7fa091eda640 1 --2- 172.21.15.5:0/2613860014 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa07006b840 0x7fa07006dd00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa088036000 tx=0x7fa08803c040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:05.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:05.235+0000 7fa0827fc640 1 -- 172.21.15.5:0/2613860014 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fa084058ff0 con 0x7fa08c156fe0 2024-03-24T10:55:05.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:05.235+0000 7fa0827fc640 1 -- 172.21.15.5:0/2613860014 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa084055290 con 0x7fa08c156fe0 2024-03-24T10:55:05.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:05.462+0000 7fa09413e640 1 -- 172.21.15.5:0/2613860014 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa08c151930 con 0x7fa08c156fe0 2024-03-24T10:55:05.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:05.476+0000 7fa0827fc640 1 -- 172.21.15.5:0/2613860014 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fa08c151930 con 0x7fa08c156fe0 2024-03-24T10:55:05.476 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:05.476 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:55:05.476 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:55:05.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:05.478+0000 7fa09413e640 1 -- 172.21.15.5:0/2613860014 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa07006b840 msgr2=0x7fa07006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:05.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:05.478+0000 7fa09413e640 1 --2- 172.21.15.5:0/2613860014 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa07006b840 0x7fa07006dd00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa088036000 tx=0x7fa08803c040 comp rx=0 tx=0).stop 2024-03-24T10:55:05.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:05.478+0000 7fa09413e640 1 -- 172.21.15.5:0/2613860014 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa08c156fe0 msgr2=0x7fa08c152150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:05.478 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:05.478+0000 7fa09413e640 1 --2- 172.21.15.5:0/2613860014 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa08c156fe0 0x7fa08c152150 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa08400c730 tx=0x7fa08400cbf0 comp rx=0 tx=0).stop 2024-03-24T10:55:05.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:05.479+0000 7fa09413e640 1 -- 172.21.15.5:0/2613860014 shutdown_connections 2024-03-24T10:55:05.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:05.479+0000 7fa09413e640 1 --2- 172.21.15.5:0/2613860014 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa07006b840 0x7fa07006dd00 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:05.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:05.479+0000 7fa09413e640 1 --2- 172.21.15.5:0/2613860014 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa08c156fe0 0x7fa08c152150 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:05.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:05.479+0000 7fa09413e640 1 --2- 172.21.15.5:0/2613860014 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa08c0c9990 0x7fa08c151c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:05.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:05.479+0000 7fa09413e640 1 --2- 172.21.15.5:0/2613860014 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa08c0c95b0 0x7fa08c14d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:05.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:05.479+0000 7fa09413e640 1 -- 172.21.15.5:0/2613860014 >> 172.21.15.5:0/2613860014 conn(0x7fa08c06c240 msgr2=0x7fa08c157430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:05.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:05.479+0000 7fa09413e640 1 -- 172.21.15.5:0/2613860014 shutdown_connections 2024-03-24T10:55:05.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:05.479+0000 7fa09413e640 1 -- 172.21.15.5:0/2613860014 wait complete. 2024-03-24T10:55:06.093 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:55:06.094 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:07.095 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:07.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.197+0000 7fda8cdb5640 1 Processor -- start 2024-03-24T10:55:07.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.197+0000 7fda8cdb5640 1 -- start start 2024-03-24T10:55:07.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.198+0000 7fda8cdb5640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fda880cf530 0x7fda880cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:07.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.198+0000 7fda8cdb5640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda880cf990 con 0x7fda880cb270 2024-03-24T10:55:07.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.198+0000 7fda8cdb5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda880ccba0 con 0x7fda880cae90 2024-03-24T10:55:07.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.198+0000 7fda8cdb5640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fda880cc9b0 con 0x7fda880cf530 2024-03-24T10:55:07.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.199+0000 7fda86d9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fda880cf530 0x7fda880cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:07.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.199+0000 7fda8659c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fda880cae90 0x7fda880cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:43780/0 (socket says 172.21.15.5:43780) 2024-03-24T10:55:07.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.199+0000 7fda8659c640 1 -- 172.21.15.5:0/669521800 learned_addr learned my addr 172.21.15.5:0/669521800 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:55:07.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.202+0000 7fda86d9d640 1 -- 172.21.15.5:0/669521800 >> v1:172.21.15.57:6789/0 conn(0x7fda880cae90 legacy=0x7fda880cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:07.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.202+0000 7fda86d9d640 1 -- 172.21.15.5:0/669521800 >> v1:172.21.15.5:6789/0 conn(0x7fda880cb270 legacy=0x7fda880cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:07.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.202+0000 7fda86d9d640 1 -- 172.21.15.5:0/669521800 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda880cce40 con 0x7fda880cf530 2024-03-24T10:55:07.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.204+0000 7fda86d9d640 1 --2- 172.21.15.5:0/669521800 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda880cf530 0x7fda880cb6f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fda80009d90 tx=0x7fda8002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b55ba35451143b25 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:07.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.209+0000 7fda85d9b640 1 -- 172.21.15.5:0/669521800 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8002cd00 con 0x7fda880cf530 2024-03-24T10:55:07.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.209+0000 7fda85d9b640 1 -- 172.21.15.5:0/669521800 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda8002ce60 con 0x7fda880cf530 2024-03-24T10:55:07.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.209+0000 7fda85d9b640 1 -- 172.21.15.5:0/669521800 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda80034660 con 0x7fda880cf530 2024-03-24T10:55:07.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.210+0000 7fda8cdb5640 1 -- 172.21.15.5:0/669521800 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda880cf530 msgr2=0x7fda880cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:07.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.210+0000 7fda8cdb5640 1 --2- 172.21.15.5:0/669521800 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda880cf530 0x7fda880cb6f0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fda80009d90 tx=0x7fda8002bf50 comp rx=0 tx=0).stop 2024-03-24T10:55:07.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.210+0000 7fda8cdb5640 1 -- 172.21.15.5:0/669521800 shutdown_connections 2024-03-24T10:55:07.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.210+0000 7fda8cdb5640 1 --2- 172.21.15.5:0/669521800 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda880cf530 0x7fda880cb6f0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:07.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.210+0000 7fda8cdb5640 1 -- 172.21.15.5:0/669521800 >> 172.21.15.5:0/669521800 conn(0x7fda880cfb80 msgr2=0x7fda880cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:07.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.210+0000 7fda8cdb5640 1 -- 172.21.15.5:0/669521800 shutdown_connections 2024-03-24T10:55:07.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.210+0000 7fda8cdb5640 1 -- 172.21.15.5:0/669521800 wait complete. 2024-03-24T10:55:07.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.212+0000 7fda8cdb5640 1 Processor -- start 2024-03-24T10:55:07.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.212+0000 7fda8cdb5640 1 -- start start 2024-03-24T10:55:07.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.213+0000 7fda8cdb5640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda880cae90 0x7fda880b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:07.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.213+0000 7fda8cdb5640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda880cb270 0x7fda880b1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:07.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.213+0000 7fda8cdb5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda880cf530 0x7fda881ef950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:07.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.213+0000 7fda8cdb5640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fda880b4b40 con 0x7fda880cae90 2024-03-24T10:55:07.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.213+0000 7fda8cdb5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fda88157990 con 0x7fda880cf530 2024-03-24T10:55:07.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.213+0000 7fda8cdb5640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fda880b5990 con 0x7fda880cb270 2024-03-24T10:55:07.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.213+0000 7fda8659c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda880cae90 0x7fda880b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:07.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.213+0000 7fda7ffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda880cb270 0x7fda880b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:07.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.213+0000 7fda86d9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda880cf530 0x7fda881ef950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:07.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.214+0000 7fda8659c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda880cae90 0x7fda880b1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:53062/0 (socket says 172.21.15.5:53062) 2024-03-24T10:55:07.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.214+0000 7fda8659c640 1 -- 172.21.15.5:0/956812432 learned_addr learned my addr 172.21.15.5:0/956812432 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:07.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.217+0000 7fda8659c640 1 -- 172.21.15.5:0/956812432 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda880cb270 msgr2=0x7fda880b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:07.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.217+0000 7fda8659c640 1 --2- 172.21.15.5:0/956812432 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda880cb270 0x7fda880b1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:07.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.217+0000 7fda8659c640 1 -- 172.21.15.5:0/956812432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda880cf530 msgr2=0x7fda881ef950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:07.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.217+0000 7fda8659c640 1 --2- 172.21.15.5:0/956812432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda880cf530 0x7fda881ef950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:07.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.217+0000 7fda8659c640 1 -- 172.21.15.5:0/956812432 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda80009a70 con 0x7fda880cae90 2024-03-24T10:55:07.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.219+0000 7fda8659c640 1 --2- 172.21.15.5:0/956812432 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda880cae90 0x7fda880b1fa0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fda7800c400 tx=0x7fda7800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:07.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.224+0000 7fda7effd640 1 -- 172.21.15.5:0/956812432 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda78005ce0 con 0x7fda880cae90 2024-03-24T10:55:07.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.224+0000 7fda7effd640 1 -- 172.21.15.5:0/956812432 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda78005e40 con 0x7fda880cae90 2024-03-24T10:55:07.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.224+0000 7fda7effd640 1 -- 172.21.15.5:0/956812432 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda780072d0 con 0x7fda880cae90 2024-03-24T10:55:07.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.224+0000 7fda8cdb5640 1 -- 172.21.15.5:0/956812432 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda88157c50 con 0x7fda880cae90 2024-03-24T10:55:07.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.224+0000 7fda8cdb5640 1 -- 172.21.15.5:0/956812432 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda880b2f90 con 0x7fda880cae90 2024-03-24T10:55:07.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.225+0000 7fda8cdb5640 1 -- 172.21.15.5:0/956812432 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda880681d0 con 0x7fda880cae90 2024-03-24T10:55:07.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.249+0000 7fda7effd640 1 -- 172.21.15.5:0/956812432 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fda780074f0 con 0x7fda880cae90 2024-03-24T10:55:07.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.250+0000 7fda7effd640 1 --2- 172.21.15.5:0/956812432 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda6806b860 0x7fda6806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:07.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.250+0000 7fda7ffff640 1 --2- 172.21.15.5:0/956812432 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda6806b860 0x7fda6806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:07.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.250+0000 7fda7ffff640 1 --2- 172.21.15.5:0/956812432 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda6806b860 0x7fda6806dd20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fda880c8270 tx=0x7fda70007e30 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:07.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.253+0000 7fda7effd640 1 -- 172.21.15.5:0/956812432 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fda780583f0 con 0x7fda880cae90 2024-03-24T10:55:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.528+0000 7fda7effd640 1 -- 172.21.15.5:0/956812432 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fda78054690 con 0x7fda880cae90 2024-03-24T10:55:07.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.754+0000 7fda8cdb5640 1 -- 172.21.15.5:0/956812432 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda881587c0 con 0x7fda880cae90 2024-03-24T10:55:07.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.767+0000 7fda7effd640 1 -- 172.21.15.5:0/956812432 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fda78017020 con 0x7fda880cae90 2024-03-24T10:55:07.767 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:07.767 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:55:07.768 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:55:07.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.769+0000 7fda8cdb5640 1 -- 172.21.15.5:0/956812432 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda6806b860 msgr2=0x7fda6806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:07.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.769+0000 7fda8cdb5640 1 --2- 172.21.15.5:0/956812432 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda6806b860 0x7fda6806dd20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fda880c8270 tx=0x7fda70007e30 comp rx=0 tx=0).stop 2024-03-24T10:55:07.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.769+0000 7fda8cdb5640 1 -- 172.21.15.5:0/956812432 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda880cae90 msgr2=0x7fda880b1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:07.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.769+0000 7fda8cdb5640 1 --2- 172.21.15.5:0/956812432 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda880cae90 0x7fda880b1fa0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fda7800c400 tx=0x7fda7800c8c0 comp rx=0 tx=0).stop 2024-03-24T10:55:07.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.769+0000 7fda8cdb5640 1 -- 172.21.15.5:0/956812432 shutdown_connections 2024-03-24T10:55:07.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.769+0000 7fda8cdb5640 1 --2- 172.21.15.5:0/956812432 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda6806b860 0x7fda6806dd20 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:07.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.769+0000 7fda8cdb5640 1 --2- 172.21.15.5:0/956812432 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda880cf530 0x7fda881ef950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:07.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.769+0000 7fda8cdb5640 1 --2- 172.21.15.5:0/956812432 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda880cb270 0x7fda880b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:07.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.769+0000 7fda8cdb5640 1 --2- 172.21.15.5:0/956812432 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda880cae90 0x7fda880b1fa0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:07.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.769+0000 7fda8cdb5640 1 -- 172.21.15.5:0/956812432 >> 172.21.15.5:0/956812432 conn(0x7fda880cfb80 msgr2=0x7fda880c8930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:07.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.770+0000 7fda8cdb5640 1 -- 172.21.15.5:0/956812432 shutdown_connections 2024-03-24T10:55:07.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:07.770+0000 7fda8cdb5640 1 -- 172.21.15.5:0/956812432 wait complete. 2024-03-24T10:55:08.385 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:55:08.386 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:09.387 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:09.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.488+0000 7fb654a15640 1 Processor -- start 2024-03-24T10:55:09.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.489+0000 7fb654a15640 1 -- start start 2024-03-24T10:55:09.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.490+0000 7fb654a15640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6501543a0 0x7fb65015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:09.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.490+0000 7fb654a15640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6500cb2f0 con 0x7fb65006bf50 2024-03-24T10:55:09.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.490+0000 7fb654a15640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6500ca7e0 con 0x7fb65006d4f0 2024-03-24T10:55:09.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.490+0000 7fb654a15640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fb6500c6750 con 0x7fb6501543a0 2024-03-24T10:55:09.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.491+0000 7fb64ed9d640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fb65006bf50 0x7fb65006c380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:53164/0 (socket says 172.21.15.5:53164) 2024-03-24T10:55:09.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.491+0000 7fb64ed9d640 1 -- 172.21.15.5:0/2185680650 learned_addr learned my addr 172.21.15.5:0/2185680650 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:55:09.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.491+0000 7fb64e59c640 1 --2- 172.21.15.5:0/2185680650 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6501543a0 0x7fb65015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:09.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.494+0000 7fb64d59a640 1 -- 172.21.15.5:0/2185680650 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1423297307 0 0) 0x7fb6500cb2f0 con 0x7fb65006bf50 2024-03-24T10:55:09.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.494+0000 7fb64d59a640 1 -- 172.21.15.5:0/2185680650 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb638003520 con 0x7fb65006bf50 2024-03-24T10:55:09.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.495+0000 7fb64d59a640 1 -- 172.21.15.5:0/2185680650 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 340202756 0 0) 0x7fb6500ca7e0 con 0x7fb65006d4f0 2024-03-24T10:55:09.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.495+0000 7fb64d59a640 1 -- 172.21.15.5:0/2185680650 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6500cb2f0 con 0x7fb65006d4f0 2024-03-24T10:55:09.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.495+0000 7fb64e59c640 1 -- 172.21.15.5:0/2185680650 >> v1:172.21.15.57:6789/0 conn(0x7fb65006d4f0 legacy=0x7fb6500cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:09.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.495+0000 7fb64e59c640 1 -- 172.21.15.5:0/2185680650 >> v1:172.21.15.5:6789/0 conn(0x7fb65006bf50 legacy=0x7fb65006c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:09.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.495+0000 7fb64e59c640 1 -- 172.21.15.5:0/2185680650 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6500cb520 con 0x7fb6501543a0 2024-03-24T10:55:09.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.500+0000 7fb64e59c640 1 --2- 172.21.15.5:0/2185680650 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6501543a0 0x7fb65015ac00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb6400097d0 tx=0x7fb64002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=133a0fadc4cee7ac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:09.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.506+0000 7fb64d59a640 1 -- 172.21.15.5:0/2185680650 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb640030880 con 0x7fb6501543a0 2024-03-24T10:55:09.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.506+0000 7fb64d59a640 1 -- 172.21.15.5:0/2185680650 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6400309e0 con 0x7fb6501543a0 2024-03-24T10:55:09.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.506+0000 7fb64d59a640 1 -- 172.21.15.5:0/2185680650 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb640030d70 con 0x7fb6501543a0 2024-03-24T10:55:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.507+0000 7fb654a15640 1 -- 172.21.15.5:0/2185680650 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6501543a0 msgr2=0x7fb65015ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.507+0000 7fb654a15640 1 --2- 172.21.15.5:0/2185680650 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6501543a0 0x7fb65015ac00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb6400097d0 tx=0x7fb64002b9c0 comp rx=0 tx=0).stop 2024-03-24T10:55:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.507+0000 7fb654a15640 1 -- 172.21.15.5:0/2185680650 shutdown_connections 2024-03-24T10:55:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.507+0000 7fb654a15640 1 --2- 172.21.15.5:0/2185680650 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6501543a0 0x7fb65015ac00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.507+0000 7fb654a15640 1 -- 172.21.15.5:0/2185680650 >> 172.21.15.5:0/2185680650 conn(0x7fb650072ef0 msgr2=0x7fb650061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.507+0000 7fb654a15640 1 -- 172.21.15.5:0/2185680650 shutdown_connections 2024-03-24T10:55:09.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.507+0000 7fb654a15640 1 -- 172.21.15.5:0/2185680650 wait complete. 2024-03-24T10:55:09.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.509+0000 7fb654a15640 1 Processor -- start 2024-03-24T10:55:09.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.509+0000 7fb654a15640 1 -- start start 2024-03-24T10:55:09.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.510+0000 7fb654a15640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb65006bf50 0x7fb6501466c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:09.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.510+0000 7fb654a15640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb65006d4f0 0x7fb6500cee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:09.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.510+0000 7fb654a15640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6501543a0 0x7fb6500cf370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:09.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.510+0000 7fb654a15640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fb65015b6f0 con 0x7fb65006d4f0 2024-03-24T10:55:09.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.510+0000 7fb654a15640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb65015e2e0 con 0x7fb65006bf50 2024-03-24T10:55:09.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.510+0000 7fb654a15640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fb65015c550 con 0x7fb6501543a0 2024-03-24T10:55:09.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.511+0000 7fb64e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb65006bf50 0x7fb6501466c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:09.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.511+0000 7fb64ed9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6501543a0 0x7fb6500cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:09.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.511+0000 7fb64e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb65006bf50 0x7fb6501466c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:60350/0 (socket says 172.21.15.5:60350) 2024-03-24T10:55:09.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.511+0000 7fb64e59c640 1 -- 172.21.15.5:0/3203795826 learned_addr learned my addr 172.21.15.5:0/3203795826 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:09.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.515+0000 7fb64e59c640 1 -- 172.21.15.5:0/3203795826 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6501543a0 msgr2=0x7fb6500cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:09.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.515+0000 7fb64e59c640 1 --2- 172.21.15.5:0/3203795826 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6501543a0 0x7fb6500cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:09.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.515+0000 7fb64e59c640 1 -- 172.21.15.5:0/3203795826 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb65006d4f0 msgr2=0x7fb6500cee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:09.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.515+0000 7fb64e59c640 1 --2- 172.21.15.5:0/3203795826 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb65006d4f0 0x7fb6500cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:09.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.515+0000 7fb64e59c640 1 -- 172.21.15.5:0/3203795826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb640009400 con 0x7fb65006bf50 2024-03-24T10:55:09.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.518+0000 7fb64e59c640 1 --2- 172.21.15.5:0/3203795826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb65006bf50 0x7fb6501466c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb64002bfd0 tx=0x7fb640004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:09.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.521+0000 7fb63effd640 1 -- 172.21.15.5:0/3203795826 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb64003e040 con 0x7fb65006bf50 2024-03-24T10:55:09.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.521+0000 7fb654a15640 1 -- 172.21.15.5:0/3203795826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6501f1500 con 0x7fb65006bf50 2024-03-24T10:55:09.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.521+0000 7fb654a15640 1 -- 172.21.15.5:0/3203795826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6501f1ac0 con 0x7fb65006bf50 2024-03-24T10:55:09.522 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.522+0000 7fb654a15640 1 -- 172.21.15.5:0/3203795826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6500681d0 con 0x7fb65006bf50 2024-03-24T10:55:09.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.731+0000 7fb63effd640 1 -- 172.21.15.5:0/3203795826 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb640039c90 con 0x7fb65006bf50 2024-03-24T10:55:09.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.823+0000 7fb63effd640 1 -- 172.21.15.5:0/3203795826 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb640043440 con 0x7fb65006bf50 2024-03-24T10:55:09.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.823+0000 7fb63effd640 1 -- 172.21.15.5:0/3203795826 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb6400423e0 con 0x7fb65006bf50 2024-03-24T10:55:09.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.824+0000 7fb63effd640 1 --2- 172.21.15.5:0/3203795826 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb63006b810 0x7fb63006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:09.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.824+0000 7fb64dd9b640 1 --2- 172.21.15.5:0/3203795826 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb63006b810 0x7fb63006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:09.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.825+0000 7fb64dd9b640 1 --2- 172.21.15.5:0/3203795826 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb63006b810 0x7fb63006dcd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb6440011e0 tx=0x7fb6440078f0 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:09.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.830+0000 7fb63effd640 1 -- 172.21.15.5:0/3203795826 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fb64007b3b0 con 0x7fb65006bf50 2024-03-24T10:55:09.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:09.847+0000 7fb63effd640 1 -- 172.21.15.5:0/3203795826 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb640077650 con 0x7fb65006bf50 2024-03-24T10:55:10.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:10.111+0000 7fb654a15640 1 -- 172.21.15.5:0/3203795826 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6501494f0 con 0x7fb65006bf50 2024-03-24T10:55:10.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:10.127+0000 7fb63effd640 1 -- 172.21.15.5:0/3203795826 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fb64007ad10 con 0x7fb65006bf50 2024-03-24T10:55:10.127 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:10.127 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:55:10.127 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:55:10.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:10.129+0000 7fb654a15640 1 -- 172.21.15.5:0/3203795826 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb63006b810 msgr2=0x7fb63006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:10.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:10.129+0000 7fb654a15640 1 --2- 172.21.15.5:0/3203795826 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb63006b810 0x7fb63006dcd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb6440011e0 tx=0x7fb6440078f0 comp rx=0 tx=0).stop 2024-03-24T10:55:10.129 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:10.129+0000 7fb654a15640 1 -- 172.21.15.5:0/3203795826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb65006bf50 msgr2=0x7fb6501466c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:10.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:10.129+0000 7fb654a15640 1 --2- 172.21.15.5:0/3203795826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb65006bf50 0x7fb6501466c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb64002bfd0 tx=0x7fb640004730 comp rx=0 tx=0).stop 2024-03-24T10:55:10.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:10.129+0000 7fb654a15640 1 -- 172.21.15.5:0/3203795826 shutdown_connections 2024-03-24T10:55:10.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:10.129+0000 7fb654a15640 1 --2- 172.21.15.5:0/3203795826 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb63006b810 0x7fb63006dcd0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:10.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:10.129+0000 7fb654a15640 1 --2- 172.21.15.5:0/3203795826 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb65006d4f0 0x7fb6500cee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:10.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:10.129+0000 7fb654a15640 1 --2- 172.21.15.5:0/3203795826 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb6501543a0 0x7fb6500cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:10.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:10.129+0000 7fb654a15640 1 --2- 172.21.15.5:0/3203795826 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb65006bf50 0x7fb6501466c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:10.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:10.129+0000 7fb654a15640 1 -- 172.21.15.5:0/3203795826 >> 172.21.15.5:0/3203795826 conn(0x7fb650072ef0 msgr2=0x7fb6500cc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:10.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:10.129+0000 7fb654a15640 1 -- 172.21.15.5:0/3203795826 shutdown_connections 2024-03-24T10:55:10.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:10.129+0000 7fb654a15640 1 -- 172.21.15.5:0/3203795826 wait complete. 2024-03-24T10:55:10.744 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:55:10.744 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:11.745 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:11.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.848+0000 7fe0532c3640 1 Processor -- start 2024-03-24T10:55:11.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.848+0000 7fe0532c3640 1 -- start start 2024-03-24T10:55:11.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.848+0000 7fe0532c3640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe04c15ac00 0x7fe04c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:11.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.849+0000 7fe0532c3640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe04c0ccaa0 0x7fe04c06bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:11.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.849+0000 7fe0532c3640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe04c0cb2f0 con 0x7fe04c1543a0 2024-03-24T10:55:11.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.849+0000 7fe0532c3640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fe04c0c6750 con 0x7fe04c15ac00 2024-03-24T10:55:11.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.849+0000 7fe0532c3640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fe04c0cc380 con 0x7fe04c0ccaa0 2024-03-24T10:55:11.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.850+0000 7fe05085e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe04c15ac00 0x7fe04c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:11.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.851+0000 7fe051860640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe04c0ccaa0 0x7fe04c06bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:11.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.852+0000 7fe051860640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe04c0ccaa0 0x7fe04c06bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:60362/0 (socket says 172.21.15.5:60362) 2024-03-24T10:55:11.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.852+0000 7fe051860640 1 -- 172.21.15.5:0/2796727881 learned_addr learned my addr 172.21.15.5:0/2796727881 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:11.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.855+0000 7fe051860640 1 -- 172.21.15.5:0/2796727881 >> v2:172.21.15.5:3300/0 conn(0x7fe04c15ac00 msgr2=0x7fe04c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:11.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.856+0000 7fe051860640 1 --2- 172.21.15.5:0/2796727881 >> v2:172.21.15.5:3300/0 conn(0x7fe04c15ac00 0x7fe04c0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:11.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.856+0000 7fe051860640 1 -- 172.21.15.5:0/2796727881 >> v1:172.21.15.5:6789/0 conn(0x7fe04c1543a0 legacy=0x7fe04c06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:11.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.856+0000 7fe051860640 1 -- 172.21.15.5:0/2796727881 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe04c15b070 con 0x7fe04c0ccaa0 2024-03-24T10:55:11.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.862+0000 7fe051860640 1 --2- 172.21.15.5:0/2796727881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe04c0ccaa0 0x7fe04c06bf50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe04c0b51c0 tx=0x7fe04402bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc77c36f43a35b3e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:11.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.867+0000 7fe043fff640 1 -- 172.21.15.5:0/2796727881 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe04402cb20 con 0x7fe04c0ccaa0 2024-03-24T10:55:11.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.867+0000 7fe043fff640 1 -- 172.21.15.5:0/2796727881 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe04402cc80 con 0x7fe04c0ccaa0 2024-03-24T10:55:11.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.867+0000 7fe043fff640 1 -- 172.21.15.5:0/2796727881 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0440343f0 con 0x7fe04c0ccaa0 2024-03-24T10:55:11.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.868+0000 7fe0532c3640 1 -- 172.21.15.5:0/2796727881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe04c0ccaa0 msgr2=0x7fe04c06bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:11.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.868+0000 7fe0532c3640 1 --2- 172.21.15.5:0/2796727881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe04c0ccaa0 0x7fe04c06bf50 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe04c0b51c0 tx=0x7fe04402bf80 comp rx=0 tx=0).stop 2024-03-24T10:55:11.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.868+0000 7fe0532c3640 1 -- 172.21.15.5:0/2796727881 shutdown_connections 2024-03-24T10:55:11.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.868+0000 7fe0532c3640 1 --2- 172.21.15.5:0/2796727881 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe04c0ccaa0 0x7fe04c06bf50 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:11.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.868+0000 7fe0532c3640 1 --2- 172.21.15.5:0/2796727881 >> v2:172.21.15.5:3300/0 conn(0x7fe04c15ac00 0x7fe04c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:11.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.868+0000 7fe0532c3640 1 -- 172.21.15.5:0/2796727881 >> 172.21.15.5:0/2796727881 conn(0x7fe04c072ef0 msgr2=0x7fe04c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:11.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.868+0000 7fe0532c3640 1 -- 172.21.15.5:0/2796727881 shutdown_connections 2024-03-24T10:55:11.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.869+0000 7fe0532c3640 1 -- 172.21.15.5:0/2796727881 wait complete. 2024-03-24T10:55:11.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.870+0000 7fe0532c3640 1 Processor -- start 2024-03-24T10:55:11.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.870+0000 7fe0532c3640 1 -- start start 2024-03-24T10:55:11.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.872+0000 7fe0532c3640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe04c0ccaa0 0x7fe04c14d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:11.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.872+0000 7fe0532c3640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe04c1543a0 0x7fe04c14d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:11.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.872+0000 7fe0532c3640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe04c15ac00 0x7fe04c144680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:11.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.872+0000 7fe0532c3640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fe04c14dee0 con 0x7fe04c1543a0 2024-03-24T10:55:11.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.872+0000 7fe0532c3640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe04c1f0060 con 0x7fe04c0ccaa0 2024-03-24T10:55:11.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.872+0000 7fe0532c3640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe04c0cf7b0 con 0x7fe04c15ac00 2024-03-24T10:55:11.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.872+0000 7fe05085e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe04c1543a0 0x7fe04c14d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:11.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.872+0000 7fe05105f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe04c0ccaa0 0x7fe04c14d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:11.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.872+0000 7fe051860640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe04c15ac00 0x7fe04c144680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:11.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.873+0000 7fe05085e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe04c1543a0 0x7fe04c14d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:53076/0 (socket says 172.21.15.5:53076) 2024-03-24T10:55:11.873 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.873+0000 7fe05085e640 1 -- 172.21.15.5:0/2273722029 learned_addr learned my addr 172.21.15.5:0/2273722029 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:11.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.876+0000 7fe05085e640 1 -- 172.21.15.5:0/2273722029 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe04c15ac00 msgr2=0x7fe04c144680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:11.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.876+0000 7fe05085e640 1 --2- 172.21.15.5:0/2273722029 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe04c15ac00 0x7fe04c144680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:11.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.876+0000 7fe05085e640 1 -- 172.21.15.5:0/2273722029 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe04c0ccaa0 msgr2=0x7fe04c14d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:11.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.876+0000 7fe05085e640 1 --2- 172.21.15.5:0/2273722029 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe04c0ccaa0 0x7fe04c14d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:11.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.876+0000 7fe05085e640 1 -- 172.21.15.5:0/2273722029 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe044009b30 con 0x7fe04c1543a0 2024-03-24T10:55:11.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.879+0000 7fe05085e640 1 --2- 172.21.15.5:0/2273722029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe04c1543a0 0x7fe04c14d9a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fe03c00e870 tx=0x7fe03c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:11.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.883+0000 7fe041ffb640 1 -- 172.21.15.5:0/2273722029 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03c0094f0 con 0x7fe04c1543a0 2024-03-24T10:55:11.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.883+0000 7fe041ffb640 1 -- 172.21.15.5:0/2273722029 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe03c00c940 con 0x7fe04c1543a0 2024-03-24T10:55:11.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.884+0000 7fe041ffb640 1 -- 172.21.15.5:0/2273722029 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03c00ccf0 con 0x7fe04c1543a0 2024-03-24T10:55:11.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.884+0000 7fe0532c3640 1 -- 172.21.15.5:0/2273722029 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe04c1f02e0 con 0x7fe04c1543a0 2024-03-24T10:55:11.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.884+0000 7fe0532c3640 1 -- 172.21.15.5:0/2273722029 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe04c1f08a0 con 0x7fe04c1543a0 2024-03-24T10:55:11.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.885+0000 7fe0532c3640 1 -- 172.21.15.5:0/2273722029 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe04c0681d0 con 0x7fe04c1543a0 2024-03-24T10:55:11.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.910+0000 7fe041ffb640 1 -- 172.21.15.5:0/2273722029 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe03c013470 con 0x7fe04c1543a0 2024-03-24T10:55:11.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.911+0000 7fe041ffb640 1 --2- 172.21.15.5:0/2273722029 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe03006b890 0x7fe03006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:11.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.911+0000 7fe05105f640 1 --2- 172.21.15.5:0/2273722029 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe03006b890 0x7fe03006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:11.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.911+0000 7fe05105f640 1 --2- 172.21.15.5:0/2273722029 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe03006b890 0x7fe03006dd50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe048001780 tx=0x7fe048009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:11.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:11.916+0000 7fe041ffb640 1 -- 172.21.15.5:0/2273722029 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fe03c058de0 con 0x7fe04c1543a0 2024-03-24T10:55:12.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:12.203+0000 7fe041ffb640 1 -- 172.21.15.5:0/2273722029 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe03c055080 con 0x7fe04c1543a0 2024-03-24T10:55:12.474 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:12.474+0000 7fe0532c3640 1 -- 172.21.15.5:0/2273722029 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe04c151670 con 0x7fe04c1543a0 2024-03-24T10:55:12.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:12.498+0000 7fe041ffb640 1 -- 172.21.15.5:0/2273722029 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fe03c017090 con 0x7fe04c1543a0 2024-03-24T10:55:12.499 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:12.499 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:55:12.499 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:55:12.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:12.500+0000 7fe0532c3640 1 -- 172.21.15.5:0/2273722029 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe03006b890 msgr2=0x7fe03006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:12.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:12.500+0000 7fe0532c3640 1 --2- 172.21.15.5:0/2273722029 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe03006b890 0x7fe03006dd50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe048001780 tx=0x7fe048009040 comp rx=0 tx=0).stop 2024-03-24T10:55:12.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:12.500+0000 7fe0532c3640 1 -- 172.21.15.5:0/2273722029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe04c1543a0 msgr2=0x7fe04c14d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:12.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:12.500+0000 7fe0532c3640 1 --2- 172.21.15.5:0/2273722029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe04c1543a0 0x7fe04c14d9a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fe03c00e870 tx=0x7fe03c00ed30 comp rx=0 tx=0).stop 2024-03-24T10:55:12.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:12.500+0000 7fe0532c3640 1 -- 172.21.15.5:0/2273722029 shutdown_connections 2024-03-24T10:55:12.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:12.500+0000 7fe0532c3640 1 --2- 172.21.15.5:0/2273722029 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe03006b890 0x7fe03006dd50 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:12.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:12.500+0000 7fe0532c3640 1 --2- 172.21.15.5:0/2273722029 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe04c1543a0 0x7fe04c14d9a0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:12.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:12.500+0000 7fe0532c3640 1 --2- 172.21.15.5:0/2273722029 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe04c15ac00 0x7fe04c144680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:12.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:12.500+0000 7fe0532c3640 1 --2- 172.21.15.5:0/2273722029 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe04c0ccaa0 0x7fe04c14d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:12.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:12.500+0000 7fe0532c3640 1 -- 172.21.15.5:0/2273722029 >> 172.21.15.5:0/2273722029 conn(0x7fe04c072ef0 msgr2=0x7fe04c157f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:12.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:12.500+0000 7fe0532c3640 1 -- 172.21.15.5:0/2273722029 shutdown_connections 2024-03-24T10:55:12.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:12.500+0000 7fe0532c3640 1 -- 172.21.15.5:0/2273722029 wait complete. 2024-03-24T10:55:13.119 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:55:13.119 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:14.120 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:14.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.224+0000 7f399d7d7640 1 Processor -- start 2024-03-24T10:55:14.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.224+0000 7f399d7d7640 1 -- start start 2024-03-24T10:55:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.224+0000 7f399d7d7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f39981543a0 0x7f399815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.224+0000 7f399d7d7640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39980cb2f0 con 0x7f399806d4f0 2024-03-24T10:55:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.224+0000 7f399d7d7640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39980ca7e0 con 0x7f399806bf50 2024-03-24T10:55:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.224+0000 7f399d7d7640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f39980c6750 con 0x7f39981543a0 2024-03-24T10:55:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.225+0000 7f3996ffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f39981543a0 0x7f399815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.225+0000 7f39977fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f399806bf50 0x7f399806c330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:43178/0 (socket says 172.21.15.5:43178) 2024-03-24T10:55:14.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.225+0000 7f39977fe640 1 -- 172.21.15.5:0/2160901484 learned_addr learned my addr 172.21.15.5:0/2160901484 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:55:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.229+0000 7f3995ffb640 1 -- 172.21.15.5:0/2160901484 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3145012107 0 0) 0x7f39980ca7e0 con 0x7f399806bf50 2024-03-24T10:55:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.229+0000 7f3995ffb640 1 -- 172.21.15.5:0/2160901484 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3980003520 con 0x7f399806bf50 2024-03-24T10:55:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.229+0000 7f3996ffd640 1 -- 172.21.15.5:0/2160901484 >> v1:172.21.15.57:6789/0 conn(0x7f399806bf50 legacy=0x7f399806c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.229+0000 7f3996ffd640 1 -- 172.21.15.5:0/2160901484 >> v1:172.21.15.5:6789/0 conn(0x7f399806d4f0 legacy=0x7f39980cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:14.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.229+0000 7f3996ffd640 1 -- 172.21.15.5:0/2160901484 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39980cb520 con 0x7f39981543a0 2024-03-24T10:55:14.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.235+0000 7f3996ffd640 1 --2- 172.21.15.5:0/2160901484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39981543a0 0x7f399815ac00 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f39880097a0 tx=0x7f398802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e97c2f500c998c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.240+0000 7f3995ffb640 1 -- 172.21.15.5:0/2160901484 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398802f810 con 0x7f39981543a0 2024-03-24T10:55:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.240+0000 7f3995ffb640 1 -- 172.21.15.5:0/2160901484 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f398802f970 con 0x7f39981543a0 2024-03-24T10:55:14.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.240+0000 7f3995ffb640 1 -- 172.21.15.5:0/2160901484 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398802fd00 con 0x7f39981543a0 2024-03-24T10:55:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.241+0000 7f399d7d7640 1 -- 172.21.15.5:0/2160901484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39981543a0 msgr2=0x7f399815ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.241+0000 7f399d7d7640 1 --2- 172.21.15.5:0/2160901484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39981543a0 0x7f399815ac00 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f39880097a0 tx=0x7f398802b9f0 comp rx=0 tx=0).stop 2024-03-24T10:55:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.241+0000 7f399d7d7640 1 -- 172.21.15.5:0/2160901484 shutdown_connections 2024-03-24T10:55:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.241+0000 7f399d7d7640 1 --2- 172.21.15.5:0/2160901484 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f39981543a0 0x7f399815ac00 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.241+0000 7f399d7d7640 1 -- 172.21.15.5:0/2160901484 >> 172.21.15.5:0/2160901484 conn(0x7f3998072ef0 msgr2=0x7f3998061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.241+0000 7f399d7d7640 1 -- 172.21.15.5:0/2160901484 shutdown_connections 2024-03-24T10:55:14.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.241+0000 7f399d7d7640 1 -- 172.21.15.5:0/2160901484 wait complete. 2024-03-24T10:55:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.242+0000 7f399d7d7640 1 Processor -- start 2024-03-24T10:55:14.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.242+0000 7f399d7d7640 1 -- start start 2024-03-24T10:55:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.243+0000 7f399d7d7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f399806bf50 0x7f3998158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.243+0000 7f399d7d7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f399806d4f0 0x7f39980b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.243+0000 7f399d7d7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39981543a0 0x7f39981f0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:14.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.243+0000 7f399d7d7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f39981f0550 con 0x7f399806d4f0 2024-03-24T10:55:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.243+0000 7f399d7d7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f39980cf5a0 con 0x7f39981543a0 2024-03-24T10:55:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.243+0000 7f399d7d7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f39981efeb0 con 0x7f399806bf50 2024-03-24T10:55:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.244+0000 7f39967fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f399806d4f0 0x7f39980b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.244+0000 7f39977fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39981543a0 0x7f39981f0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:14.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.244+0000 7f39967fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f399806d4f0 0x7f39980b42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:41178/0 (socket says 172.21.15.5:41178) 2024-03-24T10:55:14.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.244+0000 7f39967fc640 1 -- 172.21.15.5:0/1156065176 learned_addr learned my addr 172.21.15.5:0/1156065176 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:14.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.244+0000 7f3996ffd640 1 --2- 172.21.15.5:0/1156065176 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f399806bf50 0x7f3998158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:14.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.247+0000 7f39967fc640 1 -- 172.21.15.5:0/1156065176 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f399806bf50 msgr2=0x7f3998158cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:14.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.248+0000 7f39967fc640 1 --2- 172.21.15.5:0/1156065176 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f399806bf50 0x7f3998158cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:14.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.248+0000 7f39967fc640 1 -- 172.21.15.5:0/1156065176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39981543a0 msgr2=0x7f39981f0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:14.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.248+0000 7f39967fc640 1 --2- 172.21.15.5:0/1156065176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39981543a0 0x7f39981f0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:14.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.248+0000 7f39967fc640 1 -- 172.21.15.5:0/1156065176 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3988009400 con 0x7f399806d4f0 2024-03-24T10:55:14.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.250+0000 7f39967fc640 1 --2- 172.21.15.5:0/1156065176 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f399806d4f0 0x7f39980b42f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f398c00c400 tx=0x7f398c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:14.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.254+0000 7f397f7fe640 1 -- 172.21.15.5:0/1156065176 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398c0052d0 con 0x7f399806d4f0 2024-03-24T10:55:14.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.254+0000 7f397f7fe640 1 -- 172.21.15.5:0/1156065176 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f398c005450 con 0x7f399806d4f0 2024-03-24T10:55:14.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.254+0000 7f399d7d7640 1 -- 172.21.15.5:0/1156065176 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39980b4930 con 0x7f399806d4f0 2024-03-24T10:55:14.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.254+0000 7f397f7fe640 1 -- 172.21.15.5:0/1156065176 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398c007260 con 0x7f399806d4f0 2024-03-24T10:55:14.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.254+0000 7f399d7d7640 1 -- 172.21.15.5:0/1156065176 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39980b4ef0 con 0x7f399806d4f0 2024-03-24T10:55:14.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.255+0000 7f399d7d7640 1 -- 172.21.15.5:0/1156065176 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39980b5080 con 0x7f399806d4f0 2024-03-24T10:55:14.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.279+0000 7f397f7fe640 1 -- 172.21.15.5:0/1156065176 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f398c0073c0 con 0x7f399806d4f0 2024-03-24T10:55:14.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.279+0000 7f397f7fe640 1 --2- 172.21.15.5:0/1156065176 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f398406b860 0x7f398406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:14.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.279+0000 7f3996ffd640 1 --2- 172.21.15.5:0/1156065176 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f398406b860 0x7f398406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:14.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.280+0000 7f3996ffd640 1 --2- 172.21.15.5:0/1156065176 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f398406b860 0x7f398406dd20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3988036000 tx=0x7f3988008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:14.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.560+0000 7f397f7fe640 1 -- 172.21.15.5:0/1156065176 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f398c007670 con 0x7f399806d4f0 2024-03-24T10:55:14.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.560+0000 7f397f7fe640 1 -- 172.21.15.5:0/1156065176 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f398c0548d0 con 0x7f399806d4f0 2024-03-24T10:55:14.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.800+0000 7f399d7d7640 1 -- 172.21.15.5:0/1156065176 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39981f0690 con 0x7f399806d4f0 2024-03-24T10:55:14.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.843+0000 7f397f7fe640 1 -- 172.21.15.5:0/1156065176 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f398c020020 con 0x7f399806d4f0 2024-03-24T10:55:14.843 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:14.844 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:55:14.844 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:55:14.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.846+0000 7f399d7d7640 1 -- 172.21.15.5:0/1156065176 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f398406b860 msgr2=0x7f398406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:14.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.846+0000 7f399d7d7640 1 --2- 172.21.15.5:0/1156065176 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f398406b860 0x7f398406dd20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3988036000 tx=0x7f3988008030 comp rx=0 tx=0).stop 2024-03-24T10:55:14.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.846+0000 7f399d7d7640 1 -- 172.21.15.5:0/1156065176 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f399806d4f0 msgr2=0x7f39980b42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:14.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.846+0000 7f399d7d7640 1 --2- 172.21.15.5:0/1156065176 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f399806d4f0 0x7f39980b42f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f398c00c400 tx=0x7f398c00c8c0 comp rx=0 tx=0).stop 2024-03-24T10:55:14.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.846+0000 7f399d7d7640 1 -- 172.21.15.5:0/1156065176 shutdown_connections 2024-03-24T10:55:14.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.846+0000 7f399d7d7640 1 --2- 172.21.15.5:0/1156065176 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f398406b860 0x7f398406dd20 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:14.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.846+0000 7f399d7d7640 1 --2- 172.21.15.5:0/1156065176 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f399806d4f0 0x7f39980b42f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:14.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.846+0000 7f399d7d7640 1 --2- 172.21.15.5:0/1156065176 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f39981543a0 0x7f39981f0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:14.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.846+0000 7f399d7d7640 1 --2- 172.21.15.5:0/1156065176 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f399806bf50 0x7f3998158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:14.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.846+0000 7f399d7d7640 1 -- 172.21.15.5:0/1156065176 >> 172.21.15.5:0/1156065176 conn(0x7f3998072ef0 msgr2=0x7f39980c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:14.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.846+0000 7f399d7d7640 1 -- 172.21.15.5:0/1156065176 shutdown_connections 2024-03-24T10:55:14.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:14.846+0000 7f399d7d7640 1 -- 172.21.15.5:0/1156065176 wait complete. 2024-03-24T10:55:15.465 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:55:15.465 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:16.467 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:16.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.569+0000 7ff34bed0640 1 Processor -- start 2024-03-24T10:55:16.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.569+0000 7ff34bed0640 1 -- start start 2024-03-24T10:55:16.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.570+0000 7ff34bed0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff3441543a0 0x7ff34415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:16.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.570+0000 7ff34bed0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff34406bf50 0x7ff34406c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:16.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.570+0000 7ff34bed0640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3440cb2f0 con 0x7ff34406d4f0 2024-03-24T10:55:16.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.570+0000 7ff34bed0640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7ff3440c6750 con 0x7ff3441543a0 2024-03-24T10:55:16.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.570+0000 7ff34bed0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff3440cc380 con 0x7ff34406bf50 2024-03-24T10:55:16.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.571+0000 7ff349c6c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff3441543a0 0x7ff34415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:16.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.571+0000 7ff34a46d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff34406bf50 0x7ff34406c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:16.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.571+0000 7ff34a46d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff34406bf50 0x7ff34406c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:56024/0 (socket says 172.21.15.5:56024) 2024-03-24T10:55:16.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.571+0000 7ff34a46d640 1 -- 172.21.15.5:0/866444084 learned_addr learned my addr 172.21.15.5:0/866444084 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:16.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.575+0000 7ff34a46d640 1 -- 172.21.15.5:0/866444084 >> v2:172.21.15.5:3300/0 conn(0x7ff3441543a0 msgr2=0x7ff34415ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:16.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.575+0000 7ff34a46d640 1 --2- 172.21.15.5:0/866444084 >> v2:172.21.15.5:3300/0 conn(0x7ff3441543a0 0x7ff34415ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:16.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.575+0000 7ff34a46d640 1 -- 172.21.15.5:0/866444084 >> v1:172.21.15.5:6789/0 conn(0x7ff34406d4f0 legacy=0x7ff3440cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:16.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.575+0000 7ff34a46d640 1 -- 172.21.15.5:0/866444084 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3440ca7e0 con 0x7ff34406bf50 2024-03-24T10:55:16.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.577+0000 7ff34a46d640 1 --2- 172.21.15.5:0/866444084 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff34406bf50 0x7ff34406c360 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff33c009e50 tx=0x7ff33c02bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4e0dc3839b30680 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:16.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.582+0000 7ff348c6a640 1 -- 172.21.15.5:0/866444084 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33c02cb90 con 0x7ff34406bf50 2024-03-24T10:55:16.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.582+0000 7ff348c6a640 1 -- 172.21.15.5:0/866444084 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff33c02ccf0 con 0x7ff34406bf50 2024-03-24T10:55:16.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.582+0000 7ff348c6a640 1 -- 172.21.15.5:0/866444084 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff33c034610 con 0x7ff34406bf50 2024-03-24T10:55:16.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.583+0000 7ff34bed0640 1 -- 172.21.15.5:0/866444084 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff34406bf50 msgr2=0x7ff34406c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:16.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.583+0000 7ff34bed0640 1 --2- 172.21.15.5:0/866444084 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff34406bf50 0x7ff34406c360 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff33c009e50 tx=0x7ff33c02bff0 comp rx=0 tx=0).stop 2024-03-24T10:55:16.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.583+0000 7ff34bed0640 1 -- 172.21.15.5:0/866444084 shutdown_connections 2024-03-24T10:55:16.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.583+0000 7ff34bed0640 1 --2- 172.21.15.5:0/866444084 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff34406bf50 0x7ff34406c360 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:16.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.583+0000 7ff34bed0640 1 --2- 172.21.15.5:0/866444084 >> v2:172.21.15.5:3300/0 conn(0x7ff3441543a0 0x7ff34415ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:16.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.583+0000 7ff34bed0640 1 -- 172.21.15.5:0/866444084 >> 172.21.15.5:0/866444084 conn(0x7ff344072ef0 msgr2=0x7ff344061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:16.583 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.583+0000 7ff34bed0640 1 -- 172.21.15.5:0/866444084 shutdown_connections 2024-03-24T10:55:16.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.583+0000 7ff34bed0640 1 -- 172.21.15.5:0/866444084 wait complete. 2024-03-24T10:55:16.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.585+0000 7ff34bed0640 1 Processor -- start 2024-03-24T10:55:16.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.585+0000 7ff34bed0640 1 -- start start 2024-03-24T10:55:16.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.586+0000 7ff34bed0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff34406bf50 0x7ff3440c7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:16.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.586+0000 7ff34bed0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff34406d4f0 0x7ff3441f0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:16.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.586+0000 7ff34bed0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3441543a0 0x7ff34414a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:16.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.586+0000 7ff34bed0640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff3441efe00 con 0x7ff34406bf50 2024-03-24T10:55:16.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.586+0000 7ff34bed0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff3441525c0 con 0x7ff3441543a0 2024-03-24T10:55:16.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.586+0000 7ff34bed0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff3440c8090 con 0x7ff34406d4f0 2024-03-24T10:55:16.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.587+0000 7ff34946b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff34406d4f0 0x7ff3441f0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:16.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.587+0000 7ff349c6c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff34406bf50 0x7ff3440c7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:16.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.587+0000 7ff34a46d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3441543a0 0x7ff34414a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:16.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.587+0000 7ff349c6c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff34406bf50 0x7ff3440c7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:41192/0 (socket says 172.21.15.5:41192) 2024-03-24T10:55:16.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.587+0000 7ff349c6c640 1 -- 172.21.15.5:0/3883067195 learned_addr learned my addr 172.21.15.5:0/3883067195 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:16.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.591+0000 7ff349c6c640 1 -- 172.21.15.5:0/3883067195 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff34406d4f0 msgr2=0x7ff3441f0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:16.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.591+0000 7ff349c6c640 1 --2- 172.21.15.5:0/3883067195 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff34406d4f0 0x7ff3441f0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:16.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.591+0000 7ff349c6c640 1 -- 172.21.15.5:0/3883067195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3441543a0 msgr2=0x7ff34414a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:16.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.591+0000 7ff349c6c640 1 --2- 172.21.15.5:0/3883067195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3441543a0 0x7ff34414a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:16.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.591+0000 7ff349c6c640 1 -- 172.21.15.5:0/3883067195 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff33c009b30 con 0x7ff34406bf50 2024-03-24T10:55:16.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.593+0000 7ff349c6c640 1 --2- 172.21.15.5:0/3883067195 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff34406bf50 0x7ff3440c7100 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff34000e8a0 tx=0x7ff34000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:16.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.597+0000 7ff33a7fc640 1 -- 172.21.15.5:0/3883067195 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3400094f0 con 0x7ff34406bf50 2024-03-24T10:55:16.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.597+0000 7ff33a7fc640 1 -- 172.21.15.5:0/3883067195 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff34000c940 con 0x7ff34406bf50 2024-03-24T10:55:16.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.597+0000 7ff34bed0640 1 -- 172.21.15.5:0/3883067195 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3440c8330 con 0x7ff34406bf50 2024-03-24T10:55:16.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.597+0000 7ff33a7fc640 1 -- 172.21.15.5:0/3883067195 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff34000ccf0 con 0x7ff34406bf50 2024-03-24T10:55:16.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.597+0000 7ff34bed0640 1 -- 172.21.15.5:0/3883067195 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3440c88f0 con 0x7ff34406bf50 2024-03-24T10:55:16.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.598+0000 7ff34bed0640 1 -- 172.21.15.5:0/3883067195 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3440681d0 con 0x7ff34406bf50 2024-03-24T10:55:16.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.619+0000 7ff33a7fc640 1 -- 172.21.15.5:0/3883067195 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff340013470 con 0x7ff34406bf50 2024-03-24T10:55:16.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.619+0000 7ff33a7fc640 1 --2- 172.21.15.5:0/3883067195 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff32806b890 0x7ff32806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:16.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.620+0000 7ff34946b640 1 --2- 172.21.15.5:0/3883067195 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff32806b890 0x7ff32806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:16.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.620+0000 7ff34946b640 1 --2- 172.21.15.5:0/3883067195 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff32806b890 0x7ff32806dd50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff3440b2460 tx=0x7ff334009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:16.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.884+0000 7ff33a7fc640 1 -- 172.21.15.5:0/3883067195 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7ff340059190 con 0x7ff34406bf50 2024-03-24T10:55:16.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:16.884+0000 7ff33a7fc640 1 -- 172.21.15.5:0/3883067195 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff340055430 con 0x7ff34406bf50 2024-03-24T10:55:17.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:17.108+0000 7ff34bed0640 1 -- 172.21.15.5:0/3883067195 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff344151080 con 0x7ff34406bf50 2024-03-24T10:55:17.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:17.121+0000 7ff33a7fc640 1 -- 172.21.15.5:0/3883067195 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7ff340021090 con 0x7ff34406bf50 2024-03-24T10:55:17.121 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:17.121 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:55:17.121 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:55:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:17.123+0000 7ff34bed0640 1 -- 172.21.15.5:0/3883067195 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff32806b890 msgr2=0x7ff32806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:17.123+0000 7ff34bed0640 1 --2- 172.21.15.5:0/3883067195 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff32806b890 0x7ff32806dd50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff3440b2460 tx=0x7ff334009040 comp rx=0 tx=0).stop 2024-03-24T10:55:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:17.123+0000 7ff34bed0640 1 -- 172.21.15.5:0/3883067195 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff34406bf50 msgr2=0x7ff3440c7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:17.123+0000 7ff34bed0640 1 --2- 172.21.15.5:0/3883067195 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff34406bf50 0x7ff3440c7100 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff34000e8a0 tx=0x7ff34000ed60 comp rx=0 tx=0).stop 2024-03-24T10:55:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:17.123+0000 7ff34bed0640 1 -- 172.21.15.5:0/3883067195 shutdown_connections 2024-03-24T10:55:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:17.123+0000 7ff34bed0640 1 --2- 172.21.15.5:0/3883067195 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff32806b890 0x7ff32806dd50 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:17.123+0000 7ff34bed0640 1 --2- 172.21.15.5:0/3883067195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff3441543a0 0x7ff34414a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:17.123+0000 7ff34bed0640 1 --2- 172.21.15.5:0/3883067195 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff34406d4f0 0x7ff3441f0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:17.123+0000 7ff34bed0640 1 --2- 172.21.15.5:0/3883067195 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff34406bf50 0x7ff3440c7100 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:17.123+0000 7ff34bed0640 1 -- 172.21.15.5:0/3883067195 >> 172.21.15.5:0/3883067195 conn(0x7ff344072ef0 msgr2=0x7ff34414e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:17.123+0000 7ff34bed0640 1 -- 172.21.15.5:0/3883067195 shutdown_connections 2024-03-24T10:55:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:17.123+0000 7ff34bed0640 1 -- 172.21.15.5:0/3883067195 wait complete. 2024-03-24T10:55:17.738 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:55:17.739 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:18.740 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:18.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.843+0000 7f2115dc7640 1 Processor -- start 2024-03-24T10:55:18.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.843+0000 7f2115dc7640 1 -- start start 2024-03-24T10:55:18.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.843+0000 7f2115dc7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21101543a0 0x7f211015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:18.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.843+0000 7f2115dc7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f211006d4f0 0x7f21100cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:18.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.843+0000 7f2115dc7640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21100cb2f0 con 0x7f21100ccaa0 2024-03-24T10:55:18.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.843+0000 7f2115dc7640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f21100c6750 con 0x7f211006d4f0 2024-03-24T10:55:18.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.843+0000 7f2115dc7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f21100cc380 con 0x7f21101543a0 2024-03-24T10:55:18.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.844+0000 7f210f7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21101543a0 0x7f211015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:18.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.844+0000 7f210ffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f21100ccaa0 0x7f211006bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:43188/0 (socket says 172.21.15.5:43188) 2024-03-24T10:55:18.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.844+0000 7f210ffff640 1 -- 172.21.15.5:0/4074842824 learned_addr learned my addr 172.21.15.5:0/4074842824 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:55:18.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.847+0000 7f210f7fe640 1 -- 172.21.15.5:0/4074842824 >> v2:172.21.15.5:3300/0 conn(0x7f211006d4f0 msgr2=0x7f21100cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:18.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.847+0000 7f210f7fe640 1 --2- 172.21.15.5:0/4074842824 >> v2:172.21.15.5:3300/0 conn(0x7f211006d4f0 0x7f21100cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:18.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.847+0000 7f210f7fe640 1 -- 172.21.15.5:0/4074842824 >> v1:172.21.15.57:6789/0 conn(0x7f21100ccaa0 legacy=0x7f211006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:18.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.847+0000 7f210f7fe640 1 -- 172.21.15.5:0/4074842824 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21100ca7e0 con 0x7f21101543a0 2024-03-24T10:55:18.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.849+0000 7f210f7fe640 1 --2- 172.21.15.5:0/4074842824 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21101543a0 0x7f211015ac00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f21000097a0 tx=0x7f210002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6032aa39bbef6f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:18.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.854+0000 7f210e7fc640 1 -- 172.21.15.5:0/4074842824 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2100030610 con 0x7f21101543a0 2024-03-24T10:55:18.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.854+0000 7f210e7fc640 1 -- 172.21.15.5:0/4074842824 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2100030770 con 0x7f21101543a0 2024-03-24T10:55:18.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.854+0000 7f210e7fc640 1 -- 172.21.15.5:0/4074842824 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2100030b00 con 0x7f21101543a0 2024-03-24T10:55:18.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.855+0000 7f2115dc7640 1 -- 172.21.15.5:0/4074842824 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21101543a0 msgr2=0x7f211015ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:18.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.855+0000 7f2115dc7640 1 --2- 172.21.15.5:0/4074842824 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21101543a0 0x7f211015ac00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f21000097a0 tx=0x7f210002b960 comp rx=0 tx=0).stop 2024-03-24T10:55:18.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.855+0000 7f2115dc7640 1 -- 172.21.15.5:0/4074842824 shutdown_connections 2024-03-24T10:55:18.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.855+0000 7f2115dc7640 1 --2- 172.21.15.5:0/4074842824 >> v2:172.21.15.5:3300/0 conn(0x7f211006d4f0 0x7f21100cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:18.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.855+0000 7f2115dc7640 1 --2- 172.21.15.5:0/4074842824 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21101543a0 0x7f211015ac00 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:18.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.855+0000 7f2115dc7640 1 -- 172.21.15.5:0/4074842824 >> 172.21.15.5:0/4074842824 conn(0x7f2110072ef0 msgr2=0x7f2110061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:18.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.855+0000 7f2115dc7640 1 -- 172.21.15.5:0/4074842824 shutdown_connections 2024-03-24T10:55:18.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.855+0000 7f2115dc7640 1 -- 172.21.15.5:0/4074842824 wait complete. 2024-03-24T10:55:18.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.856+0000 7f2115dc7640 1 Processor -- start 2024-03-24T10:55:18.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.856+0000 7f2115dc7640 1 -- start start 2024-03-24T10:55:18.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.857+0000 7f2115dc7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f211006d4f0 0x7f211015b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:18.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.857+0000 7f2115dc7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21100ccaa0 0x7f211015d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:18.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.857+0000 7f2115dc7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21101543a0 0x7f21101f3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:18.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.857+0000 7f2115dc7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f211015b4c0 con 0x7f21100ccaa0 2024-03-24T10:55:18.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.857+0000 7f2115dc7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f211015be90 con 0x7f21101543a0 2024-03-24T10:55:18.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.857+0000 7f2115dc7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f2110157910 con 0x7f211006d4f0 2024-03-24T10:55:18.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.858+0000 7f210ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21101543a0 0x7f21101f3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:18.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.858+0000 7f210f7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f211006d4f0 0x7f211015b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:18.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.858+0000 7f210f7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f211006d4f0 0x7f211015b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:56058/0 (socket says 172.21.15.5:56058) 2024-03-24T10:55:18.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.858+0000 7f210f7fe640 1 -- 172.21.15.5:0/1933202979 learned_addr learned my addr 172.21.15.5:0/1933202979 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:18.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.859+0000 7f210effd640 1 --2- 172.21.15.5:0/1933202979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21100ccaa0 0x7f211015d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:18.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.862+0000 7f210f7fe640 1 -- 172.21.15.5:0/1933202979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21101543a0 msgr2=0x7f21101f3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:18.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.862+0000 7f210f7fe640 1 --2- 172.21.15.5:0/1933202979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21101543a0 0x7f21101f3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:18.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.862+0000 7f210f7fe640 1 -- 172.21.15.5:0/1933202979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21100ccaa0 msgr2=0x7f211015d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:18.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.862+0000 7f210f7fe640 1 --2- 172.21.15.5:0/1933202979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21100ccaa0 0x7f211015d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:18.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.862+0000 7f210f7fe640 1 -- 172.21.15.5:0/1933202979 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2100009400 con 0x7f211006d4f0 2024-03-24T10:55:18.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.866+0000 7f210f7fe640 1 --2- 172.21.15.5:0/1933202979 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f211006d4f0 0x7f211015b950 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f21000098c0 tx=0x7f2100004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:18.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.869+0000 7f20f7fff640 1 -- 172.21.15.5:0/1933202979 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210003e040 con 0x7f211006d4f0 2024-03-24T10:55:18.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.869+0000 7f2115dc7640 1 -- 172.21.15.5:0/1933202979 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21100b46f0 con 0x7f211006d4f0 2024-03-24T10:55:18.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.869+0000 7f2115dc7640 1 -- 172.21.15.5:0/1933202979 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f21100b4cb0 con 0x7f211006d4f0 2024-03-24T10:55:18.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:18.870+0000 7f2115dc7640 1 -- 172.21.15.5:0/1933202979 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21100b4ec0 con 0x7f211006d4f0 2024-03-24T10:55:19.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.162+0000 7f20f7fff640 1 -- 172.21.15.5:0/1933202979 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2100039c50 con 0x7f211006d4f0 2024-03-24T10:55:19.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.162+0000 7f20f7fff640 1 -- 172.21.15.5:0/1933202979 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2100005990 con 0x7f211006d4f0 2024-03-24T10:55:19.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.163+0000 7f20f7fff640 1 -- 172.21.15.5:0/1933202979 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f210004e440 con 0x7f211006d4f0 2024-03-24T10:55:19.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.163+0000 7f20f7fff640 1 --2- 172.21.15.5:0/1933202979 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f20fc06b840 0x7f20fc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:19.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.163+0000 7f210effd640 1 --2- 172.21.15.5:0/1933202979 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f20fc06b840 0x7f20fc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:19.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.164+0000 7f210effd640 1 --2- 172.21.15.5:0/1933202979 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f20fc06b840 0x7f20fc06dd00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f211006c7f0 tx=0x7f2104009060 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:19.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.169+0000 7f20f7fff640 1 -- 172.21.15.5:0/1933202979 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f210007af40 con 0x7f211006d4f0 2024-03-24T10:55:19.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.188+0000 7f20f7fff640 1 -- 172.21.15.5:0/1933202979 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f21000771e0 con 0x7f211006d4f0 2024-03-24T10:55:19.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.412+0000 7f2115dc7640 1 -- 172.21.15.5:0/1933202979 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f211015b2b0 con 0x7f211006d4f0 2024-03-24T10:55:19.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.435+0000 7f20f7fff640 1 -- 172.21.15.5:0/1933202979 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f2100034070 con 0x7f211006d4f0 2024-03-24T10:55:19.435 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:19.435 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:55:19.435 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:55:19.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.437+0000 7f2115dc7640 1 -- 172.21.15.5:0/1933202979 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f20fc06b840 msgr2=0x7f20fc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:19.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.437+0000 7f2115dc7640 1 --2- 172.21.15.5:0/1933202979 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f20fc06b840 0x7f20fc06dd00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f211006c7f0 tx=0x7f2104009060 comp rx=0 tx=0).stop 2024-03-24T10:55:19.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.437+0000 7f2115dc7640 1 -- 172.21.15.5:0/1933202979 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f211006d4f0 msgr2=0x7f211015b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:19.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.437+0000 7f2115dc7640 1 --2- 172.21.15.5:0/1933202979 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f211006d4f0 0x7f211015b950 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f21000098c0 tx=0x7f2100004730 comp rx=0 tx=0).stop 2024-03-24T10:55:19.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.437+0000 7f2115dc7640 1 -- 172.21.15.5:0/1933202979 shutdown_connections 2024-03-24T10:55:19.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.437+0000 7f2115dc7640 1 --2- 172.21.15.5:0/1933202979 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f20fc06b840 0x7f20fc06dd00 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:19.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.437+0000 7f2115dc7640 1 --2- 172.21.15.5:0/1933202979 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21100ccaa0 0x7f211015d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:19.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.437+0000 7f2115dc7640 1 --2- 172.21.15.5:0/1933202979 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21101543a0 0x7f21101f3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:19.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.437+0000 7f2115dc7640 1 --2- 172.21.15.5:0/1933202979 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f211006d4f0 0x7f211015b950 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:19.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.437+0000 7f2115dc7640 1 -- 172.21.15.5:0/1933202979 >> 172.21.15.5:0/1933202979 conn(0x7f2110072ef0 msgr2=0x7f211014f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:19.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.437+0000 7f2115dc7640 1 -- 172.21.15.5:0/1933202979 shutdown_connections 2024-03-24T10:55:19.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:19.437+0000 7f2115dc7640 1 -- 172.21.15.5:0/1933202979 wait complete. 2024-03-24T10:55:20.053 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:55:20.053 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:21.054 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:21.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.159+0000 7f7194081640 1 Processor -- start 2024-03-24T10:55:21.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.159+0000 7f7194081640 1 -- start start 2024-03-24T10:55:21.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.161+0000 7f7194081640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f718c0cf110 0x7f718c153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:21.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.161+0000 7f7194081640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f718c15bce0 con 0x7f718c06c580 2024-03-24T10:55:21.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.161+0000 7f7194081640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f718c0cd890 con 0x7f718c0cc8f0 2024-03-24T10:55:21.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.161+0000 7f7194081640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f718c06cae0 con 0x7f718c0cf110 2024-03-24T10:55:21.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.161+0000 7f719261e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f718c0cf110 0x7f718c153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:21.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.162+0000 7f7191e1d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f718c0cc8f0 0x7f718c15b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:43204/0 (socket says 172.21.15.5:43204) 2024-03-24T10:55:21.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.162+0000 7f7191e1d640 1 -- 172.21.15.5:0/2618152846 learned_addr learned my addr 172.21.15.5:0/2618152846 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:55:21.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.165+0000 7f7190e1b640 1 -- 172.21.15.5:0/2618152846 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3299087403 0 0) 0x7f718c0cd890 con 0x7f718c0cc8f0 2024-03-24T10:55:21.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.165+0000 7f7190e1b640 1 -- 172.21.15.5:0/2618152846 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7178003520 con 0x7f718c0cc8f0 2024-03-24T10:55:21.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.165+0000 7f719261e640 1 -- 172.21.15.5:0/2618152846 >> v1:172.21.15.57:6789/0 conn(0x7f718c0cc8f0 legacy=0x7f718c15b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:21.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.165+0000 7f719261e640 1 -- 172.21.15.5:0/2618152846 >> v1:172.21.15.5:6789/0 conn(0x7f718c06c580 legacy=0x7f718c1538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:21.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.165+0000 7f719261e640 1 -- 172.21.15.5:0/2618152846 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f718c0cdb50 con 0x7f718c0cf110 2024-03-24T10:55:21.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.170+0000 7f719261e640 1 --2- 172.21.15.5:0/2618152846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f718c0cf110 0x7f718c153fc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7184009d90 tx=0x7f718402bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cfbfc4532f0971a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:21.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.176+0000 7f7190e1b640 1 -- 172.21.15.5:0/2618152846 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f718402cd80 con 0x7f718c0cf110 2024-03-24T10:55:21.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.176+0000 7f7190e1b640 1 -- 172.21.15.5:0/2618152846 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7184034440 con 0x7f718c0cf110 2024-03-24T10:55:21.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.176+0000 7f7190e1b640 1 -- 172.21.15.5:0/2618152846 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71840347d0 con 0x7f718c0cf110 2024-03-24T10:55:21.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.176+0000 7f7194081640 1 -- 172.21.15.5:0/2618152846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f718c0cf110 msgr2=0x7f718c153fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:21.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.176+0000 7f7194081640 1 --2- 172.21.15.5:0/2618152846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f718c0cf110 0x7f718c153fc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f7184009d90 tx=0x7f718402bfd0 comp rx=0 tx=0).stop 2024-03-24T10:55:21.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.177+0000 7f7194081640 1 -- 172.21.15.5:0/2618152846 shutdown_connections 2024-03-24T10:55:21.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.177+0000 7f7194081640 1 --2- 172.21.15.5:0/2618152846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f718c0cf110 0x7f718c153fc0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:21.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.177+0000 7f7194081640 1 -- 172.21.15.5:0/2618152846 >> 172.21.15.5:0/2618152846 conn(0x7f718c0b1820 msgr2=0x7f718c0b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:21.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.177+0000 7f7194081640 1 -- 172.21.15.5:0/2618152846 shutdown_connections 2024-03-24T10:55:21.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.177+0000 7f7194081640 1 -- 172.21.15.5:0/2618152846 wait complete. 2024-03-24T10:55:21.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.178+0000 7f7194081640 1 Processor -- start 2024-03-24T10:55:21.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.179+0000 7f7194081640 1 -- start start 2024-03-24T10:55:21.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.179+0000 7f7194081640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f718c06c580 0x7f718c1581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:21.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.179+0000 7f7194081640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f718c0cc8f0 0x7f718c1efc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:21.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.179+0000 7f7194081640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f718c0cf110 0x7f718c14f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:21.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.179+0000 7f7194081640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f718c14ca70 con 0x7f718c0cf110 2024-03-24T10:55:21.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.179+0000 7f7194081640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f718c0c9770 con 0x7f718c06c580 2024-03-24T10:55:21.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.179+0000 7f7194081640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f718c0cc3c0 con 0x7f718c0cc8f0 2024-03-24T10:55:21.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.180+0000 7f719261e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f718c0cf110 0x7f718c14f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:21.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.180+0000 7f719161c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f718c0cc8f0 0x7f718c1efc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:21.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.180+0000 7f7191e1d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f718c06c580 0x7f718c1581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:21.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.180+0000 7f719261e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f718c0cf110 0x7f718c14f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:41234/0 (socket says 172.21.15.5:41234) 2024-03-24T10:55:21.180 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.180+0000 7f719261e640 1 -- 172.21.15.5:0/957809209 learned_addr learned my addr 172.21.15.5:0/957809209 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:21.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.184+0000 7f719261e640 1 -- 172.21.15.5:0/957809209 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f718c0cc8f0 msgr2=0x7f718c1efc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:21.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.184+0000 7f719261e640 1 --2- 172.21.15.5:0/957809209 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f718c0cc8f0 0x7f718c1efc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:21.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.184+0000 7f719261e640 1 -- 172.21.15.5:0/957809209 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f718c06c580 msgr2=0x7f718c1581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:21.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.184+0000 7f719261e640 1 --2- 172.21.15.5:0/957809209 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f718c06c580 0x7f718c1581b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:21.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.184+0000 7f719261e640 1 -- 172.21.15.5:0/957809209 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7184009a70 con 0x7f718c0cf110 2024-03-24T10:55:21.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.187+0000 7f719261e640 1 --2- 172.21.15.5:0/957809209 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f718c0cf110 0x7f718c14f480 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7184009d90 tx=0x7f7184004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:21.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.191+0000 7f71827fc640 1 -- 172.21.15.5:0/957809209 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7184040650 con 0x7f718c0cf110 2024-03-24T10:55:21.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.191+0000 7f71827fc640 1 -- 172.21.15.5:0/957809209 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71840407b0 con 0x7f718c0cf110 2024-03-24T10:55:21.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.192+0000 7f71827fc640 1 -- 172.21.15.5:0/957809209 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7184040b60 con 0x7f718c0cf110 2024-03-24T10:55:21.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.192+0000 7f7194081640 1 -- 172.21.15.5:0/957809209 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f718c0c8880 con 0x7f718c0cf110 2024-03-24T10:55:21.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.193+0000 7f7194081640 1 -- 172.21.15.5:0/957809209 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f718c0c8e40 con 0x7f718c0cf110 2024-03-24T10:55:21.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.193+0000 7f7194081640 1 -- 172.21.15.5:0/957809209 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f718c0ce910 con 0x7f718c0cf110 2024-03-24T10:55:21.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.486+0000 7f71827fc640 1 -- 172.21.15.5:0/957809209 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7184040cc0 con 0x7f718c0cf110 2024-03-24T10:55:21.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.486+0000 7f71827fc640 1 --2- 172.21.15.5:0/957809209 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f717006b810 0x7f717006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:21.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.486+0000 7f7191e1d640 1 --2- 172.21.15.5:0/957809209 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f717006b810 0x7f717006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:21.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.487+0000 7f7191e1d640 1 --2- 172.21.15.5:0/957809209 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f717006b810 0x7f717006dcd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f71880011e0 tx=0x7f7188007d30 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:21.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.511+0000 7f71827fc640 1 -- 172.21.15.5:0/957809209 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f718407bc50 con 0x7f718c0cf110 2024-03-24T10:55:21.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.511+0000 7f71827fc640 1 -- 172.21.15.5:0/957809209 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7184077ef0 con 0x7f718c0cf110 2024-03-24T10:55:21.736 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.736+0000 7f7194081640 1 -- 172.21.15.5:0/957809209 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f718c0681d0 con 0x7f718c0cf110 2024-03-24T10:55:21.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.751+0000 7f71827fc640 1 -- 172.21.15.5:0/957809209 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f7184044090 con 0x7f718c0cf110 2024-03-24T10:55:21.751 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:21.751 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:55:21.752 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:55:21.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.753+0000 7f7194081640 1 -- 172.21.15.5:0/957809209 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f717006b810 msgr2=0x7f717006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:21.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.753+0000 7f7194081640 1 --2- 172.21.15.5:0/957809209 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f717006b810 0x7f717006dcd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f71880011e0 tx=0x7f7188007d30 comp rx=0 tx=0).stop 2024-03-24T10:55:21.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.753+0000 7f7194081640 1 -- 172.21.15.5:0/957809209 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f718c0cf110 msgr2=0x7f718c14f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:21.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.753+0000 7f7194081640 1 --2- 172.21.15.5:0/957809209 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f718c0cf110 0x7f718c14f480 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f7184009d90 tx=0x7f7184004a40 comp rx=0 tx=0).stop 2024-03-24T10:55:21.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.753+0000 7f7194081640 1 -- 172.21.15.5:0/957809209 shutdown_connections 2024-03-24T10:55:21.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.753+0000 7f7194081640 1 --2- 172.21.15.5:0/957809209 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f717006b810 0x7f717006dcd0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:21.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.753+0000 7f7194081640 1 --2- 172.21.15.5:0/957809209 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f718c0cf110 0x7f718c14f480 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:21.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.753+0000 7f7194081640 1 --2- 172.21.15.5:0/957809209 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f718c0cc8f0 0x7f718c1efc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:21.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.753+0000 7f7194081640 1 --2- 172.21.15.5:0/957809209 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f718c06c580 0x7f718c1581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:21.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.753+0000 7f7194081640 1 -- 172.21.15.5:0/957809209 >> 172.21.15.5:0/957809209 conn(0x7f718c0b1820 msgr2=0x7f718c0b3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:21.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.753+0000 7f7194081640 1 -- 172.21.15.5:0/957809209 shutdown_connections 2024-03-24T10:55:21.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:21.753+0000 7f7194081640 1 -- 172.21.15.5:0/957809209 wait complete. 2024-03-24T10:55:22.368 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:55:22.368 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:23.369 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:23.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.470+0000 7f659870c640 1 Processor -- start 2024-03-24T10:55:23.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.471+0000 7f659870c640 1 -- start start 2024-03-24T10:55:23.471 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.471+0000 7f659870c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f65901543a0 0x7f659015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:23.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.471+0000 7f659870c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f659006d4f0 0x7f65900cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:23.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.471+0000 7f659870c640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65900cb2f0 con 0x7f65900ccaa0 2024-03-24T10:55:23.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.471+0000 7f659870c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f65900c6750 con 0x7f659006d4f0 2024-03-24T10:55:23.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.471+0000 7f659870c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f65900cc380 con 0x7f65901543a0 2024-03-24T10:55:23.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.472+0000 7f6596ca9640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f65900ccaa0 0x7f659006bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60560/0 (socket says 172.21.15.5:60560) 2024-03-24T10:55:23.472 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.472+0000 7f6596ca9640 1 -- 172.21.15.5:0/1807706893 learned_addr learned my addr 172.21.15.5:0/1807706893 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:55:23.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.472+0000 7f6595ca7640 1 --2- 172.21.15.5:0/1807706893 >> v2:172.21.15.57:3300/0 conn(0x7f659006d4f0 0x7f65900cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:23.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.472+0000 7f65964a8640 1 --2- 172.21.15.5:0/1807706893 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f65901543a0 0x7f659015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:23.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.475+0000 7f65954a6640 1 -- 172.21.15.5:0/1807706893 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1914944475 0 0) 0x7f65900cb2f0 con 0x7f65900ccaa0 2024-03-24T10:55:23.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.475+0000 7f65954a6640 1 -- 172.21.15.5:0/1807706893 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f657c003520 con 0x7f65900ccaa0 2024-03-24T10:55:23.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.476+0000 7f65964a8640 1 -- 172.21.15.5:0/1807706893 >> v2:172.21.15.57:3300/0 conn(0x7f659006d4f0 msgr2=0x7f65900cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:23.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.476+0000 7f65964a8640 1 --2- 172.21.15.5:0/1807706893 >> v2:172.21.15.57:3300/0 conn(0x7f659006d4f0 0x7f65900cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:23.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.476+0000 7f65964a8640 1 -- 172.21.15.5:0/1807706893 >> v1:172.21.15.5:6789/0 conn(0x7f65900ccaa0 legacy=0x7f659006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:23.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.476+0000 7f65964a8640 1 -- 172.21.15.5:0/1807706893 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65900ca7e0 con 0x7f65901543a0 2024-03-24T10:55:23.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.479+0000 7f65964a8640 1 --2- 172.21.15.5:0/1807706893 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f65901543a0 0x7f659015ac00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f658c0097a0 tx=0x7f658c02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee1e586d5c095e8a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:23.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.481+0000 7f65954a6640 1 -- 172.21.15.5:0/1807706893 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658c030610 con 0x7f65901543a0 2024-03-24T10:55:23.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.483+0000 7f65954a6640 1 -- 172.21.15.5:0/1807706893 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f658c030770 con 0x7f65901543a0 2024-03-24T10:55:23.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.483+0000 7f65954a6640 1 -- 172.21.15.5:0/1807706893 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658c030ae0 con 0x7f65901543a0 2024-03-24T10:55:23.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.484+0000 7f659870c640 1 -- 172.21.15.5:0/1807706893 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f65901543a0 msgr2=0x7f659015ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:23.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.484+0000 7f659870c640 1 --2- 172.21.15.5:0/1807706893 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f65901543a0 0x7f659015ac00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f658c0097a0 tx=0x7f658c02b960 comp rx=0 tx=0).stop 2024-03-24T10:55:23.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.484+0000 7f659870c640 1 -- 172.21.15.5:0/1807706893 shutdown_connections 2024-03-24T10:55:23.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.484+0000 7f659870c640 1 --2- 172.21.15.5:0/1807706893 >> v2:172.21.15.57:3300/0 conn(0x7f659006d4f0 0x7f65900cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:23.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.484+0000 7f659870c640 1 --2- 172.21.15.5:0/1807706893 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f65901543a0 0x7f659015ac00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:23.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.484+0000 7f659870c640 1 -- 172.21.15.5:0/1807706893 >> 172.21.15.5:0/1807706893 conn(0x7f6590072ef0 msgr2=0x7f6590061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:23.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.484+0000 7f659870c640 1 -- 172.21.15.5:0/1807706893 shutdown_connections 2024-03-24T10:55:23.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.484+0000 7f659870c640 1 -- 172.21.15.5:0/1807706893 wait complete. 2024-03-24T10:55:23.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.485+0000 7f659870c640 1 Processor -- start 2024-03-24T10:55:23.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.485+0000 7f659870c640 1 -- start start 2024-03-24T10:55:23.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.486+0000 7f659870c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f659006d4f0 0x7f6590156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:23.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.486+0000 7f659870c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65900ccaa0 0x7f6590154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:23.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.486+0000 7f659870c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65901543a0 0x7f65901f0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:23.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.486+0000 7f659870c640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f6590155a60 con 0x7f65900ccaa0 2024-03-24T10:55:23.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.486+0000 7f659870c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6590159f10 con 0x7f65901543a0 2024-03-24T10:55:23.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.486+0000 7f659870c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f6590156610 con 0x7f659006d4f0 2024-03-24T10:55:23.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.487+0000 7f65964a8640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f659006d4f0 0x7f6590156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:23.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.487+0000 7f6595ca7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65900ccaa0 0x7f6590154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:23.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.487+0000 7f6596ca9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65901543a0 0x7f65901f0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:23.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.487+0000 7f6595ca7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65900ccaa0 0x7f6590154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36178/0 (socket says 172.21.15.5:36178) 2024-03-24T10:55:23.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.487+0000 7f6595ca7640 1 -- 172.21.15.5:0/1735675345 learned_addr learned my addr 172.21.15.5:0/1735675345 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:23.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.491+0000 7f6596ca9640 1 -- 172.21.15.5:0/1735675345 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f659006d4f0 msgr2=0x7f6590156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:23.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.491+0000 7f6596ca9640 1 --2- 172.21.15.5:0/1735675345 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f659006d4f0 0x7f6590156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:23.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.491+0000 7f6596ca9640 1 -- 172.21.15.5:0/1735675345 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65900ccaa0 msgr2=0x7f6590154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:23.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.491+0000 7f6596ca9640 1 --2- 172.21.15.5:0/1735675345 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65900ccaa0 0x7f6590154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:23.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.491+0000 7f6596ca9640 1 -- 172.21.15.5:0/1735675345 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f658c009400 con 0x7f65901543a0 2024-03-24T10:55:23.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.493+0000 7f6596ca9640 1 --2- 172.21.15.5:0/1735675345 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65901543a0 0x7f65901f0db0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f658800cab0 tx=0x7f658800cf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:23.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.497+0000 7f6586ffd640 1 -- 172.21.15.5:0/1735675345 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658800abd0 con 0x7f65901543a0 2024-03-24T10:55:23.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.497+0000 7f6586ffd640 1 -- 172.21.15.5:0/1735675345 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f658800ad30 con 0x7f65901543a0 2024-03-24T10:55:23.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.497+0000 7f659870c640 1 -- 172.21.15.5:0/1735675345 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6590158fb0 con 0x7f65901543a0 2024-03-24T10:55:23.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.497+0000 7f6586ffd640 1 -- 172.21.15.5:0/1735675345 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6588012600 con 0x7f65901543a0 2024-03-24T10:55:23.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.497+0000 7f659870c640 1 -- 172.21.15.5:0/1735675345 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6590159570 con 0x7f65901543a0 2024-03-24T10:55:23.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.498+0000 7f659870c640 1 -- 172.21.15.5:0/1735675345 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f659015a050 con 0x7f65901543a0 2024-03-24T10:55:23.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.523+0000 7f6586ffd640 1 -- 172.21.15.5:0/1735675345 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6588007070 con 0x7f65901543a0 2024-03-24T10:55:23.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.524+0000 7f6586ffd640 1 --2- 172.21.15.5:0/1735675345 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f657406b860 0x7f657406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:23.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.524+0000 7f65964a8640 1 --2- 172.21.15.5:0/1735675345 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f657406b860 0x7f657406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:23.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.524+0000 7f65964a8640 1 --2- 172.21.15.5:0/1735675345 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f657406b860 0x7f657406dd20 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f658c036000 tx=0x7f658c008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:23.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.781+0000 7f6586ffd640 1 -- 172.21.15.5:0/1735675345 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f6588058ba0 con 0x7f65901543a0 2024-03-24T10:55:23.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:23.800+0000 7f6586ffd640 1 -- 172.21.15.5:0/1735675345 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6588054e40 con 0x7f65901543a0 2024-03-24T10:55:24.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:24.027+0000 7f659870c640 1 -- 172.21.15.5:0/1735675345 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f659015a050 con 0x7f65901543a0 2024-03-24T10:55:24.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:24.042+0000 7f6586ffd640 1 -- 172.21.15.5:0/1735675345 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f659015a050 con 0x7f65901543a0 2024-03-24T10:55:24.042 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:24.042 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:55:24.042 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:55:24.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:24.044+0000 7f659870c640 1 -- 172.21.15.5:0/1735675345 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f657406b860 msgr2=0x7f657406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:24.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:24.044+0000 7f659870c640 1 --2- 172.21.15.5:0/1735675345 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f657406b860 0x7f657406dd20 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f658c036000 tx=0x7f658c008030 comp rx=0 tx=0).stop 2024-03-24T10:55:24.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:24.044+0000 7f659870c640 1 -- 172.21.15.5:0/1735675345 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65901543a0 msgr2=0x7f65901f0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:24.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:24.044+0000 7f659870c640 1 --2- 172.21.15.5:0/1735675345 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65901543a0 0x7f65901f0db0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f658800cab0 tx=0x7f658800cf70 comp rx=0 tx=0).stop 2024-03-24T10:55:24.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:24.044+0000 7f659870c640 1 -- 172.21.15.5:0/1735675345 shutdown_connections 2024-03-24T10:55:24.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:24.044+0000 7f659870c640 1 --2- 172.21.15.5:0/1735675345 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f657406b860 0x7f657406dd20 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:24.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:24.044+0000 7f659870c640 1 --2- 172.21.15.5:0/1735675345 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f65900ccaa0 0x7f6590154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:24.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:24.044+0000 7f659870c640 1 --2- 172.21.15.5:0/1735675345 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f65901543a0 0x7f65901f0db0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:24.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:24.044+0000 7f659870c640 1 --2- 172.21.15.5:0/1735675345 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f659006d4f0 0x7f6590156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:24.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:24.044+0000 7f659870c640 1 -- 172.21.15.5:0/1735675345 >> 172.21.15.5:0/1735675345 conn(0x7f6590072ef0 msgr2=0x7f65900c4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:24.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:24.044+0000 7f659870c640 1 -- 172.21.15.5:0/1735675345 shutdown_connections 2024-03-24T10:55:24.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:24.044+0000 7f659870c640 1 -- 172.21.15.5:0/1735675345 wait complete. 2024-03-24T10:55:24.660 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:55:24.660 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:25.660 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:25.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.765+0000 7fc6149e9640 1 Processor -- start 2024-03-24T10:55:25.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.765+0000 7fc6149e9640 1 -- start start 2024-03-24T10:55:25.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.766+0000 7fc6149e9640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc6100cc8f0 0x7fc61015b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:25.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.766+0000 7fc6149e9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc61006c580 0x7fc6101538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:25.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.766+0000 7fc6149e9640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6100cd890 con 0x7fc6100cf110 2024-03-24T10:55:25.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.766+0000 7fc6149e9640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fc61006cae0 con 0x7fc6100cc8f0 2024-03-24T10:55:25.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.766+0000 7fc6149e9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc6100cdb50 con 0x7fc61006c580 2024-03-24T10:55:25.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.766+0000 7fc60e59c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc6100cc8f0 0x7fc61015b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:25.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.766+0000 7fc60dd9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc61006c580 0x7fc6101538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:25.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.767+0000 7fc60e59c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc6100cc8f0 0x7fc61015b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36182/0 (socket says 172.21.15.5:36182) 2024-03-24T10:55:25.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.767+0000 7fc60e59c640 1 -- 172.21.15.5:0/2664907290 learned_addr learned my addr 172.21.15.5:0/2664907290 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:25.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.769+0000 7fc60e59c640 1 -- 172.21.15.5:0/2664907290 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc61006c580 msgr2=0x7fc6101538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:25.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.769+0000 7fc60e59c640 1 --2- 172.21.15.5:0/2664907290 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc61006c580 0x7fc6101538b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:25.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.770+0000 7fc60e59c640 1 -- 172.21.15.5:0/2664907290 >> v1:172.21.15.5:6789/0 conn(0x7fc6100cf110 legacy=0x7fc610153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:25.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.770+0000 7fc60e59c640 1 -- 172.21.15.5:0/2664907290 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6100cde30 con 0x7fc6100cc8f0 2024-03-24T10:55:25.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.772+0000 7fc60e59c640 1 --2- 172.21.15.5:0/2664907290 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6100cc8f0 0x7fc61015b5d0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc600009870 tx=0x7fc60002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16864c763bf31683 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:25.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.779+0000 7fc60d59a640 1 -- 172.21.15.5:0/2664907290 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc600031640 con 0x7fc6100cc8f0 2024-03-24T10:55:25.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.779+0000 7fc60d59a640 1 -- 172.21.15.5:0/2664907290 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6000317a0 con 0x7fc6100cc8f0 2024-03-24T10:55:25.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.779+0000 7fc60d59a640 1 -- 172.21.15.5:0/2664907290 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc600031b30 con 0x7fc6100cc8f0 2024-03-24T10:55:25.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.779+0000 7fc6149e9640 1 -- 172.21.15.5:0/2664907290 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6100cc8f0 msgr2=0x7fc61015b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:25.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.779+0000 7fc6149e9640 1 --2- 172.21.15.5:0/2664907290 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6100cc8f0 0x7fc61015b5d0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc600009870 tx=0x7fc60002b8f0 comp rx=0 tx=0).stop 2024-03-24T10:55:25.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.780+0000 7fc6149e9640 1 -- 172.21.15.5:0/2664907290 shutdown_connections 2024-03-24T10:55:25.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.780+0000 7fc6149e9640 1 --2- 172.21.15.5:0/2664907290 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc61006c580 0x7fc6101538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:25.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.780+0000 7fc6149e9640 1 --2- 172.21.15.5:0/2664907290 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6100cc8f0 0x7fc61015b5d0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:25.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.780+0000 7fc6149e9640 1 -- 172.21.15.5:0/2664907290 >> 172.21.15.5:0/2664907290 conn(0x7fc6100b1820 msgr2=0x7fc6100b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:25.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.780+0000 7fc6149e9640 1 -- 172.21.15.5:0/2664907290 shutdown_connections 2024-03-24T10:55:25.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.780+0000 7fc6149e9640 1 -- 172.21.15.5:0/2664907290 wait complete. 2024-03-24T10:55:25.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.781+0000 7fc6149e9640 1 Processor -- start 2024-03-24T10:55:25.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.781+0000 7fc6149e9640 1 -- start start 2024-03-24T10:55:25.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.782+0000 7fc6149e9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc61006c580 0x7fc6101f0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:25.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.782+0000 7fc6149e9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc6100cc8f0 0x7fc610153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:25.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.782+0000 7fc6149e9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6100cf110 0x7fc6100c9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:25.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.782+0000 7fc6149e9640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fc6100cbe00 con 0x7fc6100cf110 2024-03-24T10:55:25.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.782+0000 7fc6149e9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc6101f0c50 con 0x7fc61006c580 2024-03-24T10:55:25.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.782+0000 7fc6149e9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc6101efc90 con 0x7fc6100cc8f0 2024-03-24T10:55:25.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.782+0000 7fc60ed9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6100cf110 0x7fc6100c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:25.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.782+0000 7fc60dd9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc6100cc8f0 0x7fc610153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:25.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.782+0000 7fc60e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc61006c580 0x7fc6101f0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:25.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.783+0000 7fc60ed9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6100cf110 0x7fc6100c9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36190/0 (socket says 172.21.15.5:36190) 2024-03-24T10:55:25.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.783+0000 7fc60ed9d640 1 -- 172.21.15.5:0/314097448 learned_addr learned my addr 172.21.15.5:0/314097448 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:25.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.786+0000 7fc60ed9d640 1 -- 172.21.15.5:0/314097448 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc6100cc8f0 msgr2=0x7fc610153170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:25.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.786+0000 7fc60ed9d640 1 --2- 172.21.15.5:0/314097448 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc6100cc8f0 0x7fc610153170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:25.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.786+0000 7fc60ed9d640 1 -- 172.21.15.5:0/314097448 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc61006c580 msgr2=0x7fc6101f0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:25.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.786+0000 7fc60ed9d640 1 --2- 172.21.15.5:0/314097448 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc61006c580 0x7fc6101f0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:25.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.786+0000 7fc60ed9d640 1 -- 172.21.15.5:0/314097448 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6000094d0 con 0x7fc6100cf110 2024-03-24T10:55:25.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.789+0000 7fc60ed9d640 1 --2- 172.21.15.5:0/314097448 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6100cf110 0x7fc6100c9470 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc60800b9a0 tx=0x7fc60800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:25.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.793+0000 7fc5feffd640 1 -- 172.21.15.5:0/314097448 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60800cb40 con 0x7fc6100cf110 2024-03-24T10:55:25.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.793+0000 7fc5feffd640 1 -- 172.21.15.5:0/314097448 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc60800cca0 con 0x7fc6100cf110 2024-03-24T10:55:25.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.793+0000 7fc6149e9640 1 -- 172.21.15.5:0/314097448 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc610148de0 con 0x7fc6100cf110 2024-03-24T10:55:25.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.793+0000 7fc5feffd640 1 -- 172.21.15.5:0/314097448 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc608014680 con 0x7fc6100cf110 2024-03-24T10:55:25.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.793+0000 7fc6149e9640 1 -- 172.21.15.5:0/314097448 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6100cc2c0 con 0x7fc6100cf110 2024-03-24T10:55:25.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.795+0000 7fc6149e9640 1 -- 172.21.15.5:0/314097448 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6101544c0 con 0x7fc6100cf110 2024-03-24T10:55:25.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.819+0000 7fc5feffd640 1 -- 172.21.15.5:0/314097448 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc608003970 con 0x7fc6100cf110 2024-03-24T10:55:25.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.820+0000 7fc5feffd640 1 --2- 172.21.15.5:0/314097448 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc5f006b860 0x7fc5f006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:25.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.820+0000 7fc60e59c640 1 --2- 172.21.15.5:0/314097448 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc5f006b860 0x7fc5f006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:25.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:25.820+0000 7fc60e59c640 1 --2- 172.21.15.5:0/314097448 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc5f006b860 0x7fc5f006dd20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc600036000 tx=0x7fc60003c040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:26.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:26.111+0000 7fc5feffd640 1 -- 172.21.15.5:0/314097448 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fc608058980 con 0x7fc6100cf110 2024-03-24T10:55:26.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:26.111+0000 7fc5feffd640 1 -- 172.21.15.5:0/314097448 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc608054c20 con 0x7fc6100cf110 2024-03-24T10:55:26.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:26.409+0000 7fc6149e9640 1 -- 172.21.15.5:0/314097448 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6100681d0 con 0x7fc6100cf110 2024-03-24T10:55:26.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:26.423+0000 7fc5feffd640 1 -- 172.21.15.5:0/314097448 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fc608019090 con 0x7fc6100cf110 2024-03-24T10:55:26.423 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:26.423 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:55:26.423 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:55:26.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:26.425+0000 7fc6149e9640 1 -- 172.21.15.5:0/314097448 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc5f006b860 msgr2=0x7fc5f006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:26.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:26.425+0000 7fc6149e9640 1 --2- 172.21.15.5:0/314097448 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc5f006b860 0x7fc5f006dd20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc600036000 tx=0x7fc60003c040 comp rx=0 tx=0).stop 2024-03-24T10:55:26.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:26.425+0000 7fc6149e9640 1 -- 172.21.15.5:0/314097448 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6100cf110 msgr2=0x7fc6100c9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:26.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:26.425+0000 7fc6149e9640 1 --2- 172.21.15.5:0/314097448 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6100cf110 0x7fc6100c9470 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc60800b9a0 tx=0x7fc60800be60 comp rx=0 tx=0).stop 2024-03-24T10:55:26.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:26.425+0000 7fc6149e9640 1 -- 172.21.15.5:0/314097448 shutdown_connections 2024-03-24T10:55:26.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:26.425+0000 7fc6149e9640 1 --2- 172.21.15.5:0/314097448 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc5f006b860 0x7fc5f006dd20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:26.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:26.425+0000 7fc6149e9640 1 --2- 172.21.15.5:0/314097448 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6100cf110 0x7fc6100c9470 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:26.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:26.425+0000 7fc6149e9640 1 --2- 172.21.15.5:0/314097448 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc6100cc8f0 0x7fc610153170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:26.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:26.425+0000 7fc6149e9640 1 --2- 172.21.15.5:0/314097448 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc61006c580 0x7fc6101f0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:26.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:26.425+0000 7fc6149e9640 1 -- 172.21.15.5:0/314097448 >> 172.21.15.5:0/314097448 conn(0x7fc6100b1820 msgr2=0x7fc610151c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:26.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:26.425+0000 7fc6149e9640 1 -- 172.21.15.5:0/314097448 shutdown_connections 2024-03-24T10:55:26.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:26.425+0000 7fc6149e9640 1 -- 172.21.15.5:0/314097448 wait complete. 2024-03-24T10:55:27.041 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:55:27.041 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:28.042 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:28.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.146+0000 7f82f8719640 1 Processor -- start 2024-03-24T10:55:28.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.146+0000 7f82f8719640 1 -- start start 2024-03-24T10:55:28.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.147+0000 7f82f8719640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f82f00cb270 0x7f82f00cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:28.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.147+0000 7f82f8719640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f82f00cd960 0x7f82f00cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:28.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.147+0000 7f82f8719640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82f00ccba0 con 0x7f82f00cae90 2024-03-24T10:55:28.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.147+0000 7f82f8719640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f82f00cc9b0 con 0x7f82f00cb270 2024-03-24T10:55:28.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.147+0000 7f82f8719640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f82f00cce40 con 0x7f82f00cd960 2024-03-24T10:55:28.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.147+0000 7f82f5cb4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f82f00cb270 0x7f82f00cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:28.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.148+0000 7f82f6cb6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f82f00cd960 0x7f82f00cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:28.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.148+0000 7f82f64b5640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f82f00cae90 0x7f82f00cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60584/0 (socket says 172.21.15.5:60584) 2024-03-24T10:55:28.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.148+0000 7f82f64b5640 1 -- 172.21.15.5:0/1328392667 learned_addr learned my addr 172.21.15.5:0/1328392667 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:55:28.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.151+0000 7f82f6cb6640 1 -- 172.21.15.5:0/1328392667 >> v2:172.21.15.57:3300/0 conn(0x7f82f00cb270 msgr2=0x7f82f00cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:28.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.151+0000 7f82f6cb6640 1 --2- 172.21.15.5:0/1328392667 >> v2:172.21.15.57:3300/0 conn(0x7f82f00cb270 0x7f82f00cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:28.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.151+0000 7f82f6cb6640 1 -- 172.21.15.5:0/1328392667 >> v1:172.21.15.5:6789/0 conn(0x7f82f00cae90 legacy=0x7f82f00cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:28.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.151+0000 7f82f6cb6640 1 -- 172.21.15.5:0/1328392667 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f015a590 con 0x7f82f00cd960 2024-03-24T10:55:28.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.154+0000 7f82f6cb6640 1 --2- 172.21.15.5:0/1328392667 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f82f00cd960 0x7f82f00cb6f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f82e8009d90 tx=0x7f82e802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=489ccf503b6b2e34 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:28.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.157+0000 7f82f54b3640 1 -- 172.21.15.5:0/1328392667 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e802cce0 con 0x7f82f00cd960 2024-03-24T10:55:28.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.161+0000 7f82f54b3640 1 -- 172.21.15.5:0/1328392667 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82e802ce40 con 0x7f82f00cd960 2024-03-24T10:55:28.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.161+0000 7f82f54b3640 1 -- 172.21.15.5:0/1328392667 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e8034660 con 0x7f82f00cd960 2024-03-24T10:55:28.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.162+0000 7f82f8719640 1 -- 172.21.15.5:0/1328392667 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f82f00cd960 msgr2=0x7f82f00cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:28.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.162+0000 7f82f8719640 1 --2- 172.21.15.5:0/1328392667 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f82f00cd960 0x7f82f00cb6f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f82e8009d90 tx=0x7f82e802bf30 comp rx=0 tx=0).stop 2024-03-24T10:55:28.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.162+0000 7f82f8719640 1 -- 172.21.15.5:0/1328392667 shutdown_connections 2024-03-24T10:55:28.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.162+0000 7f82f8719640 1 --2- 172.21.15.5:0/1328392667 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f82f00cd960 0x7f82f00cb6f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:28.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.162+0000 7f82f8719640 1 --2- 172.21.15.5:0/1328392667 >> v2:172.21.15.57:3300/0 conn(0x7f82f00cb270 0x7f82f00cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:28.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.162+0000 7f82f8719640 1 -- 172.21.15.5:0/1328392667 >> 172.21.15.5:0/1328392667 conn(0x7f82f00cfb80 msgr2=0x7f82f00cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:28.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.162+0000 7f82f8719640 1 -- 172.21.15.5:0/1328392667 shutdown_connections 2024-03-24T10:55:28.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.162+0000 7f82f8719640 1 -- 172.21.15.5:0/1328392667 wait complete. 2024-03-24T10:55:28.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.163+0000 7f82f8719640 1 Processor -- start 2024-03-24T10:55:28.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.163+0000 7f82f8719640 1 -- start start 2024-03-24T10:55:28.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.164+0000 7f82f8719640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f82f00cae90 0x7f82f0154020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:28.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.164+0000 7f82f8719640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82f00cb270 0x7f82f0155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:28.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.164+0000 7f82f8719640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f00cd960 0x7f82f00b0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:28.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.164+0000 7f82f8719640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f82f00cc620 con 0x7f82f00cb270 2024-03-24T10:55:28.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.164+0000 7f82f8719640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f82f0158bb0 con 0x7f82f00cd960 2024-03-24T10:55:28.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.164+0000 7f82f8719640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f82f0154560 con 0x7f82f00cae90 2024-03-24T10:55:28.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.165+0000 7f82f5cb4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82f00cb270 0x7f82f0155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:28.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.165+0000 7f82f6cb6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f00cd960 0x7f82f00b0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:28.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.165+0000 7f82f64b5640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f82f00cae90 0x7f82f0154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:28.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.165+0000 7f82f5cb4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82f00cb270 0x7f82f0155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36206/0 (socket says 172.21.15.5:36206) 2024-03-24T10:55:28.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.165+0000 7f82f5cb4640 1 -- 172.21.15.5:0/702625079 learned_addr learned my addr 172.21.15.5:0/702625079 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:28.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.169+0000 7f82f5cb4640 1 -- 172.21.15.5:0/702625079 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f82f00cae90 msgr2=0x7f82f0154020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:28.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.169+0000 7f82f5cb4640 1 --2- 172.21.15.5:0/702625079 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f82f00cae90 0x7f82f0154020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:28.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.169+0000 7f82f5cb4640 1 -- 172.21.15.5:0/702625079 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f00cd960 msgr2=0x7f82f00b0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:28.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.169+0000 7f82f5cb4640 1 --2- 172.21.15.5:0/702625079 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f00cd960 0x7f82f00b0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:28.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.169+0000 7f82f5cb4640 1 -- 172.21.15.5:0/702625079 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e8009a70 con 0x7f82f00cb270 2024-03-24T10:55:28.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.171+0000 7f82f5cb4640 1 --2- 172.21.15.5:0/702625079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82f00cb270 0x7f82f0155c60 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f82e000e9e0 tx=0x7f82e000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:28.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.176+0000 7f82e6ffd640 1 -- 172.21.15.5:0/702625079 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e0009670 con 0x7f82f00cb270 2024-03-24T10:55:28.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.176+0000 7f82e6ffd640 1 -- 172.21.15.5:0/702625079 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82e000c970 con 0x7f82f00cb270 2024-03-24T10:55:28.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.176+0000 7f82f8719640 1 -- 172.21.15.5:0/702625079 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82f01564a0 con 0x7f82f00cb270 2024-03-24T10:55:28.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.176+0000 7f82e6ffd640 1 -- 172.21.15.5:0/702625079 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e000cd20 con 0x7f82f00cb270 2024-03-24T10:55:28.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.176+0000 7f82f8719640 1 -- 172.21.15.5:0/702625079 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82f01f37f0 con 0x7f82f00cb270 2024-03-24T10:55:28.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.177+0000 7f82f8719640 1 -- 172.21.15.5:0/702625079 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82f00681d0 con 0x7f82f00cb270 2024-03-24T10:55:28.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.201+0000 7f82e6ffd640 1 -- 172.21.15.5:0/702625079 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f82e0013470 con 0x7f82f00cb270 2024-03-24T10:55:28.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.202+0000 7f82e6ffd640 1 --2- 172.21.15.5:0/702625079 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f82d406b860 0x7f82d406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:28.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.202+0000 7f82f64b5640 1 --2- 172.21.15.5:0/702625079 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f82d406b860 0x7f82d406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:28.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.202+0000 7f82f64b5640 1 --2- 172.21.15.5:0/702625079 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f82d406b860 0x7f82d406dd20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f82ec000f60 tx=0x7f82ec007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:28.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.205+0000 7f82e6ffd640 1 -- 172.21.15.5:0/702625079 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f82e000ce80 con 0x7f82f00cb270 2024-03-24T10:55:28.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.485+0000 7f82e6ffd640 1 -- 172.21.15.5:0/702625079 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f82e0055670 con 0x7f82f00cb270 2024-03-24T10:55:28.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.710+0000 7f82f8719640 1 -- 172.21.15.5:0/702625079 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82f006d7b0 con 0x7f82f00cb270 2024-03-24T10:55:28.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.757+0000 7f82e6ffd640 1 -- 172.21.15.5:0/702625079 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f82e0021020 con 0x7f82f00cb270 2024-03-24T10:55:28.757 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:28.757 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:55:28.758 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:55:28.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.759+0000 7f82f8719640 1 -- 172.21.15.5:0/702625079 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f82d406b860 msgr2=0x7f82d406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:28.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.759+0000 7f82f8719640 1 --2- 172.21.15.5:0/702625079 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f82d406b860 0x7f82d406dd20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f82ec000f60 tx=0x7f82ec007e80 comp rx=0 tx=0).stop 2024-03-24T10:55:28.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.759+0000 7f82f8719640 1 -- 172.21.15.5:0/702625079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82f00cb270 msgr2=0x7f82f0155c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:28.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.759+0000 7f82f8719640 1 --2- 172.21.15.5:0/702625079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82f00cb270 0x7f82f0155c60 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f82e000e9e0 tx=0x7f82e000eea0 comp rx=0 tx=0).stop 2024-03-24T10:55:28.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.759+0000 7f82f8719640 1 -- 172.21.15.5:0/702625079 shutdown_connections 2024-03-24T10:55:28.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.759+0000 7f82f8719640 1 --2- 172.21.15.5:0/702625079 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f82d406b860 0x7f82d406dd20 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:28.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.759+0000 7f82f8719640 1 --2- 172.21.15.5:0/702625079 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f82f00cb270 0x7f82f0155c60 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:28.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.759+0000 7f82f8719640 1 --2- 172.21.15.5:0/702625079 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f82f00cd960 0x7f82f00b0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:28.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.759+0000 7f82f8719640 1 --2- 172.21.15.5:0/702625079 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f82f00cae90 0x7f82f0154020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:28.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.759+0000 7f82f8719640 1 -- 172.21.15.5:0/702625079 >> 172.21.15.5:0/702625079 conn(0x7f82f00cfb80 msgr2=0x7f82f0159d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:28.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.759+0000 7f82f8719640 1 -- 172.21.15.5:0/702625079 shutdown_connections 2024-03-24T10:55:28.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:28.759+0000 7f82f8719640 1 -- 172.21.15.5:0/702625079 wait complete. 2024-03-24T10:55:29.376 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:55:29.376 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:30.377 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:30.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.480+0000 7f080d2af640 1 Processor -- start 2024-03-24T10:55:30.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.480+0000 7f080d2af640 1 -- start start 2024-03-24T10:55:30.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.481+0000 7f080d2af640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f08081543a0 0x7f080815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:30.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.481+0000 7f080d2af640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f080806bf50 0x7f080806c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:30.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.481+0000 7f080d2af640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08080cb2f0 con 0x7f080806d4f0 2024-03-24T10:55:30.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.481+0000 7f080d2af640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f08080c6750 con 0x7f08081543a0 2024-03-24T10:55:30.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.481+0000 7f080d2af640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f08080cc380 con 0x7f080806bf50 2024-03-24T10:55:30.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.482+0000 7f0806ffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f08081543a0 0x7f080815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:30.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.482+0000 7f08077fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f080806bf50 0x7f080806c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:30.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.482+0000 7f08067fc640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f080806d4f0 0x7f08080cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:60594/0 (socket says 172.21.15.5:60594) 2024-03-24T10:55:30.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.482+0000 7f08067fc640 1 -- 172.21.15.5:0/3495427627 learned_addr learned my addr 172.21.15.5:0/3495427627 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:55:30.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.485+0000 7f08077fe640 1 -- 172.21.15.5:0/3495427627 >> v2:172.21.15.5:3300/0 conn(0x7f08081543a0 msgr2=0x7f080815ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:30.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.485+0000 7f08077fe640 1 --2- 172.21.15.5:0/3495427627 >> v2:172.21.15.5:3300/0 conn(0x7f08081543a0 0x7f080815ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:30.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.485+0000 7f08077fe640 1 -- 172.21.15.5:0/3495427627 >> v1:172.21.15.5:6789/0 conn(0x7f080806d4f0 legacy=0x7f08080cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:30.485 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.485+0000 7f08077fe640 1 -- 172.21.15.5:0/3495427627 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08080ca7e0 con 0x7f080806bf50 2024-03-24T10:55:30.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.487+0000 7f08077fe640 1 --2- 172.21.15.5:0/3495427627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080806bf50 0x7f080806c330 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f08080b51c0 tx=0x7f080002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e69a2dd38e181e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:30.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.491+0000 7f0805ffb640 1 -- 172.21.15.5:0/3495427627 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080002cd00 con 0x7f080806bf50 2024-03-24T10:55:30.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.491+0000 7f0805ffb640 1 -- 172.21.15.5:0/3495427627 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f080002ce60 con 0x7f080806bf50 2024-03-24T10:55:30.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.491+0000 7f0805ffb640 1 -- 172.21.15.5:0/3495427627 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0800034660 con 0x7f080806bf50 2024-03-24T10:55:30.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.492+0000 7f080d2af640 1 -- 172.21.15.5:0/3495427627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080806bf50 msgr2=0x7f080806c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:30.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.492+0000 7f080d2af640 1 --2- 172.21.15.5:0/3495427627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080806bf50 0x7f080806c330 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f08080b51c0 tx=0x7f080002bf50 comp rx=0 tx=0).stop 2024-03-24T10:55:30.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.492+0000 7f080d2af640 1 -- 172.21.15.5:0/3495427627 shutdown_connections 2024-03-24T10:55:30.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.492+0000 7f080d2af640 1 --2- 172.21.15.5:0/3495427627 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080806bf50 0x7f080806c330 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:30.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.492+0000 7f080d2af640 1 --2- 172.21.15.5:0/3495427627 >> v2:172.21.15.5:3300/0 conn(0x7f08081543a0 0x7f080815ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:30.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.492+0000 7f080d2af640 1 -- 172.21.15.5:0/3495427627 >> 172.21.15.5:0/3495427627 conn(0x7f0808072ef0 msgr2=0x7f0808061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:30.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.492+0000 7f080d2af640 1 -- 172.21.15.5:0/3495427627 shutdown_connections 2024-03-24T10:55:30.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.492+0000 7f080d2af640 1 -- 172.21.15.5:0/3495427627 wait complete. 2024-03-24T10:55:30.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.493+0000 7f080d2af640 1 Processor -- start 2024-03-24T10:55:30.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.493+0000 7f080d2af640 1 -- start start 2024-03-24T10:55:30.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.494+0000 7f080d2af640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f080806bf50 0x7f08081f3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:30.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.494+0000 7f080d2af640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080806d4f0 0x7f080814e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:30.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.494+0000 7f080d2af640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08081543a0 0x7f08080cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:30.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.494+0000 7f080d2af640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f080814f5f0 con 0x7f08081543a0 2024-03-24T10:55:30.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.494+0000 7f080d2af640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0808153610 con 0x7f080806d4f0 2024-03-24T10:55:30.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.494+0000 7f080d2af640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0808153050 con 0x7f080806bf50 2024-03-24T10:55:30.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.495+0000 7f08067fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080806d4f0 0x7f080814e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:30.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.495+0000 7f0806ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f080806bf50 0x7f08081f3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:30.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.495+0000 7f08077fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08081543a0 0x7f08080cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:30.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.495+0000 7f08067fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080806d4f0 0x7f080814e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:48674/0 (socket says 172.21.15.5:48674) 2024-03-24T10:55:30.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.495+0000 7f08067fc640 1 -- 172.21.15.5:0/2714232181 learned_addr learned my addr 172.21.15.5:0/2714232181 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:30.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.499+0000 7f08077fe640 1 -- 172.21.15.5:0/2714232181 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f080806bf50 msgr2=0x7f08081f3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:30.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.499+0000 7f08077fe640 1 --2- 172.21.15.5:0/2714232181 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f080806bf50 0x7f08081f3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:30.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.499+0000 7f08077fe640 1 -- 172.21.15.5:0/2714232181 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080806d4f0 msgr2=0x7f080814e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:30.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.499+0000 7f08077fe640 1 --2- 172.21.15.5:0/2714232181 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080806d4f0 0x7f080814e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:30.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.499+0000 7f08077fe640 1 -- 172.21.15.5:0/2714232181 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0800009a70 con 0x7f08081543a0 2024-03-24T10:55:30.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.501+0000 7f08077fe640 1 --2- 172.21.15.5:0/2714232181 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08081543a0 0x7f08080cf1d0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f080815a0f0 tx=0x7f0800004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:30.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.505+0000 7f07ef7fe640 1 -- 172.21.15.5:0/2714232181 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080003e610 con 0x7f08081543a0 2024-03-24T10:55:30.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.505+0000 7f07ef7fe640 1 -- 172.21.15.5:0/2714232181 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f080003e770 con 0x7f08081543a0 2024-03-24T10:55:30.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.505+0000 7f080d2af640 1 -- 172.21.15.5:0/2714232181 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0808153330 con 0x7f08081543a0 2024-03-24T10:55:30.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.505+0000 7f07ef7fe640 1 -- 172.21.15.5:0/2714232181 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f080003eb20 con 0x7f08081543a0 2024-03-24T10:55:30.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.505+0000 7f080d2af640 1 -- 172.21.15.5:0/2714232181 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f080814f210 con 0x7f08081543a0 2024-03-24T10:55:30.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.506+0000 7f080d2af640 1 -- 172.21.15.5:0/2714232181 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f080814f730 con 0x7f08081543a0 2024-03-24T10:55:30.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.525+0000 7f07ef7fe640 1 -- 172.21.15.5:0/2714232181 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f080003b050 con 0x7f08081543a0 2024-03-24T10:55:30.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.526+0000 7f07ef7fe640 1 --2- 172.21.15.5:0/2714232181 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f07f406b810 0x7f07f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:30.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.526+0000 7f0806ffd640 1 --2- 172.21.15.5:0/2714232181 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f07f406b810 0x7f07f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:30.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.527+0000 7f0806ffd640 1 --2- 172.21.15.5:0/2714232181 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f07f406b810 0x7f07f406dcd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f07f8009aa0 tx=0x7f07f8009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:30.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.785+0000 7f07ef7fe640 1 -- 172.21.15.5:0/2714232181 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f080007b430 con 0x7f08081543a0 2024-03-24T10:55:30.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:30.785+0000 7f07ef7fe640 1 -- 172.21.15.5:0/2714232181 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f08000776d0 con 0x7f08081543a0 2024-03-24T10:55:31.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:31.014+0000 7f080d2af640 1 -- 172.21.15.5:0/2714232181 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f080814f730 con 0x7f08081543a0 2024-03-24T10:55:31.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:31.028+0000 7f07ef7fe640 1 -- 172.21.15.5:0/2714232181 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f080814f730 con 0x7f08081543a0 2024-03-24T10:55:31.029 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:31.029 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:55:31.029 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:55:31.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:31.031+0000 7f080d2af640 1 -- 172.21.15.5:0/2714232181 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f07f406b810 msgr2=0x7f07f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:31.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:31.031+0000 7f080d2af640 1 --2- 172.21.15.5:0/2714232181 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f07f406b810 0x7f07f406dcd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f07f8009aa0 tx=0x7f07f8009190 comp rx=0 tx=0).stop 2024-03-24T10:55:31.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:31.031+0000 7f080d2af640 1 -- 172.21.15.5:0/2714232181 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08081543a0 msgr2=0x7f08080cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:31.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:31.031+0000 7f080d2af640 1 --2- 172.21.15.5:0/2714232181 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08081543a0 0x7f08080cf1d0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f080815a0f0 tx=0x7f0800004a40 comp rx=0 tx=0).stop 2024-03-24T10:55:31.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:31.031+0000 7f080d2af640 1 -- 172.21.15.5:0/2714232181 shutdown_connections 2024-03-24T10:55:31.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:31.031+0000 7f080d2af640 1 --2- 172.21.15.5:0/2714232181 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f07f406b810 0x7f07f406dcd0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:31.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:31.031+0000 7f080d2af640 1 --2- 172.21.15.5:0/2714232181 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f08081543a0 0x7f08080cf1d0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:31.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:31.031+0000 7f080d2af640 1 --2- 172.21.15.5:0/2714232181 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f080806d4f0 0x7f080814e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:31.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:31.031+0000 7f080d2af640 1 --2- 172.21.15.5:0/2714232181 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f080806bf50 0x7f08081f3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:31.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:31.031+0000 7f080d2af640 1 -- 172.21.15.5:0/2714232181 >> 172.21.15.5:0/2714232181 conn(0x7f0808072ef0 msgr2=0x7f08080cc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:31.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:31.031+0000 7f080d2af640 1 -- 172.21.15.5:0/2714232181 shutdown_connections 2024-03-24T10:55:31.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:31.031+0000 7f080d2af640 1 -- 172.21.15.5:0/2714232181 wait complete. 2024-03-24T10:55:31.645 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:55:31.645 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:32.607 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:55:32.609 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:55:32.646 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:32.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.752+0000 7fa9b4313640 1 Processor -- start 2024-03-24T10:55:32.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.752+0000 7fa9b4313640 1 -- start start 2024-03-24T10:55:32.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.753+0000 7fa9b4313640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa9ac15ac00 0x7fa9ac0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:32.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.753+0000 7fa9b4313640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa9ac0ccaa0 0x7fa9ac06bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:32.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.753+0000 7fa9b4313640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9ac0cb2f0 con 0x7fa9ac1543a0 2024-03-24T10:55:32.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.753+0000 7fa9b4313640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fa9ac0c6750 con 0x7fa9ac15ac00 2024-03-24T10:55:32.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.753+0000 7fa9b4313640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fa9ac0cc380 con 0x7fa9ac0ccaa0 2024-03-24T10:55:32.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.754+0000 7fa9b18ae640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa9ac15ac00 0x7fa9ac0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:32.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.754+0000 7fa9b28b0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa9ac0ccaa0 0x7fa9ac06bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:32.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.754+0000 7fa9b18ae640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa9ac15ac00 0x7fa9ac0cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:37496/0 (socket says 172.21.15.5:37496) 2024-03-24T10:55:32.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.754+0000 7fa9b18ae640 1 -- 172.21.15.5:0/1264386427 learned_addr learned my addr 172.21.15.5:0/1264386427 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:32.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.757+0000 7fa9b18ae640 1 -- 172.21.15.5:0/1264386427 >> v2:172.21.15.57:3300/0 conn(0x7fa9ac0ccaa0 msgr2=0x7fa9ac06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:32.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.757+0000 7fa9b18ae640 1 --2- 172.21.15.5:0/1264386427 >> v2:172.21.15.57:3300/0 conn(0x7fa9ac0ccaa0 0x7fa9ac06bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:32.757 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.757+0000 7fa9b18ae640 1 -- 172.21.15.5:0/1264386427 >> v1:172.21.15.57:6789/0 conn(0x7fa9ac1543a0 legacy=0x7fa9ac06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:32.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.757+0000 7fa9b18ae640 1 -- 172.21.15.5:0/1264386427 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9ac15b070 con 0x7fa9ac15ac00 2024-03-24T10:55:32.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.760+0000 7fa9b18ae640 1 --2- 172.21.15.5:0/1264386427 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa9ac15ac00 0x7fa9ac0cc560 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0b51c0 tx=0x7fa99c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=886a9ba17c76f8a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:32.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.765+0000 7fa9b10ad640 1 -- 172.21.15.5:0/1264386427 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99c030860 con 0x7fa9ac15ac00 2024-03-24T10:55:32.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.765+0000 7fa9b10ad640 1 -- 172.21.15.5:0/1264386427 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa99c0309c0 con 0x7fa9ac15ac00 2024-03-24T10:55:32.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.765+0000 7fa9b10ad640 1 -- 172.21.15.5:0/1264386427 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99c030d50 con 0x7fa9ac15ac00 2024-03-24T10:55:32.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.766+0000 7fa9b4313640 1 -- 172.21.15.5:0/1264386427 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa9ac15ac00 msgr2=0x7fa9ac0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:32.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.766+0000 7fa9b4313640 1 --2- 172.21.15.5:0/1264386427 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa9ac15ac00 0x7fa9ac0cc560 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0b51c0 tx=0x7fa99c02b9a0 comp rx=0 tx=0).stop 2024-03-24T10:55:32.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.766+0000 7fa9b4313640 1 -- 172.21.15.5:0/1264386427 shutdown_connections 2024-03-24T10:55:32.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.766+0000 7fa9b4313640 1 --2- 172.21.15.5:0/1264386427 >> v2:172.21.15.57:3300/0 conn(0x7fa9ac0ccaa0 0x7fa9ac06bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:32.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.766+0000 7fa9b4313640 1 --2- 172.21.15.5:0/1264386427 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa9ac15ac00 0x7fa9ac0cc560 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:32.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.766+0000 7fa9b4313640 1 -- 172.21.15.5:0/1264386427 >> 172.21.15.5:0/1264386427 conn(0x7fa9ac072ef0 msgr2=0x7fa9ac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:32.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.766+0000 7fa9b4313640 1 -- 172.21.15.5:0/1264386427 shutdown_connections 2024-03-24T10:55:32.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.766+0000 7fa9b4313640 1 -- 172.21.15.5:0/1264386427 wait complete. 2024-03-24T10:55:32.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.767+0000 7fa9b4313640 1 Processor -- start 2024-03-24T10:55:32.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.767+0000 7fa9b4313640 1 -- start start 2024-03-24T10:55:32.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.768+0000 7fa9b4313640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa9ac0ccaa0 0x7fa9ac0b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:32.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.768+0000 7fa9b4313640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9ac1543a0 0x7fa9ac15b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:32.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.768+0000 7fa9b4313640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa9ac15ac00 0x7fa9ac1568f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:32.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.768+0000 7fa9b4313640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ac156e30 con 0x7fa9ac15ac00 2024-03-24T10:55:32.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.768+0000 7fa9b4313640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ac1f03c0 con 0x7fa9ac1543a0 2024-03-24T10:55:32.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.768+0000 7fa9b4313640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fa9ac1f08f0 con 0x7fa9ac0ccaa0 2024-03-24T10:55:32.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.769+0000 7fa9b28b0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa9ac15ac00 0x7fa9ac1568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:32.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.769+0000 7fa9b18ae640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9ac1543a0 0x7fa9ac15b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:32.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.769+0000 7fa9b20af640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa9ac0ccaa0 0x7fa9ac0b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:32.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.769+0000 7fa9b28b0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa9ac15ac00 0x7fa9ac1568f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:37508/0 (socket says 172.21.15.5:37508) 2024-03-24T10:55:32.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.769+0000 7fa9b28b0640 1 -- 172.21.15.5:0/3020566741 learned_addr learned my addr 172.21.15.5:0/3020566741 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:32.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.774+0000 7fa9b28b0640 1 -- 172.21.15.5:0/3020566741 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa9ac0ccaa0 msgr2=0x7fa9ac0b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:32.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.774+0000 7fa9b28b0640 1 --2- 172.21.15.5:0/3020566741 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa9ac0ccaa0 0x7fa9ac0b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:32.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.774+0000 7fa9b28b0640 1 -- 172.21.15.5:0/3020566741 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9ac1543a0 msgr2=0x7fa9ac15b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:32.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.774+0000 7fa9b28b0640 1 --2- 172.21.15.5:0/3020566741 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9ac1543a0 0x7fa9ac15b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:32.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.774+0000 7fa9b28b0640 1 -- 172.21.15.5:0/3020566741 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa99c0094d0 con 0x7fa9ac15ac00 2024-03-24T10:55:32.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.774+0000 7fa9b18ae640 1 --2- 172.21.15.5:0/3020566741 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9ac1543a0 0x7fa9ac15b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:55:32.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.777+0000 7fa9b28b0640 1 --2- 172.21.15.5:0/3020566741 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa9ac15ac00 0x7fa9ac1568f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa9a400efc0 tx=0x7fa9a400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:32.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.781+0000 7fa9a2ffd640 1 -- 172.21.15.5:0/3020566741 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4014480 con 0x7fa9ac15ac00 2024-03-24T10:55:32.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.781+0000 7fa9a2ffd640 1 -- 172.21.15.5:0/3020566741 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9a400f040 con 0x7fa9ac15ac00 2024-03-24T10:55:32.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.781+0000 7fa9b4313640 1 -- 172.21.15.5:0/3020566741 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9ac153b80 con 0x7fa9ac15ac00 2024-03-24T10:55:32.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.781+0000 7fa9a2ffd640 1 -- 172.21.15.5:0/3020566741 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4014820 con 0x7fa9ac15ac00 2024-03-24T10:55:32.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.781+0000 7fa9b4313640 1 -- 172.21.15.5:0/3020566741 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9ac155e10 con 0x7fa9ac15ac00 2024-03-24T10:55:32.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:32.782+0000 7fa9b4313640 1 -- 172.21.15.5:0/3020566741 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9ac0681d0 con 0x7fa9ac15ac00 2024-03-24T10:55:33.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:33.062+0000 7fa9a2ffd640 1 -- 172.21.15.5:0/3020566741 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa9a4014980 con 0x7fa9ac15ac00 2024-03-24T10:55:33.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:33.062+0000 7fa9a2ffd640 1 --2- 172.21.15.5:0/3020566741 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa99006b810 0x7fa99006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:33.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:33.062+0000 7fa9b20af640 1 --2- 172.21.15.5:0/3020566741 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa99006b810 0x7fa99006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:33.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:33.063+0000 7fa9b20af640 1 --2- 172.21.15.5:0/3020566741 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa99006b810 0x7fa99006dcd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8001780 tx=0x7fa9a8009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:33.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:33.087+0000 7fa9a2ffd640 1 -- 172.21.15.5:0/3020566741 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fa9a4059d00 con 0x7fa9ac15ac00 2024-03-24T10:55:33.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:33.087+0000 7fa9a2ffd640 1 -- 172.21.15.5:0/3020566741 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa9a4055fa0 con 0x7fa9ac15ac00 2024-03-24T10:55:33.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:33.315+0000 7fa9b4313640 1 -- 172.21.15.5:0/3020566741 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9ac0c4f50 con 0x7fa9ac15ac00 2024-03-24T10:55:33.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:33.329+0000 7fa9a2ffd640 1 -- 172.21.15.5:0/3020566741 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fa9a4018090 con 0x7fa9ac15ac00 2024-03-24T10:55:33.329 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:33.329 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:55:33.329 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:55:33.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:33.331+0000 7fa9b4313640 1 -- 172.21.15.5:0/3020566741 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa99006b810 msgr2=0x7fa99006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:33.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:33.331+0000 7fa9b4313640 1 --2- 172.21.15.5:0/3020566741 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa99006b810 0x7fa99006dcd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8001780 tx=0x7fa9a8009040 comp rx=0 tx=0).stop 2024-03-24T10:55:33.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:33.331+0000 7fa9b4313640 1 -- 172.21.15.5:0/3020566741 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa9ac15ac00 msgr2=0x7fa9ac1568f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:33.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:33.331+0000 7fa9b4313640 1 --2- 172.21.15.5:0/3020566741 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa9ac15ac00 0x7fa9ac1568f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa9a400efc0 tx=0x7fa9a400c490 comp rx=0 tx=0).stop 2024-03-24T10:55:33.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:33.331+0000 7fa9b4313640 1 -- 172.21.15.5:0/3020566741 shutdown_connections 2024-03-24T10:55:33.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:33.331+0000 7fa9b4313640 1 --2- 172.21.15.5:0/3020566741 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa99006b810 0x7fa99006dcd0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:33.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:33.331+0000 7fa9b4313640 1 --2- 172.21.15.5:0/3020566741 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa9ac15ac00 0x7fa9ac1568f0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:33.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:33.331+0000 7fa9b4313640 1 --2- 172.21.15.5:0/3020566741 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa9ac1543a0 0x7fa9ac15b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:33.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:33.331+0000 7fa9b4313640 1 --2- 172.21.15.5:0/3020566741 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa9ac0ccaa0 0x7fa9ac0b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:33.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:33.331+0000 7fa9b4313640 1 -- 172.21.15.5:0/3020566741 >> 172.21.15.5:0/3020566741 conn(0x7fa9ac072ef0 msgr2=0x7fa9ac14a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:33.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:33.332+0000 7fa9b4313640 1 -- 172.21.15.5:0/3020566741 shutdown_connections 2024-03-24T10:55:33.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:33.332+0000 7fa9b4313640 1 -- 172.21.15.5:0/3020566741 wait complete. 2024-03-24T10:55:33.951 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:55:33.951 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:34.952 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:35.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.055+0000 7fce2cd01640 1 Processor -- start 2024-03-24T10:55:35.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.055+0000 7fce2cd01640 1 -- start start 2024-03-24T10:55:35.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.056+0000 7fce2cd01640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fce280cc8f0 0x7fce2815b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:35.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.056+0000 7fce2cd01640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fce2806c580 0x7fce281538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:35.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.056+0000 7fce2cd01640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce280cd890 con 0x7fce280cf110 2024-03-24T10:55:35.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.056+0000 7fce2cd01640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fce2806cae0 con 0x7fce2806c580 2024-03-24T10:55:35.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.056+0000 7fce2cd01640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fce280cdb50 con 0x7fce280cc8f0 2024-03-24T10:55:35.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.056+0000 7fce2659c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fce280cc8f0 0x7fce2815b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:35.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.057+0000 7fce1ffff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fce2806c580 0x7fce281538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:35.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.057+0000 7fce1ffff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fce2806c580 0x7fce281538b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:37510/0 (socket says 172.21.15.5:37510) 2024-03-24T10:55:35.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.057+0000 7fce1ffff640 1 -- 172.21.15.5:0/1731039493 learned_addr learned my addr 172.21.15.5:0/1731039493 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:35.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.059+0000 7fce1ffff640 1 -- 172.21.15.5:0/1731039493 >> v2:172.21.15.57:3300/0 conn(0x7fce280cc8f0 msgr2=0x7fce2815b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:35.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.059+0000 7fce1ffff640 1 --2- 172.21.15.5:0/1731039493 >> v2:172.21.15.57:3300/0 conn(0x7fce280cc8f0 0x7fce2815b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:35.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.059+0000 7fce1ffff640 1 -- 172.21.15.5:0/1731039493 >> v1:172.21.15.57:6789/0 conn(0x7fce280cf110 legacy=0x7fce28153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:35.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.059+0000 7fce1ffff640 1 -- 172.21.15.5:0/1731039493 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce280cde30 con 0x7fce2806c580 2024-03-24T10:55:35.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.061+0000 7fce1ffff640 1 --2- 172.21.15.5:0/1731039493 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fce2806c580 0x7fce281538b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fce280cd030 tx=0x7fce1002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f38f365cb20769 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:35.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.066+0000 7fce25d9b640 1 -- 172.21.15.5:0/1731039493 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1002f850 con 0x7fce2806c580 2024-03-24T10:55:35.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.066+0000 7fce25d9b640 1 -- 172.21.15.5:0/1731039493 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce1002f9b0 con 0x7fce2806c580 2024-03-24T10:55:35.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.066+0000 7fce25d9b640 1 -- 172.21.15.5:0/1731039493 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1002fd40 con 0x7fce2806c580 2024-03-24T10:55:35.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.067+0000 7fce2cd01640 1 -- 172.21.15.5:0/1731039493 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fce2806c580 msgr2=0x7fce281538b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:35.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.067+0000 7fce2cd01640 1 --2- 172.21.15.5:0/1731039493 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fce2806c580 0x7fce281538b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fce280cd030 tx=0x7fce1002ba30 comp rx=0 tx=0).stop 2024-03-24T10:55:35.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.067+0000 7fce2cd01640 1 -- 172.21.15.5:0/1731039493 shutdown_connections 2024-03-24T10:55:35.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.067+0000 7fce2cd01640 1 --2- 172.21.15.5:0/1731039493 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fce2806c580 0x7fce281538b0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:35.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.067+0000 7fce2cd01640 1 --2- 172.21.15.5:0/1731039493 >> v2:172.21.15.57:3300/0 conn(0x7fce280cc8f0 0x7fce2815b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:35.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.067+0000 7fce2cd01640 1 -- 172.21.15.5:0/1731039493 >> 172.21.15.5:0/1731039493 conn(0x7fce280b1820 msgr2=0x7fce280b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:35.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.067+0000 7fce2cd01640 1 -- 172.21.15.5:0/1731039493 shutdown_connections 2024-03-24T10:55:35.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.067+0000 7fce2cd01640 1 -- 172.21.15.5:0/1731039493 wait complete. 2024-03-24T10:55:35.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.069+0000 7fce2cd01640 1 Processor -- start 2024-03-24T10:55:35.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.069+0000 7fce2cd01640 1 -- start start 2024-03-24T10:55:35.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.070+0000 7fce2cd01640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fce2806c580 0x7fce280c5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:35.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.070+0000 7fce2cd01640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce280cc8f0 0x7fce281e5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:35.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.070+0000 7fce2cd01640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fce280cf110 0x7fce2814bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:35.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.070+0000 7fce2cd01640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fce280c46f0 con 0x7fce280cf110 2024-03-24T10:55:35.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.070+0000 7fce2cd01640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fce280cb5d0 con 0x7fce280cc8f0 2024-03-24T10:55:35.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.070+0000 7fce2cd01640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fce280c5970 con 0x7fce2806c580 2024-03-24T10:55:35.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.071+0000 7fce26d9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fce280cf110 0x7fce2814bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:35.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.071+0000 7fce1ffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce280cc8f0 0x7fce281e5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:35.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.071+0000 7fce26d9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fce280cf110 0x7fce2814bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:37526/0 (socket says 172.21.15.5:37526) 2024-03-24T10:55:35.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.071+0000 7fce26d9d640 1 -- 172.21.15.5:0/562892953 learned_addr learned my addr 172.21.15.5:0/562892953 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:35.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.071+0000 7fce2659c640 1 --2- 172.21.15.5:0/562892953 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fce2806c580 0x7fce280c5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:35.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.074+0000 7fce26d9d640 1 -- 172.21.15.5:0/562892953 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fce2806c580 msgr2=0x7fce280c5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:35.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.074+0000 7fce26d9d640 1 --2- 172.21.15.5:0/562892953 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fce2806c580 0x7fce280c5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:35.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.074+0000 7fce26d9d640 1 -- 172.21.15.5:0/562892953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce280cc8f0 msgr2=0x7fce281e5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:35.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.074+0000 7fce26d9d640 1 --2- 172.21.15.5:0/562892953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce280cc8f0 0x7fce281e5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:35.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.074+0000 7fce26d9d640 1 -- 172.21.15.5:0/562892953 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce100094d0 con 0x7fce280cf110 2024-03-24T10:55:35.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.077+0000 7fce26d9d640 1 --2- 172.21.15.5:0/562892953 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fce280cf110 0x7fce2814bc80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fce2000c6b0 tx=0x7fce2000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:35.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.082+0000 7fce1effd640 1 -- 172.21.15.5:0/562892953 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce20003b70 con 0x7fce280cf110 2024-03-24T10:55:35.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.082+0000 7fce1effd640 1 -- 172.21.15.5:0/562892953 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce20003cf0 con 0x7fce280cf110 2024-03-24T10:55:35.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.082+0000 7fce1effd640 1 -- 172.21.15.5:0/562892953 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce20003ea0 con 0x7fce280cf110 2024-03-24T10:55:35.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.082+0000 7fce2cd01640 1 -- 172.21.15.5:0/562892953 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce280cbb40 con 0x7fce280cf110 2024-03-24T10:55:35.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.082+0000 7fce2cd01640 1 -- 172.21.15.5:0/562892953 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce280c4c30 con 0x7fce280cf110 2024-03-24T10:55:35.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.083+0000 7fce2cd01640 1 -- 172.21.15.5:0/562892953 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce280cee60 con 0x7fce280cf110 2024-03-24T10:55:35.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.368+0000 7fce1effd640 1 -- 172.21.15.5:0/562892953 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fce20005310 con 0x7fce280cf110 2024-03-24T10:55:35.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.368+0000 7fce1effd640 1 --2- 172.21.15.5:0/562892953 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fce0806b860 0x7fce0806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:35.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.368+0000 7fce2659c640 1 --2- 172.21.15.5:0/562892953 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fce0806b860 0x7fce0806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:35.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.369+0000 7fce2659c640 1 --2- 172.21.15.5:0/562892953 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fce0806b860 0x7fce0806dd20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fce18009b40 tx=0x7fce18009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:35.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.384+0000 7fce1effd640 1 -- 172.21.15.5:0/562892953 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fce20059180 con 0x7fce280cf110 2024-03-24T10:55:35.384 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.384+0000 7fce1effd640 1 -- 172.21.15.5:0/562892953 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fce20055420 con 0x7fce280cf110 2024-03-24T10:55:35.657 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.657+0000 7fce2cd01640 1 -- 172.21.15.5:0/562892953 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce280732f0 con 0x7fce280cf110 2024-03-24T10:55:35.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.672+0000 7fce1effd640 1 -- 172.21.15.5:0/562892953 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7fce20058ae0 con 0x7fce280cf110 2024-03-24T10:55:35.672 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:35.672 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:55:35.673 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:55:35.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.674+0000 7fce2cd01640 1 -- 172.21.15.5:0/562892953 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fce0806b860 msgr2=0x7fce0806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:35.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.674+0000 7fce2cd01640 1 --2- 172.21.15.5:0/562892953 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fce0806b860 0x7fce0806dd20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fce18009b40 tx=0x7fce18009190 comp rx=0 tx=0).stop 2024-03-24T10:55:35.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.674+0000 7fce2cd01640 1 -- 172.21.15.5:0/562892953 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fce280cf110 msgr2=0x7fce2814bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:35.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.674+0000 7fce2cd01640 1 --2- 172.21.15.5:0/562892953 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fce280cf110 0x7fce2814bc80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fce2000c6b0 tx=0x7fce2000cb70 comp rx=0 tx=0).stop 2024-03-24T10:55:35.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.674+0000 7fce2cd01640 1 -- 172.21.15.5:0/562892953 shutdown_connections 2024-03-24T10:55:35.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.674+0000 7fce2cd01640 1 --2- 172.21.15.5:0/562892953 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fce0806b860 0x7fce0806dd20 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:35.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.674+0000 7fce2cd01640 1 --2- 172.21.15.5:0/562892953 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fce280cf110 0x7fce2814bc80 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:35.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.674+0000 7fce2cd01640 1 --2- 172.21.15.5:0/562892953 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fce280cc8f0 0x7fce281e5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:35.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.674+0000 7fce2cd01640 1 --2- 172.21.15.5:0/562892953 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fce2806c580 0x7fce280c5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:35.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.674+0000 7fce2cd01640 1 -- 172.21.15.5:0/562892953 >> 172.21.15.5:0/562892953 conn(0x7fce280b1820 msgr2=0x7fce2806bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:35.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.674+0000 7fce2cd01640 1 -- 172.21.15.5:0/562892953 shutdown_connections 2024-03-24T10:55:35.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:35.674+0000 7fce2cd01640 1 -- 172.21.15.5:0/562892953 wait complete. 2024-03-24T10:55:36.289 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:55:36.290 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:37.290 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:37.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.394+0000 7f3c98d9a640 1 Processor -- start 2024-03-24T10:55:37.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.395+0000 7f3c98d9a640 1 -- start start 2024-03-24T10:55:37.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.395+0000 7f3c98d9a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3c941543a0 0x7f3c9415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:37.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.395+0000 7f3c98d9a640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c940cb2f0 con 0x7f3c9406d4f0 2024-03-24T10:55:37.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.395+0000 7f3c98d9a640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c940ca7e0 con 0x7f3c9406bf50 2024-03-24T10:55:37.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.395+0000 7f3c98d9a640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f3c940c6750 con 0x7f3c941543a0 2024-03-24T10:55:37.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.396+0000 7f3c8bfff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f3c9406d4f0 0x7f3c940cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:34030/0 (socket says 172.21.15.5:34030) 2024-03-24T10:55:37.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.396+0000 7f3c8bfff640 1 -- 172.21.15.5:0/710877177 learned_addr learned my addr 172.21.15.5:0/710877177 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:55:37.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.396+0000 7f3c9259c640 1 --2- 172.21.15.5:0/710877177 >> v2:172.21.15.5:3300/0 conn(0x7f3c941543a0 0x7f3c9415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:37.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.400+0000 7f3c91d9b640 1 -- 172.21.15.5:0/710877177 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 923796848 0 0) 0x7f3c940cb2f0 con 0x7f3c9406d4f0 2024-03-24T10:55:37.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.400+0000 7f3c91d9b640 1 -- 172.21.15.5:0/710877177 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c80003520 con 0x7f3c9406d4f0 2024-03-24T10:55:37.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.402+0000 7f3c9259c640 1 -- 172.21.15.5:0/710877177 >> v1:172.21.15.57:6789/0 conn(0x7f3c9406bf50 legacy=0x7f3c9406c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:37.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.402+0000 7f3c9259c640 1 -- 172.21.15.5:0/710877177 >> v1:172.21.15.5:6789/0 conn(0x7f3c9406d4f0 legacy=0x7f3c940cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:37.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.402+0000 7f3c91d9b640 1 -- 172.21.15.5:0/710877177 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2714374097 0 0) 0x7f3c940ca7e0 con 0x7f3c9406bf50 2024-03-24T10:55:37.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.402+0000 7f3c9259c640 1 -- 172.21.15.5:0/710877177 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c940cb520 con 0x7f3c941543a0 2024-03-24T10:55:37.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.405+0000 7f3c9259c640 1 --2- 172.21.15.5:0/710877177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c941543a0 0x7f3c9415ac00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3c840097a0 tx=0x7f3c8402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca675318e1673055 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:37.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.411+0000 7f3c91d9b640 1 -- 172.21.15.5:0/710877177 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8402f810 con 0x7f3c941543a0 2024-03-24T10:55:37.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.411+0000 7f3c91d9b640 1 -- 172.21.15.5:0/710877177 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c8402f970 con 0x7f3c941543a0 2024-03-24T10:55:37.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.411+0000 7f3c91d9b640 1 -- 172.21.15.5:0/710877177 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8402fd00 con 0x7f3c941543a0 2024-03-24T10:55:37.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.412+0000 7f3c98d9a640 1 -- 172.21.15.5:0/710877177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c941543a0 msgr2=0x7f3c9415ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:37.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.412+0000 7f3c98d9a640 1 --2- 172.21.15.5:0/710877177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c941543a0 0x7f3c9415ac00 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f3c840097a0 tx=0x7f3c8402b9f0 comp rx=0 tx=0).stop 2024-03-24T10:55:37.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.412+0000 7f3c98d9a640 1 -- 172.21.15.5:0/710877177 shutdown_connections 2024-03-24T10:55:37.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.412+0000 7f3c98d9a640 1 --2- 172.21.15.5:0/710877177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c941543a0 0x7f3c9415ac00 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:37.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.412+0000 7f3c98d9a640 1 -- 172.21.15.5:0/710877177 >> 172.21.15.5:0/710877177 conn(0x7f3c94072ef0 msgr2=0x7f3c94061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:37.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.412+0000 7f3c98d9a640 1 -- 172.21.15.5:0/710877177 shutdown_connections 2024-03-24T10:55:37.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.412+0000 7f3c98d9a640 1 -- 172.21.15.5:0/710877177 wait complete. 2024-03-24T10:55:37.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.414+0000 7f3c98d9a640 1 Processor -- start 2024-03-24T10:55:37.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.414+0000 7f3c98d9a640 1 -- start start 2024-03-24T10:55:37.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.414+0000 7f3c98d9a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c9406bf50 0x7f3c94158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:37.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.414+0000 7f3c98d9a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c9406d4f0 0x7f3c94158a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:37.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.414+0000 7f3c98d9a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3c941543a0 0x7f3c94157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:37.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.414+0000 7f3c98d9a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f3c94158090 con 0x7f3c9406bf50 2024-03-24T10:55:37.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.414+0000 7f3c98d9a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3c94151640 con 0x7f3c9406d4f0 2024-03-24T10:55:37.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.414+0000 7f3c98d9a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f3c94150ac0 con 0x7f3c941543a0 2024-03-24T10:55:37.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.415+0000 7f3c9259c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c9406bf50 0x7f3c94158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:37.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.415+0000 7f3c8bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c9406d4f0 0x7f3c94158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:37.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.415+0000 7f3c92d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3c941543a0 0x7f3c94157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:37.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.415+0000 7f3c9259c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c9406bf50 0x7f3c94158520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:37542/0 (socket says 172.21.15.5:37542) 2024-03-24T10:55:37.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.415+0000 7f3c9259c640 1 -- 172.21.15.5:0/1792179758 learned_addr learned my addr 172.21.15.5:0/1792179758 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:37.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.419+0000 7f3c92d9d640 1 -- 172.21.15.5:0/1792179758 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c9406d4f0 msgr2=0x7f3c94158a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:37.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.419+0000 7f3c92d9d640 1 --2- 172.21.15.5:0/1792179758 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c9406d4f0 0x7f3c94158a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:37.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.419+0000 7f3c92d9d640 1 -- 172.21.15.5:0/1792179758 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c9406bf50 msgr2=0x7f3c94158520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:37.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.419+0000 7f3c92d9d640 1 --2- 172.21.15.5:0/1792179758 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c9406bf50 0x7f3c94158520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:37.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.419+0000 7f3c92d9d640 1 -- 172.21.15.5:0/1792179758 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c84009400 con 0x7f3c941543a0 2024-03-24T10:55:37.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.422+0000 7f3c92d9d640 1 --2- 172.21.15.5:0/1792179758 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3c941543a0 0x7f3c94157020 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00c6a0 tx=0x7f3c8c00cb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:37.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.426+0000 7f3c8affd640 1 -- 172.21.15.5:0/1792179758 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c003b70 con 0x7f3c941543a0 2024-03-24T10:55:37.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.426+0000 7f3c8affd640 1 -- 172.21.15.5:0/1792179758 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c8c003cf0 con 0x7f3c941543a0 2024-03-24T10:55:37.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.426+0000 7f3c8affd640 1 -- 172.21.15.5:0/1792179758 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8c0052b0 con 0x7f3c941543a0 2024-03-24T10:55:37.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.426+0000 7f3c98d9a640 1 -- 172.21.15.5:0/1792179758 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c94150da0 con 0x7f3c941543a0 2024-03-24T10:55:37.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.426+0000 7f3c98d9a640 1 -- 172.21.15.5:0/1792179758 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c94151360 con 0x7f3c941543a0 2024-03-24T10:55:37.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.428+0000 7f3c98d9a640 1 -- 172.21.15.5:0/1792179758 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c94158240 con 0x7f3c941543a0 2024-03-24T10:55:37.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.447+0000 7f3c8affd640 1 -- 172.21.15.5:0/1792179758 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3c8c005410 con 0x7f3c941543a0 2024-03-24T10:55:37.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.448+0000 7f3c8affd640 1 --2- 172.21.15.5:0/1792179758 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3c7406b890 0x7f3c7406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:37.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.448+0000 7f3c9259c640 1 --2- 172.21.15.5:0/1792179758 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3c7406b890 0x7f3c7406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:37.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.448+0000 7f3c9259c640 1 --2- 172.21.15.5:0/1792179758 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3c7406b890 0x7f3c7406dd50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3c8402ba30 tx=0x7f3c84008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:37.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.451+0000 7f3c8affd640 1 -- 172.21.15.5:0/1792179758 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f3c8c058f70 con 0x7f3c941543a0 2024-03-24T10:55:37.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.709+0000 7f3c8affd640 1 -- 172.21.15.5:0/1792179758 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3c8c055210 con 0x7f3c941543a0 2024-03-24T10:55:37.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:37.935+0000 7f3c98d9a640 1 -- 172.21.15.5:0/1792179758 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c94158200 con 0x7f3c941543a0 2024-03-24T10:55:38.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:38.024+0000 7f3c8affd640 1 -- 172.21.15.5:0/1792179758 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f3c8c003e60 con 0x7f3c941543a0 2024-03-24T10:55:38.025 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:38.025 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:55:38.025 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:55:38.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:38.026+0000 7f3c98d9a640 1 -- 172.21.15.5:0/1792179758 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3c7406b890 msgr2=0x7f3c7406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:38.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:38.026+0000 7f3c98d9a640 1 --2- 172.21.15.5:0/1792179758 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3c7406b890 0x7f3c7406dd50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3c8402ba30 tx=0x7f3c84008030 comp rx=0 tx=0).stop 2024-03-24T10:55:38.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:38.026+0000 7f3c98d9a640 1 -- 172.21.15.5:0/1792179758 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3c941543a0 msgr2=0x7f3c94157020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:38.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:38.026+0000 7f3c98d9a640 1 --2- 172.21.15.5:0/1792179758 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3c941543a0 0x7f3c94157020 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c00c6a0 tx=0x7f3c8c00cb60 comp rx=0 tx=0).stop 2024-03-24T10:55:38.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:38.026+0000 7f3c98d9a640 1 -- 172.21.15.5:0/1792179758 shutdown_connections 2024-03-24T10:55:38.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:38.026+0000 7f3c98d9a640 1 --2- 172.21.15.5:0/1792179758 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3c7406b890 0x7f3c7406dd50 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:38.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:38.027+0000 7f3c98d9a640 1 --2- 172.21.15.5:0/1792179758 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3c941543a0 0x7f3c94157020 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:38.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:38.027+0000 7f3c98d9a640 1 --2- 172.21.15.5:0/1792179758 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3c9406d4f0 0x7f3c94158a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:38.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:38.027+0000 7f3c98d9a640 1 --2- 172.21.15.5:0/1792179758 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3c9406bf50 0x7f3c94158520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:38.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:38.027+0000 7f3c98d9a640 1 -- 172.21.15.5:0/1792179758 >> 172.21.15.5:0/1792179758 conn(0x7f3c94072ef0 msgr2=0x7f3c941522d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:38.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:38.027+0000 7f3c98d9a640 1 -- 172.21.15.5:0/1792179758 shutdown_connections 2024-03-24T10:55:38.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:38.027+0000 7f3c98d9a640 1 -- 172.21.15.5:0/1792179758 wait complete. 2024-03-24T10:55:38.646 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:55:38.646 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:39.647 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:39.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.749+0000 7f0c32bfd640 1 Processor -- start 2024-03-24T10:55:39.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.749+0000 7f0c32bfd640 1 -- start start 2024-03-24T10:55:39.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.750+0000 7f0c32bfd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0c2c15ac00 0x7f0c2c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:39.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.750+0000 7f0c32bfd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c2c0ccaa0 0x7f0c2c06bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:39.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.750+0000 7f0c32bfd640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c2c0cb2f0 con 0x7f0c2c1543a0 2024-03-24T10:55:39.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.750+0000 7f0c32bfd640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f0c2c0c6750 con 0x7f0c2c15ac00 2024-03-24T10:55:39.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.750+0000 7f0c32bfd640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0c2c0cceb0 con 0x7f0c2c0ccaa0 2024-03-24T10:55:39.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.751+0000 7f0c2bfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0c2c15ac00 0x7f0c2c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:39.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.751+0000 7f0c3119a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c2c0ccaa0 0x7f0c2c06bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:39.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.752+0000 7f0c3119a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c2c0ccaa0 0x7f0c2c06bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:36954/0 (socket says 172.21.15.5:36954) 2024-03-24T10:55:39.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.752+0000 7f0c3119a640 1 -- 172.21.15.5:0/3549955689 learned_addr learned my addr 172.21.15.5:0/3549955689 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:39.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.754+0000 7f0c3119a640 1 -- 172.21.15.5:0/3549955689 >> v2:172.21.15.57:3300/0 conn(0x7f0c2c15ac00 msgr2=0x7f0c2c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:39.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.754+0000 7f0c3119a640 1 --2- 172.21.15.5:0/3549955689 >> v2:172.21.15.57:3300/0 conn(0x7f0c2c15ac00 0x7f0c2c0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:39.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.754+0000 7f0c3119a640 1 -- 172.21.15.5:0/3549955689 >> v1:172.21.15.57:6789/0 conn(0x7f0c2c1543a0 legacy=0x7f0c2c06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:39.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.754+0000 7f0c3119a640 1 -- 172.21.15.5:0/3549955689 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c2c15b070 con 0x7f0c2c0ccaa0 2024-03-24T10:55:39.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.756+0000 7f0c3119a640 1 --2- 172.21.15.5:0/3549955689 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c2c0ccaa0 0x7f0c2c06bf50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0c24009e50 tx=0x7f0c2402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1df03eb897c6ecf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:39.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.764+0000 7f0c2b7fe640 1 -- 172.21.15.5:0/3549955689 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2402cb00 con 0x7f0c2c0ccaa0 2024-03-24T10:55:39.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.764+0000 7f0c2b7fe640 1 -- 172.21.15.5:0/3549955689 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c2402cc60 con 0x7f0c2c0ccaa0 2024-03-24T10:55:39.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.764+0000 7f0c2b7fe640 1 -- 172.21.15.5:0/3549955689 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c240343f0 con 0x7f0c2c0ccaa0 2024-03-24T10:55:39.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.765+0000 7f0c32bfd640 1 -- 172.21.15.5:0/3549955689 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c2c0ccaa0 msgr2=0x7f0c2c06bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:39.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.765+0000 7f0c32bfd640 1 --2- 172.21.15.5:0/3549955689 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c2c0ccaa0 0x7f0c2c06bf50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0c24009e50 tx=0x7f0c2402bf60 comp rx=0 tx=0).stop 2024-03-24T10:55:39.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.765+0000 7f0c32bfd640 1 -- 172.21.15.5:0/3549955689 shutdown_connections 2024-03-24T10:55:39.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.765+0000 7f0c32bfd640 1 --2- 172.21.15.5:0/3549955689 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c2c0ccaa0 0x7f0c2c06bf50 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:39.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.765+0000 7f0c32bfd640 1 --2- 172.21.15.5:0/3549955689 >> v2:172.21.15.57:3300/0 conn(0x7f0c2c15ac00 0x7f0c2c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:39.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.765+0000 7f0c32bfd640 1 -- 172.21.15.5:0/3549955689 >> 172.21.15.5:0/3549955689 conn(0x7f0c2c072ef0 msgr2=0x7f0c2c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:39.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.765+0000 7f0c32bfd640 1 -- 172.21.15.5:0/3549955689 shutdown_connections 2024-03-24T10:55:39.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.765+0000 7f0c32bfd640 1 -- 172.21.15.5:0/3549955689 wait complete. 2024-03-24T10:55:39.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.766+0000 7f0c32bfd640 1 Processor -- start 2024-03-24T10:55:39.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.766+0000 7f0c32bfd640 1 -- start start 2024-03-24T10:55:39.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.767+0000 7f0c32bfd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c2c0ccaa0 0x7f0c2c1ef840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:39.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.767+0000 7f0c32bfd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c2c1543a0 0x7f0c2c0b4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:39.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.767+0000 7f0c32bfd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c2c15ac00 0x7f0c2c159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:39.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.767+0000 7f0c32bfd640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f0c2c0b4560 con 0x7f0c2c1543a0 2024-03-24T10:55:39.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.767+0000 7f0c32bfd640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0c2c0b39e0 con 0x7f0c2c0ccaa0 2024-03-24T10:55:39.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.767+0000 7f0c32bfd640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0c2c0b3cc0 con 0x7f0c2c15ac00 2024-03-24T10:55:39.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.767+0000 7f0c3119a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c2c15ac00 0x7f0c2c159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:39.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.767+0000 7f0c2bfff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c2c1543a0 0x7f0c2c0b4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:39.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.767+0000 7f0c30999640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c2c0ccaa0 0x7f0c2c1ef840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:39.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.768+0000 7f0c3119a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c2c15ac00 0x7f0c2c159150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:36960/0 (socket says 172.21.15.5:36960) 2024-03-24T10:55:39.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.768+0000 7f0c3119a640 1 -- 172.21.15.5:0/1928698466 learned_addr learned my addr 172.21.15.5:0/1928698466 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:39.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.771+0000 7f0c3119a640 1 -- 172.21.15.5:0/1928698466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c2c0ccaa0 msgr2=0x7f0c2c1ef840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:39.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.771+0000 7f0c3119a640 1 --2- 172.21.15.5:0/1928698466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c2c0ccaa0 0x7f0c2c1ef840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:39.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.771+0000 7f0c3119a640 1 -- 172.21.15.5:0/1928698466 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c2c1543a0 msgr2=0x7f0c2c0b4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:39.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.771+0000 7f0c3119a640 1 --2- 172.21.15.5:0/1928698466 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c2c1543a0 0x7f0c2c0b4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:39.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.771+0000 7f0c3119a640 1 -- 172.21.15.5:0/1928698466 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c24009b30 con 0x7f0c2c15ac00 2024-03-24T10:55:39.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.774+0000 7f0c3119a640 1 --2- 172.21.15.5:0/1928698466 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c2c15ac00 0x7f0c2c159150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0c24036040 tx=0x7f0c24004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:39.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.778+0000 7f0c297fa640 1 -- 172.21.15.5:0/1928698466 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c24009510 con 0x7f0c2c15ac00 2024-03-24T10:55:39.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.778+0000 7f0c297fa640 1 -- 172.21.15.5:0/1928698466 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c2403e450 con 0x7f0c2c15ac00 2024-03-24T10:55:39.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.778+0000 7f0c297fa640 1 -- 172.21.15.5:0/1928698466 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c2403e800 con 0x7f0c2c15ac00 2024-03-24T10:55:39.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.778+0000 7f0c32bfd640 1 -- 172.21.15.5:0/1928698466 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c2c0b3fa0 con 0x7f0c2c15ac00 2024-03-24T10:55:39.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.778+0000 7f0c32bfd640 1 -- 172.21.15.5:0/1928698466 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c2c158c30 con 0x7f0c2c15ac00 2024-03-24T10:55:39.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.779+0000 7f0c32bfd640 1 -- 172.21.15.5:0/1928698466 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c2c158a90 con 0x7f0c2c15ac00 2024-03-24T10:55:39.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.804+0000 7f0c297fa640 1 -- 172.21.15.5:0/1928698466 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0c2403e960 con 0x7f0c2c15ac00 2024-03-24T10:55:39.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.804+0000 7f0c297fa640 1 --2- 172.21.15.5:0/1928698466 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0c1006b810 0x7f0c1006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:39.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.805+0000 7f0c30999640 1 --2- 172.21.15.5:0/1928698466 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0c1006b810 0x7f0c1006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:39.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:39.805+0000 7f0c30999640 1 --2- 172.21.15.5:0/1928698466 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0c1006b810 0x7f0c1006dcd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0c20001780 tx=0x7f0c20009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:40.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:40.069+0000 7f0c297fa640 1 -- 172.21.15.5:0/1928698466 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f0c2407b860 con 0x7f0c2c15ac00 2024-03-24T10:55:40.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:40.080+0000 7f0c297fa640 1 -- 172.21.15.5:0/1928698466 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0c24077b00 con 0x7f0c2c15ac00 2024-03-24T10:55:40.381 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:40.381+0000 7f0c32bfd640 1 -- 172.21.15.5:0/1928698466 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c2c158a90 con 0x7f0c2c15ac00 2024-03-24T10:55:40.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:40.398+0000 7f0c297fa640 1 -- 172.21.15.5:0/1928698466 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6258 (secure 0 0 0) 0x7f0c2c158a90 con 0x7f0c2c15ac00 2024-03-24T10:55:40.399 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:40.399 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:46:42.995574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:42.995573+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:55:40.399 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 22 2024-03-24T10:55:40.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:40.400+0000 7f0c32bfd640 1 -- 172.21.15.5:0/1928698466 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0c1006b810 msgr2=0x7f0c1006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:40.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:40.400+0000 7f0c32bfd640 1 --2- 172.21.15.5:0/1928698466 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0c1006b810 0x7f0c1006dcd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0c20001780 tx=0x7f0c20009040 comp rx=0 tx=0).stop 2024-03-24T10:55:40.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:40.400+0000 7f0c32bfd640 1 -- 172.21.15.5:0/1928698466 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c2c15ac00 msgr2=0x7f0c2c159150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:40.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:40.400+0000 7f0c32bfd640 1 --2- 172.21.15.5:0/1928698466 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c2c15ac00 0x7f0c2c159150 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0c24036040 tx=0x7f0c24004a40 comp rx=0 tx=0).stop 2024-03-24T10:55:40.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:40.401+0000 7f0c32bfd640 1 -- 172.21.15.5:0/1928698466 shutdown_connections 2024-03-24T10:55:40.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:40.401+0000 7f0c32bfd640 1 --2- 172.21.15.5:0/1928698466 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0c1006b810 0x7f0c1006dcd0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:40.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:40.401+0000 7f0c32bfd640 1 --2- 172.21.15.5:0/1928698466 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0c2c1543a0 0x7f0c2c0b4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:40.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:40.401+0000 7f0c32bfd640 1 --2- 172.21.15.5:0/1928698466 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0c2c15ac00 0x7f0c2c159150 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:40.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:40.401+0000 7f0c32bfd640 1 --2- 172.21.15.5:0/1928698466 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0c2c0ccaa0 0x7f0c2c1ef840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:40.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:40.401+0000 7f0c32bfd640 1 -- 172.21.15.5:0/1928698466 >> 172.21.15.5:0/1928698466 conn(0x7f0c2c072ef0 msgr2=0x7f0c2c0c5740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:40.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:40.401+0000 7f0c32bfd640 1 -- 172.21.15.5:0/1928698466 shutdown_connections 2024-03-24T10:55:40.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:40.401+0000 7f0c32bfd640 1 -- 172.21.15.5:0/1928698466 wait complete. 2024-03-24T10:55:41.017 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:46:42.995574+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:42.995573+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:55:41.017 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:42.019 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:42.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.121+0000 7fe367864640 1 Processor -- start 2024-03-24T10:55:42.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.121+0000 7fe367864640 1 -- start start 2024-03-24T10:55:42.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.122+0000 7fe367864640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe3601543a0 0x7fe36015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:42.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.122+0000 7fe367864640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe36006d4f0 0x7fe3600cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:42.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.122+0000 7fe367864640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3600cb2f0 con 0x7fe3600ccaa0 2024-03-24T10:55:42.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.122+0000 7fe367864640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fe3600c6750 con 0x7fe3601543a0 2024-03-24T10:55:42.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.122+0000 7fe367864640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe3600cc380 con 0x7fe36006d4f0 2024-03-24T10:55:42.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.122+0000 7fe365600640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe3601543a0 0x7fe36015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:42.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.122+0000 7fe364dff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe36006d4f0 0x7fe3600cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:42.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.122+0000 7fe365600640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe3601543a0 0x7fe36015ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:37554/0 (socket says 172.21.15.5:37554) 2024-03-24T10:55:42.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.122+0000 7fe365600640 1 -- 172.21.15.5:0/4112119652 learned_addr learned my addr 172.21.15.5:0/4112119652 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:42.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.125+0000 7fe365600640 1 -- 172.21.15.5:0/4112119652 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe36006d4f0 msgr2=0x7fe3600cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:42.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.125+0000 7fe365600640 1 --2- 172.21.15.5:0/4112119652 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe36006d4f0 0x7fe3600cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:42.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.125+0000 7fe365600640 1 -- 172.21.15.5:0/4112119652 >> v1:172.21.15.57:6789/0 conn(0x7fe3600ccaa0 legacy=0x7fe36006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:42.125 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.125+0000 7fe365600640 1 -- 172.21.15.5:0/4112119652 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3600ca7e0 con 0x7fe3601543a0 2024-03-24T10:55:42.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.127+0000 7fe365600640 1 --2- 172.21.15.5:0/4112119652 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3601543a0 0x7fe36015ac00 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe35c009870 tx=0x7fe35c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=202c368aecb147d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:42.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.131+0000 7fe357fff640 1 -- 172.21.15.5:0/4112119652 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c02f840 con 0x7fe3601543a0 2024-03-24T10:55:42.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.132+0000 7fe357fff640 1 -- 172.21.15.5:0/4112119652 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe35c02f9a0 con 0x7fe3601543a0 2024-03-24T10:55:42.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.132+0000 7fe357fff640 1 -- 172.21.15.5:0/4112119652 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c02fd30 con 0x7fe3601543a0 2024-03-24T10:55:42.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.132+0000 7fe367864640 1 -- 172.21.15.5:0/4112119652 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3601543a0 msgr2=0x7fe36015ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:42.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.132+0000 7fe367864640 1 --2- 172.21.15.5:0/4112119652 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3601543a0 0x7fe36015ac00 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe35c009870 tx=0x7fe35c02ba20 comp rx=0 tx=0).stop 2024-03-24T10:55:42.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.133+0000 7fe367864640 1 -- 172.21.15.5:0/4112119652 shutdown_connections 2024-03-24T10:55:42.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.133+0000 7fe367864640 1 --2- 172.21.15.5:0/4112119652 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe36006d4f0 0x7fe3600cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:42.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.133+0000 7fe367864640 1 --2- 172.21.15.5:0/4112119652 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3601543a0 0x7fe36015ac00 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:42.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.133+0000 7fe367864640 1 -- 172.21.15.5:0/4112119652 >> 172.21.15.5:0/4112119652 conn(0x7fe360072ef0 msgr2=0x7fe360061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:42.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.133+0000 7fe367864640 1 -- 172.21.15.5:0/4112119652 shutdown_connections 2024-03-24T10:55:42.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.133+0000 7fe367864640 1 -- 172.21.15.5:0/4112119652 wait complete. 2024-03-24T10:55:42.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.135+0000 7fe367864640 1 Processor -- start 2024-03-24T10:55:42.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.135+0000 7fe367864640 1 -- start start 2024-03-24T10:55:42.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.136+0000 7fe367864640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe36006d4f0 0x7fe360155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:42.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.136+0000 7fe367864640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3600ccaa0 0x7fe360157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:42.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.136+0000 7fe367864640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3601543a0 0x7fe3601575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:42.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.136+0000 7fe367864640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fe360155c90 con 0x7fe3600ccaa0 2024-03-24T10:55:42.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.136+0000 7fe367864640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe360156be0 con 0x7fe3601543a0 2024-03-24T10:55:42.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.136+0000 7fe367864640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe360145050 con 0x7fe36006d4f0 2024-03-24T10:55:42.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.136+0000 7fe364dff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3600ccaa0 0x7fe360157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:42.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.137+0000 7fe365600640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe36006d4f0 0x7fe360155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:42.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.137+0000 7fe365e01640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3601543a0 0x7fe3601575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:42.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.137+0000 7fe364dff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3600ccaa0 0x7fe360157070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:37564/0 (socket says 172.21.15.5:37564) 2024-03-24T10:55:42.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.137+0000 7fe364dff640 1 -- 172.21.15.5:0/997082577 learned_addr learned my addr 172.21.15.5:0/997082577 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:42.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.140+0000 7fe364dff640 1 -- 172.21.15.5:0/997082577 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe36006d4f0 msgr2=0x7fe360155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:42.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.140+0000 7fe364dff640 1 --2- 172.21.15.5:0/997082577 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe36006d4f0 0x7fe360155750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:42.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.140+0000 7fe364dff640 1 -- 172.21.15.5:0/997082577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3601543a0 msgr2=0x7fe3601575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:42.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.140+0000 7fe364dff640 1 --2- 172.21.15.5:0/997082577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3601543a0 0x7fe3601575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:42.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.140+0000 7fe364dff640 1 -- 172.21.15.5:0/997082577 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe35c0094d0 con 0x7fe3600ccaa0 2024-03-24T10:55:42.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.143+0000 7fe364dff640 1 --2- 172.21.15.5:0/997082577 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3600ccaa0 0x7fe360157070 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe35000e9b0 tx=0x7fe35000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:42.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.147+0000 7fe355ffb640 1 -- 172.21.15.5:0/997082577 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe350009670 con 0x7fe3600ccaa0 2024-03-24T10:55:42.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.147+0000 7fe355ffb640 1 -- 172.21.15.5:0/997082577 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe35000c9a0 con 0x7fe3600ccaa0 2024-03-24T10:55:42.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.147+0000 7fe355ffb640 1 -- 172.21.15.5:0/997082577 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35000cd50 con 0x7fe3600ccaa0 2024-03-24T10:55:42.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.147+0000 7fe367864640 1 -- 172.21.15.5:0/997082577 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe360156320 con 0x7fe3600ccaa0 2024-03-24T10:55:42.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.148+0000 7fe367864640 1 -- 172.21.15.5:0/997082577 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe360156590 con 0x7fe3600ccaa0 2024-03-24T10:55:42.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.148+0000 7fe367864640 1 -- 172.21.15.5:0/997082577 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3600681d0 con 0x7fe3600ccaa0 2024-03-24T10:55:42.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.416+0000 7fe355ffb640 1 -- 172.21.15.5:0/997082577 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe350013470 con 0x7fe3600ccaa0 2024-03-24T10:55:42.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.417+0000 7fe355ffb640 1 --2- 172.21.15.5:0/997082577 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe34406b810 0x7fe34406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:42.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.417+0000 7fe365600640 1 --2- 172.21.15.5:0/997082577 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe34406b810 0x7fe34406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:42.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.417+0000 7fe365600640 1 --2- 172.21.15.5:0/997082577 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe34406b810 0x7fe34406dcd0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe35c036000 tx=0x7fe35c008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:42.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.430+0000 7fe355ffb640 1 -- 172.21.15.5:0/997082577 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fe3500593b0 con 0x7fe3600ccaa0 2024-03-24T10:55:42.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.430+0000 7fe355ffb640 1 -- 172.21.15.5:0/997082577 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe350055650 con 0x7fe3600ccaa0 2024-03-24T10:55:42.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.681+0000 7fe367864640 1 -- 172.21.15.5:0/997082577 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe360156d50 con 0x7fe3600ccaa0 2024-03-24T10:55:42.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.704+0000 7fe355ffb640 1 -- 172.21.15.5:0/997082577 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7fe350021090 con 0x7fe3600ccaa0 2024-03-24T10:55:42.704 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:42.705 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:55:42.705 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:55:42.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.706+0000 7fe367864640 1 -- 172.21.15.5:0/997082577 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe34406b810 msgr2=0x7fe34406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:42.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.706+0000 7fe367864640 1 --2- 172.21.15.5:0/997082577 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe34406b810 0x7fe34406dcd0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe35c036000 tx=0x7fe35c008030 comp rx=0 tx=0).stop 2024-03-24T10:55:42.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.706+0000 7fe367864640 1 -- 172.21.15.5:0/997082577 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3600ccaa0 msgr2=0x7fe360157070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:42.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.706+0000 7fe367864640 1 --2- 172.21.15.5:0/997082577 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3600ccaa0 0x7fe360157070 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe35000e9b0 tx=0x7fe35000ee70 comp rx=0 tx=0).stop 2024-03-24T10:55:42.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.706+0000 7fe367864640 1 -- 172.21.15.5:0/997082577 shutdown_connections 2024-03-24T10:55:42.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.706+0000 7fe367864640 1 --2- 172.21.15.5:0/997082577 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe34406b810 0x7fe34406dcd0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:42.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.706+0000 7fe367864640 1 --2- 172.21.15.5:0/997082577 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe3600ccaa0 0x7fe360157070 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:42.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.706+0000 7fe367864640 1 --2- 172.21.15.5:0/997082577 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3601543a0 0x7fe3601575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:42.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.706+0000 7fe367864640 1 --2- 172.21.15.5:0/997082577 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe36006d4f0 0x7fe360155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:42.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.706+0000 7fe367864640 1 -- 172.21.15.5:0/997082577 >> 172.21.15.5:0/997082577 conn(0x7fe360072ef0 msgr2=0x7fe3600cabe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:42.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.707+0000 7fe367864640 1 -- 172.21.15.5:0/997082577 shutdown_connections 2024-03-24T10:55:42.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:42.707+0000 7fe367864640 1 -- 172.21.15.5:0/997082577 wait complete. 2024-03-24T10:55:43.323 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:55:43.323 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:44.324 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:44.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.426+0000 7f33965f7640 1 Processor -- start 2024-03-24T10:55:44.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.426+0000 7f33965f7640 1 -- start start 2024-03-24T10:55:44.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.427+0000 7f33965f7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f33900cb270 0x7f33900cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:44.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.427+0000 7f33965f7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33900cd960 0x7f33900cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:44.427 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.427+0000 7f33965f7640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33900ccba0 con 0x7f33900cae90 2024-03-24T10:55:44.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.427+0000 7f33965f7640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f33900cc9b0 con 0x7f33900cb270 2024-03-24T10:55:44.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.427+0000 7f33965f7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f33900cce40 con 0x7f33900cd960 2024-03-24T10:55:44.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.428+0000 7f3394b94640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33900cd960 0x7f33900cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:44.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.428+0000 7f338f7fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f33900cb270 0x7f33900cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:44.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.428+0000 7f338ffff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f33900cae90 0x7f33900cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:41372/0 (socket says 172.21.15.5:41372) 2024-03-24T10:55:44.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.428+0000 7f338ffff640 1 -- 172.21.15.5:0/1497410974 learned_addr learned my addr 172.21.15.5:0/1497410974 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:55:44.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.432+0000 7f3394b94640 1 -- 172.21.15.5:0/1497410974 >> v2:172.21.15.5:3300/0 conn(0x7f33900cb270 msgr2=0x7f33900cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:44.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.432+0000 7f3394b94640 1 --2- 172.21.15.5:0/1497410974 >> v2:172.21.15.5:3300/0 conn(0x7f33900cb270 0x7f33900cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:44.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.432+0000 7f3394b94640 1 -- 172.21.15.5:0/1497410974 >> v1:172.21.15.5:6789/0 conn(0x7f33900cae90 legacy=0x7f33900cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:44.432 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.432+0000 7f3394b94640 1 -- 172.21.15.5:0/1497410974 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f339015a590 con 0x7f33900cd960 2024-03-24T10:55:44.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.435+0000 7f3394b94640 1 --2- 172.21.15.5:0/1497410974 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33900cd960 0x7f33900cb6f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3388009d90 tx=0x7f338802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cef0e15314f720c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:44.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.440+0000 7f338effd640 1 -- 172.21.15.5:0/1497410974 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f338802cce0 con 0x7f33900cd960 2024-03-24T10:55:44.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.440+0000 7f338effd640 1 -- 172.21.15.5:0/1497410974 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f338802ce40 con 0x7f33900cd960 2024-03-24T10:55:44.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.440+0000 7f338effd640 1 -- 172.21.15.5:0/1497410974 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3388034660 con 0x7f33900cd960 2024-03-24T10:55:44.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.440+0000 7f33965f7640 1 -- 172.21.15.5:0/1497410974 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33900cd960 msgr2=0x7f33900cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:44.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.440+0000 7f33965f7640 1 --2- 172.21.15.5:0/1497410974 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33900cd960 0x7f33900cb6f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3388009d90 tx=0x7f338802bf30 comp rx=0 tx=0).stop 2024-03-24T10:55:44.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.440+0000 7f33965f7640 1 -- 172.21.15.5:0/1497410974 shutdown_connections 2024-03-24T10:55:44.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.440+0000 7f33965f7640 1 --2- 172.21.15.5:0/1497410974 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33900cd960 0x7f33900cb6f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:44.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.440+0000 7f33965f7640 1 --2- 172.21.15.5:0/1497410974 >> v2:172.21.15.5:3300/0 conn(0x7f33900cb270 0x7f33900cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:44.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.440+0000 7f33965f7640 1 -- 172.21.15.5:0/1497410974 >> 172.21.15.5:0/1497410974 conn(0x7f33900cfb80 msgr2=0x7f33900cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:44.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.441+0000 7f33965f7640 1 -- 172.21.15.5:0/1497410974 shutdown_connections 2024-03-24T10:55:44.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.441+0000 7f33965f7640 1 -- 172.21.15.5:0/1497410974 wait complete. 2024-03-24T10:55:44.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.442+0000 7f33965f7640 1 Processor -- start 2024-03-24T10:55:44.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.442+0000 7f33965f7640 1 -- start start 2024-03-24T10:55:44.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.443+0000 7f33965f7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33900cae90 0x7f3390154020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:44.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.443+0000 7f33965f7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33900cb270 0x7f3390155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:44.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.443+0000 7f33965f7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33900cd960 0x7f33900b0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:44.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.443+0000 7f33965f7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f33900cc620 con 0x7f33900cae90 2024-03-24T10:55:44.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.443+0000 7f33965f7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3390158bb0 con 0x7f33900cb270 2024-03-24T10:55:44.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.443+0000 7f33965f7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f3390154560 con 0x7f33900cd960 2024-03-24T10:55:44.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.443+0000 7f338ffff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33900cae90 0x7f3390154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:44.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.444+0000 7f338ffff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33900cae90 0x7f3390154020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:47186/0 (socket says 172.21.15.5:47186) 2024-03-24T10:55:44.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.444+0000 7f338ffff640 1 -- 172.21.15.5:0/2530618932 learned_addr learned my addr 172.21.15.5:0/2530618932 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:44.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.445+0000 7f338f7fe640 1 --2- 172.21.15.5:0/2530618932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33900cb270 0x7f3390155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:44.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.446+0000 7f3394b94640 1 --2- 172.21.15.5:0/2530618932 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33900cd960 0x7f33900b0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:44.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.448+0000 7f338ffff640 1 -- 172.21.15.5:0/2530618932 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33900cd960 msgr2=0x7f33900b0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:44.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.448+0000 7f338ffff640 1 --2- 172.21.15.5:0/2530618932 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33900cd960 0x7f33900b0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:44.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.448+0000 7f338ffff640 1 -- 172.21.15.5:0/2530618932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33900cb270 msgr2=0x7f3390155c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:44.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.448+0000 7f338ffff640 1 --2- 172.21.15.5:0/2530618932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33900cb270 0x7f3390155c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:44.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.448+0000 7f338ffff640 1 -- 172.21.15.5:0/2530618932 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3388009a70 con 0x7f33900cae90 2024-03-24T10:55:44.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.452+0000 7f338ffff640 1 --2- 172.21.15.5:0/2530618932 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33900cae90 0x7f3390154020 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f338000c430 tx=0x7f338000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:44.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.455+0000 7f338cff9640 1 -- 172.21.15.5:0/2530618932 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3380005ce0 con 0x7f33900cae90 2024-03-24T10:55:44.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.456+0000 7f33965f7640 1 -- 172.21.15.5:0/2530618932 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33901564a0 con 0x7f33900cae90 2024-03-24T10:55:44.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.456+0000 7f33965f7640 1 -- 172.21.15.5:0/2530618932 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33901f37f0 con 0x7f33900cae90 2024-03-24T10:55:44.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.457+0000 7f33965f7640 1 -- 172.21.15.5:0/2530618932 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33901f4400 con 0x7f33900cae90 2024-03-24T10:55:44.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.674+0000 7f338cff9640 1 -- 172.21.15.5:0/2530618932 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3380005e40 con 0x7f33900cae90 2024-03-24T10:55:44.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.674+0000 7f338cff9640 1 -- 172.21.15.5:0/2530618932 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3380007270 con 0x7f33900cae90 2024-03-24T10:55:44.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.749+0000 7f338cff9640 1 -- 172.21.15.5:0/2530618932 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3380007490 con 0x7f33900cae90 2024-03-24T10:55:44.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.750+0000 7f338cff9640 1 --2- 172.21.15.5:0/2530618932 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f337c06b810 0x7f337c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:44.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.750+0000 7f338f7fe640 1 --2- 172.21.15.5:0/2530618932 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f337c06b810 0x7f337c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:44.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.750+0000 7f338f7fe640 1 --2- 172.21.15.5:0/2530618932 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f337c06b810 0x7f337c06dcd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f33900cde60 tx=0x7f3384009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:44.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.765+0000 7f338cff9640 1 -- 172.21.15.5:0/2530618932 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f33800583c0 con 0x7f33900cae90 2024-03-24T10:55:44.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:44.765+0000 7f338cff9640 1 -- 172.21.15.5:0/2530618932 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3380054660 con 0x7f33900cae90 2024-03-24T10:55:45.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:45.006+0000 7f33965f7640 1 -- 172.21.15.5:0/2530618932 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33900681d0 con 0x7f33900cae90 2024-03-24T10:55:45.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:45.049+0000 7f338cff9640 1 -- 172.21.15.5:0/2530618932 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f3380017070 con 0x7f33900cae90 2024-03-24T10:55:45.050 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:45.050 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:55:45.050 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:55:45.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:45.051+0000 7f33965f7640 1 -- 172.21.15.5:0/2530618932 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f337c06b810 msgr2=0x7f337c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:45.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:45.051+0000 7f33965f7640 1 --2- 172.21.15.5:0/2530618932 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f337c06b810 0x7f337c06dcd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f33900cde60 tx=0x7f3384009190 comp rx=0 tx=0).stop 2024-03-24T10:55:45.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:45.051+0000 7f33965f7640 1 -- 172.21.15.5:0/2530618932 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33900cae90 msgr2=0x7f3390154020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:45.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:45.051+0000 7f33965f7640 1 --2- 172.21.15.5:0/2530618932 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33900cae90 0x7f3390154020 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f338000c430 tx=0x7f338000c8f0 comp rx=0 tx=0).stop 2024-03-24T10:55:45.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:45.051+0000 7f33965f7640 1 -- 172.21.15.5:0/2530618932 shutdown_connections 2024-03-24T10:55:45.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:45.051+0000 7f33965f7640 1 --2- 172.21.15.5:0/2530618932 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f337c06b810 0x7f337c06dcd0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:45.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:45.051+0000 7f33965f7640 1 --2- 172.21.15.5:0/2530618932 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33900cd960 0x7f33900b0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:45.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:45.051+0000 7f33965f7640 1 --2- 172.21.15.5:0/2530618932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33900cb270 0x7f3390155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:45.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:45.051+0000 7f33965f7640 1 --2- 172.21.15.5:0/2530618932 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33900cae90 0x7f3390154020 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:45.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:45.051+0000 7f33965f7640 1 -- 172.21.15.5:0/2530618932 >> 172.21.15.5:0/2530618932 conn(0x7f33900cfb80 msgr2=0x7f3390159d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:45.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:45.051+0000 7f33965f7640 1 -- 172.21.15.5:0/2530618932 shutdown_connections 2024-03-24T10:55:45.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:45.051+0000 7f33965f7640 1 -- 172.21.15.5:0/2530618932 wait complete. 2024-03-24T10:55:45.666 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:55:45.667 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:46.668 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:46.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.769+0000 7f80dba9d640 1 Processor -- start 2024-03-24T10:55:46.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.770+0000 7f80dba9d640 1 -- start start 2024-03-24T10:55:46.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.771+0000 7f80dba9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f80d40cb270 0x7f80d40cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:46.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.771+0000 7f80dba9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f80d40cd960 0x7f80d40cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:46.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.771+0000 7f80dba9d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80d40ccba0 con 0x7f80d40cae90 2024-03-24T10:55:46.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.771+0000 7f80dba9d640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f80d40cc9b0 con 0x7f80d40cd960 2024-03-24T10:55:46.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.771+0000 7f80dba9d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f80d40cce40 con 0x7f80d40cb270 2024-03-24T10:55:46.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.772+0000 7f80da03a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f80d40cd960 0x7f80d40cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:46.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.772+0000 7f80d9038640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f80d40cb270 0x7f80d40cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:46.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.772+0000 7f80da03a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f80d40cd960 0x7f80d40cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:47190/0 (socket says 172.21.15.5:47190) 2024-03-24T10:55:46.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.772+0000 7f80da03a640 1 -- 172.21.15.5:0/289528006 learned_addr learned my addr 172.21.15.5:0/289528006 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:46.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.774+0000 7f80da03a640 1 -- 172.21.15.5:0/289528006 >> v2:172.21.15.57:3300/0 conn(0x7f80d40cb270 msgr2=0x7f80d40cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:46.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.774+0000 7f80da03a640 1 --2- 172.21.15.5:0/289528006 >> v2:172.21.15.57:3300/0 conn(0x7f80d40cb270 0x7f80d40cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:46.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.774+0000 7f80da03a640 1 -- 172.21.15.5:0/289528006 >> v1:172.21.15.57:6789/0 conn(0x7f80d40cae90 legacy=0x7f80d40cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:46.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.774+0000 7f80da03a640 1 -- 172.21.15.5:0/289528006 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d415a590 con 0x7f80d40cd960 2024-03-24T10:55:46.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.776+0000 7f80da03a640 1 --2- 172.21.15.5:0/289528006 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80d40cd960 0x7f80d40cb6f0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f80d40cd680 tx=0x7f80cc02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70f06170a75a781c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:46.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.781+0000 7f80d8837640 1 -- 172.21.15.5:0/289528006 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc02cbb0 con 0x7f80d40cd960 2024-03-24T10:55:46.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.781+0000 7f80d8837640 1 -- 172.21.15.5:0/289528006 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80cc02cd10 con 0x7f80d40cd960 2024-03-24T10:55:46.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.781+0000 7f80d8837640 1 -- 172.21.15.5:0/289528006 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc034610 con 0x7f80d40cd960 2024-03-24T10:55:46.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.782+0000 7f80dba9d640 1 -- 172.21.15.5:0/289528006 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80d40cd960 msgr2=0x7f80d40cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:46.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.782+0000 7f80dba9d640 1 --2- 172.21.15.5:0/289528006 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80d40cd960 0x7f80d40cb6f0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f80d40cd680 tx=0x7f80cc02c010 comp rx=0 tx=0).stop 2024-03-24T10:55:46.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.782+0000 7f80dba9d640 1 -- 172.21.15.5:0/289528006 shutdown_connections 2024-03-24T10:55:46.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.782+0000 7f80dba9d640 1 --2- 172.21.15.5:0/289528006 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80d40cd960 0x7f80d40cb6f0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:46.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.782+0000 7f80dba9d640 1 --2- 172.21.15.5:0/289528006 >> v2:172.21.15.57:3300/0 conn(0x7f80d40cb270 0x7f80d40cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:46.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.782+0000 7f80dba9d640 1 -- 172.21.15.5:0/289528006 >> 172.21.15.5:0/289528006 conn(0x7f80d40cfb80 msgr2=0x7f80d40cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:46.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.782+0000 7f80dba9d640 1 -- 172.21.15.5:0/289528006 shutdown_connections 2024-03-24T10:55:46.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.782+0000 7f80dba9d640 1 -- 172.21.15.5:0/289528006 wait complete. 2024-03-24T10:55:46.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.784+0000 7f80dba9d640 1 Processor -- start 2024-03-24T10:55:46.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.784+0000 7f80dba9d640 1 -- start start 2024-03-24T10:55:46.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.785+0000 7f80dba9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f80d40cae90 0x7f80d415e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:46.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.785+0000 7f80dba9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80d40cb270 0x7f80d415b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:46.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.785+0000 7f80dba9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80d40cd960 0x7f80d41572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:46.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.785+0000 7f80dba9d640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f80d40c3480 con 0x7f80d40cb270 2024-03-24T10:55:46.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.785+0000 7f80dba9d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f80d415dbc0 con 0x7f80d40cd960 2024-03-24T10:55:46.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.785+0000 7f80dba9d640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f80d41f09e0 con 0x7f80d40cae90 2024-03-24T10:55:46.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.786+0000 7f80d9038640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80d40cb270 0x7f80d415b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:46.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.786+0000 7f80da03a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80d40cd960 0x7f80d41572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:46.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.786+0000 7f80d9038640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80d40cb270 0x7f80d415b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:47192/0 (socket says 172.21.15.5:47192) 2024-03-24T10:55:46.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.786+0000 7f80d9038640 1 -- 172.21.15.5:0/3751449655 learned_addr learned my addr 172.21.15.5:0/3751449655 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:46.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.786+0000 7f80d9839640 1 --2- 172.21.15.5:0/3751449655 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f80d40cae90 0x7f80d415e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:46.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.789+0000 7f80d9038640 1 -- 172.21.15.5:0/3751449655 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f80d40cae90 msgr2=0x7f80d415e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:46.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.789+0000 7f80d9038640 1 --2- 172.21.15.5:0/3751449655 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f80d40cae90 0x7f80d415e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:46.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.789+0000 7f80d9038640 1 -- 172.21.15.5:0/3751449655 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80d40cd960 msgr2=0x7f80d41572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:46.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.789+0000 7f80d9038640 1 --2- 172.21.15.5:0/3751449655 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80d40cd960 0x7f80d41572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:46.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.790+0000 7f80d9038640 1 -- 172.21.15.5:0/3751449655 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80cc009b30 con 0x7f80d40cb270 2024-03-24T10:55:46.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.792+0000 7f80d9038640 1 --2- 172.21.15.5:0/3751449655 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80d40cb270 0x7f80d415b5e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f80c400e9e0 tx=0x7f80c400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:46.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.796+0000 7f80ca7fc640 1 -- 172.21.15.5:0/3751449655 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c4009670 con 0x7f80d40cb270 2024-03-24T10:55:46.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.796+0000 7f80ca7fc640 1 -- 172.21.15.5:0/3751449655 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80c400c9d0 con 0x7f80d40cb270 2024-03-24T10:55:46.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.796+0000 7f80dba9d640 1 -- 172.21.15.5:0/3751449655 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80d41f0c40 con 0x7f80d40cb270 2024-03-24T10:55:46.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.796+0000 7f80ca7fc640 1 -- 172.21.15.5:0/3751449655 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c400cd80 con 0x7f80d40cb270 2024-03-24T10:55:46.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.796+0000 7f80dba9d640 1 -- 172.21.15.5:0/3751449655 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80d415d570 con 0x7f80d40cb270 2024-03-24T10:55:46.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:46.798+0000 7f80dba9d640 1 -- 172.21.15.5:0/3751449655 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80d40c9710 con 0x7f80d40cb270 2024-03-24T10:55:47.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:47.066+0000 7f80ca7fc640 1 -- 172.21.15.5:0/3751449655 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f80c4013470 con 0x7f80d40cb270 2024-03-24T10:55:47.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:47.067+0000 7f80ca7fc640 1 --2- 172.21.15.5:0/3751449655 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f80b806b810 0x7f80b806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:47.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:47.067+0000 7f80d9839640 1 --2- 172.21.15.5:0/3751449655 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f80b806b810 0x7f80b806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:47.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:47.067+0000 7f80d9839640 1 --2- 172.21.15.5:0/3751449655 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f80b806b810 0x7f80b806dcd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f80d0000f60 tx=0x7f80d0009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:47.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:47.083+0000 7f80ca7fc640 1 -- 172.21.15.5:0/3751449655 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f80c4059400 con 0x7f80d40cb270 2024-03-24T10:55:47.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:47.083+0000 7f80ca7fc640 1 -- 172.21.15.5:0/3751449655 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f80c40556a0 con 0x7f80d40cb270 2024-03-24T10:55:47.333 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:47.332+0000 7f80dba9d640 1 -- 172.21.15.5:0/3751449655 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80d40b1e60 con 0x7f80d40cb270 2024-03-24T10:55:47.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:47.346+0000 7f80ca7fc640 1 -- 172.21.15.5:0/3751449655 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f80c4021020 con 0x7f80d40cb270 2024-03-24T10:55:47.347 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:47.347 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:55:47.347 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:55:47.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:47.348+0000 7f80dba9d640 1 -- 172.21.15.5:0/3751449655 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f80b806b810 msgr2=0x7f80b806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:47.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:47.348+0000 7f80dba9d640 1 --2- 172.21.15.5:0/3751449655 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f80b806b810 0x7f80b806dcd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f80d0000f60 tx=0x7f80d0009040 comp rx=0 tx=0).stop 2024-03-24T10:55:47.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:47.348+0000 7f80dba9d640 1 -- 172.21.15.5:0/3751449655 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80d40cb270 msgr2=0x7f80d415b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:47.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:47.348+0000 7f80dba9d640 1 --2- 172.21.15.5:0/3751449655 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80d40cb270 0x7f80d415b5e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f80c400e9e0 tx=0x7f80c400eea0 comp rx=0 tx=0).stop 2024-03-24T10:55:47.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:47.348+0000 7f80dba9d640 1 -- 172.21.15.5:0/3751449655 shutdown_connections 2024-03-24T10:55:47.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:47.348+0000 7f80dba9d640 1 --2- 172.21.15.5:0/3751449655 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f80b806b810 0x7f80b806dcd0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:47.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:47.348+0000 7f80dba9d640 1 --2- 172.21.15.5:0/3751449655 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f80d40cb270 0x7f80d415b5e0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:47.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:47.348+0000 7f80dba9d640 1 --2- 172.21.15.5:0/3751449655 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f80d40cd960 0x7f80d41572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:47.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:47.348+0000 7f80dba9d640 1 --2- 172.21.15.5:0/3751449655 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f80d40cae90 0x7f80d415e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:47.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:47.348+0000 7f80dba9d640 1 -- 172.21.15.5:0/3751449655 >> 172.21.15.5:0/3751449655 conn(0x7f80d40cfb80 msgr2=0x7f80d406c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:47.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:47.348+0000 7f80dba9d640 1 -- 172.21.15.5:0/3751449655 shutdown_connections 2024-03-24T10:55:47.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:47.348+0000 7f80dba9d640 1 -- 172.21.15.5:0/3751449655 wait complete. 2024-03-24T10:55:47.965 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:55:47.965 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:48.966 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:49.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.069+0000 7f0eee409640 1 Processor -- start 2024-03-24T10:55:49.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.069+0000 7f0eee409640 1 -- start start 2024-03-24T10:55:49.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.070+0000 7f0eee409640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0ee80c2ea0 0x7f0ee80caab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:49.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.070+0000 7f0eee409640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0ee80c9610 0x7f0ee80c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:49.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.070+0000 7f0eee409640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee80ca000 con 0x7f0ee80c58c0 2024-03-24T10:55:49.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.070+0000 7f0eee409640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee80cb210 con 0x7f0ee80c2ea0 2024-03-24T10:55:49.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.070+0000 7f0eee409640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee8156db0 con 0x7f0ee80c9610 2024-03-24T10:55:49.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.070+0000 7f0eec9a6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0ee80c9610 0x7f0ee80c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:49.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.071+0000 7f0ee7fff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0ee80c2ea0 0x7f0ee80caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:49.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.071+0000 7f0ee77fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f0ee80c58c0 0x7f0ee80c3280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:41384/0 (socket says 172.21.15.5:41384) 2024-03-24T10:55:49.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.071+0000 7f0ee77fe640 1 -- 172.21.15.5:0/1082511001 learned_addr learned my addr 172.21.15.5:0/1082511001 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:55:49.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.074+0000 7f0eec9a6640 1 -- 172.21.15.5:0/1082511001 >> v2:172.21.15.5:3300/0 conn(0x7f0ee80c2ea0 msgr2=0x7f0ee80caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:49.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.074+0000 7f0eec9a6640 1 --2- 172.21.15.5:0/1082511001 >> v2:172.21.15.5:3300/0 conn(0x7f0ee80c2ea0 0x7f0ee80caab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:49.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.074+0000 7f0eec9a6640 1 -- 172.21.15.5:0/1082511001 >> v1:172.21.15.5:6789/0 conn(0x7f0ee80c58c0 legacy=0x7f0ee80c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:49.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.074+0000 7f0eec9a6640 1 -- 172.21.15.5:0/1082511001 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee8156fc0 con 0x7f0ee80c9610 2024-03-24T10:55:49.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.077+0000 7f0eec9a6640 1 --2- 172.21.15.5:0/1082511001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ee80c9610 0x7f0ee80c9ac0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0ee80c69a0 tx=0x7f0ee002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7dbfb40cf28e7bfd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:49.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.082+0000 7f0ee6ffd640 1 -- 172.21.15.5:0/1082511001 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee002cd00 con 0x7f0ee80c9610 2024-03-24T10:55:49.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.082+0000 7f0ee6ffd640 1 -- 172.21.15.5:0/1082511001 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ee002ce60 con 0x7f0ee80c9610 2024-03-24T10:55:49.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.082+0000 7f0ee6ffd640 1 -- 172.21.15.5:0/1082511001 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0034660 con 0x7f0ee80c9610 2024-03-24T10:55:49.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.083+0000 7f0eee409640 1 -- 172.21.15.5:0/1082511001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ee80c9610 msgr2=0x7f0ee80c9ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:49.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.083+0000 7f0eee409640 1 --2- 172.21.15.5:0/1082511001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ee80c9610 0x7f0ee80c9ac0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f0ee80c69a0 tx=0x7f0ee002bf50 comp rx=0 tx=0).stop 2024-03-24T10:55:49.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.083+0000 7f0eee409640 1 -- 172.21.15.5:0/1082511001 shutdown_connections 2024-03-24T10:55:49.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.083+0000 7f0eee409640 1 --2- 172.21.15.5:0/1082511001 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ee80c9610 0x7f0ee80c9ac0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:49.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.083+0000 7f0eee409640 1 --2- 172.21.15.5:0/1082511001 >> v2:172.21.15.5:3300/0 conn(0x7f0ee80c2ea0 0x7f0ee80caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:49.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.083+0000 7f0eee409640 1 -- 172.21.15.5:0/1082511001 >> 172.21.15.5:0/1082511001 conn(0x7f0ee806b130 msgr2=0x7f0ee806b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:49.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.083+0000 7f0eee409640 1 -- 172.21.15.5:0/1082511001 shutdown_connections 2024-03-24T10:55:49.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.083+0000 7f0eee409640 1 -- 172.21.15.5:0/1082511001 wait complete. 2024-03-24T10:55:49.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.084+0000 7f0eee409640 1 Processor -- start 2024-03-24T10:55:49.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.084+0000 7f0eee409640 1 -- start start 2024-03-24T10:55:49.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.085+0000 7f0eee409640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ee80c2ea0 0x7f0ee81eee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:49.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.085+0000 7f0eee409640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ee80c58c0 0x7f0ee81437a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:49.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.085+0000 7f0eee409640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0ee80c9610 0x7f0ee81ee480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:49.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.085+0000 7f0eee409640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee8143380 con 0x7f0ee80c2ea0 2024-03-24T10:55:49.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.085+0000 7f0eee409640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee81ee9c0 con 0x7f0ee80c58c0 2024-03-24T10:55:49.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.085+0000 7f0eee409640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee8142460 con 0x7f0ee80c9610 2024-03-24T10:55:49.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.085+0000 7f0ee7fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ee80c2ea0 0x7f0ee81eee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:49.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.085+0000 7f0ee77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ee80c58c0 0x7f0ee81437a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:49.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.085+0000 7f0ee7fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ee80c2ea0 0x7f0ee81eee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:47208/0 (socket says 172.21.15.5:47208) 2024-03-24T10:55:49.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.085+0000 7f0ee7fff640 1 -- 172.21.15.5:0/288234647 learned_addr learned my addr 172.21.15.5:0/288234647 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:49.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.085+0000 7f0eec9a6640 1 --2- 172.21.15.5:0/288234647 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0ee80c9610 0x7f0ee81ee480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:49.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.089+0000 7f0ee7fff640 1 -- 172.21.15.5:0/288234647 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0ee80c9610 msgr2=0x7f0ee81ee480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:49.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.089+0000 7f0ee7fff640 1 --2- 172.21.15.5:0/288234647 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0ee80c9610 0x7f0ee81ee480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:49.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.089+0000 7f0ee7fff640 1 -- 172.21.15.5:0/288234647 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ee80c58c0 msgr2=0x7f0ee81437a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:49.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.089+0000 7f0ee7fff640 1 --2- 172.21.15.5:0/288234647 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ee80c58c0 0x7f0ee81437a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:49.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.089+0000 7f0ee7fff640 1 -- 172.21.15.5:0/288234647 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee0009a70 con 0x7f0ee80c2ea0 2024-03-24T10:55:49.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.091+0000 7f0ee7fff640 1 --2- 172.21.15.5:0/288234647 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ee80c2ea0 0x7f0ee81eee70 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f0ed800e870 tx=0x7f0ed800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:49.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.095+0000 7f0ee4ff9640 1 -- 172.21.15.5:0/288234647 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed80094f0 con 0x7f0ee80c2ea0 2024-03-24T10:55:49.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.095+0000 7f0ee4ff9640 1 -- 172.21.15.5:0/288234647 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ed800c940 con 0x7f0ee80c2ea0 2024-03-24T10:55:49.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.095+0000 7f0ee4ff9640 1 -- 172.21.15.5:0/288234647 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed800ccf0 con 0x7f0ee80c2ea0 2024-03-24T10:55:49.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.096+0000 7f0eee409640 1 -- 172.21.15.5:0/288234647 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ee8142700 con 0x7f0ee80c2ea0 2024-03-24T10:55:49.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.096+0000 7f0eee409640 1 -- 172.21.15.5:0/288234647 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ee8142cc0 con 0x7f0ee80c2ea0 2024-03-24T10:55:49.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.097+0000 7f0eee409640 1 -- 172.21.15.5:0/288234647 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ee81435f0 con 0x7f0ee80c2ea0 2024-03-24T10:55:49.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.374+0000 7f0ee4ff9640 1 -- 172.21.15.5:0/288234647 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0ed8013470 con 0x7f0ee80c2ea0 2024-03-24T10:55:49.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.375+0000 7f0ee4ff9640 1 --2- 172.21.15.5:0/288234647 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0ed406b860 0x7f0ed406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:49.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.375+0000 7f0ee77fe640 1 --2- 172.21.15.5:0/288234647 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0ed406b860 0x7f0ed406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:49.376 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.375+0000 7f0ee77fe640 1 --2- 172.21.15.5:0/288234647 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0ed406b860 0x7f0ed406dd20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8156b10 tx=0x7f0edc007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:49.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.415+0000 7f0ee4ff9640 1 -- 172.21.15.5:0/288234647 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f0ed8059190 con 0x7f0ee80c2ea0 2024-03-24T10:55:49.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.425+0000 7f0ee4ff9640 1 -- 172.21.15.5:0/288234647 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0ed8055430 con 0x7f0ee80c2ea0 2024-03-24T10:55:49.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.671+0000 7f0eee409640 1 -- 172.21.15.5:0/288234647 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ee8062a00 con 0x7f0ee80c2ea0 2024-03-24T10:55:49.685 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.684+0000 7f0ee4ff9640 1 -- 172.21.15.5:0/288234647 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f0ed8021090 con 0x7f0ee80c2ea0 2024-03-24T10:55:49.685 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:49.685 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:55:49.685 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:55:49.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.686+0000 7f0eee409640 1 -- 172.21.15.5:0/288234647 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0ed406b860 msgr2=0x7f0ed406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:49.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.686+0000 7f0eee409640 1 --2- 172.21.15.5:0/288234647 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0ed406b860 0x7f0ed406dd20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8156b10 tx=0x7f0edc007e80 comp rx=0 tx=0).stop 2024-03-24T10:55:49.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.686+0000 7f0eee409640 1 -- 172.21.15.5:0/288234647 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ee80c2ea0 msgr2=0x7f0ee81eee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:49.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.686+0000 7f0eee409640 1 --2- 172.21.15.5:0/288234647 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ee80c2ea0 0x7f0ee81eee70 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f0ed800e870 tx=0x7f0ed800ed30 comp rx=0 tx=0).stop 2024-03-24T10:55:49.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.686+0000 7f0eee409640 1 -- 172.21.15.5:0/288234647 shutdown_connections 2024-03-24T10:55:49.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.686+0000 7f0eee409640 1 --2- 172.21.15.5:0/288234647 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0ed406b860 0x7f0ed406dd20 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:49.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.686+0000 7f0eee409640 1 --2- 172.21.15.5:0/288234647 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0ee80c9610 0x7f0ee81ee480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:49.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.686+0000 7f0eee409640 1 --2- 172.21.15.5:0/288234647 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ee80c58c0 0x7f0ee81437a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:49.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.686+0000 7f0eee409640 1 --2- 172.21.15.5:0/288234647 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ee80c2ea0 0x7f0ee81eee70 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:49.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.686+0000 7f0eee409640 1 -- 172.21.15.5:0/288234647 >> 172.21.15.5:0/288234647 conn(0x7f0ee806b130 msgr2=0x7f0ee80c8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:49.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.686+0000 7f0eee409640 1 -- 172.21.15.5:0/288234647 shutdown_connections 2024-03-24T10:55:49.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:49.686+0000 7f0eee409640 1 -- 172.21.15.5:0/288234647 wait complete. 2024-03-24T10:55:50.303 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:55:50.303 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:51.303 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:51.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.406+0000 7fdd485e2640 1 Processor -- start 2024-03-24T10:55:51.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.406+0000 7fdd485e2640 1 -- start start 2024-03-24T10:55:51.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.407+0000 7fdd485e2640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fdd400cf530 0x7fdd400cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:51.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.407+0000 7fdd485e2640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd400cf990 con 0x7fdd400cb270 2024-03-24T10:55:51.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.407+0000 7fdd485e2640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd400ccba0 con 0x7fdd400cae90 2024-03-24T10:55:51.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.407+0000 7fdd485e2640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fdd400cc9b0 con 0x7fdd400cf530 2024-03-24T10:55:51.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.407+0000 7fdd45b7d640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fdd400cb270 0x7fdd400cd960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:41398/0 (socket says 172.21.15.5:41398) 2024-03-24T10:55:51.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.407+0000 7fdd45b7d640 1 -- 172.21.15.5:0/2954220101 learned_addr learned my addr 172.21.15.5:0/2954220101 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:55:51.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.407+0000 7fdd46b7f640 1 --2- 172.21.15.5:0/2954220101 >> v2:172.21.15.57:3300/0 conn(0x7fdd400cf530 0x7fdd400cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:51.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.410+0000 7fdd4537c640 1 -- 172.21.15.5:0/2954220101 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3901020506 0 0) 0x7fdd400cf990 con 0x7fdd400cb270 2024-03-24T10:55:51.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.410+0000 7fdd4537c640 1 -- 172.21.15.5:0/2954220101 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd2c003520 con 0x7fdd400cb270 2024-03-24T10:55:51.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.411+0000 7fdd46b7f640 1 -- 172.21.15.5:0/2954220101 >> v1:172.21.15.57:6789/0 conn(0x7fdd400cae90 legacy=0x7fdd400cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:51.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.411+0000 7fdd46b7f640 1 -- 172.21.15.5:0/2954220101 >> v1:172.21.15.5:6789/0 conn(0x7fdd400cb270 legacy=0x7fdd400cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:51.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.411+0000 7fdd46b7f640 1 -- 172.21.15.5:0/2954220101 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd400cce40 con 0x7fdd400cf530 2024-03-24T10:55:51.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.414+0000 7fdd46b7f640 1 --2- 172.21.15.5:0/2954220101 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd400cf530 0x7fdd400cb6f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009d90 tx=0x7fdd3802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0d3c3881a487398 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:51.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.419+0000 7fdd4537c640 1 -- 172.21.15.5:0/2954220101 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3802cd00 con 0x7fdd400cf530 2024-03-24T10:55:51.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.420+0000 7fdd4537c640 1 -- 172.21.15.5:0/2954220101 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd3802ce60 con 0x7fdd400cf530 2024-03-24T10:55:51.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.420+0000 7fdd4537c640 1 -- 172.21.15.5:0/2954220101 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38034660 con 0x7fdd400cf530 2024-03-24T10:55:51.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.420+0000 7fdd485e2640 1 -- 172.21.15.5:0/2954220101 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd400cf530 msgr2=0x7fdd400cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:51.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.420+0000 7fdd485e2640 1 --2- 172.21.15.5:0/2954220101 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd400cf530 0x7fdd400cb6f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009d90 tx=0x7fdd3802bf50 comp rx=0 tx=0).stop 2024-03-24T10:55:51.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.420+0000 7fdd485e2640 1 -- 172.21.15.5:0/2954220101 shutdown_connections 2024-03-24T10:55:51.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.420+0000 7fdd485e2640 1 --2- 172.21.15.5:0/2954220101 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd400cf530 0x7fdd400cb6f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:51.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.420+0000 7fdd485e2640 1 -- 172.21.15.5:0/2954220101 >> 172.21.15.5:0/2954220101 conn(0x7fdd400cfb80 msgr2=0x7fdd400cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:51.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.420+0000 7fdd485e2640 1 -- 172.21.15.5:0/2954220101 shutdown_connections 2024-03-24T10:55:51.422 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.420+0000 7fdd485e2640 1 -- 172.21.15.5:0/2954220101 wait complete. 2024-03-24T10:55:51.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.422+0000 7fdd485e2640 1 Processor -- start 2024-03-24T10:55:51.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.422+0000 7fdd485e2640 1 -- start start 2024-03-24T10:55:51.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.422+0000 7fdd485e2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd400cae90 0x7fdd400ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:51.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.422+0000 7fdd485e2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd400cb270 0x7fdd400b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:51.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.422+0000 7fdd485e2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd400cf530 0x7fdd400b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:51.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.422+0000 7fdd485e2640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fdd40157550 con 0x7fdd400cae90 2024-03-24T10:55:51.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.422+0000 7fdd485e2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fdd401f09d0 con 0x7fdd400cb270 2024-03-24T10:55:51.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.422+0000 7fdd485e2640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fdd401f0c90 con 0x7fdd400cf530 2024-03-24T10:55:51.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.423+0000 7fdd4637e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd400cae90 0x7fdd400ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:51.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.423+0000 7fdd4637e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd400cae90 0x7fdd400ae980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:47216/0 (socket says 172.21.15.5:47216) 2024-03-24T10:55:51.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.423+0000 7fdd4637e640 1 -- 172.21.15.5:0/491188174 learned_addr learned my addr 172.21.15.5:0/491188174 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:51.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.424+0000 7fdd45b7d640 1 --2- 172.21.15.5:0/491188174 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd400cb270 0x7fdd400b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:51.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.424+0000 7fdd46b7f640 1 --2- 172.21.15.5:0/491188174 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd400cf530 0x7fdd400b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:51.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.427+0000 7fdd4637e640 1 -- 172.21.15.5:0/491188174 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd400cf530 msgr2=0x7fdd400b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:51.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.427+0000 7fdd4637e640 1 --2- 172.21.15.5:0/491188174 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd400cf530 0x7fdd400b42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:51.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.427+0000 7fdd4637e640 1 -- 172.21.15.5:0/491188174 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd400cb270 msgr2=0x7fdd400b07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:51.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.427+0000 7fdd4637e640 1 --2- 172.21.15.5:0/491188174 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd400cb270 0x7fdd400b07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:51.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.427+0000 7fdd4637e640 1 -- 172.21.15.5:0/491188174 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd38009a70 con 0x7fdd400cae90 2024-03-24T10:55:51.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.429+0000 7fdd4637e640 1 --2- 172.21.15.5:0/491188174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd400cae90 0x7fdd400ae980 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00c430 tx=0x7fdd3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:51.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.434+0000 7fdd36ffd640 1 -- 172.21.15.5:0/491188174 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c011790 con 0x7fdd400cae90 2024-03-24T10:55:51.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.434+0000 7fdd36ffd640 1 -- 172.21.15.5:0/491188174 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd3c0118f0 con 0x7fdd400cae90 2024-03-24T10:55:51.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.434+0000 7fdd485e2640 1 -- 172.21.15.5:0/491188174 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd400af5a0 con 0x7fdd400cae90 2024-03-24T10:55:51.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.434+0000 7fdd36ffd640 1 -- 172.21.15.5:0/491188174 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c011ce0 con 0x7fdd400cae90 2024-03-24T10:55:51.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.434+0000 7fdd485e2640 1 -- 172.21.15.5:0/491188174 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd400afb60 con 0x7fdd400cae90 2024-03-24T10:55:51.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.435+0000 7fdd485e2640 1 -- 172.21.15.5:0/491188174 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd400ca1d0 con 0x7fdd400cae90 2024-03-24T10:55:51.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.456+0000 7fdd36ffd640 1 -- 172.21.15.5:0/491188174 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdd3c01c460 con 0x7fdd400cae90 2024-03-24T10:55:51.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.456+0000 7fdd36ffd640 1 --2- 172.21.15.5:0/491188174 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdd2406b890 0x7fdd2406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:51.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.456+0000 7fdd45b7d640 1 --2- 172.21.15.5:0/491188174 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdd2406b890 0x7fdd2406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:51.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.457+0000 7fdd45b7d640 1 --2- 172.21.15.5:0/491188174 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdd2406b890 0x7fdd2406dd50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fdd401eda70 tx=0x7fdd30009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:51.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.717+0000 7fdd36ffd640 1 -- 172.21.15.5:0/491188174 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fdd3c011e40 con 0x7fdd400cae90 2024-03-24T10:55:51.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.717+0000 7fdd36ffd640 1 -- 172.21.15.5:0/491188174 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdd3c055560 con 0x7fdd400cae90 2024-03-24T10:55:51.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.947+0000 7fdd485e2640 1 -- 172.21.15.5:0/491188174 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd4006dad0 con 0x7fdd400cae90 2024-03-24T10:55:51.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.970+0000 7fdd36ffd640 1 -- 172.21.15.5:0/491188174 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7fdd3c058c20 con 0x7fdd400cae90 2024-03-24T10:55:51.971 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:51.971 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:55:51.971 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:55:51.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.973+0000 7fdd485e2640 1 -- 172.21.15.5:0/491188174 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdd2406b890 msgr2=0x7fdd2406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:51.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.973+0000 7fdd485e2640 1 --2- 172.21.15.5:0/491188174 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdd2406b890 0x7fdd2406dd50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fdd401eda70 tx=0x7fdd30009040 comp rx=0 tx=0).stop 2024-03-24T10:55:51.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.973+0000 7fdd485e2640 1 -- 172.21.15.5:0/491188174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd400cae90 msgr2=0x7fdd400ae980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:51.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.973+0000 7fdd485e2640 1 --2- 172.21.15.5:0/491188174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd400cae90 0x7fdd400ae980 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00c430 tx=0x7fdd3c00c8f0 comp rx=0 tx=0).stop 2024-03-24T10:55:51.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.973+0000 7fdd485e2640 1 -- 172.21.15.5:0/491188174 shutdown_connections 2024-03-24T10:55:51.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.973+0000 7fdd485e2640 1 --2- 172.21.15.5:0/491188174 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdd2406b890 0x7fdd2406dd50 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:51.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.973+0000 7fdd485e2640 1 --2- 172.21.15.5:0/491188174 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdd400cf530 0x7fdd400b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:51.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.973+0000 7fdd485e2640 1 --2- 172.21.15.5:0/491188174 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdd400cb270 0x7fdd400b07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:51.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.973+0000 7fdd485e2640 1 --2- 172.21.15.5:0/491188174 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdd400cae90 0x7fdd400ae980 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:51.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.973+0000 7fdd485e2640 1 -- 172.21.15.5:0/491188174 >> 172.21.15.5:0/491188174 conn(0x7fdd400cfb80 msgr2=0x7fdd400c7750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:51.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.973+0000 7fdd485e2640 1 -- 172.21.15.5:0/491188174 shutdown_connections 2024-03-24T10:55:51.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:51.973+0000 7fdd485e2640 1 -- 172.21.15.5:0/491188174 wait complete. 2024-03-24T10:55:52.591 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:55:52.591 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:53.592 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:53.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.693+0000 7f685f62e640 1 Processor -- start 2024-03-24T10:55:53.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.693+0000 7f685f62e640 1 -- start start 2024-03-24T10:55:53.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.694+0000 7f685f62e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f68581543a0 0x7f685815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:53.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.694+0000 7f685f62e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f685806bf50 0x7f685806c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:53.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.694+0000 7f685f62e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68580cb2f0 con 0x7f685806d4f0 2024-03-24T10:55:53.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.694+0000 7f685f62e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f68580c6750 con 0x7f68581543a0 2024-03-24T10:55:53.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.694+0000 7f685f62e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f68580cc380 con 0x7f685806bf50 2024-03-24T10:55:53.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.694+0000 7f685dbcb640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f685806bf50 0x7f685806c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:53.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.695+0000 7f685d3ca640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f68581543a0 0x7f685815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:53.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.695+0000 7f685cbc9640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f685806d4f0 0x7f68580cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:44922/0 (socket says 172.21.15.5:44922) 2024-03-24T10:55:53.696 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.695+0000 7f685cbc9640 1 -- 172.21.15.5:0/3173492518 learned_addr learned my addr 172.21.15.5:0/3173492518 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:55:53.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.697+0000 7f685dbcb640 1 -- 172.21.15.5:0/3173492518 >> v2:172.21.15.5:3300/0 conn(0x7f68581543a0 msgr2=0x7f685815ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:53.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.697+0000 7f685dbcb640 1 --2- 172.21.15.5:0/3173492518 >> v2:172.21.15.5:3300/0 conn(0x7f68581543a0 0x7f685815ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:53.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.697+0000 7f685dbcb640 1 -- 172.21.15.5:0/3173492518 >> v1:172.21.15.5:6789/0 conn(0x7f685806d4f0 legacy=0x7f68580cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:53.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.697+0000 7f685dbcb640 1 -- 172.21.15.5:0/3173492518 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68580ca7e0 con 0x7f685806bf50 2024-03-24T10:55:53.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.699+0000 7f685dbcb640 1 --2- 172.21.15.5:0/3173492518 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f685806bf50 0x7f685806c330 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f68580b51c0 tx=0x7f685002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d0c01364a17ff2c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:53.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.704+0000 7f684ffff640 1 -- 172.21.15.5:0/3173492518 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685002cd00 con 0x7f685806bf50 2024-03-24T10:55:53.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.704+0000 7f684ffff640 1 -- 172.21.15.5:0/3173492518 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f685002ce60 con 0x7f685806bf50 2024-03-24T10:55:53.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.704+0000 7f684ffff640 1 -- 172.21.15.5:0/3173492518 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6850034660 con 0x7f685806bf50 2024-03-24T10:55:53.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.704+0000 7f685f62e640 1 -- 172.21.15.5:0/3173492518 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f685806bf50 msgr2=0x7f685806c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:53.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.704+0000 7f685f62e640 1 --2- 172.21.15.5:0/3173492518 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f685806bf50 0x7f685806c330 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f68580b51c0 tx=0x7f685002bf50 comp rx=0 tx=0).stop 2024-03-24T10:55:53.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.705+0000 7f685f62e640 1 -- 172.21.15.5:0/3173492518 shutdown_connections 2024-03-24T10:55:53.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.705+0000 7f685f62e640 1 --2- 172.21.15.5:0/3173492518 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f685806bf50 0x7f685806c330 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:53.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.705+0000 7f685f62e640 1 --2- 172.21.15.5:0/3173492518 >> v2:172.21.15.5:3300/0 conn(0x7f68581543a0 0x7f685815ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:53.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.705+0000 7f685f62e640 1 -- 172.21.15.5:0/3173492518 >> 172.21.15.5:0/3173492518 conn(0x7f6858072ef0 msgr2=0x7f6858061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:53.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.705+0000 7f685f62e640 1 -- 172.21.15.5:0/3173492518 shutdown_connections 2024-03-24T10:55:53.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.705+0000 7f685f62e640 1 -- 172.21.15.5:0/3173492518 wait complete. 2024-03-24T10:55:53.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.707+0000 7f685f62e640 1 Processor -- start 2024-03-24T10:55:53.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.707+0000 7f685f62e640 1 -- start start 2024-03-24T10:55:53.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.708+0000 7f685f62e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f685806bf50 0x7f68580ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:53.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.708+0000 7f685f62e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f685806d4f0 0x7f68581f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:53.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.708+0000 7f685f62e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f68581543a0 0x7f68581f0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:53.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.708+0000 7f685f62e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f6858149140 con 0x7f685806d4f0 2024-03-24T10:55:53.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.708+0000 7f685f62e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f68581f34d0 con 0x7f685806bf50 2024-03-24T10:55:53.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.708+0000 7f685f62e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f6858148390 con 0x7f68581543a0 2024-03-24T10:55:53.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.708+0000 7f685cbc9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f685806d4f0 0x7f68581f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:53.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.708+0000 7f685d3ca640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f685806bf50 0x7f68580ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:53.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.708+0000 7f685dbcb640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f68581543a0 0x7f68581f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:53.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.709+0000 7f685cbc9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f685806d4f0 0x7f68581f3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:34940/0 (socket says 172.21.15.5:34940) 2024-03-24T10:55:53.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.709+0000 7f685cbc9640 1 -- 172.21.15.5:0/2748996654 learned_addr learned my addr 172.21.15.5:0/2748996654 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:53.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.712+0000 7f685cbc9640 1 -- 172.21.15.5:0/2748996654 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f68581543a0 msgr2=0x7f68581f0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:53.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.712+0000 7f685cbc9640 1 --2- 172.21.15.5:0/2748996654 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f68581543a0 0x7f68581f0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:53.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.712+0000 7f685cbc9640 1 -- 172.21.15.5:0/2748996654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f685806bf50 msgr2=0x7f68580ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:53.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.712+0000 7f685cbc9640 1 --2- 172.21.15.5:0/2748996654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f685806bf50 0x7f68580ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:53.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.712+0000 7f685cbc9640 1 -- 172.21.15.5:0/2748996654 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6850009a70 con 0x7f685806d4f0 2024-03-24T10:55:53.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.714+0000 7f685cbc9640 1 --2- 172.21.15.5:0/2748996654 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f685806d4f0 0x7f68581f3be0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f684800c400 tx=0x7f684800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:53.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.718+0000 7f684dffb640 1 -- 172.21.15.5:0/2748996654 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6848005ce0 con 0x7f685806d4f0 2024-03-24T10:55:53.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.719+0000 7f684dffb640 1 -- 172.21.15.5:0/2748996654 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6848005e40 con 0x7f685806d4f0 2024-03-24T10:55:53.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.719+0000 7f684dffb640 1 -- 172.21.15.5:0/2748996654 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68480072f0 con 0x7f685806d4f0 2024-03-24T10:55:53.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.719+0000 7f685f62e640 1 -- 172.21.15.5:0/2748996654 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68581f3750 con 0x7f685806d4f0 2024-03-24T10:55:53.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.719+0000 7f685f62e640 1 -- 172.21.15.5:0/2748996654 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68581effb0 con 0x7f685806d4f0 2024-03-24T10:55:53.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.721+0000 7f685f62e640 1 -- 172.21.15.5:0/2748996654 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6858157f60 con 0x7f685806d4f0 2024-03-24T10:55:53.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.990+0000 7f684dffb640 1 -- 172.21.15.5:0/2748996654 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6848007510 con 0x7f685806d4f0 2024-03-24T10:55:53.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.991+0000 7f684dffb640 1 --2- 172.21.15.5:0/2748996654 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f683c06b810 0x7f683c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:53.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.991+0000 7f685d3ca640 1 --2- 172.21.15.5:0/2748996654 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f683c06b810 0x7f683c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:53.992 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:53.991+0000 7f685d3ca640 1 --2- 172.21.15.5:0/2748996654 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f683c06b810 0x7f683c06dcd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6854009aa0 tx=0x7f6854009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:54.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:54.007+0000 7f684dffb640 1 -- 172.21.15.5:0/2748996654 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f6848058430 con 0x7f685806d4f0 2024-03-24T10:55:54.008 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:54.007+0000 7f684dffb640 1 -- 172.21.15.5:0/2748996654 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f68480546d0 con 0x7f685806d4f0 2024-03-24T10:55:54.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:54.235+0000 7f685f62e640 1 -- 172.21.15.5:0/2748996654 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68580681d0 con 0x7f685806d4f0 2024-03-24T10:55:54.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:54.250+0000 7f684dffb640 1 -- 172.21.15.5:0/2748996654 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f6848017030 con 0x7f685806d4f0 2024-03-24T10:55:54.250 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:54.250 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:55:54.250 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:55:54.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:54.252+0000 7f685f62e640 1 -- 172.21.15.5:0/2748996654 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f683c06b810 msgr2=0x7f683c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:54.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:54.252+0000 7f685f62e640 1 --2- 172.21.15.5:0/2748996654 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f683c06b810 0x7f683c06dcd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6854009aa0 tx=0x7f6854009190 comp rx=0 tx=0).stop 2024-03-24T10:55:54.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:54.252+0000 7f685f62e640 1 -- 172.21.15.5:0/2748996654 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f685806d4f0 msgr2=0x7f68581f3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:54.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:54.252+0000 7f685f62e640 1 --2- 172.21.15.5:0/2748996654 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f685806d4f0 0x7f68581f3be0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f684800c400 tx=0x7f684800c8c0 comp rx=0 tx=0).stop 2024-03-24T10:55:54.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:54.252+0000 7f685f62e640 1 -- 172.21.15.5:0/2748996654 shutdown_connections 2024-03-24T10:55:54.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:54.252+0000 7f685f62e640 1 --2- 172.21.15.5:0/2748996654 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f683c06b810 0x7f683c06dcd0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:54.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:54.252+0000 7f685f62e640 1 --2- 172.21.15.5:0/2748996654 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f685806d4f0 0x7f68581f3be0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:54.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:54.252+0000 7f685f62e640 1 --2- 172.21.15.5:0/2748996654 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f68581543a0 0x7f68581f0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:54.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:54.252+0000 7f685f62e640 1 --2- 172.21.15.5:0/2748996654 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f685806bf50 0x7f68580ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:54.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:54.252+0000 7f685f62e640 1 -- 172.21.15.5:0/2748996654 >> 172.21.15.5:0/2748996654 conn(0x7f6858072ef0 msgr2=0x7f68580c5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:54.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:54.252+0000 7f685f62e640 1 -- 172.21.15.5:0/2748996654 shutdown_connections 2024-03-24T10:55:54.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:54.252+0000 7f685f62e640 1 -- 172.21.15.5:0/2748996654 wait complete. 2024-03-24T10:55:54.868 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:55:54.868 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:55.869 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:55.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.971+0000 7f1d4a02e640 1 Processor -- start 2024-03-24T10:55:55.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.971+0000 7f1d4a02e640 1 -- start start 2024-03-24T10:55:55.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.972+0000 7f1d4a02e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1d440cae90 0x7f1d440cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:55.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.973+0000 7f1d4a02e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d440ccba0 con 0x7f1d440cb270 2024-03-24T10:55:55.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.973+0000 7f1d4a02e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d440cce40 con 0x7f1d440cd960 2024-03-24T10:55:55.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.973+0000 7f1d4a02e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f1d440cc9b0 con 0x7f1d440cae90 2024-03-24T10:55:55.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.973+0000 7f1d42ffd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f1d440cb270 0x7f1d440cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:44936/0 (socket says 172.21.15.5:44936) 2024-03-24T10:55:55.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.973+0000 7f1d42ffd640 1 -- 172.21.15.5:0/1335569462 learned_addr learned my addr 172.21.15.5:0/1335569462 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:55:55.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.973+0000 7f1d437fe640 1 --2- 172.21.15.5:0/1335569462 >> v2:172.21.15.57:3300/0 conn(0x7f1d440cae90 0x7f1d440cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:55.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.976+0000 7f1d427fc640 1 -- 172.21.15.5:0/1335569462 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1401614056 0 0) 0x7f1d440ccba0 con 0x7f1d440cb270 2024-03-24T10:55:55.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.976+0000 7f1d427fc640 1 -- 172.21.15.5:0/1335569462 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d2c003520 con 0x7f1d440cb270 2024-03-24T10:55:55.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.977+0000 7f1d437fe640 1 -- 172.21.15.5:0/1335569462 >> v1:172.21.15.57:6789/0 conn(0x7f1d440cd960 legacy=0x7f1d440cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:55.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.977+0000 7f1d437fe640 1 -- 172.21.15.5:0/1335569462 >> v1:172.21.15.5:6789/0 conn(0x7f1d440cb270 legacy=0x7f1d440cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:55.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.977+0000 7f1d437fe640 1 -- 172.21.15.5:0/1335569462 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d4415a590 con 0x7f1d440cae90 2024-03-24T10:55:55.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.983+0000 7f1d437fe640 1 --2- 172.21.15.5:0/1335569462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d440cae90 0x7f1d440cf530 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1d34009d90 tx=0x7f1d3402bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=566d7abc81754f3d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:55.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.987+0000 7f1d427fc640 1 -- 172.21.15.5:0/1335569462 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3402cd80 con 0x7f1d440cae90 2024-03-24T10:55:55.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.987+0000 7f1d427fc640 1 -- 172.21.15.5:0/1335569462 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d34034440 con 0x7f1d440cae90 2024-03-24T10:55:55.987 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.987+0000 7f1d427fc640 1 -- 172.21.15.5:0/1335569462 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d340347d0 con 0x7f1d440cae90 2024-03-24T10:55:55.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.988+0000 7f1d4a02e640 1 -- 172.21.15.5:0/1335569462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d440cae90 msgr2=0x7f1d440cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:55.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.988+0000 7f1d4a02e640 1 --2- 172.21.15.5:0/1335569462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d440cae90 0x7f1d440cf530 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1d34009d90 tx=0x7f1d3402bfd0 comp rx=0 tx=0).stop 2024-03-24T10:55:55.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.988+0000 7f1d4a02e640 1 -- 172.21.15.5:0/1335569462 shutdown_connections 2024-03-24T10:55:55.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.988+0000 7f1d4a02e640 1 --2- 172.21.15.5:0/1335569462 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d440cae90 0x7f1d440cf530 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:55.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.988+0000 7f1d4a02e640 1 -- 172.21.15.5:0/1335569462 >> 172.21.15.5:0/1335569462 conn(0x7f1d440cfb80 msgr2=0x7f1d440cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:55.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.988+0000 7f1d4a02e640 1 -- 172.21.15.5:0/1335569462 shutdown_connections 2024-03-24T10:55:55.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.988+0000 7f1d4a02e640 1 -- 172.21.15.5:0/1335569462 wait complete. 2024-03-24T10:55:55.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.990+0000 7f1d4a02e640 1 Processor -- start 2024-03-24T10:55:55.990 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.990+0000 7f1d4a02e640 1 -- start start 2024-03-24T10:55:55.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.991+0000 7f1d4a02e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d440cae90 0x7f1d4415dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:55.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.991+0000 7f1d4a02e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1d440cb270 0x7f1d4414ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:55.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.991+0000 7f1d4a02e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d440cd960 0x7f1d440b2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:55.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.992+0000 7f1d4a02e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f1d44150690 con 0x7f1d440cd960 2024-03-24T10:55:55.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.992+0000 7f1d4a02e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1d4414dd50 con 0x7f1d440cae90 2024-03-24T10:55:55.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.992+0000 7f1d4a02e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1d440b58e0 con 0x7f1d440cb270 2024-03-24T10:55:55.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.992+0000 7f1d43fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d440cd960 0x7f1d440b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:55.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.992+0000 7f1d437fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d440cae90 0x7f1d4415dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:55.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.992+0000 7f1d42ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1d440cb270 0x7f1d4414ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:55.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.992+0000 7f1d43fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d440cd960 0x7f1d440b2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:34952/0 (socket says 172.21.15.5:34952) 2024-03-24T10:55:55.994 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.992+0000 7f1d43fff640 1 -- 172.21.15.5:0/674577765 learned_addr learned my addr 172.21.15.5:0/674577765 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:55.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.996+0000 7f1d43fff640 1 -- 172.21.15.5:0/674577765 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1d440cb270 msgr2=0x7f1d4414ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:55.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.996+0000 7f1d43fff640 1 --2- 172.21.15.5:0/674577765 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1d440cb270 0x7f1d4414ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:55.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.996+0000 7f1d43fff640 1 -- 172.21.15.5:0/674577765 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d440cae90 msgr2=0x7f1d4415dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:55.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.996+0000 7f1d43fff640 1 --2- 172.21.15.5:0/674577765 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d440cae90 0x7f1d4415dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:55.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.996+0000 7f1d43fff640 1 -- 172.21.15.5:0/674577765 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d34009a70 con 0x7f1d440cd960 2024-03-24T10:55:55.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:55.999+0000 7f1d43fff640 1 --2- 172.21.15.5:0/674577765 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d440cd960 0x7f1d440b2750 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c00c390 tx=0x7f1d3c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:56.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.002+0000 7f1d2bfff640 1 -- 172.21.15.5:0/674577765 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c0052d0 con 0x7f1d440cd960 2024-03-24T10:55:56.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.002+0000 7f1d2bfff640 1 -- 172.21.15.5:0/674577765 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d3c005430 con 0x7f1d440cd960 2024-03-24T10:55:56.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.002+0000 7f1d2bfff640 1 -- 172.21.15.5:0/674577765 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c003810 con 0x7f1d440cd960 2024-03-24T10:55:56.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.002+0000 7f1d4a02e640 1 -- 172.21.15.5:0/674577765 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d441f4000 con 0x7f1d440cd960 2024-03-24T10:55:56.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.002+0000 7f1d4a02e640 1 -- 172.21.15.5:0/674577765 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d4414e280 con 0x7f1d440cd960 2024-03-24T10:55:56.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.004+0000 7f1d4a02e640 1 -- 172.21.15.5:0/674577765 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d4414ea80 con 0x7f1d440cd960 2024-03-24T10:55:56.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.272+0000 7f1d2bfff640 1 -- 172.21.15.5:0/674577765 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1d3c003970 con 0x7f1d440cd960 2024-03-24T10:55:56.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.272+0000 7f1d2bfff640 1 --2- 172.21.15.5:0/674577765 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1d3006b860 0x7f1d3006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:56.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.273+0000 7f1d437fe640 1 --2- 172.21.15.5:0/674577765 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1d3006b860 0x7f1d3006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:56.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.273+0000 7f1d437fe640 1 --2- 172.21.15.5:0/674577765 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1d3006b860 0x7f1d3006dd20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1d34036000 tx=0x7f1d34030000 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:56.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.297+0000 7f1d2bfff640 1 -- 172.21.15.5:0/674577765 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f1d3c057f80 con 0x7f1d440cd960 2024-03-24T10:55:56.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.297+0000 7f1d2bfff640 1 -- 172.21.15.5:0/674577765 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1d3c054220 con 0x7f1d440cd960 2024-03-24T10:55:56.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.539+0000 7f1d4a02e640 1 -- 172.21.15.5:0/674577765 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d4414ea80 con 0x7f1d440cd960 2024-03-24T10:55:56.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.553+0000 7f1d2bfff640 1 -- 172.21.15.5:0/674577765 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f1d4414ea80 con 0x7f1d440cd960 2024-03-24T10:55:56.553 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:56.553 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:55:56.554 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:55:56.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.555+0000 7f1d4a02e640 1 -- 172.21.15.5:0/674577765 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1d3006b860 msgr2=0x7f1d3006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:56.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.555+0000 7f1d4a02e640 1 --2- 172.21.15.5:0/674577765 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1d3006b860 0x7f1d3006dd20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1d34036000 tx=0x7f1d34030000 comp rx=0 tx=0).stop 2024-03-24T10:55:56.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.555+0000 7f1d4a02e640 1 -- 172.21.15.5:0/674577765 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d440cd960 msgr2=0x7f1d440b2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:56.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.555+0000 7f1d4a02e640 1 --2- 172.21.15.5:0/674577765 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d440cd960 0x7f1d440b2750 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c00c390 tx=0x7f1d3c00c850 comp rx=0 tx=0).stop 2024-03-24T10:55:56.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.555+0000 7f1d4a02e640 1 -- 172.21.15.5:0/674577765 shutdown_connections 2024-03-24T10:55:56.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.555+0000 7f1d4a02e640 1 --2- 172.21.15.5:0/674577765 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1d3006b860 0x7f1d3006dd20 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:56.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.555+0000 7f1d4a02e640 1 --2- 172.21.15.5:0/674577765 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d440cd960 0x7f1d440b2750 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:56.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.555+0000 7f1d4a02e640 1 --2- 172.21.15.5:0/674577765 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1d440cb270 0x7f1d4414ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:56.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.555+0000 7f1d4a02e640 1 --2- 172.21.15.5:0/674577765 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d440cae90 0x7f1d4415dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:56.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.555+0000 7f1d4a02e640 1 -- 172.21.15.5:0/674577765 >> 172.21.15.5:0/674577765 conn(0x7f1d440cfb80 msgr2=0x7f1d440cddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:56.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.555+0000 7f1d4a02e640 1 -- 172.21.15.5:0/674577765 shutdown_connections 2024-03-24T10:55:56.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:56.555+0000 7f1d4a02e640 1 -- 172.21.15.5:0/674577765 wait complete. 2024-03-24T10:55:57.173 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:55:57.173 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:55:58.174 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:55:58.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.276+0000 7f0ad1fb5640 1 Processor -- start 2024-03-24T10:55:58.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.276+0000 7f0ad1fb5640 1 -- start start 2024-03-24T10:55:58.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.277+0000 7f0ad1fb5640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0acc1543a0 0x7f0acc15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:58.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.277+0000 7f0ad1fb5640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0acc06bf50 0x7f0acc06c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:58.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.277+0000 7f0ad1fb5640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0acc0cb2f0 con 0x7f0acc06d4f0 2024-03-24T10:55:58.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.277+0000 7f0ad1fb5640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f0acc0c6750 con 0x7f0acc1543a0 2024-03-24T10:55:58.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.277+0000 7f0ad1fb5640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f0acc0cc380 con 0x7f0acc06bf50 2024-03-24T10:55:58.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.277+0000 7f0acbfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0acc06bf50 0x7f0acc06c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:58.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.277+0000 7f0acb7fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0acc1543a0 0x7f0acc15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:58.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.278+0000 7f0acbfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0acc06bf50 0x7f0acc06c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:43976/0 (socket says 172.21.15.5:43976) 2024-03-24T10:55:58.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.278+0000 7f0acbfff640 1 -- 172.21.15.5:0/1953424697 learned_addr learned my addr 172.21.15.5:0/1953424697 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:58.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.281+0000 7f0acbfff640 1 -- 172.21.15.5:0/1953424697 >> v2:172.21.15.5:3300/0 conn(0x7f0acc1543a0 msgr2=0x7f0acc15ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:55:58.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.281+0000 7f0acbfff640 1 --2- 172.21.15.5:0/1953424697 >> v2:172.21.15.5:3300/0 conn(0x7f0acc1543a0 0x7f0acc15ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:58.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.281+0000 7f0acbfff640 1 -- 172.21.15.5:0/1953424697 >> v1:172.21.15.57:6789/0 conn(0x7f0acc06d4f0 legacy=0x7f0acc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:58.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.281+0000 7f0acbfff640 1 -- 172.21.15.5:0/1953424697 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0acc0ca7e0 con 0x7f0acc06bf50 2024-03-24T10:55:58.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.287+0000 7f0acbfff640 1 --2- 172.21.15.5:0/1953424697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc06bf50 0x7f0acc06c330 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0b51c0 tx=0x7f0ac402ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f7f02fff9423cbc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:58.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.294+0000 7f0aca7fc640 1 -- 172.21.15.5:0/1953424697 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac402f850 con 0x7f0acc06bf50 2024-03-24T10:55:58.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.294+0000 7f0aca7fc640 1 -- 172.21.15.5:0/1953424697 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ac402f9b0 con 0x7f0acc06bf50 2024-03-24T10:55:58.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.294+0000 7f0aca7fc640 1 -- 172.21.15.5:0/1953424697 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac402fd40 con 0x7f0acc06bf50 2024-03-24T10:55:58.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.294+0000 7f0ad1fb5640 1 -- 172.21.15.5:0/1953424697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc06bf50 msgr2=0x7f0acc06c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:58.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.294+0000 7f0ad1fb5640 1 --2- 172.21.15.5:0/1953424697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc06bf50 0x7f0acc06c330 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0acc0b51c0 tx=0x7f0ac402ba30 comp rx=0 tx=0).stop 2024-03-24T10:55:58.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.294+0000 7f0ad1fb5640 1 -- 172.21.15.5:0/1953424697 shutdown_connections 2024-03-24T10:55:58.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.294+0000 7f0ad1fb5640 1 --2- 172.21.15.5:0/1953424697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc06bf50 0x7f0acc06c330 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:58.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.294+0000 7f0ad1fb5640 1 --2- 172.21.15.5:0/1953424697 >> v2:172.21.15.5:3300/0 conn(0x7f0acc1543a0 0x7f0acc15ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:58.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.294+0000 7f0ad1fb5640 1 -- 172.21.15.5:0/1953424697 >> 172.21.15.5:0/1953424697 conn(0x7f0acc072ef0 msgr2=0x7f0acc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:58.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.294+0000 7f0ad1fb5640 1 -- 172.21.15.5:0/1953424697 shutdown_connections 2024-03-24T10:55:58.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.294+0000 7f0ad1fb5640 1 -- 172.21.15.5:0/1953424697 wait complete. 2024-03-24T10:55:58.295 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.296+0000 7f0ad1fb5640 1 Processor -- start 2024-03-24T10:55:58.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.296+0000 7f0ad1fb5640 1 -- start start 2024-03-24T10:55:58.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.296+0000 7f0ad1fb5640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc06bf50 0x7f0acc152da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:58.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.296+0000 7f0ad1fb5640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0acc06d4f0 0x7f0acc1f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:58.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.296+0000 7f0ad1fb5640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0acc1543a0 0x7f0acc1486f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:58.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.296+0000 7f0ad1fb5640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc1f3f30 con 0x7f0acc1543a0 2024-03-24T10:55:58.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.296+0000 7f0ad1fb5640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc1534a0 con 0x7f0acc06bf50 2024-03-24T10:55:58.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.296+0000 7f0ad1fb5640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0acc153740 con 0x7f0acc06d4f0 2024-03-24T10:55:58.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.297+0000 7f0acb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc06bf50 0x7f0acc152da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:58.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.297+0000 7f0acbfff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0acc1543a0 0x7f0acc1486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:58.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.297+0000 7f0acaffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0acc06d4f0 0x7f0acc1f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:58.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.297+0000 7f0acb7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc06bf50 0x7f0acc152da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:43988/0 (socket says 172.21.15.5:43988) 2024-03-24T10:55:58.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.297+0000 7f0acb7fe640 1 -- 172.21.15.5:0/1535648404 learned_addr learned my addr 172.21.15.5:0/1535648404 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:55:58.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.301+0000 7f0acb7fe640 1 -- 172.21.15.5:0/1535648404 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0acc06d4f0 msgr2=0x7f0acc1f39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:58.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.301+0000 7f0acb7fe640 1 --2- 172.21.15.5:0/1535648404 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0acc06d4f0 0x7f0acc1f39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:58.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.301+0000 7f0acb7fe640 1 -- 172.21.15.5:0/1535648404 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0acc1543a0 msgr2=0x7f0acc1486f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:58.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.301+0000 7f0acb7fe640 1 --2- 172.21.15.5:0/1535648404 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0acc1543a0 0x7f0acc1486f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:58.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.301+0000 7f0acb7fe640 1 -- 172.21.15.5:0/1535648404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac40094d0 con 0x7f0acc06bf50 2024-03-24T10:55:58.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.304+0000 7f0acb7fe640 1 --2- 172.21.15.5:0/1535648404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc06bf50 0x7f0acc152da0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0abc00efc0 tx=0x7f0abc00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:58.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.308+0000 7f0ab3fff640 1 -- 172.21.15.5:0/1535648404 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc009b80 con 0x7f0acc06bf50 2024-03-24T10:55:58.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.308+0000 7f0ab3fff640 1 -- 172.21.15.5:0/1535648404 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0abc00f040 con 0x7f0acc06bf50 2024-03-24T10:55:58.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.308+0000 7f0ad1fb5640 1 -- 172.21.15.5:0/1535648404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0acc14f0c0 con 0x7f0acc06bf50 2024-03-24T10:55:58.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.308+0000 7f0ab3fff640 1 -- 172.21.15.5:0/1535648404 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0abc0146a0 con 0x7f0acc06bf50 2024-03-24T10:55:58.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.308+0000 7f0ad1fb5640 1 -- 172.21.15.5:0/1535648404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0acc1f31f0 con 0x7f0acc06bf50 2024-03-24T10:55:58.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.309+0000 7f0ad1fb5640 1 -- 172.21.15.5:0/1535648404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0acc153c20 con 0x7f0acc06bf50 2024-03-24T10:55:58.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.329+0000 7f0ab3fff640 1 -- 172.21.15.5:0/1535648404 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0abc014800 con 0x7f0acc06bf50 2024-03-24T10:55:58.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.329+0000 7f0ab3fff640 1 --2- 172.21.15.5:0/1535648404 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0ab806b860 0x7f0ab806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:55:58.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.330+0000 7f0acaffd640 1 --2- 172.21.15.5:0/1535648404 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0ab806b860 0x7f0ab806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:55:58.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.330+0000 7f0acaffd640 1 --2- 172.21.15.5:0/1535648404 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0ab806b860 0x7f0ab806dd20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0acc062260 tx=0x7f0ac0009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:55:58.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.335+0000 7f0ab3fff640 1 -- 172.21.15.5:0/1535648404 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f0abc059b80 con 0x7f0acc06bf50 2024-03-24T10:55:58.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.597+0000 7f0ab3fff640 1 -- 172.21.15.5:0/1535648404 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0abc055e20 con 0x7f0acc06bf50 2024-03-24T10:55:58.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.884+0000 7f0ad1fb5640 1 -- 172.21.15.5:0/1535648404 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0acc0681d0 con 0x7f0acc06bf50 2024-03-24T10:55:58.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.907+0000 7f0ab3fff640 1 -- 172.21.15.5:0/1535648404 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f0abc0594e0 con 0x7f0acc06bf50 2024-03-24T10:55:58.907 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:55:58.907 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:55:58.907 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:55:58.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.910+0000 7f0ad1fb5640 1 -- 172.21.15.5:0/1535648404 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0ab806b860 msgr2=0x7f0ab806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:58.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.910+0000 7f0ad1fb5640 1 --2- 172.21.15.5:0/1535648404 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0ab806b860 0x7f0ab806dd20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f0acc062260 tx=0x7f0ac0009040 comp rx=0 tx=0).stop 2024-03-24T10:55:58.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.910+0000 7f0ad1fb5640 1 -- 172.21.15.5:0/1535648404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc06bf50 msgr2=0x7f0acc152da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:55:58.910 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.910+0000 7f0ad1fb5640 1 --2- 172.21.15.5:0/1535648404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc06bf50 0x7f0acc152da0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0abc00efc0 tx=0x7f0abc00c490 comp rx=0 tx=0).stop 2024-03-24T10:55:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.910+0000 7f0ad1fb5640 1 -- 172.21.15.5:0/1535648404 shutdown_connections 2024-03-24T10:55:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.910+0000 7f0ad1fb5640 1 --2- 172.21.15.5:0/1535648404 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0ab806b860 0x7f0ab806dd20 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.910+0000 7f0ad1fb5640 1 --2- 172.21.15.5:0/1535648404 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0acc1543a0 0x7f0acc1486f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.910+0000 7f0ad1fb5640 1 --2- 172.21.15.5:0/1535648404 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0acc06d4f0 0x7f0acc1f39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.910+0000 7f0ad1fb5640 1 --2- 172.21.15.5:0/1535648404 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0acc06bf50 0x7f0acc152da0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:55:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.910+0000 7f0ad1fb5640 1 -- 172.21.15.5:0/1535648404 >> 172.21.15.5:0/1535648404 conn(0x7f0acc072ef0 msgr2=0x7f0acc158850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:55:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.910+0000 7f0ad1fb5640 1 -- 172.21.15.5:0/1535648404 shutdown_connections 2024-03-24T10:55:58.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:55:58.910+0000 7f0ad1fb5640 1 -- 172.21.15.5:0/1535648404 wait complete. 2024-03-24T10:55:59.529 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:55:59.530 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:00.531 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:00.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.633+0000 7fe31549d640 1 Processor -- start 2024-03-24T10:56:00.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.633+0000 7fe31549d640 1 -- start start 2024-03-24T10:56:00.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.634+0000 7fe31549d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe31015ac00 0x7fe3100cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:00.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.634+0000 7fe31549d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3100ccaa0 0x7fe31006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:00.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.634+0000 7fe31549d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3100cb2f0 con 0x7fe3101543a0 2024-03-24T10:56:00.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.634+0000 7fe31549d640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fe3100c6750 con 0x7fe31015ac00 2024-03-24T10:56:00.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.634+0000 7fe31549d640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe3100cceb0 con 0x7fe3100ccaa0 2024-03-24T10:56:00.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.635+0000 7fe30e7fc640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe31015ac00 0x7fe3100cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:00.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.635+0000 7fe30e7fc640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe31015ac00 0x7fe3100cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:34984/0 (socket says 172.21.15.5:34984) 2024-03-24T10:56:00.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.635+0000 7fe30e7fc640 1 -- 172.21.15.5:0/1045208871 learned_addr learned my addr 172.21.15.5:0/1045208871 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:00.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.635+0000 7fe30f7fe640 1 --2- 172.21.15.5:0/1045208871 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3100ccaa0 0x7fe31006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:00.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.637+0000 7fe30e7fc640 1 -- 172.21.15.5:0/1045208871 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3100ccaa0 msgr2=0x7fe31006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:00.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.637+0000 7fe30e7fc640 1 --2- 172.21.15.5:0/1045208871 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3100ccaa0 0x7fe31006bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:00.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.637+0000 7fe30e7fc640 1 -- 172.21.15.5:0/1045208871 >> v1:172.21.15.57:6789/0 conn(0x7fe3101543a0 legacy=0x7fe31006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:00.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.637+0000 7fe30e7fc640 1 -- 172.21.15.5:0/1045208871 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe31015b070 con 0x7fe31015ac00 2024-03-24T10:56:00.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.639+0000 7fe30e7fc640 1 --2- 172.21.15.5:0/1045208871 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe31015ac00 0x7fe3100cc560 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe304009870 tx=0x7fe30402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d01a7c0a35245ed7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:00.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.644+0000 7fe30dffb640 1 -- 172.21.15.5:0/1045208871 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304030840 con 0x7fe31015ac00 2024-03-24T10:56:00.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.644+0000 7fe30dffb640 1 -- 172.21.15.5:0/1045208871 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3040309a0 con 0x7fe31015ac00 2024-03-24T10:56:00.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.644+0000 7fe30dffb640 1 -- 172.21.15.5:0/1045208871 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304030d30 con 0x7fe31015ac00 2024-03-24T10:56:00.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.645+0000 7fe31549d640 1 -- 172.21.15.5:0/1045208871 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe31015ac00 msgr2=0x7fe3100cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:00.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.645+0000 7fe31549d640 1 --2- 172.21.15.5:0/1045208871 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe31015ac00 0x7fe3100cc560 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe304009870 tx=0x7fe30402b980 comp rx=0 tx=0).stop 2024-03-24T10:56:00.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.645+0000 7fe31549d640 1 -- 172.21.15.5:0/1045208871 shutdown_connections 2024-03-24T10:56:00.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.645+0000 7fe31549d640 1 --2- 172.21.15.5:0/1045208871 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3100ccaa0 0x7fe31006bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:00.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.645+0000 7fe31549d640 1 --2- 172.21.15.5:0/1045208871 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe31015ac00 0x7fe3100cc560 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:00.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.645+0000 7fe31549d640 1 -- 172.21.15.5:0/1045208871 >> 172.21.15.5:0/1045208871 conn(0x7fe310072ef0 msgr2=0x7fe310061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:00.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.645+0000 7fe31549d640 1 -- 172.21.15.5:0/1045208871 shutdown_connections 2024-03-24T10:56:00.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.645+0000 7fe31549d640 1 -- 172.21.15.5:0/1045208871 wait complete. 2024-03-24T10:56:00.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.646+0000 7fe31549d640 1 Processor -- start 2024-03-24T10:56:00.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.646+0000 7fe31549d640 1 -- start start 2024-03-24T10:56:00.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.647+0000 7fe31549d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3100ccaa0 0x7fe310146200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:00.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.647+0000 7fe31549d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3101543a0 0x7fe310146740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:00.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.647+0000 7fe31549d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe31015ac00 0x7fe3100b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:00.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.647+0000 7fe31549d640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fe3101586c0 con 0x7fe31015ac00 2024-03-24T10:56:00.647 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.647+0000 7fe31549d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe310159f10 con 0x7fe3100ccaa0 2024-03-24T10:56:00.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.647+0000 7fe31549d640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe310143fb0 con 0x7fe3101543a0 2024-03-24T10:56:00.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.648+0000 7fe30f7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe31015ac00 0x7fe3100b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:00.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.648+0000 7fe30e7fc640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3101543a0 0x7fe310146740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:00.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.648+0000 7fe30effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3100ccaa0 0x7fe310146200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:00.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.648+0000 7fe30f7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe31015ac00 0x7fe3100b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:35000/0 (socket says 172.21.15.5:35000) 2024-03-24T10:56:00.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.648+0000 7fe30f7fe640 1 -- 172.21.15.5:0/2130285940 learned_addr learned my addr 172.21.15.5:0/2130285940 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:00.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.653+0000 7fe30f7fe640 1 -- 172.21.15.5:0/2130285940 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3101543a0 msgr2=0x7fe310146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:00.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.653+0000 7fe30f7fe640 1 --2- 172.21.15.5:0/2130285940 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3101543a0 0x7fe310146740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:00.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.653+0000 7fe30f7fe640 1 -- 172.21.15.5:0/2130285940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3100ccaa0 msgr2=0x7fe310146200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:00.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.653+0000 7fe30f7fe640 1 --2- 172.21.15.5:0/2130285940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3100ccaa0 0x7fe310146200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:00.653 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.653+0000 7fe30f7fe640 1 -- 172.21.15.5:0/2130285940 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3040094d0 con 0x7fe31015ac00 2024-03-24T10:56:00.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.656+0000 7fe30f7fe640 1 --2- 172.21.15.5:0/2130285940 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe31015ac00 0x7fe3100b3e30 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe30800efc0 tx=0x7fe30800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:00.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.660+0000 7fe2f77fe640 1 -- 172.21.15.5:0/2130285940 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe308014480 con 0x7fe31015ac00 2024-03-24T10:56:00.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.660+0000 7fe2f77fe640 1 -- 172.21.15.5:0/2130285940 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe30800f040 con 0x7fe31015ac00 2024-03-24T10:56:00.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.660+0000 7fe2f77fe640 1 -- 172.21.15.5:0/2130285940 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe308014820 con 0x7fe31015ac00 2024-03-24T10:56:00.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.660+0000 7fe31549d640 1 -- 172.21.15.5:0/2130285940 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe310159c30 con 0x7fe31015ac00 2024-03-24T10:56:00.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.660+0000 7fe31549d640 1 -- 172.21.15.5:0/2130285940 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3101583e0 con 0x7fe31015ac00 2024-03-24T10:56:00.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.661+0000 7fe31549d640 1 -- 172.21.15.5:0/2130285940 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3100681d0 con 0x7fe31015ac00 2024-03-24T10:56:00.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.942+0000 7fe2f77fe640 1 -- 172.21.15.5:0/2130285940 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe308014980 con 0x7fe31015ac00 2024-03-24T10:56:00.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.943+0000 7fe2f77fe640 1 --2- 172.21.15.5:0/2130285940 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe2fc06b860 0x7fe2fc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:00.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.943+0000 7fe30effd640 1 --2- 172.21.15.5:0/2130285940 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe2fc06b860 0x7fe2fc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:00.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.943+0000 7fe30effd640 1 --2- 172.21.15.5:0/2130285940 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe2fc06b860 0x7fe2fc06dd20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe300001780 tx=0x7fe300009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:00.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.958+0000 7fe2f77fe640 1 -- 172.21.15.5:0/2130285940 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fe308059c60 con 0x7fe31015ac00 2024-03-24T10:56:00.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:00.958+0000 7fe2f77fe640 1 -- 172.21.15.5:0/2130285940 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe308055f00 con 0x7fe31015ac00 2024-03-24T10:56:01.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:01.181+0000 7fe31549d640 1 -- 172.21.15.5:0/2130285940 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe310061cd0 con 0x7fe31015ac00 2024-03-24T10:56:01.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:01.211+0000 7fe2f77fe640 1 -- 172.21.15.5:0/2130285940 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7fe308018020 con 0x7fe31015ac00 2024-03-24T10:56:01.211 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:01.211 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:01.212 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:01.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:01.213+0000 7fe31549d640 1 -- 172.21.15.5:0/2130285940 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe2fc06b860 msgr2=0x7fe2fc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:01.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:01.213+0000 7fe31549d640 1 --2- 172.21.15.5:0/2130285940 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe2fc06b860 0x7fe2fc06dd20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fe300001780 tx=0x7fe300009040 comp rx=0 tx=0).stop 2024-03-24T10:56:01.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:01.213+0000 7fe31549d640 1 -- 172.21.15.5:0/2130285940 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe31015ac00 msgr2=0x7fe3100b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:01.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:01.213+0000 7fe31549d640 1 --2- 172.21.15.5:0/2130285940 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe31015ac00 0x7fe3100b3e30 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fe30800efc0 tx=0x7fe30800c490 comp rx=0 tx=0).stop 2024-03-24T10:56:01.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:01.213+0000 7fe31549d640 1 -- 172.21.15.5:0/2130285940 shutdown_connections 2024-03-24T10:56:01.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:01.213+0000 7fe31549d640 1 --2- 172.21.15.5:0/2130285940 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe2fc06b860 0x7fe2fc06dd20 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:01.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:01.213+0000 7fe31549d640 1 --2- 172.21.15.5:0/2130285940 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe31015ac00 0x7fe3100b3e30 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:01.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:01.213+0000 7fe31549d640 1 --2- 172.21.15.5:0/2130285940 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe3101543a0 0x7fe310146740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:01.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:01.213+0000 7fe31549d640 1 --2- 172.21.15.5:0/2130285940 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe3100ccaa0 0x7fe310146200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:01.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:01.213+0000 7fe31549d640 1 -- 172.21.15.5:0/2130285940 >> 172.21.15.5:0/2130285940 conn(0x7fe310072ef0 msgr2=0x7fe310153470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:01.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:01.213+0000 7fe31549d640 1 -- 172.21.15.5:0/2130285940 shutdown_connections 2024-03-24T10:56:01.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:01.213+0000 7fe31549d640 1 -- 172.21.15.5:0/2130285940 wait complete. 2024-03-24T10:56:01.829 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:01.829 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:02.669 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:56:02.672 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:56:02.830 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:02.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.935+0000 7f02f963b640 1 Processor -- start 2024-03-24T10:56:02.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.935+0000 7f02f963b640 1 -- start start 2024-03-24T10:56:02.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.936+0000 7f02f963b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02f40cbe10 0x7f02f40ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:02.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.936+0000 7f02f963b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f02f40caee0 0x7f02f415a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:02.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.936+0000 7f02f963b640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02f415af60 con 0x7f02f41543c0 2024-03-24T10:56:02.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.936+0000 7f02f963b640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f02f40c5110 con 0x7f02f40caee0 2024-03-24T10:56:02.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.936+0000 7f02f963b640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f02f415b240 con 0x7f02f40cbe10 2024-03-24T10:56:02.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.937+0000 7f02ebfff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02f40cbe10 0x7f02f40ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:02.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.937+0000 7f02f37fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f02f40caee0 0x7f02f415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:02.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.937+0000 7f02f2ffd640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f02f41543c0 0x7f02f40c4a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:45440/0 (socket says 172.21.15.5:45440) 2024-03-24T10:56:02.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.937+0000 7f02f2ffd640 1 -- 172.21.15.5:0/3056067501 learned_addr learned my addr 172.21.15.5:0/3056067501 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:56:02.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.941+0000 7f02f27fc640 1 -- 172.21.15.5:0/3056067501 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1010220315 0 0) 0x7f02f415af60 con 0x7f02f41543c0 2024-03-24T10:56:02.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.941+0000 7f02f37fe640 1 -- 172.21.15.5:0/3056067501 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02f40cbe10 msgr2=0x7f02f40ca9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:02.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.941+0000 7f02f37fe640 1 --2- 172.21.15.5:0/3056067501 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02f40cbe10 0x7f02f40ca9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:02.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.941+0000 7f02f37fe640 1 -- 172.21.15.5:0/3056067501 >> v1:172.21.15.57:6789/0 conn(0x7f02f41543c0 legacy=0x7f02f40c4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:02.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.941+0000 7f02f37fe640 1 -- 172.21.15.5:0/3056067501 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f415d970 con 0x7f02f40caee0 2024-03-24T10:56:02.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.941+0000 7f02ebfff640 1 --2- 172.21.15.5:0/3056067501 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02f40cbe10 0x7f02f40ca9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:56:02.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.943+0000 7f02f37fe640 1 --2- 172.21.15.5:0/3056067501 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02f40caee0 0x7f02f415a380 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f02ec009d90 tx=0x7f02ec02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a2e173d6737ef41 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:02.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.947+0000 7f02f27fc640 1 -- 172.21.15.5:0/3056067501 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec02cc50 con 0x7f02f40caee0 2024-03-24T10:56:02.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.947+0000 7f02f27fc640 1 -- 172.21.15.5:0/3056067501 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02ec02cdb0 con 0x7f02f40caee0 2024-03-24T10:56:02.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.947+0000 7f02f27fc640 1 -- 172.21.15.5:0/3056067501 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec034660 con 0x7f02f40caee0 2024-03-24T10:56:02.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.948+0000 7f02f963b640 1 -- 172.21.15.5:0/3056067501 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02f40caee0 msgr2=0x7f02f415a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:02.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.948+0000 7f02f963b640 1 --2- 172.21.15.5:0/3056067501 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02f40caee0 0x7f02f415a380 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f02ec009d90 tx=0x7f02ec02bea0 comp rx=0 tx=0).stop 2024-03-24T10:56:02.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.948+0000 7f02f963b640 1 -- 172.21.15.5:0/3056067501 shutdown_connections 2024-03-24T10:56:02.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.948+0000 7f02f963b640 1 --2- 172.21.15.5:0/3056067501 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02f40cbe10 0x7f02f40ca9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:02.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.948+0000 7f02f963b640 1 --2- 172.21.15.5:0/3056067501 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02f40caee0 0x7f02f415a380 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:02.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.948+0000 7f02f963b640 1 -- 172.21.15.5:0/3056067501 >> 172.21.15.5:0/3056067501 conn(0x7f02f4061b50 msgr2=0x7f02f4061f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:02.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.948+0000 7f02f963b640 1 -- 172.21.15.5:0/3056067501 shutdown_connections 2024-03-24T10:56:02.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.948+0000 7f02f963b640 1 -- 172.21.15.5:0/3056067501 wait complete. 2024-03-24T10:56:02.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.949+0000 7f02f963b640 1 Processor -- start 2024-03-24T10:56:02.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.950+0000 7f02f963b640 1 -- start start 2024-03-24T10:56:02.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.950+0000 7f02f963b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02f40caee0 0x7f02f41f3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:02.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.950+0000 7f02f963b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f40cbe10 0x7f02f40b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:02.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.950+0000 7f02f963b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02f41543c0 0x7f02f415ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:02.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.950+0000 7f02f963b640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f02f40c76e0 con 0x7f02f40caee0 2024-03-24T10:56:02.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.950+0000 7f02f963b640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f02f40b3d60 con 0x7f02f40cbe10 2024-03-24T10:56:02.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.950+0000 7f02f963b640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f02f41f35b0 con 0x7f02f41543c0 2024-03-24T10:56:02.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.951+0000 7f02f2ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02f40caee0 0x7f02f41f3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:02.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.951+0000 7f02f37fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02f41543c0 0x7f02f415ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:02.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.951+0000 7f02ebfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f40cbe10 0x7f02f40b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:02.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.951+0000 7f02f2ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02f40caee0 0x7f02f41f3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:33096/0 (socket says 172.21.15.5:33096) 2024-03-24T10:56:02.952 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.951+0000 7f02f2ffd640 1 -- 172.21.15.5:0/3025364649 learned_addr learned my addr 172.21.15.5:0/3025364649 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:02.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.955+0000 7f02ebfff640 1 -- 172.21.15.5:0/3025364649 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02f41543c0 msgr2=0x7f02f415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:02.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.955+0000 7f02ebfff640 1 --2- 172.21.15.5:0/3025364649 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02f41543c0 0x7f02f415ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:02.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.955+0000 7f02ebfff640 1 -- 172.21.15.5:0/3025364649 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02f40caee0 msgr2=0x7f02f41f3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:02.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.955+0000 7f02ebfff640 1 --2- 172.21.15.5:0/3025364649 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02f40caee0 0x7f02f41f3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:02.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.955+0000 7f02ebfff640 1 -- 172.21.15.5:0/3025364649 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02dc009400 con 0x7f02f40cbe10 2024-03-24T10:56:02.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.958+0000 7f02ebfff640 1 --2- 172.21.15.5:0/3025364649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f40cbe10 0x7f02f40b47b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f02dc00e8d0 tx=0x7f02dc00ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:02.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.962+0000 7f02eb7fe640 1 -- 172.21.15.5:0/3025364649 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc00c940 con 0x7f02f40cbe10 2024-03-24T10:56:02.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.963+0000 7f02eb7fe640 1 -- 172.21.15.5:0/3025364649 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02dc00caa0 con 0x7f02f40cbe10 2024-03-24T10:56:02.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.963+0000 7f02eb7fe640 1 -- 172.21.15.5:0/3025364649 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02dc00ce30 con 0x7f02f40cbe10 2024-03-24T10:56:02.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.963+0000 7f02f963b640 1 -- 172.21.15.5:0/3025364649 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02ec009a70 con 0x7f02f40cbe10 2024-03-24T10:56:02.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.963+0000 7f02f963b640 1 -- 172.21.15.5:0/3025364649 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f41f3820 con 0x7f02f40cbe10 2024-03-24T10:56:02.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.964+0000 7f02f963b640 1 -- 172.21.15.5:0/3025364649 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02f4153f00 con 0x7f02f40cbe10 2024-03-24T10:56:02.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.989+0000 7f02eb7fe640 1 -- 172.21.15.5:0/3025364649 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f02dc013470 con 0x7f02f40cbe10 2024-03-24T10:56:02.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.989+0000 7f02eb7fe640 1 --2- 172.21.15.5:0/3025364649 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f02d406b860 0x7f02d406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:02.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.989+0000 7f02f2ffd640 1 --2- 172.21.15.5:0/3025364649 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f02d406b860 0x7f02d406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:02.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:02.990+0000 7f02f2ffd640 1 --2- 172.21.15.5:0/3025364649 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f02d406b860 0x7f02d406dd20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f02e4000f60 tx=0x7f02e4007240 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:03.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:03.272+0000 7f02eb7fe640 1 -- 172.21.15.5:0/3025364649 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f02dc059370 con 0x7f02f40cbe10 2024-03-24T10:56:03.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:03.293+0000 7f02eb7fe640 1 -- 172.21.15.5:0/3025364649 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f02dc055610 con 0x7f02f40cbe10 2024-03-24T10:56:03.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:03.514+0000 7f02f963b640 1 -- 172.21.15.5:0/3025364649 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02f40688e0 con 0x7f02f40cbe10 2024-03-24T10:56:03.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:03.529+0000 7f02eb7fe640 1 -- 172.21.15.5:0/3025364649 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f02dc017020 con 0x7f02f40cbe10 2024-03-24T10:56:03.529 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:03.529 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:03.529 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:03.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:03.531+0000 7f02f963b640 1 -- 172.21.15.5:0/3025364649 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f02d406b860 msgr2=0x7f02d406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:03.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:03.531+0000 7f02f963b640 1 --2- 172.21.15.5:0/3025364649 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f02d406b860 0x7f02d406dd20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f02e4000f60 tx=0x7f02e4007240 comp rx=0 tx=0).stop 2024-03-24T10:56:03.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:03.531+0000 7f02f963b640 1 -- 172.21.15.5:0/3025364649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f40cbe10 msgr2=0x7f02f40b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:03.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:03.531+0000 7f02f963b640 1 --2- 172.21.15.5:0/3025364649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f40cbe10 0x7f02f40b47b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f02dc00e8d0 tx=0x7f02dc00ed90 comp rx=0 tx=0).stop 2024-03-24T10:56:03.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:03.531+0000 7f02f963b640 1 -- 172.21.15.5:0/3025364649 shutdown_connections 2024-03-24T10:56:03.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:03.531+0000 7f02f963b640 1 --2- 172.21.15.5:0/3025364649 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f02d406b860 0x7f02d406dd20 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:03.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:03.531+0000 7f02f963b640 1 --2- 172.21.15.5:0/3025364649 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02f41543c0 0x7f02f415ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:03.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:03.531+0000 7f02f963b640 1 --2- 172.21.15.5:0/3025364649 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02f40cbe10 0x7f02f40b47b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:03.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:03.531+0000 7f02f963b640 1 --2- 172.21.15.5:0/3025364649 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02f40caee0 0x7f02f41f3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:03.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:03.531+0000 7f02f963b640 1 -- 172.21.15.5:0/3025364649 >> 172.21.15.5:0/3025364649 conn(0x7f02f4061b50 msgr2=0x7f02f40c4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:03.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:03.531+0000 7f02f963b640 1 -- 172.21.15.5:0/3025364649 shutdown_connections 2024-03-24T10:56:03.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:03.531+0000 7f02f963b640 1 -- 172.21.15.5:0/3025364649 wait complete. 2024-03-24T10:56:04.146 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:04.146 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:05.147 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:05.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.249+0000 7fa8f66d0640 1 Processor -- start 2024-03-24T10:56:05.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.249+0000 7fa8f66d0640 1 -- start start 2024-03-24T10:56:05.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.250+0000 7fa8f66d0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa8f01543a0 0x7fa8f015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:05.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.251+0000 7fa8f66d0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa8f006bf50 0x7fa8f006c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:05.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.251+0000 7fa8f66d0640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8f00cb2f0 con 0x7fa8f006d4f0 2024-03-24T10:56:05.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.251+0000 7fa8f66d0640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fa8f00c6750 con 0x7fa8f01543a0 2024-03-24T10:56:05.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.251+0000 7fa8f66d0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fa8f00cc380 con 0x7fa8f006bf50 2024-03-24T10:56:05.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.251+0000 7fa8effff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa8f01543a0 0x7fa8f015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:05.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.252+0000 7fa8ef7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa8f006d4f0 0x7fa8f00cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:45444/0 (socket says 172.21.15.5:45444) 2024-03-24T10:56:05.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.252+0000 7fa8ef7fe640 1 -- 172.21.15.5:0/1315137116 learned_addr learned my addr 172.21.15.5:0/1315137116 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:56:05.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.252+0000 7fa8f4c6d640 1 --2- 172.21.15.5:0/1315137116 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa8f006bf50 0x7fa8f006c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:05.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.255+0000 7fa8eeffd640 1 -- 172.21.15.5:0/1315137116 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2191650411 0 0) 0x7fa8f00cb2f0 con 0x7fa8f006d4f0 2024-03-24T10:56:05.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.255+0000 7fa8eeffd640 1 -- 172.21.15.5:0/1315137116 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8d8003520 con 0x7fa8f006d4f0 2024-03-24T10:56:05.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.255+0000 7fa8effff640 1 -- 172.21.15.5:0/1315137116 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa8f006bf50 msgr2=0x7fa8f006c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:05.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.255+0000 7fa8effff640 1 --2- 172.21.15.5:0/1315137116 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa8f006bf50 0x7fa8f006c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:05.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.255+0000 7fa8effff640 1 -- 172.21.15.5:0/1315137116 >> v1:172.21.15.57:6789/0 conn(0x7fa8f006d4f0 legacy=0x7fa8f00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:05.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.255+0000 7fa8effff640 1 -- 172.21.15.5:0/1315137116 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f00ca7e0 con 0x7fa8f01543a0 2024-03-24T10:56:05.257 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.257+0000 7fa8effff640 1 --2- 172.21.15.5:0/1315137116 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8f01543a0 0x7fa8f015ac00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00097a0 tx=0x7fa8e002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf85e7938238d07e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:05.262 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.262+0000 7fa8eeffd640 1 -- 172.21.15.5:0/1315137116 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0030610 con 0x7fa8f01543a0 2024-03-24T10:56:05.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.262+0000 7fa8eeffd640 1 -- 172.21.15.5:0/1315137116 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8e0030770 con 0x7fa8f01543a0 2024-03-24T10:56:05.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.262+0000 7fa8eeffd640 1 -- 172.21.15.5:0/1315137116 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0030b00 con 0x7fa8f01543a0 2024-03-24T10:56:05.263 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.263+0000 7fa8f66d0640 1 -- 172.21.15.5:0/1315137116 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8f01543a0 msgr2=0x7fa8f015ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:05.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.263+0000 7fa8f66d0640 1 --2- 172.21.15.5:0/1315137116 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8f01543a0 0x7fa8f015ac00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00097a0 tx=0x7fa8e002b960 comp rx=0 tx=0).stop 2024-03-24T10:56:05.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.263+0000 7fa8f66d0640 1 -- 172.21.15.5:0/1315137116 shutdown_connections 2024-03-24T10:56:05.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.263+0000 7fa8f66d0640 1 --2- 172.21.15.5:0/1315137116 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa8f006bf50 0x7fa8f006c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:05.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.263+0000 7fa8f66d0640 1 --2- 172.21.15.5:0/1315137116 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8f01543a0 0x7fa8f015ac00 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:05.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.263+0000 7fa8f66d0640 1 -- 172.21.15.5:0/1315137116 >> 172.21.15.5:0/1315137116 conn(0x7fa8f0072ef0 msgr2=0x7fa8f0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:05.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.263+0000 7fa8f66d0640 1 -- 172.21.15.5:0/1315137116 shutdown_connections 2024-03-24T10:56:05.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.263+0000 7fa8f66d0640 1 -- 172.21.15.5:0/1315137116 wait complete. 2024-03-24T10:56:05.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.264+0000 7fa8f66d0640 1 Processor -- start 2024-03-24T10:56:05.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.265+0000 7fa8f66d0640 1 -- start start 2024-03-24T10:56:05.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.265+0000 7fa8f66d0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8f006bf50 0x7fa8f00b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:05.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.265+0000 7fa8f66d0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa8f006d4f0 0x7fa8f01f0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:05.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.265+0000 7fa8f66d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8f01543a0 0x7fa8f00ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:05.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.265+0000 7fa8f66d0640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f00cf8c0 con 0x7fa8f006bf50 2024-03-24T10:56:05.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.265+0000 7fa8f66d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f01eff70 con 0x7fa8f01543a0 2024-03-24T10:56:05.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.265+0000 7fa8f66d0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fa8f00cf620 con 0x7fa8f006d4f0 2024-03-24T10:56:05.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.266+0000 7fa8effff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8f006bf50 0x7fa8f00b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:05.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.266+0000 7fa8ef7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa8f006d4f0 0x7fa8f01f0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:05.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.266+0000 7fa8effff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8f006bf50 0x7fa8f00b2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:33106/0 (socket says 172.21.15.5:33106) 2024-03-24T10:56:05.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.266+0000 7fa8effff640 1 -- 172.21.15.5:0/3768009491 learned_addr learned my addr 172.21.15.5:0/3768009491 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:05.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.267+0000 7fa8f4c6d640 1 --2- 172.21.15.5:0/3768009491 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8f01543a0 0x7fa8f00ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:05.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.271+0000 7fa8effff640 1 -- 172.21.15.5:0/3768009491 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa8f006d4f0 msgr2=0x7fa8f01f0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:05.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.271+0000 7fa8effff640 1 --2- 172.21.15.5:0/3768009491 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa8f006d4f0 0x7fa8f01f0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:05.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.271+0000 7fa8effff640 1 -- 172.21.15.5:0/3768009491 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8f01543a0 msgr2=0x7fa8f00ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:05.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.271+0000 7fa8effff640 1 --2- 172.21.15.5:0/3768009491 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8f01543a0 0x7fa8f00ceb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:05.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.271+0000 7fa8effff640 1 -- 172.21.15.5:0/3768009491 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e0009400 con 0x7fa8f006bf50 2024-03-24T10:56:05.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.275+0000 7fa8effff640 1 --2- 172.21.15.5:0/3768009491 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8f006bf50 0x7fa8f00b2f60 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa8e002bf70 tx=0x7fa8e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:05.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.282+0000 7fa8ecff9640 1 -- 172.21.15.5:0/3768009491 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e003e070 con 0x7fa8f006bf50 2024-03-24T10:56:05.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.282+0000 7fa8ecff9640 1 -- 172.21.15.5:0/3768009491 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8e00397a0 con 0x7fa8f006bf50 2024-03-24T10:56:05.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.282+0000 7fa8ecff9640 1 -- 172.21.15.5:0/3768009491 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e0039b70 con 0x7fa8f006bf50 2024-03-24T10:56:05.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.283+0000 7fa8f66d0640 1 -- 172.21.15.5:0/3768009491 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8f01ef940 con 0x7fa8f006bf50 2024-03-24T10:56:05.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.283+0000 7fa8f66d0640 1 -- 172.21.15.5:0/3768009491 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8f00b4cb0 con 0x7fa8f006bf50 2024-03-24T10:56:05.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.284+0000 7fa8f66d0640 1 -- 172.21.15.5:0/3768009491 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8f00c5c70 con 0x7fa8f006bf50 2024-03-24T10:56:05.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.567+0000 7fa8ecff9640 1 -- 172.21.15.5:0/3768009491 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa8e0039cd0 con 0x7fa8f006bf50 2024-03-24T10:56:05.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.567+0000 7fa8ecff9640 1 --2- 172.21.15.5:0/3768009491 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa8dc06b890 0x7fa8dc06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:05.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.567+0000 7fa8ef7fe640 1 --2- 172.21.15.5:0/3768009491 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa8dc06b890 0x7fa8dc06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:05.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.568+0000 7fa8ef7fe640 1 --2- 172.21.15.5:0/3768009491 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa8dc06b890 0x7fa8dc06dd50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0158b00 tx=0x7fa8e4009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:05.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.591+0000 7fa8ecff9640 1 -- 172.21.15.5:0/3768009491 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fa8e007ab70 con 0x7fa8f006bf50 2024-03-24T10:56:05.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.591+0000 7fa8ecff9640 1 -- 172.21.15.5:0/3768009491 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa8e0076e10 con 0x7fa8f006bf50 2024-03-24T10:56:05.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.819+0000 7fa8f66d0640 1 -- 172.21.15.5:0/3768009491 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8f00681d0 con 0x7fa8f006bf50 2024-03-24T10:56:05.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.840+0000 7fa8ecff9640 1 -- 172.21.15.5:0/3768009491 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7fa8e0048090 con 0x7fa8f006bf50 2024-03-24T10:56:05.840 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:05.840 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:05.841 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:05.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.842+0000 7fa8f66d0640 1 -- 172.21.15.5:0/3768009491 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa8dc06b890 msgr2=0x7fa8dc06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:05.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.842+0000 7fa8f66d0640 1 --2- 172.21.15.5:0/3768009491 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa8dc06b890 0x7fa8dc06dd50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0158b00 tx=0x7fa8e4009040 comp rx=0 tx=0).stop 2024-03-24T10:56:05.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.842+0000 7fa8f66d0640 1 -- 172.21.15.5:0/3768009491 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8f006bf50 msgr2=0x7fa8f00b2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:05.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.842+0000 7fa8f66d0640 1 --2- 172.21.15.5:0/3768009491 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8f006bf50 0x7fa8f00b2f60 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa8e002bf70 tx=0x7fa8e00046c0 comp rx=0 tx=0).stop 2024-03-24T10:56:05.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.842+0000 7fa8f66d0640 1 -- 172.21.15.5:0/3768009491 shutdown_connections 2024-03-24T10:56:05.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.842+0000 7fa8f66d0640 1 --2- 172.21.15.5:0/3768009491 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa8dc06b890 0x7fa8dc06dd50 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:05.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.842+0000 7fa8f66d0640 1 --2- 172.21.15.5:0/3768009491 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa8f01543a0 0x7fa8f00ceb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:05.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.842+0000 7fa8f66d0640 1 --2- 172.21.15.5:0/3768009491 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa8f006d4f0 0x7fa8f01f0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:05.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.842+0000 7fa8f66d0640 1 --2- 172.21.15.5:0/3768009491 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa8f006bf50 0x7fa8f00b2f60 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:05.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.842+0000 7fa8f66d0640 1 -- 172.21.15.5:0/3768009491 >> 172.21.15.5:0/3768009491 conn(0x7fa8f0072ef0 msgr2=0x7fa8f00cc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:05.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.842+0000 7fa8f66d0640 1 -- 172.21.15.5:0/3768009491 shutdown_connections 2024-03-24T10:56:05.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:05.842+0000 7fa8f66d0640 1 -- 172.21.15.5:0/3768009491 wait complete. 2024-03-24T10:56:06.457 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:06.457 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:07.458 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.561+0000 7f704faed640 1 Processor -- start 2024-03-24T10:56:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.562+0000 7f704faed640 1 -- start start 2024-03-24T10:56:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.562+0000 7f704faed640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f704815aac0 0x7f704815aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.562+0000 7f704faed640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f704806c230 0x7f704806c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.562+0000 7f704faed640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f704806cd50 con 0x7f70480cebc0 2024-03-24T10:56:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.562+0000 7f704faed640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f70480ce790 con 0x7f704806c230 2024-03-24T10:56:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.562+0000 7f704faed640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f704815b3e0 con 0x7f704815aac0 2024-03-24T10:56:07.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.563+0000 7f704d088640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f704806c230 0x7f704806c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.563+0000 7f704d889640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f704815aac0 0x7f704815aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.563+0000 7f704d088640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f704806c230 0x7f704806c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:33110/0 (socket says 172.21.15.5:33110) 2024-03-24T10:56:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.563+0000 7f704d088640 1 -- 172.21.15.5:0/599153520 learned_addr learned my addr 172.21.15.5:0/599153520 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:07.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.567+0000 7f704d088640 1 -- 172.21.15.5:0/599153520 >> v2:172.21.15.57:3300/0 conn(0x7f704815aac0 msgr2=0x7f704815aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:07.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.567+0000 7f704d088640 1 --2- 172.21.15.5:0/599153520 >> v2:172.21.15.57:3300/0 conn(0x7f704815aac0 0x7f704815aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:07.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.567+0000 7f704d088640 1 -- 172.21.15.5:0/599153520 >> v1:172.21.15.57:6789/0 conn(0x7f70480cebc0 legacy=0x7f70480ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:07.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.567+0000 7f704d088640 1 -- 172.21.15.5:0/599153520 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70480cdb10 con 0x7f704806c230 2024-03-24T10:56:07.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.569+0000 7f704d088640 1 --2- 172.21.15.5:0/599153520 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f704806c230 0x7f704806c670 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f70480b2780 tx=0x7f703802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fafb305f9660ef55 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:07.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.575+0000 7f704c887640 1 -- 172.21.15.5:0/599153520 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703802f840 con 0x7f704806c230 2024-03-24T10:56:07.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.575+0000 7f704c887640 1 -- 172.21.15.5:0/599153520 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f703802f9a0 con 0x7f704806c230 2024-03-24T10:56:07.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.575+0000 7f704c887640 1 -- 172.21.15.5:0/599153520 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703802fd30 con 0x7f704806c230 2024-03-24T10:56:07.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.575+0000 7f704faed640 1 -- 172.21.15.5:0/599153520 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f704806c230 msgr2=0x7f704806c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:07.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.575+0000 7f704faed640 1 --2- 172.21.15.5:0/599153520 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f704806c230 0x7f704806c670 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f70480b2780 tx=0x7f703802ba20 comp rx=0 tx=0).stop 2024-03-24T10:56:07.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.575+0000 7f704faed640 1 -- 172.21.15.5:0/599153520 shutdown_connections 2024-03-24T10:56:07.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.575+0000 7f704faed640 1 --2- 172.21.15.5:0/599153520 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f704806c230 0x7f704806c670 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:07.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.575+0000 7f704faed640 1 --2- 172.21.15.5:0/599153520 >> v2:172.21.15.57:3300/0 conn(0x7f704815aac0 0x7f704815aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:07.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.575+0000 7f704faed640 1 -- 172.21.15.5:0/599153520 >> 172.21.15.5:0/599153520 conn(0x7f70480cfb90 msgr2=0x7f70480cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:07.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.575+0000 7f704faed640 1 -- 172.21.15.5:0/599153520 shutdown_connections 2024-03-24T10:56:07.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.575+0000 7f704faed640 1 -- 172.21.15.5:0/599153520 wait complete. 2024-03-24T10:56:07.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.577+0000 7f704faed640 1 Processor -- start 2024-03-24T10:56:07.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.577+0000 7f704faed640 1 -- start start 2024-03-24T10:56:07.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.577+0000 7f704faed640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f704806c230 0x7f70481569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:07.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.577+0000 7f704faed640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70480cebc0 0x7f70480b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:07.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.578+0000 7f704faed640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f704815aac0 0x7f70480c36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:07.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.578+0000 7f704faed640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f70481f0a40 con 0x7f704815aac0 2024-03-24T10:56:07.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.578+0000 7f704faed640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f70481efb70 con 0x7f70480cebc0 2024-03-24T10:56:07.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.578+0000 7f704faed640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f70481efe30 con 0x7f704806c230 2024-03-24T10:56:07.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.578+0000 7f704e08a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f704815aac0 0x7f70480c36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:07.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.578+0000 7f704d088640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70480cebc0 0x7f70480b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:07.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.578+0000 7f704d889640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f704806c230 0x7f70481569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:07.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.578+0000 7f704e08a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f704815aac0 0x7f70480c36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:33126/0 (socket says 172.21.15.5:33126) 2024-03-24T10:56:07.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.578+0000 7f704e08a640 1 -- 172.21.15.5:0/4177744273 learned_addr learned my addr 172.21.15.5:0/4177744273 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:07.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.582+0000 7f704e08a640 1 -- 172.21.15.5:0/4177744273 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f704806c230 msgr2=0x7f70481569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:07.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.582+0000 7f704e08a640 1 --2- 172.21.15.5:0/4177744273 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f704806c230 0x7f70481569a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:07.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.582+0000 7f704e08a640 1 -- 172.21.15.5:0/4177744273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70480cebc0 msgr2=0x7f70480b07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:07.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.582+0000 7f704e08a640 1 --2- 172.21.15.5:0/4177744273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70480cebc0 0x7f70480b07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:07.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.582+0000 7f704e08a640 1 -- 172.21.15.5:0/4177744273 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70380094d0 con 0x7f704815aac0 2024-03-24T10:56:07.584 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.584+0000 7f704e08a640 1 --2- 172.21.15.5:0/4177744273 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f704815aac0 0x7f70480c36d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f704000c7a0 tx=0x7f704000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:07.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.589+0000 7f703e7fc640 1 -- 172.21.15.5:0/4177744273 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7040003b70 con 0x7f704815aac0 2024-03-24T10:56:07.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.589+0000 7f703e7fc640 1 -- 172.21.15.5:0/4177744273 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7040003cf0 con 0x7f704815aac0 2024-03-24T10:56:07.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.589+0000 7f703e7fc640 1 -- 172.21.15.5:0/4177744273 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7040003ea0 con 0x7f704815aac0 2024-03-24T10:56:07.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.589+0000 7f704faed640 1 -- 172.21.15.5:0/4177744273 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70481f0110 con 0x7f704815aac0 2024-03-24T10:56:07.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.589+0000 7f704faed640 1 -- 172.21.15.5:0/4177744273 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70481f06d0 con 0x7f704815aac0 2024-03-24T10:56:07.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.590+0000 7f704faed640 1 -- 172.21.15.5:0/4177744273 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70481f0860 con 0x7f704815aac0 2024-03-24T10:56:07.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.615+0000 7f703e7fc640 1 -- 172.21.15.5:0/4177744273 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f704000aa90 con 0x7f704815aac0 2024-03-24T10:56:07.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.616+0000 7f703e7fc640 1 --2- 172.21.15.5:0/4177744273 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f702c06b810 0x7f702c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:07.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.616+0000 7f704d889640 1 --2- 172.21.15.5:0/4177744273 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f702c06b810 0x7f702c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:07.616 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.616+0000 7f704d889640 1 --2- 172.21.15.5:0/4177744273 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f702c06b810 0x7f702c06dcd0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7044009aa0 tx=0x7f7044009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:07.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.901+0000 7f703e7fc640 1 -- 172.21.15.5:0/4177744273 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f7040059210 con 0x7f704815aac0 2024-03-24T10:56:07.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:07.901+0000 7f703e7fc640 1 -- 172.21.15.5:0/4177744273 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f70400554b0 con 0x7f704815aac0 2024-03-24T10:56:08.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:08.148+0000 7f704faed640 1 -- 172.21.15.5:0/4177744273 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70480681d0 con 0x7f704815aac0 2024-03-24T10:56:08.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:08.164+0000 7f703e7fc640 1 -- 172.21.15.5:0/4177744273 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f7040058b70 con 0x7f704815aac0 2024-03-24T10:56:08.164 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:08.164 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:08.164 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:08.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:08.166+0000 7f704faed640 1 -- 172.21.15.5:0/4177744273 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f702c06b810 msgr2=0x7f702c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:08.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:08.166+0000 7f704faed640 1 --2- 172.21.15.5:0/4177744273 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f702c06b810 0x7f702c06dcd0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7044009aa0 tx=0x7f7044009190 comp rx=0 tx=0).stop 2024-03-24T10:56:08.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:08.166+0000 7f704faed640 1 -- 172.21.15.5:0/4177744273 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f704815aac0 msgr2=0x7f70480c36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:08.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:08.166+0000 7f704faed640 1 --2- 172.21.15.5:0/4177744273 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f704815aac0 0x7f70480c36d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f704000c7a0 tx=0x7f704000cc60 comp rx=0 tx=0).stop 2024-03-24T10:56:08.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:08.166+0000 7f704faed640 1 -- 172.21.15.5:0/4177744273 shutdown_connections 2024-03-24T10:56:08.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:08.166+0000 7f704faed640 1 --2- 172.21.15.5:0/4177744273 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f702c06b810 0x7f702c06dcd0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:08.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:08.166+0000 7f704faed640 1 --2- 172.21.15.5:0/4177744273 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f704815aac0 0x7f70480c36d0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:08.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:08.166+0000 7f704faed640 1 --2- 172.21.15.5:0/4177744273 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f70480cebc0 0x7f70480b07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:08.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:08.166+0000 7f704faed640 1 --2- 172.21.15.5:0/4177744273 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f704806c230 0x7f70481569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:08.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:08.166+0000 7f704faed640 1 -- 172.21.15.5:0/4177744273 >> 172.21.15.5:0/4177744273 conn(0x7f70480cfb90 msgr2=0x7f70480c6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:08.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:08.166+0000 7f704faed640 1 -- 172.21.15.5:0/4177744273 shutdown_connections 2024-03-24T10:56:08.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:08.166+0000 7f704faed640 1 -- 172.21.15.5:0/4177744273 wait complete. 2024-03-24T10:56:08.783 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:08.783 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:09.784 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:09.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.886+0000 7f3080956640 1 Processor -- start 2024-03-24T10:56:09.886 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.886+0000 7f3080956640 1 -- start start 2024-03-24T10:56:09.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.887+0000 7f3080956640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f307c0cb270 0x7f307c0cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:09.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.887+0000 7f3080956640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f307c0cd960 0x7f307c0cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:09.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.887+0000 7f3080956640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f307c0ccba0 con 0x7f307c0cae90 2024-03-24T10:56:09.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.887+0000 7f3080956640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f307c0cc9b0 con 0x7f307c0cd960 2024-03-24T10:56:09.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.887+0000 7f3080956640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f307c0cce40 con 0x7f307c0cb270 2024-03-24T10:56:09.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.887+0000 7f307ad9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f307c0cd960 0x7f307c0cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:09.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.888+0000 7f307ad9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f307c0cd960 0x7f307c0cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:33136/0 (socket says 172.21.15.5:33136) 2024-03-24T10:56:09.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.888+0000 7f307ad9d640 1 -- 172.21.15.5:0/3739217346 learned_addr learned my addr 172.21.15.5:0/3739217346 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:09.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.888+0000 7f3079d9b640 1 --2- 172.21.15.5:0/3739217346 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f307c0cb270 0x7f307c0cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:09.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.891+0000 7f307ad9d640 1 -- 172.21.15.5:0/3739217346 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f307c0cb270 msgr2=0x7f307c0cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:09.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.891+0000 7f307ad9d640 1 --2- 172.21.15.5:0/3739217346 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f307c0cb270 0x7f307c0cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:09.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.891+0000 7f307ad9d640 1 -- 172.21.15.5:0/3739217346 >> v1:172.21.15.5:6789/0 conn(0x7f307c0cae90 legacy=0x7f307c0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:09.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.891+0000 7f307ad9d640 1 -- 172.21.15.5:0/3739217346 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f307c15a590 con 0x7f307c0cd960 2024-03-24T10:56:09.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.894+0000 7f307ad9d640 1 --2- 172.21.15.5:0/3739217346 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f307c0cd960 0x7f307c0cb6f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3074009e50 tx=0x7f307402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df55909f57c9dbe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:09.896 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.897+0000 7f307959a640 1 -- 172.21.15.5:0/3739217346 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f307402cb00 con 0x7f307c0cd960 2024-03-24T10:56:09.898 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.899+0000 7f307959a640 1 -- 172.21.15.5:0/3739217346 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f307402cc60 con 0x7f307c0cd960 2024-03-24T10:56:09.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.899+0000 7f307959a640 1 -- 172.21.15.5:0/3739217346 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30740343f0 con 0x7f307c0cd960 2024-03-24T10:56:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.900+0000 7f3080956640 1 -- 172.21.15.5:0/3739217346 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f307c0cd960 msgr2=0x7f307c0cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.900+0000 7f3080956640 1 --2- 172.21.15.5:0/3739217346 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f307c0cd960 0x7f307c0cb6f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3074009e50 tx=0x7f307402bf60 comp rx=0 tx=0).stop 2024-03-24T10:56:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.900+0000 7f3080956640 1 -- 172.21.15.5:0/3739217346 shutdown_connections 2024-03-24T10:56:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.900+0000 7f3080956640 1 --2- 172.21.15.5:0/3739217346 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f307c0cd960 0x7f307c0cb6f0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.900+0000 7f3080956640 1 --2- 172.21.15.5:0/3739217346 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f307c0cb270 0x7f307c0cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.900+0000 7f3080956640 1 -- 172.21.15.5:0/3739217346 >> 172.21.15.5:0/3739217346 conn(0x7f307c0cfb80 msgr2=0x7f307c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.900+0000 7f3080956640 1 -- 172.21.15.5:0/3739217346 shutdown_connections 2024-03-24T10:56:09.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.900+0000 7f3080956640 1 -- 172.21.15.5:0/3739217346 wait complete. 2024-03-24T10:56:09.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.902+0000 7f3080956640 1 Processor -- start 2024-03-24T10:56:09.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.902+0000 7f3080956640 1 -- start start 2024-03-24T10:56:09.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.903+0000 7f3080956640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f307c0cae90 0x7f307c15e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:09.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.903+0000 7f3080956640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f307c0cb270 0x7f307c0c3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:09.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.903+0000 7f3080956640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307c0cd960 0x7f307c1efe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:09.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.903+0000 7f3080956640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f307c15c900 con 0x7f307c0cb270 2024-03-24T10:56:09.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.903+0000 7f3080956640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f307c15ced0 con 0x7f307c0cd960 2024-03-24T10:56:09.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.903+0000 7f3080956640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f307c15d1b0 con 0x7f307c0cae90 2024-03-24T10:56:09.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.903+0000 7f3079d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f307c0cb270 0x7f307c0c3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:09.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.904+0000 7f307ad9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307c0cd960 0x7f307c1efe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:09.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.904+0000 7f307a59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f307c0cae90 0x7f307c15e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:09.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.904+0000 7f3079d9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f307c0cb270 0x7f307c0c3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:33146/0 (socket says 172.21.15.5:33146) 2024-03-24T10:56:09.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.904+0000 7f3079d9b640 1 -- 172.21.15.5:0/461217374 learned_addr learned my addr 172.21.15.5:0/461217374 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:09.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.909+0000 7f3079d9b640 1 -- 172.21.15.5:0/461217374 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f307c0cae90 msgr2=0x7f307c15e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:09.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.909+0000 7f3079d9b640 1 --2- 172.21.15.5:0/461217374 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f307c0cae90 0x7f307c15e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:09.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.909+0000 7f3079d9b640 1 -- 172.21.15.5:0/461217374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307c0cd960 msgr2=0x7f307c1efe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:09.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.909+0000 7f3079d9b640 1 --2- 172.21.15.5:0/461217374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307c0cd960 0x7f307c1efe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:09.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.909+0000 7f3079d9b640 1 -- 172.21.15.5:0/461217374 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3074009b30 con 0x7f307c0cb270 2024-03-24T10:56:09.912 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.912+0000 7f3079d9b640 1 --2- 172.21.15.5:0/461217374 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f307c0cb270 0x7f307c0c3750 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f307000e9b0 tx=0x7f307000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:09.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.916+0000 7f306affd640 1 -- 172.21.15.5:0/461217374 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3070009670 con 0x7f307c0cb270 2024-03-24T10:56:09.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.916+0000 7f306affd640 1 -- 172.21.15.5:0/461217374 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f307000c9a0 con 0x7f307c0cb270 2024-03-24T10:56:09.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.916+0000 7f306affd640 1 -- 172.21.15.5:0/461217374 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f307000cd50 con 0x7f307c0cb270 2024-03-24T10:56:09.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.916+0000 7f3080956640 1 -- 172.21.15.5:0/461217374 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f307c15d490 con 0x7f307c0cb270 2024-03-24T10:56:09.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.916+0000 7f3080956640 1 -- 172.21.15.5:0/461217374 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f307c15da50 con 0x7f307c0cb270 2024-03-24T10:56:09.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:09.917+0000 7f3080956640 1 -- 172.21.15.5:0/461217374 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f307c06d5a0 con 0x7f307c0cb270 2024-03-24T10:56:10.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:10.196+0000 7f306affd640 1 -- 172.21.15.5:0/461217374 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3070013470 con 0x7f307c0cb270 2024-03-24T10:56:10.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:10.197+0000 7f306affd640 1 --2- 172.21.15.5:0/461217374 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f305c06b810 0x7f305c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:10.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:10.197+0000 7f307a59c640 1 --2- 172.21.15.5:0/461217374 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f305c06b810 0x7f305c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:10.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:10.198+0000 7f307a59c640 1 --2- 172.21.15.5:0/461217374 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f305c06b810 0x7f305c06dcd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f306c006fd0 tx=0x7f306c009060 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:10.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:10.225+0000 7f306affd640 1 -- 172.21.15.5:0/461217374 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f3070059420 con 0x7f307c0cb270 2024-03-24T10:56:10.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:10.225+0000 7f306affd640 1 -- 172.21.15.5:0/461217374 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f30700556c0 con 0x7f307c0cb270 2024-03-24T10:56:10.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:10.536+0000 7f3080956640 1 -- 172.21.15.5:0/461217374 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f307c0681d0 con 0x7f307c0cb270 2024-03-24T10:56:10.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:10.549+0000 7f306affd640 1 -- 172.21.15.5:0/461217374 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f3070021020 con 0x7f307c0cb270 2024-03-24T10:56:10.549 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:10.549 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:10.550 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:10.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:10.551+0000 7f3080956640 1 -- 172.21.15.5:0/461217374 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f305c06b810 msgr2=0x7f305c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:10.551+0000 7f3080956640 1 --2- 172.21.15.5:0/461217374 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f305c06b810 0x7f305c06dcd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f306c006fd0 tx=0x7f306c009060 comp rx=0 tx=0).stop 2024-03-24T10:56:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:10.551+0000 7f3080956640 1 -- 172.21.15.5:0/461217374 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f307c0cb270 msgr2=0x7f307c0c3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:10.551+0000 7f3080956640 1 --2- 172.21.15.5:0/461217374 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f307c0cb270 0x7f307c0c3750 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f307000e9b0 tx=0x7f307000ee70 comp rx=0 tx=0).stop 2024-03-24T10:56:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:10.551+0000 7f3080956640 1 -- 172.21.15.5:0/461217374 shutdown_connections 2024-03-24T10:56:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:10.551+0000 7f3080956640 1 --2- 172.21.15.5:0/461217374 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f305c06b810 0x7f305c06dcd0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:10.551+0000 7f3080956640 1 --2- 172.21.15.5:0/461217374 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f307c0cb270 0x7f307c0c3750 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:10.551+0000 7f3080956640 1 --2- 172.21.15.5:0/461217374 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f307c0cd960 0x7f307c1efe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:10.551+0000 7f3080956640 1 --2- 172.21.15.5:0/461217374 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f307c0cae90 0x7f307c15e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:10.551+0000 7f3080956640 1 -- 172.21.15.5:0/461217374 >> 172.21.15.5:0/461217374 conn(0x7f307c0cfb80 msgr2=0x7f307c0c89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:10.551+0000 7f3080956640 1 -- 172.21.15.5:0/461217374 shutdown_connections 2024-03-24T10:56:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:10.551+0000 7f3080956640 1 -- 172.21.15.5:0/461217374 wait complete. 2024-03-24T10:56:11.166 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:11.167 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:12.167 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:12.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.271+0000 7f7b3e437640 1 Processor -- start 2024-03-24T10:56:12.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.272+0000 7f7b3e437640 1 -- start start 2024-03-24T10:56:12.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.272+0000 7f7b3e437640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7b3806c580 0x7f7b381538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:12.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.272+0000 7f7b3e437640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7b380cf110 0x7f7b38153df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:12.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.272+0000 7f7b3e437640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b3815bce0 con 0x7f7b380cc8f0 2024-03-24T10:56:12.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.272+0000 7f7b3e437640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f7b380cd890 con 0x7f7b3806c580 2024-03-24T10:56:12.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.272+0000 7f7b3e437640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7b3806cae0 con 0x7f7b380cf110 2024-03-24T10:56:12.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.273+0000 7f7b3c9d4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7b380cf110 0x7f7b38153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:12.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.273+0000 7f7b377fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f7b3806c580 0x7f7b381538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:12.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.274+0000 7f7b3c9d4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7b380cf110 0x7f7b38153df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:59304/0 (socket says 172.21.15.5:59304) 2024-03-24T10:56:12.273 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.274+0000 7f7b3c9d4640 1 -- 172.21.15.5:0/435636833 learned_addr learned my addr 172.21.15.5:0/435636833 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:12.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.276+0000 7f7b3c9d4640 1 -- 172.21.15.5:0/435636833 >> v2:172.21.15.57:3300/0 conn(0x7f7b3806c580 msgr2=0x7f7b381538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:12.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.276+0000 7f7b3c9d4640 1 --2- 172.21.15.5:0/435636833 >> v2:172.21.15.57:3300/0 conn(0x7f7b3806c580 0x7f7b381538b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:12.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.276+0000 7f7b3c9d4640 1 -- 172.21.15.5:0/435636833 >> v1:172.21.15.57:6789/0 conn(0x7f7b380cc8f0 legacy=0x7f7b3815b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:12.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.276+0000 7f7b3c9d4640 1 -- 172.21.15.5:0/435636833 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b380cdb50 con 0x7f7b380cf110 2024-03-24T10:56:12.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.278+0000 7f7b3c9d4640 1 --2- 172.21.15.5:0/435636833 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7b380cf110 0x7f7b38153df0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009e50 tx=0x7f7b3002bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=795ada8d921b5a20 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:12.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.283+0000 7f7b36ffd640 1 -- 172.21.15.5:0/435636833 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3002cb90 con 0x7f7b380cf110 2024-03-24T10:56:12.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.283+0000 7f7b36ffd640 1 -- 172.21.15.5:0/435636833 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b3002ccf0 con 0x7f7b380cf110 2024-03-24T10:56:12.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.283+0000 7f7b36ffd640 1 -- 172.21.15.5:0/435636833 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b30034610 con 0x7f7b380cf110 2024-03-24T10:56:12.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.284+0000 7f7b3e437640 1 -- 172.21.15.5:0/435636833 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7b380cf110 msgr2=0x7f7b38153df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:12.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.284+0000 7f7b3e437640 1 --2- 172.21.15.5:0/435636833 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7b380cf110 0x7f7b38153df0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009e50 tx=0x7f7b3002bff0 comp rx=0 tx=0).stop 2024-03-24T10:56:12.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.284+0000 7f7b3e437640 1 -- 172.21.15.5:0/435636833 shutdown_connections 2024-03-24T10:56:12.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.284+0000 7f7b3e437640 1 --2- 172.21.15.5:0/435636833 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7b380cf110 0x7f7b38153df0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:12.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.284+0000 7f7b3e437640 1 --2- 172.21.15.5:0/435636833 >> v2:172.21.15.57:3300/0 conn(0x7f7b3806c580 0x7f7b381538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:12.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.284+0000 7f7b3e437640 1 -- 172.21.15.5:0/435636833 >> 172.21.15.5:0/435636833 conn(0x7f7b380b1820 msgr2=0x7f7b380b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:12.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.284+0000 7f7b3e437640 1 -- 172.21.15.5:0/435636833 shutdown_connections 2024-03-24T10:56:12.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.284+0000 7f7b3e437640 1 -- 172.21.15.5:0/435636833 wait complete. 2024-03-24T10:56:12.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.285+0000 7f7b3e437640 1 Processor -- start 2024-03-24T10:56:12.285 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.285+0000 7f7b3e437640 1 -- start start 2024-03-24T10:56:12.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.286+0000 7f7b3e437640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7b3806c580 0x7f7b380c8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:12.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.286+0000 7f7b3e437640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b380cc8f0 0x7f7b380c9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:12.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.286+0000 7f7b3e437640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7b380cf110 0x7f7b381e4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:12.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.286+0000 7f7b3e437640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f7b380c7c40 con 0x7f7b380cf110 2024-03-24T10:56:12.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.286+0000 7f7b3e437640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f7b381f0730 con 0x7f7b380cc8f0 2024-03-24T10:56:12.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.286+0000 7f7b3e437640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f7b381efb40 con 0x7f7b3806c580 2024-03-24T10:56:12.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.286+0000 7f7b37fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7b3806c580 0x7f7b380c8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:12.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.286+0000 7f7b3c9d4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7b380cf110 0x7f7b381e4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:12.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.287+0000 7f7b377fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b380cc8f0 0x7f7b380c9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:12.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.287+0000 7f7b3c9d4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7b380cf110 0x7f7b381e4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:33154/0 (socket says 172.21.15.5:33154) 2024-03-24T10:56:12.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.287+0000 7f7b3c9d4640 1 -- 172.21.15.5:0/2726770872 learned_addr learned my addr 172.21.15.5:0/2726770872 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:12.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.290+0000 7f7b3c9d4640 1 -- 172.21.15.5:0/2726770872 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7b3806c580 msgr2=0x7f7b380c8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:12.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.290+0000 7f7b3c9d4640 1 --2- 172.21.15.5:0/2726770872 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7b3806c580 0x7f7b380c8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:12.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.290+0000 7f7b3c9d4640 1 -- 172.21.15.5:0/2726770872 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b380cc8f0 msgr2=0x7f7b380c9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:12.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.290+0000 7f7b3c9d4640 1 --2- 172.21.15.5:0/2726770872 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b380cc8f0 0x7f7b380c9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:12.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.290+0000 7f7b3c9d4640 1 -- 172.21.15.5:0/2726770872 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b30009b30 con 0x7f7b380cf110 2024-03-24T10:56:12.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.293+0000 7f7b3c9d4640 1 --2- 172.21.15.5:0/2726770872 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7b380cf110 0x7f7b381e4240 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009e50 tx=0x7f7b30004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.297+0000 7f7b34ff9640 1 -- 172.21.15.5:0/2726770872 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b30009510 con 0x7f7b380cf110 2024-03-24T10:56:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.297+0000 7f7b34ff9640 1 -- 172.21.15.5:0/2726770872 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b3003e450 con 0x7f7b380cf110 2024-03-24T10:56:12.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.297+0000 7f7b34ff9640 1 -- 172.21.15.5:0/2726770872 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b3003e800 con 0x7f7b380cf110 2024-03-24T10:56:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.297+0000 7f7b3e437640 1 -- 172.21.15.5:0/2726770872 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b380c9fb0 con 0x7f7b380cf110 2024-03-24T10:56:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.297+0000 7f7b3e437640 1 -- 172.21.15.5:0/2726770872 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b381f00e0 con 0x7f7b380cf110 2024-03-24T10:56:12.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.299+0000 7f7b3e437640 1 -- 172.21.15.5:0/2726770872 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b380cee60 con 0x7f7b380cf110 2024-03-24T10:56:12.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.325+0000 7f7b34ff9640 1 -- 172.21.15.5:0/2726770872 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7b3003e960 con 0x7f7b380cf110 2024-03-24T10:56:12.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.325+0000 7f7b34ff9640 1 --2- 172.21.15.5:0/2726770872 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7b2406b810 0x7f7b2406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:12.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.325+0000 7f7b37fff640 1 --2- 172.21.15.5:0/2726770872 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7b2406b810 0x7f7b2406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:12.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.326+0000 7f7b37fff640 1 --2- 172.21.15.5:0/2726770872 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7b2406b810 0x7f7b2406dcd0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7b28001780 tx=0x7f7b28009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:12.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.332+0000 7f7b34ff9640 1 -- 172.21.15.5:0/2726770872 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f7b3007b860 con 0x7f7b380cf110 2024-03-24T10:56:12.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.601+0000 7f7b34ff9640 1 -- 172.21.15.5:0/2726770872 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7b30077b00 con 0x7f7b380cf110 2024-03-24T10:56:12.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.835+0000 7f7b3e437640 1 -- 172.21.15.5:0/2726770872 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b381f08a0 con 0x7f7b380cf110 2024-03-24T10:56:12.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.848+0000 7f7b34ff9640 1 -- 172.21.15.5:0/2726770872 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f7b3002ce60 con 0x7f7b380cf110 2024-03-24T10:56:12.849 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:12.849 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:12.849 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:12.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.850+0000 7f7b3e437640 1 -- 172.21.15.5:0/2726770872 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7b2406b810 msgr2=0x7f7b2406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:12.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.850+0000 7f7b3e437640 1 --2- 172.21.15.5:0/2726770872 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7b2406b810 0x7f7b2406dcd0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7b28001780 tx=0x7f7b28009040 comp rx=0 tx=0).stop 2024-03-24T10:56:12.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.850+0000 7f7b3e437640 1 -- 172.21.15.5:0/2726770872 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7b380cf110 msgr2=0x7f7b381e4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:12.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.850+0000 7f7b3e437640 1 --2- 172.21.15.5:0/2726770872 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7b380cf110 0x7f7b381e4240 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009e50 tx=0x7f7b30004a40 comp rx=0 tx=0).stop 2024-03-24T10:56:12.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.850+0000 7f7b3e437640 1 -- 172.21.15.5:0/2726770872 shutdown_connections 2024-03-24T10:56:12.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.850+0000 7f7b3e437640 1 --2- 172.21.15.5:0/2726770872 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f7b2406b810 0x7f7b2406dcd0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:12.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.850+0000 7f7b3e437640 1 --2- 172.21.15.5:0/2726770872 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f7b380cf110 0x7f7b381e4240 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:12.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.850+0000 7f7b3e437640 1 --2- 172.21.15.5:0/2726770872 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7b380cc8f0 0x7f7b380c9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:12.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.850+0000 7f7b3e437640 1 --2- 172.21.15.5:0/2726770872 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f7b3806c580 0x7f7b380c8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:12.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.850+0000 7f7b3e437640 1 -- 172.21.15.5:0/2726770872 >> 172.21.15.5:0/2726770872 conn(0x7f7b380b1820 msgr2=0x7f7b38159880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:12.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.850+0000 7f7b3e437640 1 -- 172.21.15.5:0/2726770872 shutdown_connections 2024-03-24T10:56:12.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:12.850+0000 7f7b3e437640 1 -- 172.21.15.5:0/2726770872 wait complete. 2024-03-24T10:56:13.465 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:13.466 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:14.467 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:14.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.569+0000 7fcafd5b6640 1 Processor -- start 2024-03-24T10:56:14.569 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.569+0000 7fcafd5b6640 1 -- start start 2024-03-24T10:56:14.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.570+0000 7fcafd5b6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcaf81543a0 0x7fcaf815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:14.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.570+0000 7fcafd5b6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcaf806bf50 0x7fcaf806c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:14.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.570+0000 7fcafd5b6640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaf80cb2f0 con 0x7fcaf806d4f0 2024-03-24T10:56:14.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.570+0000 7fcafd5b6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fcaf80c6750 con 0x7fcaf81543a0 2024-03-24T10:56:14.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.570+0000 7fcafd5b6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fcaf80cc380 con 0x7fcaf806bf50 2024-03-24T10:56:14.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.570+0000 7fcaf77fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcaf806bf50 0x7fcaf806c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:14.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.571+0000 7fcaf6ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcaf81543a0 0x7fcaf815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:14.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.571+0000 7fcaf77fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcaf806bf50 0x7fcaf806c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:58828/0 (socket says 172.21.15.5:58828) 2024-03-24T10:56:14.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.571+0000 7fcaf77fe640 1 -- 172.21.15.5:0/3201585805 learned_addr learned my addr 172.21.15.5:0/3201585805 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:14.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.574+0000 7fcaf6ffd640 1 -- 172.21.15.5:0/3201585805 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcaf806bf50 msgr2=0x7fcaf806c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:14.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.574+0000 7fcaf6ffd640 1 --2- 172.21.15.5:0/3201585805 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcaf806bf50 0x7fcaf806c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:14.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.575+0000 7fcaf6ffd640 1 -- 172.21.15.5:0/3201585805 >> v1:172.21.15.57:6789/0 conn(0x7fcaf806d4f0 legacy=0x7fcaf80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:14.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.575+0000 7fcaf6ffd640 1 -- 172.21.15.5:0/3201585805 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf80ca7e0 con 0x7fcaf81543a0 2024-03-24T10:56:14.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.580+0000 7fcaf6ffd640 1 --2- 172.21.15.5:0/3201585805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcaf81543a0 0x7fcaf815ac00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fcae80097a0 tx=0x7fcae802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6266fa2523c6eae0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:14.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.585+0000 7fcaf5ffb640 1 -- 172.21.15.5:0/3201585805 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae8030610 con 0x7fcaf81543a0 2024-03-24T10:56:14.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.585+0000 7fcaf5ffb640 1 -- 172.21.15.5:0/3201585805 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcae8030770 con 0x7fcaf81543a0 2024-03-24T10:56:14.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.585+0000 7fcaf5ffb640 1 -- 172.21.15.5:0/3201585805 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae8030b00 con 0x7fcaf81543a0 2024-03-24T10:56:14.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.585+0000 7fcafd5b6640 1 -- 172.21.15.5:0/3201585805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcaf81543a0 msgr2=0x7fcaf815ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:14.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.585+0000 7fcafd5b6640 1 --2- 172.21.15.5:0/3201585805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcaf81543a0 0x7fcaf815ac00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fcae80097a0 tx=0x7fcae802b960 comp rx=0 tx=0).stop 2024-03-24T10:56:14.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.585+0000 7fcafd5b6640 1 -- 172.21.15.5:0/3201585805 shutdown_connections 2024-03-24T10:56:14.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.585+0000 7fcafd5b6640 1 --2- 172.21.15.5:0/3201585805 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcaf806bf50 0x7fcaf806c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:14.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.585+0000 7fcafd5b6640 1 --2- 172.21.15.5:0/3201585805 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcaf81543a0 0x7fcaf815ac00 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:14.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.585+0000 7fcafd5b6640 1 -- 172.21.15.5:0/3201585805 >> 172.21.15.5:0/3201585805 conn(0x7fcaf8072ef0 msgr2=0x7fcaf8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:14.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.585+0000 7fcafd5b6640 1 -- 172.21.15.5:0/3201585805 shutdown_connections 2024-03-24T10:56:14.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.585+0000 7fcafd5b6640 1 -- 172.21.15.5:0/3201585805 wait complete. 2024-03-24T10:56:14.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.587+0000 7fcafd5b6640 1 Processor -- start 2024-03-24T10:56:14.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.587+0000 7fcafd5b6640 1 -- start start 2024-03-24T10:56:14.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.587+0000 7fcafd5b6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcaf806bf50 0x7fcaf8157f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:14.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.587+0000 7fcafd5b6640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcaf806d4f0 0x7fcaf81f0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:14.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.587+0000 7fcafd5b6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcaf81543a0 0x7fcaf80cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:14.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.587+0000 7fcafd5b6640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf81584a0 con 0x7fcaf806d4f0 2024-03-24T10:56:14.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.587+0000 7fcafd5b6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf81f0dd0 con 0x7fcaf81543a0 2024-03-24T10:56:14.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.587+0000 7fcafd5b6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fcaf81586a0 con 0x7fcaf806bf50 2024-03-24T10:56:14.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.588+0000 7fcaf67fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcaf806d4f0 0x7fcaf81f0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:14.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.588+0000 7fcaf77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcaf81543a0 0x7fcaf80cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:14.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.588+0000 7fcaf6ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcaf806bf50 0x7fcaf8157f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:14.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.588+0000 7fcaf67fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcaf806d4f0 0x7fcaf81f0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:33946/0 (socket says 172.21.15.5:33946) 2024-03-24T10:56:14.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.588+0000 7fcaf67fc640 1 -- 172.21.15.5:0/1649916785 learned_addr learned my addr 172.21.15.5:0/1649916785 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:14.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.592+0000 7fcaf67fc640 1 -- 172.21.15.5:0/1649916785 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcaf806bf50 msgr2=0x7fcaf8157f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:14.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.592+0000 7fcaf67fc640 1 --2- 172.21.15.5:0/1649916785 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcaf806bf50 0x7fcaf8157f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:14.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.592+0000 7fcaf67fc640 1 -- 172.21.15.5:0/1649916785 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcaf81543a0 msgr2=0x7fcaf80cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:14.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.592+0000 7fcaf67fc640 1 --2- 172.21.15.5:0/1649916785 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcaf81543a0 0x7fcaf80cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:14.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.592+0000 7fcaf67fc640 1 -- 172.21.15.5:0/1649916785 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae8009400 con 0x7fcaf806d4f0 2024-03-24T10:56:14.594 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.594+0000 7fcaf67fc640 1 --2- 172.21.15.5:0/1649916785 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcaf806d4f0 0x7fcaf81f0890 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fcaec00c400 tx=0x7fcaec00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:14.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.598+0000 7fcadf7fe640 1 -- 172.21.15.5:0/1649916785 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaec0052d0 con 0x7fcaf806d4f0 2024-03-24T10:56:14.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.598+0000 7fcadf7fe640 1 -- 172.21.15.5:0/1649916785 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaec005450 con 0x7fcaf806d4f0 2024-03-24T10:56:14.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.598+0000 7fcafd5b6640 1 -- 172.21.15.5:0/1649916785 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaf8158980 con 0x7fcaf806d4f0 2024-03-24T10:56:14.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.598+0000 7fcadf7fe640 1 -- 172.21.15.5:0/1649916785 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaec007260 con 0x7fcaf806d4f0 2024-03-24T10:56:14.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.598+0000 7fcafd5b6640 1 -- 172.21.15.5:0/1649916785 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaf81e6910 con 0x7fcaf806d4f0 2024-03-24T10:56:14.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.599+0000 7fcafd5b6640 1 -- 172.21.15.5:0/1649916785 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaf81532c0 con 0x7fcaf806d4f0 2024-03-24T10:56:14.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.619+0000 7fcadf7fe640 1 -- 172.21.15.5:0/1649916785 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcaec0073c0 con 0x7fcaf806d4f0 2024-03-24T10:56:14.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.619+0000 7fcadf7fe640 1 --2- 172.21.15.5:0/1649916785 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcae406b860 0x7fcae406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:14.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.619+0000 7fcaf6ffd640 1 --2- 172.21.15.5:0/1649916785 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcae406b860 0x7fcae406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:14.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.620+0000 7fcaf6ffd640 1 --2- 172.21.15.5:0/1649916785 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcae406b860 0x7fcae406dd20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fcae8036000 tx=0x7fcae803b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:14.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.623+0000 7fcadf7fe640 1 -- 172.21.15.5:0/1649916785 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fcaec007670 con 0x7fcaf806d4f0 2024-03-24T10:56:14.883 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:14.883+0000 7fcadf7fe640 1 -- 172.21.15.5:0/1649916785 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcaec0548c0 con 0x7fcaf806d4f0 2024-03-24T10:56:15.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:15.102+0000 7fcafd5b6640 1 -- 172.21.15.5:0/1649916785 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcaf814e420 con 0x7fcaf806d4f0 2024-03-24T10:56:15.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:15.115+0000 7fcadf7fe640 1 -- 172.21.15.5:0/1649916785 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7fcaec020020 con 0x7fcaf806d4f0 2024-03-24T10:56:15.115 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:15.116 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:15.116 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:15.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:15.118+0000 7fcafd5b6640 1 -- 172.21.15.5:0/1649916785 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcae406b860 msgr2=0x7fcae406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:15.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:15.118+0000 7fcafd5b6640 1 --2- 172.21.15.5:0/1649916785 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcae406b860 0x7fcae406dd20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fcae8036000 tx=0x7fcae803b040 comp rx=0 tx=0).stop 2024-03-24T10:56:15.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:15.118+0000 7fcafd5b6640 1 -- 172.21.15.5:0/1649916785 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcaf806d4f0 msgr2=0x7fcaf81f0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:15.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:15.118+0000 7fcafd5b6640 1 --2- 172.21.15.5:0/1649916785 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcaf806d4f0 0x7fcaf81f0890 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fcaec00c400 tx=0x7fcaec00c8c0 comp rx=0 tx=0).stop 2024-03-24T10:56:15.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:15.118+0000 7fcafd5b6640 1 -- 172.21.15.5:0/1649916785 shutdown_connections 2024-03-24T10:56:15.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:15.118+0000 7fcafd5b6640 1 --2- 172.21.15.5:0/1649916785 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcae406b860 0x7fcae406dd20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:15.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:15.118+0000 7fcafd5b6640 1 --2- 172.21.15.5:0/1649916785 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcaf806d4f0 0x7fcaf81f0890 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:15.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:15.118+0000 7fcafd5b6640 1 --2- 172.21.15.5:0/1649916785 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcaf81543a0 0x7fcaf80cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:15.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:15.118+0000 7fcafd5b6640 1 --2- 172.21.15.5:0/1649916785 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcaf806bf50 0x7fcaf8157f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:15.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:15.118+0000 7fcafd5b6640 1 -- 172.21.15.5:0/1649916785 >> 172.21.15.5:0/1649916785 conn(0x7fcaf8072ef0 msgr2=0x7fcaf814ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:15.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:15.118+0000 7fcafd5b6640 1 -- 172.21.15.5:0/1649916785 shutdown_connections 2024-03-24T10:56:15.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:15.118+0000 7fcafd5b6640 1 -- 172.21.15.5:0/1649916785 wait complete. 2024-03-24T10:56:15.738 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:15.738 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:16.739 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:16.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.841+0000 7fe9ebe64640 1 Processor -- start 2024-03-24T10:56:16.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.842+0000 7fe9ebe64640 1 -- start start 2024-03-24T10:56:16.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.843+0000 7fe9ebe64640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9e415ac00 0x7fe9e40cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:16.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.843+0000 7fe9ebe64640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9e40cb2f0 con 0x7fe9e40ccaa0 2024-03-24T10:56:16.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.843+0000 7fe9ebe64640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9e415b070 con 0x7fe9e41543a0 2024-03-24T10:56:16.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.843+0000 7fe9ebe64640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe9e40c6750 con 0x7fe9e415ac00 2024-03-24T10:56:16.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.844+0000 7fe9ea401640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fe9e40ccaa0 0x7fe9e406bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:49374/0 (socket says 172.21.15.5:49374) 2024-03-24T10:56:16.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.844+0000 7fe9ea401640 1 -- 172.21.15.5:0/856119524 learned_addr learned my addr 172.21.15.5:0/856119524 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:56:16.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.844+0000 7fe9e93ff640 1 --2- 172.21.15.5:0/856119524 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9e415ac00 0x7fe9e40cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:16.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.846+0000 7fe9e8bfe640 1 -- 172.21.15.5:0/856119524 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 298154498 0 0) 0x7fe9e40cb2f0 con 0x7fe9e40ccaa0 2024-03-24T10:56:16.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.846+0000 7fe9e8bfe640 1 -- 172.21.15.5:0/856119524 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9d0003520 con 0x7fe9e40ccaa0 2024-03-24T10:56:16.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.847+0000 7fe9e93ff640 1 -- 172.21.15.5:0/856119524 >> v1:172.21.15.57:6789/0 conn(0x7fe9e41543a0 legacy=0x7fe9e406d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:16.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.847+0000 7fe9e93ff640 1 -- 172.21.15.5:0/856119524 >> v1:172.21.15.5:6789/0 conn(0x7fe9e40ccaa0 legacy=0x7fe9e406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:16.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.847+0000 7fe9e93ff640 1 -- 172.21.15.5:0/856119524 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e40ca7e0 con 0x7fe9e415ac00 2024-03-24T10:56:16.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.849+0000 7fe9e93ff640 1 --2- 172.21.15.5:0/856119524 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9e415ac00 0x7fe9e40cc560 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe9d40097d0 tx=0x7fe9d402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1b6e215d4910dc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:16.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.853+0000 7fe9e8bfe640 1 -- 172.21.15.5:0/856119524 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d4030800 con 0x7fe9e415ac00 2024-03-24T10:56:16.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.853+0000 7fe9e8bfe640 1 -- 172.21.15.5:0/856119524 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9d4030960 con 0x7fe9e415ac00 2024-03-24T10:56:16.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.853+0000 7fe9e8bfe640 1 -- 172.21.15.5:0/856119524 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d4030cf0 con 0x7fe9e415ac00 2024-03-24T10:56:16.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.854+0000 7fe9ebe64640 1 -- 172.21.15.5:0/856119524 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9e415ac00 msgr2=0x7fe9e40cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:16.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.854+0000 7fe9ebe64640 1 --2- 172.21.15.5:0/856119524 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9e415ac00 0x7fe9e40cc560 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe9d40097d0 tx=0x7fe9d402b940 comp rx=0 tx=0).stop 2024-03-24T10:56:16.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.854+0000 7fe9ebe64640 1 -- 172.21.15.5:0/856119524 shutdown_connections 2024-03-24T10:56:16.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.854+0000 7fe9ebe64640 1 --2- 172.21.15.5:0/856119524 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9e415ac00 0x7fe9e40cc560 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:16.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.854+0000 7fe9ebe64640 1 -- 172.21.15.5:0/856119524 >> 172.21.15.5:0/856119524 conn(0x7fe9e4072ef0 msgr2=0x7fe9e4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:16.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.854+0000 7fe9ebe64640 1 -- 172.21.15.5:0/856119524 shutdown_connections 2024-03-24T10:56:16.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.854+0000 7fe9ebe64640 1 -- 172.21.15.5:0/856119524 wait complete. 2024-03-24T10:56:16.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.856+0000 7fe9ebe64640 1 Processor -- start 2024-03-24T10:56:16.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.856+0000 7fe9ebe64640 1 -- start start 2024-03-24T10:56:16.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.857+0000 7fe9ebe64640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9e40ccaa0 0x7fe9e4159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:16.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.857+0000 7fe9ebe64640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9e41543a0 0x7fe9e4149ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:16.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.857+0000 7fe9ebe64640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9e415ac00 0x7fe9e414a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:16.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.857+0000 7fe9ebe64640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e4159990 con 0x7fe9e415ac00 2024-03-24T10:56:16.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.857+0000 7fe9ebe64640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e4159b20 con 0x7fe9e41543a0 2024-03-24T10:56:16.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.857+0000 7fe9ebe64640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe9e4159d70 con 0x7fe9e40ccaa0 2024-03-24T10:56:16.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.858+0000 7fe9e9c00640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9e40ccaa0 0x7fe9e4159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:16.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.858+0000 7fe9ea401640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9e415ac00 0x7fe9e414a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:16.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.858+0000 7fe9e93ff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9e41543a0 0x7fe9e4149ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:16.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.858+0000 7fe9e9c00640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9e40ccaa0 0x7fe9e4159450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:58854/0 (socket says 172.21.15.5:58854) 2024-03-24T10:56:16.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.858+0000 7fe9e9c00640 1 -- 172.21.15.5:0/554625959 learned_addr learned my addr 172.21.15.5:0/554625959 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:16.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.863+0000 7fe9ea401640 1 -- 172.21.15.5:0/554625959 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9e40ccaa0 msgr2=0x7fe9e4159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:16.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.863+0000 7fe9ea401640 1 --2- 172.21.15.5:0/554625959 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9e40ccaa0 0x7fe9e4159450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:16.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.863+0000 7fe9ea401640 1 -- 172.21.15.5:0/554625959 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9e41543a0 msgr2=0x7fe9e4149ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:16.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.863+0000 7fe9ea401640 1 --2- 172.21.15.5:0/554625959 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9e41543a0 0x7fe9e4149ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:16.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.863+0000 7fe9ea401640 1 -- 172.21.15.5:0/554625959 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9d4009400 con 0x7fe9e415ac00 2024-03-24T10:56:16.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.866+0000 7fe9ea401640 1 --2- 172.21.15.5:0/554625959 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9e415ac00 0x7fe9e414a010 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc00ca80 tx=0x7fe9dc00cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:16.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.870+0000 7fe9da7fc640 1 -- 172.21.15.5:0/554625959 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc00ac30 con 0x7fe9e415ac00 2024-03-24T10:56:16.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.870+0000 7fe9da7fc640 1 -- 172.21.15.5:0/554625959 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9dc00ad90 con 0x7fe9e415ac00 2024-03-24T10:56:16.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.870+0000 7fe9da7fc640 1 -- 172.21.15.5:0/554625959 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc012600 con 0x7fe9e415ac00 2024-03-24T10:56:16.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.870+0000 7fe9ebe64640 1 -- 172.21.15.5:0/554625959 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9e415a050 con 0x7fe9e415ac00 2024-03-24T10:56:16.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.870+0000 7fe9ebe64640 1 -- 172.21.15.5:0/554625959 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9e4158c50 con 0x7fe9e415ac00 2024-03-24T10:56:16.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:16.871+0000 7fe9ebe64640 1 -- 172.21.15.5:0/554625959 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9e40681d0 con 0x7fe9e415ac00 2024-03-24T10:56:17.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:17.154+0000 7fe9da7fc640 1 -- 172.21.15.5:0/554625959 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe9dc007070 con 0x7fe9e415ac00 2024-03-24T10:56:17.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:17.154+0000 7fe9da7fc640 1 --2- 172.21.15.5:0/554625959 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe9c806b860 0x7fe9c806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:17.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:17.154+0000 7fe9e9c00640 1 --2- 172.21.15.5:0/554625959 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe9c806b860 0x7fe9c806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:17.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:17.155+0000 7fe9e9c00640 1 --2- 172.21.15.5:0/554625959 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe9c806b860 0x7fe9c806dd20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0007fd0 tx=0x7fe9e0009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:17.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:17.178+0000 7fe9da7fc640 1 -- 172.21.15.5:0/554625959 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fe9dc058e00 con 0x7fe9e415ac00 2024-03-24T10:56:17.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:17.178+0000 7fe9da7fc640 1 -- 172.21.15.5:0/554625959 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe9dc0550a0 con 0x7fe9e415ac00 2024-03-24T10:56:17.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:17.400+0000 7fe9ebe64640 1 -- 172.21.15.5:0/554625959 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9e415e510 con 0x7fe9e415ac00 2024-03-24T10:56:17.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:17.414+0000 7fe9da7fc640 1 -- 172.21.15.5:0/554625959 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7fe9dc058760 con 0x7fe9e415ac00 2024-03-24T10:56:17.415 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:17.415 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:17.415 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:17.417+0000 7fe9ebe64640 1 -- 172.21.15.5:0/554625959 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe9c806b860 msgr2=0x7fe9c806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:17.417+0000 7fe9ebe64640 1 --2- 172.21.15.5:0/554625959 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe9c806b860 0x7fe9c806dd20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0007fd0 tx=0x7fe9e0009040 comp rx=0 tx=0).stop 2024-03-24T10:56:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:17.417+0000 7fe9ebe64640 1 -- 172.21.15.5:0/554625959 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9e415ac00 msgr2=0x7fe9e414a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:17.417+0000 7fe9ebe64640 1 --2- 172.21.15.5:0/554625959 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9e415ac00 0x7fe9e414a010 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc00ca80 tx=0x7fe9dc00cf40 comp rx=0 tx=0).stop 2024-03-24T10:56:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:17.417+0000 7fe9ebe64640 1 -- 172.21.15.5:0/554625959 shutdown_connections 2024-03-24T10:56:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:17.417+0000 7fe9ebe64640 1 --2- 172.21.15.5:0/554625959 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe9c806b860 0x7fe9c806dd20 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:17.417+0000 7fe9ebe64640 1 --2- 172.21.15.5:0/554625959 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe9e415ac00 0x7fe9e414a010 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:17.417+0000 7fe9ebe64640 1 --2- 172.21.15.5:0/554625959 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe9e41543a0 0x7fe9e4149ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:17.417+0000 7fe9ebe64640 1 --2- 172.21.15.5:0/554625959 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe9e40ccaa0 0x7fe9e4159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:17.417+0000 7fe9ebe64640 1 -- 172.21.15.5:0/554625959 >> 172.21.15.5:0/554625959 conn(0x7fe9e4072ef0 msgr2=0x7fe9e41507f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:17.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:17.417+0000 7fe9ebe64640 1 -- 172.21.15.5:0/554625959 shutdown_connections 2024-03-24T10:56:17.419 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:17.417+0000 7fe9ebe64640 1 -- 172.21.15.5:0/554625959 wait complete. 2024-03-24T10:56:18.034 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:18.034 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:19.035 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:19.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.137+0000 7f33a9325640 1 Processor -- start 2024-03-24T10:56:19.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.137+0000 7f33a9325640 1 -- start start 2024-03-24T10:56:19.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.138+0000 7f33a9325640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33a406bf50 0x7f33a406c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:19.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.138+0000 7f33a9325640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33a40cb2f0 con 0x7f33a415ac00 2024-03-24T10:56:19.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.138+0000 7f33a9325640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33a415b070 con 0x7f33a41543a0 2024-03-24T10:56:19.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.138+0000 7f33a9325640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f33a40c6750 con 0x7f33a406bf50 2024-03-24T10:56:19.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.138+0000 7f33a27fc640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f33a415ac00 0x7f33a40cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:49380/0 (socket says 172.21.15.5:49380) 2024-03-24T10:56:19.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.138+0000 7f33a27fc640 1 -- 172.21.15.5:0/1321957943 learned_addr learned my addr 172.21.15.5:0/1321957943 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:56:19.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.138+0000 7f33a37fe640 1 --2- 172.21.15.5:0/1321957943 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33a406bf50 0x7f33a406c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:19.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.141+0000 7f33a1ffb640 1 -- 172.21.15.5:0/1321957943 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2312000153 0 0) 0x7f33a40cb2f0 con 0x7f33a415ac00 2024-03-24T10:56:19.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.141+0000 7f33a1ffb640 1 -- 172.21.15.5:0/1321957943 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f338c003520 con 0x7f33a415ac00 2024-03-24T10:56:19.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.141+0000 7f33a1ffb640 1 -- 172.21.15.5:0/1321957943 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1790760376 0 0) 0x7f33a415b070 con 0x7f33a41543a0 2024-03-24T10:56:19.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.141+0000 7f33a1ffb640 1 -- 172.21.15.5:0/1321957943 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33a40cb2f0 con 0x7f33a41543a0 2024-03-24T10:56:19.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.142+0000 7f33a37fe640 1 -- 172.21.15.5:0/1321957943 >> v1:172.21.15.57:6789/0 conn(0x7f33a41543a0 legacy=0x7f33a406d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:19.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.142+0000 7f33a37fe640 1 -- 172.21.15.5:0/1321957943 >> v1:172.21.15.5:6789/0 conn(0x7f33a415ac00 legacy=0x7f33a40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:19.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.142+0000 7f33a37fe640 1 -- 172.21.15.5:0/1321957943 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33a40ca7e0 con 0x7f33a406bf50 2024-03-24T10:56:19.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.145+0000 7f33a37fe640 1 --2- 172.21.15.5:0/1321957943 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33a406bf50 0x7f33a406c360 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f339c009dc0 tx=0x7f339c02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac78a42f16e640b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:19.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.150+0000 7f33a1ffb640 1 -- 172.21.15.5:0/1321957943 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339c02ccd0 con 0x7f33a406bf50 2024-03-24T10:56:19.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.150+0000 7f33a1ffb640 1 -- 172.21.15.5:0/1321957943 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f339c02ce30 con 0x7f33a406bf50 2024-03-24T10:56:19.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.150+0000 7f33a1ffb640 1 -- 172.21.15.5:0/1321957943 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f339c034660 con 0x7f33a406bf50 2024-03-24T10:56:19.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.151+0000 7f33a9325640 1 -- 172.21.15.5:0/1321957943 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33a406bf50 msgr2=0x7f33a406c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:19.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.151+0000 7f33a9325640 1 --2- 172.21.15.5:0/1321957943 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33a406bf50 0x7f33a406c360 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f339c009dc0 tx=0x7f339c02bf20 comp rx=0 tx=0).stop 2024-03-24T10:56:19.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.151+0000 7f33a9325640 1 -- 172.21.15.5:0/1321957943 shutdown_connections 2024-03-24T10:56:19.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.151+0000 7f33a9325640 1 --2- 172.21.15.5:0/1321957943 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33a406bf50 0x7f33a406c360 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:19.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.151+0000 7f33a9325640 1 -- 172.21.15.5:0/1321957943 >> 172.21.15.5:0/1321957943 conn(0x7f33a4072ef0 msgr2=0x7f33a4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:19.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.151+0000 7f33a9325640 1 -- 172.21.15.5:0/1321957943 shutdown_connections 2024-03-24T10:56:19.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.151+0000 7f33a9325640 1 -- 172.21.15.5:0/1321957943 wait complete. 2024-03-24T10:56:19.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.152+0000 7f33a9325640 1 Processor -- start 2024-03-24T10:56:19.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.152+0000 7f33a9325640 1 -- start start 2024-03-24T10:56:19.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.153+0000 7f33a9325640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33a406bf50 0x7f33a40ce450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:19.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.153+0000 7f33a9325640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33a41543a0 0x7f33a415c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:19.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.153+0000 7f33a9325640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33a415ac00 0x7f33a40b4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:19.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.153+0000 7f33a9325640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f33a415b870 con 0x7f33a406bf50 2024-03-24T10:56:19.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.153+0000 7f33a9325640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f33a40cf0c0 con 0x7f33a41543a0 2024-03-24T10:56:19.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.153+0000 7f33a9325640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f33a41f03f0 con 0x7f33a415ac00 2024-03-24T10:56:19.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.154+0000 7f33a2ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33a406bf50 0x7f33a40ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:19.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.154+0000 7f33a37fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33a415ac00 0x7f33a40b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:19.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.154+0000 7f33a27fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33a41543a0 0x7f33a415c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:19.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.154+0000 7f33a2ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33a406bf50 0x7f33a40ce450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:33966/0 (socket says 172.21.15.5:33966) 2024-03-24T10:56:19.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.154+0000 7f33a2ffd640 1 -- 172.21.15.5:0/2441989191 learned_addr learned my addr 172.21.15.5:0/2441989191 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:19.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.158+0000 7f33a2ffd640 1 -- 172.21.15.5:0/2441989191 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33a415ac00 msgr2=0x7f33a40b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:19.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.158+0000 7f33a2ffd640 1 --2- 172.21.15.5:0/2441989191 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33a415ac00 0x7f33a40b4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:19.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.158+0000 7f33a2ffd640 1 -- 172.21.15.5:0/2441989191 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33a41543a0 msgr2=0x7f33a415c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:19.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.158+0000 7f33a2ffd640 1 --2- 172.21.15.5:0/2441989191 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33a41543a0 0x7f33a415c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:19.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.158+0000 7f33a2ffd640 1 -- 172.21.15.5:0/2441989191 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f339c009a70 con 0x7f33a406bf50 2024-03-24T10:56:19.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.160+0000 7f33a2ffd640 1 --2- 172.21.15.5:0/2441989191 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33a406bf50 0x7f33a40ce450 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f339400c490 tx=0x7f339400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:19.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.164+0000 7f338b7fe640 1 -- 172.21.15.5:0/2441989191 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33940052d0 con 0x7f33a406bf50 2024-03-24T10:56:19.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.164+0000 7f338b7fe640 1 -- 172.21.15.5:0/2441989191 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3394005430 con 0x7f33a406bf50 2024-03-24T10:56:19.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.164+0000 7f33a9325640 1 -- 172.21.15.5:0/2441989191 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33a40cd970 con 0x7f33a406bf50 2024-03-24T10:56:19.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.164+0000 7f338b7fe640 1 -- 172.21.15.5:0/2441989191 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3394012870 con 0x7f33a406bf50 2024-03-24T10:56:19.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.165+0000 7f33a9325640 1 -- 172.21.15.5:0/2441989191 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33a40cdf30 con 0x7f33a406bf50 2024-03-24T10:56:19.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.166+0000 7f33a9325640 1 -- 172.21.15.5:0/2441989191 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33a40ce120 con 0x7f33a406bf50 2024-03-24T10:56:19.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.465+0000 7f338b7fe640 1 -- 172.21.15.5:0/2441989191 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f33940129d0 con 0x7f33a406bf50 2024-03-24T10:56:19.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.466+0000 7f338b7fe640 1 --2- 172.21.15.5:0/2441989191 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f339006b860 0x7f339006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:19.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.466+0000 7f33a27fc640 1 --2- 172.21.15.5:0/2441989191 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f339006b860 0x7f339006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:19.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.467+0000 7f33a27fc640 1 --2- 172.21.15.5:0/2441989191 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f339006b860 0x7f339006dd20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f33a40b51a0 tx=0x7f3398007d30 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:19.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.482+0000 7f338b7fe640 1 -- 172.21.15.5:0/2441989191 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f3394059000 con 0x7f33a406bf50 2024-03-24T10:56:19.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.482+0000 7f338b7fe640 1 -- 172.21.15.5:0/2441989191 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f33940552a0 con 0x7f33a406bf50 2024-03-24T10:56:19.738 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.738+0000 7f33a9325640 1 -- 172.21.15.5:0/2441989191 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33a40681d0 con 0x7f33a406bf50 2024-03-24T10:56:19.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.759+0000 7f338b7fe640 1 -- 172.21.15.5:0/2441989191 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f3394058960 con 0x7f33a406bf50 2024-03-24T10:56:19.759 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:19.759 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:19.759 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:19.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.761+0000 7f33a9325640 1 -- 172.21.15.5:0/2441989191 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f339006b860 msgr2=0x7f339006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:19.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.761+0000 7f33a9325640 1 --2- 172.21.15.5:0/2441989191 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f339006b860 0x7f339006dd20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f33a40b51a0 tx=0x7f3398007d30 comp rx=0 tx=0).stop 2024-03-24T10:56:19.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.761+0000 7f33a9325640 1 -- 172.21.15.5:0/2441989191 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33a406bf50 msgr2=0x7f33a40ce450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:19.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.761+0000 7f33a9325640 1 --2- 172.21.15.5:0/2441989191 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33a406bf50 0x7f33a40ce450 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f339400c490 tx=0x7f339400c950 comp rx=0 tx=0).stop 2024-03-24T10:56:19.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.761+0000 7f33a9325640 1 -- 172.21.15.5:0/2441989191 shutdown_connections 2024-03-24T10:56:19.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.761+0000 7f33a9325640 1 --2- 172.21.15.5:0/2441989191 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f339006b860 0x7f339006dd20 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:19.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.761+0000 7f33a9325640 1 --2- 172.21.15.5:0/2441989191 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f33a415ac00 0x7f33a40b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:19.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.761+0000 7f33a9325640 1 --2- 172.21.15.5:0/2441989191 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f33a41543a0 0x7f33a415c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:19.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.761+0000 7f33a9325640 1 --2- 172.21.15.5:0/2441989191 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f33a406bf50 0x7f33a40ce450 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:19.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.761+0000 7f33a9325640 1 -- 172.21.15.5:0/2441989191 >> 172.21.15.5:0/2441989191 conn(0x7f33a4072ef0 msgr2=0x7f33a4153640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:19.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.761+0000 7f33a9325640 1 -- 172.21.15.5:0/2441989191 shutdown_connections 2024-03-24T10:56:19.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:19.761+0000 7f33a9325640 1 -- 172.21.15.5:0/2441989191 wait complete. 2024-03-24T10:56:20.376 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:20.377 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:21.378 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:21.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.483+0000 7f2dcf0d4640 1 Processor -- start 2024-03-24T10:56:21.483 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.483+0000 7f2dcf0d4640 1 -- start start 2024-03-24T10:56:21.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.483+0000 7f2dcf0d4640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2dc80cc8f0 0x7f2dc815b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:21.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.483+0000 7f2dcf0d4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2dc806c580 0x7f2dc81538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:21.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.483+0000 7f2dcf0d4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dc80cd890 con 0x7f2dc80cf110 2024-03-24T10:56:21.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.483+0000 7f2dcf0d4640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f2dc806cae0 con 0x7f2dc80cc8f0 2024-03-24T10:56:21.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.483+0000 7f2dcf0d4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f2dc80cdb50 con 0x7f2dc806c580 2024-03-24T10:56:21.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.484+0000 7f2dcce70640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2dc80cc8f0 0x7f2dc815b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:21.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.484+0000 7f2dc7fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2dc806c580 0x7f2dc81538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:21.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.484+0000 7f2dcd671640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f2dc80cf110 0x7f2dc8153df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:59712/0 (socket says 172.21.15.5:59712) 2024-03-24T10:56:21.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.484+0000 7f2dcd671640 1 -- 172.21.15.5:0/1096765235 learned_addr learned my addr 172.21.15.5:0/1096765235 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:56:21.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.487+0000 7f2dcce70640 1 -- 172.21.15.5:0/1096765235 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2dc806c580 msgr2=0x7f2dc81538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:21.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.487+0000 7f2dcce70640 1 --2- 172.21.15.5:0/1096765235 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2dc806c580 0x7f2dc81538b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:21.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.487+0000 7f2dcce70640 1 -- 172.21.15.5:0/1096765235 >> v1:172.21.15.57:6789/0 conn(0x7f2dc80cf110 legacy=0x7f2dc8153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:21.487 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.487+0000 7f2dcce70640 1 -- 172.21.15.5:0/1096765235 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dc80cde30 con 0x7f2dc80cc8f0 2024-03-24T10:56:21.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.489+0000 7f2dcce70640 1 --2- 172.21.15.5:0/1096765235 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2dc80cc8f0 0x7f2dc815b5d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc0097a0 tx=0x7f2dbc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8165117c1f5d505 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:21.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.493+0000 7f2dc77fe640 1 -- 172.21.15.5:0/1096765235 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc031830 con 0x7f2dc80cc8f0 2024-03-24T10:56:21.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.493+0000 7f2dc77fe640 1 -- 172.21.15.5:0/1096765235 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dbc031990 con 0x7f2dc80cc8f0 2024-03-24T10:56:21.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.493+0000 7f2dc77fe640 1 -- 172.21.15.5:0/1096765235 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc031d20 con 0x7f2dc80cc8f0 2024-03-24T10:56:21.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.494+0000 7f2dcf0d4640 1 -- 172.21.15.5:0/1096765235 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2dc80cc8f0 msgr2=0x7f2dc815b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:21.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.494+0000 7f2dcf0d4640 1 --2- 172.21.15.5:0/1096765235 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2dc80cc8f0 0x7f2dc815b5d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc0097a0 tx=0x7f2dbc02b8d0 comp rx=0 tx=0).stop 2024-03-24T10:56:21.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.494+0000 7f2dcf0d4640 1 -- 172.21.15.5:0/1096765235 shutdown_connections 2024-03-24T10:56:21.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.494+0000 7f2dcf0d4640 1 --2- 172.21.15.5:0/1096765235 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2dc806c580 0x7f2dc81538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:21.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.494+0000 7f2dcf0d4640 1 --2- 172.21.15.5:0/1096765235 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2dc80cc8f0 0x7f2dc815b5d0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:21.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.494+0000 7f2dcf0d4640 1 -- 172.21.15.5:0/1096765235 >> 172.21.15.5:0/1096765235 conn(0x7f2dc80b1820 msgr2=0x7f2dc80b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:21.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.494+0000 7f2dcf0d4640 1 -- 172.21.15.5:0/1096765235 shutdown_connections 2024-03-24T10:56:21.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.494+0000 7f2dcf0d4640 1 -- 172.21.15.5:0/1096765235 wait complete. 2024-03-24T10:56:21.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.495+0000 7f2dcf0d4640 1 Processor -- start 2024-03-24T10:56:21.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.495+0000 7f2dcf0d4640 1 -- start start 2024-03-24T10:56:21.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.496+0000 7f2dcf0d4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2dc806c580 0x7f2dc814f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:21.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.496+0000 7f2dcf0d4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2dc80cc8f0 0x7f2dc8157550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:21.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.496+0000 7f2dcf0d4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2dc80cf110 0x7f2dc814de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:21.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.496+0000 7f2dcf0d4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc81f4010 con 0x7f2dc806c580 2024-03-24T10:56:21.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.496+0000 7f2dcf0d4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2dc814c9d0 con 0x7f2dc80cf110 2024-03-24T10:56:21.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.496+0000 7f2dcf0d4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f2dc814fe10 con 0x7f2dc80cc8f0 2024-03-24T10:56:21.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.497+0000 7f2dcce70640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2dc806c580 0x7f2dc814f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:21.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.497+0000 7f2dc7fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2dc80cc8f0 0x7f2dc8157550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:21.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.497+0000 7f2dcd671640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2dc80cf110 0x7f2dc814de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:21.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.497+0000 7f2dcce70640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2dc806c580 0x7f2dc814f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:33996/0 (socket says 172.21.15.5:33996) 2024-03-24T10:56:21.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.497+0000 7f2dcce70640 1 -- 172.21.15.5:0/2643239699 learned_addr learned my addr 172.21.15.5:0/2643239699 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:21.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.501+0000 7f2dcce70640 1 -- 172.21.15.5:0/2643239699 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2dc80cc8f0 msgr2=0x7f2dc8157550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:21.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.501+0000 7f2dcce70640 1 --2- 172.21.15.5:0/2643239699 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2dc80cc8f0 0x7f2dc8157550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:21.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.501+0000 7f2dcce70640 1 -- 172.21.15.5:0/2643239699 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2dc80cf110 msgr2=0x7f2dc814de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:21.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.501+0000 7f2dcce70640 1 --2- 172.21.15.5:0/2643239699 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2dc80cf110 0x7f2dc814de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:21.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.501+0000 7f2dcce70640 1 -- 172.21.15.5:0/2643239699 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dbc009400 con 0x7f2dc806c580 2024-03-24T10:56:21.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.503+0000 7f2dcce70640 1 --2- 172.21.15.5:0/2643239699 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2dc806c580 0x7f2dc814f8d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc009890 tx=0x7f2dbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:21.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.505+0000 7f2dc57fa640 1 -- 172.21.15.5:0/2643239699 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc03ba20 con 0x7f2dc806c580 2024-03-24T10:56:21.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.505+0000 7f2dcf0d4640 1 -- 172.21.15.5:0/2643239699 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dc814ce90 con 0x7f2dc806c580 2024-03-24T10:56:21.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.505+0000 7f2dcf0d4640 1 -- 172.21.15.5:0/2643239699 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dc81f3d30 con 0x7f2dc806c580 2024-03-24T10:56:21.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.507+0000 7f2dcf0d4640 1 -- 172.21.15.5:0/2643239699 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dc80d09e0 con 0x7f2dc806c580 2024-03-24T10:56:21.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.778+0000 7f2dc57fa640 1 -- 172.21.15.5:0/2643239699 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dbc03bb80 con 0x7f2dc806c580 2024-03-24T10:56:21.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.778+0000 7f2dc57fa640 1 -- 172.21.15.5:0/2643239699 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dbc04d440 con 0x7f2dc806c580 2024-03-24T10:56:21.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.779+0000 7f2dc57fa640 1 -- 172.21.15.5:0/2643239699 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2dbc0433f0 con 0x7f2dc806c580 2024-03-24T10:56:21.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.779+0000 7f2dc57fa640 1 --2- 172.21.15.5:0/2643239699 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2dac06b810 0x7f2dac06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:21.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.779+0000 7f2dc7fff640 1 --2- 172.21.15.5:0/2643239699 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2dac06b810 0x7f2dac06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:21.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.780+0000 7f2dc7fff640 1 --2- 172.21.15.5:0/2643239699 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2dac06b810 0x7f2dac06dcd0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2dc81e6020 tx=0x7f2db4009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:21.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.796+0000 7f2dc57fa640 1 -- 172.21.15.5:0/2643239699 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f2dbc07aa30 con 0x7f2dc806c580 2024-03-24T10:56:21.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:21.796+0000 7f2dc57fa640 1 -- 172.21.15.5:0/2643239699 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2dbc076cd0 con 0x7f2dc806c580 2024-03-24T10:56:22.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:22.044+0000 7f2dcf0d4640 1 -- 172.21.15.5:0/2643239699 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2dc80681d0 con 0x7f2dc806c580 2024-03-24T10:56:22.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:22.116+0000 7f2dc57fa640 1 -- 172.21.15.5:0/2643239699 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f2dbc048020 con 0x7f2dc806c580 2024-03-24T10:56:22.117 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:22.117 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:22.117 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:22.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:22.118+0000 7f2dcf0d4640 1 -- 172.21.15.5:0/2643239699 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2dac06b810 msgr2=0x7f2dac06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:22.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:22.118+0000 7f2dcf0d4640 1 --2- 172.21.15.5:0/2643239699 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2dac06b810 0x7f2dac06dcd0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2dc81e6020 tx=0x7f2db4009190 comp rx=0 tx=0).stop 2024-03-24T10:56:22.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:22.118+0000 7f2dcf0d4640 1 -- 172.21.15.5:0/2643239699 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2dc806c580 msgr2=0x7f2dc814f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:22.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:22.118+0000 7f2dcf0d4640 1 --2- 172.21.15.5:0/2643239699 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2dc806c580 0x7f2dc814f8d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2dbc009890 tx=0x7f2dbc0046c0 comp rx=0 tx=0).stop 2024-03-24T10:56:22.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:22.119+0000 7f2dcf0d4640 1 -- 172.21.15.5:0/2643239699 shutdown_connections 2024-03-24T10:56:22.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:22.119+0000 7f2dcf0d4640 1 --2- 172.21.15.5:0/2643239699 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2dac06b810 0x7f2dac06dcd0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:22.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:22.119+0000 7f2dcf0d4640 1 --2- 172.21.15.5:0/2643239699 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2dc80cf110 0x7f2dc814de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:22.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:22.119+0000 7f2dcf0d4640 1 --2- 172.21.15.5:0/2643239699 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2dc80cc8f0 0x7f2dc8157550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:22.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:22.119+0000 7f2dcf0d4640 1 --2- 172.21.15.5:0/2643239699 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2dc806c580 0x7f2dc814f8d0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:22.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:22.119+0000 7f2dcf0d4640 1 -- 172.21.15.5:0/2643239699 >> 172.21.15.5:0/2643239699 conn(0x7f2dc80b1820 msgr2=0x7f2dc81e6590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:22.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:22.119+0000 7f2dcf0d4640 1 -- 172.21.15.5:0/2643239699 shutdown_connections 2024-03-24T10:56:22.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:22.119+0000 7f2dcf0d4640 1 -- 172.21.15.5:0/2643239699 wait complete. 2024-03-24T10:56:22.735 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:22.735 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:23.736 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:23.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.837+0000 7fe86d793640 1 Processor -- start 2024-03-24T10:56:23.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.838+0000 7fe86d793640 1 -- start start 2024-03-24T10:56:23.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.838+0000 7fe86d793640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8681543a0 0x7fe86815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:23.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.838+0000 7fe86d793640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe86806d4f0 0x7fe8680cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:23.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.838+0000 7fe86d793640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe8680ccaa0 0x7fe86806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:23.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.838+0000 7fe86d793640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fe8680c6750 con 0x7fe86806d4f0 2024-03-24T10:56:23.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.838+0000 7fe86d793640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fe8680cc380 con 0x7fe8680ccaa0 2024-03-24T10:56:23.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.838+0000 7fe86d793640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe8680cb2f0 con 0x7fe8681543a0 2024-03-24T10:56:23.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.839+0000 7fe8667fc640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe86806d4f0 0x7fe8680cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:23.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.839+0000 7fe866ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8681543a0 0x7fe86815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:23.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.839+0000 7fe8677fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe8680ccaa0 0x7fe86806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:23.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.839+0000 7fe8667fc640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe86806d4f0 0x7fe8680cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52182/0 (socket says 172.21.15.5:52182) 2024-03-24T10:56:23.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.839+0000 7fe8667fc640 1 -- 172.21.15.5:0/4211639599 learned_addr learned my addr 172.21.15.5:0/4211639599 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:23.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.843+0000 7fe8667fc640 1 -- 172.21.15.5:0/4211639599 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8681543a0 msgr2=0x7fe86815ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:23.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.843+0000 7fe8667fc640 1 --2- 172.21.15.5:0/4211639599 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8681543a0 0x7fe86815ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:23.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.843+0000 7fe8667fc640 1 -- 172.21.15.5:0/4211639599 >> v2:172.21.15.57:3300/0 conn(0x7fe8680ccaa0 msgr2=0x7fe86806bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:23.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.843+0000 7fe8667fc640 1 --2- 172.21.15.5:0/4211639599 >> v2:172.21.15.57:3300/0 conn(0x7fe8680ccaa0 0x7fe86806bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:23.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.843+0000 7fe8667fc640 1 -- 172.21.15.5:0/4211639599 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8680ca7e0 con 0x7fe86806d4f0 2024-03-24T10:56:23.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.845+0000 7fe8667fc640 1 --2- 172.21.15.5:0/4211639599 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe86806d4f0 0x7fe8680cc560 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe86806de60 tx=0x7fe85c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ccb7fa075236bad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:23.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.850+0000 7fe865ffb640 1 -- 172.21.15.5:0/4211639599 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c030860 con 0x7fe86806d4f0 2024-03-24T10:56:23.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.850+0000 7fe865ffb640 1 -- 172.21.15.5:0/4211639599 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe85c0309c0 con 0x7fe86806d4f0 2024-03-24T10:56:23.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.850+0000 7fe865ffb640 1 -- 172.21.15.5:0/4211639599 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c030d50 con 0x7fe86806d4f0 2024-03-24T10:56:23.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.851+0000 7fe86d793640 1 -- 172.21.15.5:0/4211639599 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe86806d4f0 msgr2=0x7fe8680cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:23.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.851+0000 7fe86d793640 1 --2- 172.21.15.5:0/4211639599 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe86806d4f0 0x7fe8680cc560 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fe86806de60 tx=0x7fe85c02b9a0 comp rx=0 tx=0).stop 2024-03-24T10:56:23.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.851+0000 7fe86d793640 1 -- 172.21.15.5:0/4211639599 shutdown_connections 2024-03-24T10:56:23.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.851+0000 7fe86d793640 1 --2- 172.21.15.5:0/4211639599 >> v2:172.21.15.57:3300/0 conn(0x7fe8680ccaa0 0x7fe86806bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:23.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.851+0000 7fe86d793640 1 --2- 172.21.15.5:0/4211639599 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe86806d4f0 0x7fe8680cc560 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:23.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.851+0000 7fe86d793640 1 --2- 172.21.15.5:0/4211639599 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8681543a0 0x7fe86815ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:23.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.851+0000 7fe86d793640 1 -- 172.21.15.5:0/4211639599 >> 172.21.15.5:0/4211639599 conn(0x7fe868072ef0 msgr2=0x7fe868061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:23.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.851+0000 7fe86d793640 1 -- 172.21.15.5:0/4211639599 shutdown_connections 2024-03-24T10:56:23.854 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.851+0000 7fe86d793640 1 -- 172.21.15.5:0/4211639599 wait complete. 2024-03-24T10:56:23.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.853+0000 7fe86d793640 1 Processor -- start 2024-03-24T10:56:23.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.853+0000 7fe86d793640 1 -- start start 2024-03-24T10:56:23.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.853+0000 7fe86d793640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe86806d4f0 0x7fe8681576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:23.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.853+0000 7fe86d793640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8680ccaa0 0x7fe868156910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:23.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.853+0000 7fe86d793640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8681543a0 0x7fe8681f0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:23.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.853+0000 7fe86d793640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fe868154a60 con 0x7fe8681543a0 2024-03-24T10:56:23.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.853+0000 7fe86d793640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe8680c7720 con 0x7fe86806d4f0 2024-03-24T10:56:23.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.853+0000 7fe86d793640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe8680c7a00 con 0x7fe8680ccaa0 2024-03-24T10:56:23.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.854+0000 7fe8667fc640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8680ccaa0 0x7fe868156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:23.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.854+0000 7fe866ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe86806d4f0 0x7fe8681576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:23.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.854+0000 7fe8677fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8681543a0 0x7fe8681f0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:23.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.854+0000 7fe866ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe86806d4f0 0x7fe8681576e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:47872/0 (socket says 172.21.15.5:47872) 2024-03-24T10:56:23.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.854+0000 7fe866ffd640 1 -- 172.21.15.5:0/4090826588 learned_addr learned my addr 172.21.15.5:0/4090826588 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:23.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.859+0000 7fe8677fe640 1 -- 172.21.15.5:0/4090826588 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8680ccaa0 msgr2=0x7fe868156910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:23.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.859+0000 7fe8677fe640 1 --2- 172.21.15.5:0/4090826588 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8680ccaa0 0x7fe868156910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:23.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.859+0000 7fe8677fe640 1 -- 172.21.15.5:0/4090826588 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe86806d4f0 msgr2=0x7fe8681576e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:23.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.859+0000 7fe8677fe640 1 --2- 172.21.15.5:0/4090826588 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe86806d4f0 0x7fe8681576e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:23.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.859+0000 7fe8677fe640 1 -- 172.21.15.5:0/4090826588 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe85c0094d0 con 0x7fe8681543a0 2024-03-24T10:56:23.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.859+0000 7fe866ffd640 1 --2- 172.21.15.5:0/4090826588 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe86806d4f0 0x7fe8681576e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:56:23.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.862+0000 7fe8677fe640 1 --2- 172.21.15.5:0/4090826588 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8681543a0 0x7fe8681f0330 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe86000efc0 tx=0x7fe86000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:23.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.867+0000 7fe84f7fe640 1 -- 172.21.15.5:0/4090826588 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe860009c00 con 0x7fe8681543a0 2024-03-24T10:56:23.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.867+0000 7fe84f7fe640 1 -- 172.21.15.5:0/4090826588 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe86000f040 con 0x7fe8681543a0 2024-03-24T10:56:23.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.867+0000 7fe86d793640 1 -- 172.21.15.5:0/4090826588 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe868154cb0 con 0x7fe8681543a0 2024-03-24T10:56:23.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.867+0000 7fe84f7fe640 1 -- 172.21.15.5:0/4090826588 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8600146c0 con 0x7fe8681543a0 2024-03-24T10:56:23.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.867+0000 7fe86d793640 1 -- 172.21.15.5:0/4090826588 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8680c7410 con 0x7fe8681543a0 2024-03-24T10:56:23.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:23.868+0000 7fe86d793640 1 -- 172.21.15.5:0/4090826588 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8681551d0 con 0x7fe8681543a0 2024-03-24T10:56:24.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:24.149+0000 7fe84f7fe640 1 -- 172.21.15.5:0/4090826588 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe860014820 con 0x7fe8681543a0 2024-03-24T10:56:24.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:24.150+0000 7fe84f7fe640 1 --2- 172.21.15.5:0/4090826588 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe85406b860 0x7fe85406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:24.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:24.150+0000 7fe866ffd640 1 --2- 172.21.15.5:0/4090826588 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe85406b860 0x7fe85406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:24.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:24.150+0000 7fe866ffd640 1 --2- 172.21.15.5:0/4090826588 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe85406b860 0x7fe85406dd20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe858009ed0 tx=0x7fe858009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:24.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:24.172+0000 7fe84f7fe640 1 -- 172.21.15.5:0/4090826588 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fe860059d20 con 0x7fe8681543a0 2024-03-24T10:56:24.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:24.172+0000 7fe84f7fe640 1 -- 172.21.15.5:0/4090826588 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe860055fc0 con 0x7fe8681543a0 2024-03-24T10:56:24.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:24.401+0000 7fe86d793640 1 -- 172.21.15.5:0/4090826588 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8680681d0 con 0x7fe8681543a0 2024-03-24T10:56:24.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:24.425+0000 7fe84f7fe640 1 -- 172.21.15.5:0/4090826588 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7fe860018090 con 0x7fe8681543a0 2024-03-24T10:56:24.426 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:24.426 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:24.426 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:24.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:24.428+0000 7fe86d793640 1 -- 172.21.15.5:0/4090826588 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe85406b860 msgr2=0x7fe85406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:24.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:24.428+0000 7fe86d793640 1 --2- 172.21.15.5:0/4090826588 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe85406b860 0x7fe85406dd20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe858009ed0 tx=0x7fe858009190 comp rx=0 tx=0).stop 2024-03-24T10:56:24.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:24.428+0000 7fe86d793640 1 -- 172.21.15.5:0/4090826588 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8681543a0 msgr2=0x7fe8681f0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:24.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:24.428+0000 7fe86d793640 1 --2- 172.21.15.5:0/4090826588 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8681543a0 0x7fe8681f0330 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fe86000efc0 tx=0x7fe86000c490 comp rx=0 tx=0).stop 2024-03-24T10:56:24.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:24.428+0000 7fe86d793640 1 -- 172.21.15.5:0/4090826588 shutdown_connections 2024-03-24T10:56:24.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:24.428+0000 7fe86d793640 1 --2- 172.21.15.5:0/4090826588 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe85406b860 0x7fe85406dd20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:24.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:24.428+0000 7fe86d793640 1 --2- 172.21.15.5:0/4090826588 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe8681543a0 0x7fe8681f0330 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:24.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:24.428+0000 7fe86d793640 1 --2- 172.21.15.5:0/4090826588 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe8680ccaa0 0x7fe868156910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:24.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:24.428+0000 7fe86d793640 1 --2- 172.21.15.5:0/4090826588 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe86806d4f0 0x7fe8681576e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:24.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:24.428+0000 7fe86d793640 1 -- 172.21.15.5:0/4090826588 >> 172.21.15.5:0/4090826588 conn(0x7fe868072ef0 msgr2=0x7fe86806ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:24.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:24.428+0000 7fe86d793640 1 -- 172.21.15.5:0/4090826588 shutdown_connections 2024-03-24T10:56:24.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:24.428+0000 7fe86d793640 1 -- 172.21.15.5:0/4090826588 wait complete. 2024-03-24T10:56:25.043 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:25.044 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:26.044 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:26.147 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.147+0000 7f97fb181640 1 Processor -- start 2024-03-24T10:56:26.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.147+0000 7f97fb181640 1 -- start start 2024-03-24T10:56:26.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.148+0000 7f97fb181640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f97f40cb270 0x7f97f40cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:26.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.148+0000 7f97fb181640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f97f40cd960 0x7f97f40cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:26.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.148+0000 7f97fb181640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97f40ccba0 con 0x7f97f40cae90 2024-03-24T10:56:26.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.148+0000 7f97fb181640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f97f40cc9b0 con 0x7f97f40cd960 2024-03-24T10:56:26.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.148+0000 7f97fb181640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f97f40cce40 con 0x7f97f40cb270 2024-03-24T10:56:26.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.149+0000 7f97f971e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f97f40cd960 0x7f97f40cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:26.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.149+0000 7f97f971e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f97f40cd960 0x7f97f40cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52210/0 (socket says 172.21.15.5:52210) 2024-03-24T10:56:26.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.149+0000 7f97f971e640 1 -- 172.21.15.5:0/2398111916 learned_addr learned my addr 172.21.15.5:0/2398111916 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:26.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.149+0000 7f97f3fff640 1 --2- 172.21.15.5:0/2398111916 >> v2:172.21.15.57:3300/0 conn(0x7f97f40cb270 0x7f97f40cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:26.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.152+0000 7f97f971e640 1 -- 172.21.15.5:0/2398111916 >> v2:172.21.15.57:3300/0 conn(0x7f97f40cb270 msgr2=0x7f97f40cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:26.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.152+0000 7f97f971e640 1 --2- 172.21.15.5:0/2398111916 >> v2:172.21.15.57:3300/0 conn(0x7f97f40cb270 0x7f97f40cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:26.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.153+0000 7f97f971e640 1 -- 172.21.15.5:0/2398111916 >> v1:172.21.15.57:6789/0 conn(0x7f97f40cae90 legacy=0x7f97f40cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:26.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.153+0000 7f97f971e640 1 -- 172.21.15.5:0/2398111916 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f415a590 con 0x7f97f40cd960 2024-03-24T10:56:26.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.155+0000 7f97f971e640 1 --2- 172.21.15.5:0/2398111916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97f40cd960 0x7f97f40cb6f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f97f40cd680 tx=0x7f97ec02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b0e14a966a60915 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:26.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.160+0000 7f97f37fe640 1 -- 172.21.15.5:0/2398111916 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ec02cbb0 con 0x7f97f40cd960 2024-03-24T10:56:26.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.160+0000 7f97f37fe640 1 -- 172.21.15.5:0/2398111916 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97ec02cd10 con 0x7f97f40cd960 2024-03-24T10:56:26.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.160+0000 7f97f37fe640 1 -- 172.21.15.5:0/2398111916 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ec034610 con 0x7f97f40cd960 2024-03-24T10:56:26.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.160+0000 7f97fb181640 1 -- 172.21.15.5:0/2398111916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97f40cd960 msgr2=0x7f97f40cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:26.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.160+0000 7f97fb181640 1 --2- 172.21.15.5:0/2398111916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97f40cd960 0x7f97f40cb6f0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f97f40cd680 tx=0x7f97ec02c010 comp rx=0 tx=0).stop 2024-03-24T10:56:26.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.160+0000 7f97fb181640 1 -- 172.21.15.5:0/2398111916 shutdown_connections 2024-03-24T10:56:26.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.160+0000 7f97fb181640 1 --2- 172.21.15.5:0/2398111916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97f40cd960 0x7f97f40cb6f0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:26.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.160+0000 7f97fb181640 1 --2- 172.21.15.5:0/2398111916 >> v2:172.21.15.57:3300/0 conn(0x7f97f40cb270 0x7f97f40cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:26.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.160+0000 7f97fb181640 1 -- 172.21.15.5:0/2398111916 >> 172.21.15.5:0/2398111916 conn(0x7f97f40cfb80 msgr2=0x7f97f40cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:26.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.161+0000 7f97fb181640 1 -- 172.21.15.5:0/2398111916 shutdown_connections 2024-03-24T10:56:26.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.161+0000 7f97fb181640 1 -- 172.21.15.5:0/2398111916 wait complete. 2024-03-24T10:56:26.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.162+0000 7f97fb181640 1 Processor -- start 2024-03-24T10:56:26.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.162+0000 7f97fb181640 1 -- start start 2024-03-24T10:56:26.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.163+0000 7f97fb181640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f97f40cae90 0x7f97f40b5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:26.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.163+0000 7f97fb181640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97f40cb270 0x7f97f40c4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:26.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.163+0000 7f97fb181640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97f40cd960 0x7f97f40c5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:26.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.163+0000 7f97fb181640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f97f415a0f0 con 0x7f97f40cd960 2024-03-24T10:56:26.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.163+0000 7f97fb181640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f97f4157970 con 0x7f97f40cb270 2024-03-24T10:56:26.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.163+0000 7f97fb181640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f97f40c67e0 con 0x7f97f40cae90 2024-03-24T10:56:26.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.163+0000 7f97f971e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97f40cd960 0x7f97f40c5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:26.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.163+0000 7f97f3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97f40cb270 0x7f97f40c4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:26.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.164+0000 7f97f8f1d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f97f40cae90 0x7f97f40b5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:26.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.164+0000 7f97f3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97f40cb270 0x7f97f40c4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:47886/0 (socket says 172.21.15.5:47886) 2024-03-24T10:56:26.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.164+0000 7f97f3fff640 1 -- 172.21.15.5:0/843693151 learned_addr learned my addr 172.21.15.5:0/843693151 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:26.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.167+0000 7f97f3fff640 1 -- 172.21.15.5:0/843693151 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f97f40cae90 msgr2=0x7f97f40b5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:26.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.167+0000 7f97f3fff640 1 --2- 172.21.15.5:0/843693151 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f97f40cae90 0x7f97f40b5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:26.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.167+0000 7f97f3fff640 1 -- 172.21.15.5:0/843693151 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97f40cd960 msgr2=0x7f97f40c5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:26.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.167+0000 7f97f3fff640 1 --2- 172.21.15.5:0/843693151 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97f40cd960 0x7f97f40c5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:26.167 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.167+0000 7f97f3fff640 1 -- 172.21.15.5:0/843693151 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97ec009b30 con 0x7f97f40cb270 2024-03-24T10:56:26.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.169+0000 7f97f3fff640 1 --2- 172.21.15.5:0/843693151 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97f40cb270 0x7f97f40c4e00 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f97e000e9b0 tx=0x7f97e000ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:26.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.174+0000 7f97f17fa640 1 -- 172.21.15.5:0/843693151 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e0009670 con 0x7f97f40cb270 2024-03-24T10:56:26.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.174+0000 7f97f17fa640 1 -- 172.21.15.5:0/843693151 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97e000c9a0 con 0x7f97f40cb270 2024-03-24T10:56:26.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.174+0000 7f97f17fa640 1 -- 172.21.15.5:0/843693151 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e000cd50 con 0x7f97f40cb270 2024-03-24T10:56:26.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.174+0000 7f97fb181640 1 -- 172.21.15.5:0/843693151 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97f4159da0 con 0x7f97f40cb270 2024-03-24T10:56:26.174 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.174+0000 7f97fb181640 1 -- 172.21.15.5:0/843693151 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97f41f03f0 con 0x7f97f40cb270 2024-03-24T10:56:26.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.175+0000 7f97fb181640 1 -- 172.21.15.5:0/843693151 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97f415a260 con 0x7f97f40cb270 2024-03-24T10:56:26.194 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.194+0000 7f97f17fa640 1 -- 172.21.15.5:0/843693151 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f97e0013470 con 0x7f97f40cb270 2024-03-24T10:56:26.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.195+0000 7f97f17fa640 1 --2- 172.21.15.5:0/843693151 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f97d806b860 0x7f97d806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:26.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.195+0000 7f97f8f1d640 1 --2- 172.21.15.5:0/843693151 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f97d806b860 0x7f97d806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:26.195 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.195+0000 7f97f8f1d640 1 --2- 172.21.15.5:0/843693151 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f97d806b860 0x7f97d806dd20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f97e8001780 tx=0x7f97e8009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:26.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.446+0000 7f97f17fa640 1 -- 172.21.15.5:0/843693151 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f97e00593c0 con 0x7f97f40cb270 2024-03-24T10:56:26.458 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.458+0000 7f97f17fa640 1 -- 172.21.15.5:0/843693151 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f97e0055660 con 0x7f97f40cb270 2024-03-24T10:56:26.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.704+0000 7f97fb181640 1 -- 172.21.15.5:0/843693151 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97f4159f30 con 0x7f97f40cb270 2024-03-24T10:56:26.727 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.727+0000 7f97f17fa640 1 -- 172.21.15.5:0/843693151 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f97f4159f30 con 0x7f97f40cb270 2024-03-24T10:56:26.727 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:26.727 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:26.727 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:26.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.729+0000 7f97fb181640 1 -- 172.21.15.5:0/843693151 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f97d806b860 msgr2=0x7f97d806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:26.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.729+0000 7f97fb181640 1 --2- 172.21.15.5:0/843693151 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f97d806b860 0x7f97d806dd20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f97e8001780 tx=0x7f97e8009040 comp rx=0 tx=0).stop 2024-03-24T10:56:26.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.729+0000 7f97fb181640 1 -- 172.21.15.5:0/843693151 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97f40cb270 msgr2=0x7f97f40c4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:26.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.729+0000 7f97fb181640 1 --2- 172.21.15.5:0/843693151 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97f40cb270 0x7f97f40c4e00 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f97e000e9b0 tx=0x7f97e000ee70 comp rx=0 tx=0).stop 2024-03-24T10:56:26.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.729+0000 7f97fb181640 1 -- 172.21.15.5:0/843693151 shutdown_connections 2024-03-24T10:56:26.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.729+0000 7f97fb181640 1 --2- 172.21.15.5:0/843693151 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f97d806b860 0x7f97d806dd20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:26.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.729+0000 7f97fb181640 1 --2- 172.21.15.5:0/843693151 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f97f40cd960 0x7f97f40c5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:26.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.729+0000 7f97fb181640 1 --2- 172.21.15.5:0/843693151 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97f40cb270 0x7f97f40c4e00 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:26.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.729+0000 7f97fb181640 1 --2- 172.21.15.5:0/843693151 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f97f40cae90 0x7f97f40b5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:26.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.729+0000 7f97fb181640 1 -- 172.21.15.5:0/843693151 >> 172.21.15.5:0/843693151 conn(0x7f97f40cfb80 msgr2=0x7f97f40cddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:26.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.729+0000 7f97fb181640 1 -- 172.21.15.5:0/843693151 shutdown_connections 2024-03-24T10:56:26.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:26.729+0000 7f97fb181640 1 -- 172.21.15.5:0/843693151 wait complete. 2024-03-24T10:56:27.344 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:27.344 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:28.345 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:28.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.447+0000 7ff19a2d4640 1 Processor -- start 2024-03-24T10:56:28.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.447+0000 7ff19a2d4640 1 -- start start 2024-03-24T10:56:28.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.448+0000 7ff19a2d4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff19415aea0 0x7ff1940cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:28.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.448+0000 7ff19a2d4640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff1940cf100 0x7ff1940ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:28.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.448+0000 7ff19a2d4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff19406cd50 con 0x7ff19415aac0 2024-03-24T10:56:28.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.448+0000 7ff19a2d4640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7ff1940ce790 con 0x7ff1940cf100 2024-03-24T10:56:28.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.448+0000 7ff19a2d4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff1940cdb10 con 0x7ff19415aea0 2024-03-24T10:56:28.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.448+0000 7ff198871640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff1940cf100 0x7ff1940ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:28.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.449+0000 7ff1937fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff19415aea0 0x7ff1940cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:28.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.449+0000 7ff1937fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff19415aea0 0x7ff1940cebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:49072/0 (socket says 172.21.15.5:49072) 2024-03-24T10:56:28.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.449+0000 7ff1937fe640 1 -- 172.21.15.5:0/2942240281 learned_addr learned my addr 172.21.15.5:0/2942240281 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:28.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.452+0000 7ff1937fe640 1 -- 172.21.15.5:0/2942240281 >> v2:172.21.15.5:3300/0 conn(0x7ff1940cf100 msgr2=0x7ff1940ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:28.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.452+0000 7ff1937fe640 1 --2- 172.21.15.5:0/2942240281 >> v2:172.21.15.5:3300/0 conn(0x7ff1940cf100 0x7ff1940ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:28.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.452+0000 7ff1937fe640 1 -- 172.21.15.5:0/2942240281 >> v1:172.21.15.57:6789/0 conn(0x7ff19415aac0 legacy=0x7ff19406c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:28.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.452+0000 7ff1937fe640 1 -- 172.21.15.5:0/2942240281 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1940cddf0 con 0x7ff19415aea0 2024-03-24T10:56:28.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.456+0000 7ff1937fe640 1 --2- 172.21.15.5:0/2942240281 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff19415aea0 0x7ff1940cebc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff188009870 tx=0x7ff18802ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ce2ae2ef8aa1e85 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:28.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.462+0000 7ff192ffd640 1 -- 172.21.15.5:0/2942240281 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18802f840 con 0x7ff19415aea0 2024-03-24T10:56:28.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.462+0000 7ff192ffd640 1 -- 172.21.15.5:0/2942240281 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff18802f9a0 con 0x7ff19415aea0 2024-03-24T10:56:28.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.462+0000 7ff192ffd640 1 -- 172.21.15.5:0/2942240281 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18802fd30 con 0x7ff19415aea0 2024-03-24T10:56:28.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.462+0000 7ff19a2d4640 1 -- 172.21.15.5:0/2942240281 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff19415aea0 msgr2=0x7ff1940cebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:28.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.462+0000 7ff19a2d4640 1 --2- 172.21.15.5:0/2942240281 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff19415aea0 0x7ff1940cebc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff188009870 tx=0x7ff18802ba20 comp rx=0 tx=0).stop 2024-03-24T10:56:28.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.462+0000 7ff19a2d4640 1 -- 172.21.15.5:0/2942240281 shutdown_connections 2024-03-24T10:56:28.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.462+0000 7ff19a2d4640 1 --2- 172.21.15.5:0/2942240281 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff19415aea0 0x7ff1940cebc0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:28.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.462+0000 7ff19a2d4640 1 --2- 172.21.15.5:0/2942240281 >> v2:172.21.15.5:3300/0 conn(0x7ff1940cf100 0x7ff1940ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:28.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.462+0000 7ff19a2d4640 1 -- 172.21.15.5:0/2942240281 >> 172.21.15.5:0/2942240281 conn(0x7ff1940cfb90 msgr2=0x7ff1940cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:28.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.462+0000 7ff19a2d4640 1 -- 172.21.15.5:0/2942240281 shutdown_connections 2024-03-24T10:56:28.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.462+0000 7ff19a2d4640 1 -- 172.21.15.5:0/2942240281 wait complete. 2024-03-24T10:56:28.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.464+0000 7ff19a2d4640 1 Processor -- start 2024-03-24T10:56:28.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.464+0000 7ff19a2d4640 1 -- start start 2024-03-24T10:56:28.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.465+0000 7ff19a2d4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff1940cf100 0x7ff1941e39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:28.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.465+0000 7ff19a2d4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff19415aac0 0x7ff1941f0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:28.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.465+0000 7ff19a2d4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff19415aea0 0x7ff19414fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:28.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.465+0000 7ff19a2d4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff19415c200 con 0x7ff19415aea0 2024-03-24T10:56:28.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.465+0000 7ff19a2d4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff19415ba20 con 0x7ff19415aac0 2024-03-24T10:56:28.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.465+0000 7ff19a2d4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff1941e5e20 con 0x7ff1940cf100 2024-03-24T10:56:28.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.465+0000 7ff198871640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff19415aea0 0x7ff19414fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:28.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.465+0000 7ff1937fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff19415aac0 0x7ff1941f0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:28.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.465+0000 7ff1937fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff19415aac0 0x7ff1941f0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:47898/0 (socket says 172.21.15.5:47898) 2024-03-24T10:56:28.466 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.465+0000 7ff1937fe640 1 -- 172.21.15.5:0/501210085 learned_addr learned my addr 172.21.15.5:0/501210085 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:28.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.467+0000 7ff193fff640 1 --2- 172.21.15.5:0/501210085 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff1940cf100 0x7ff1941e39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:28.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.470+0000 7ff1937fe640 1 -- 172.21.15.5:0/501210085 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff1940cf100 msgr2=0x7ff1941e39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:28.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.470+0000 7ff1937fe640 1 --2- 172.21.15.5:0/501210085 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff1940cf100 0x7ff1941e39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:28.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.470+0000 7ff1937fe640 1 -- 172.21.15.5:0/501210085 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff19415aea0 msgr2=0x7ff19414fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:28.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.470+0000 7ff1937fe640 1 --2- 172.21.15.5:0/501210085 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff19415aea0 0x7ff19414fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:28.470 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.470+0000 7ff1937fe640 1 -- 172.21.15.5:0/501210085 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1880094d0 con 0x7ff19415aac0 2024-03-24T10:56:28.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.473+0000 7ff1937fe640 1 --2- 172.21.15.5:0/501210085 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff19415aac0 0x7ff1941f0140 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff188000c00 tx=0x7ff188004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:28.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.478+0000 7ff190ff9640 1 -- 172.21.15.5:0/501210085 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188037c40 con 0x7ff19415aac0 2024-03-24T10:56:28.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.479+0000 7ff190ff9640 1 -- 172.21.15.5:0/501210085 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff188037da0 con 0x7ff19415aac0 2024-03-24T10:56:28.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.479+0000 7ff19a2d4640 1 -- 172.21.15.5:0/501210085 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1940b1090 con 0x7ff19415aac0 2024-03-24T10:56:28.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.479+0000 7ff190ff9640 1 -- 172.21.15.5:0/501210085 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188041640 con 0x7ff19415aac0 2024-03-24T10:56:28.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.479+0000 7ff19a2d4640 1 -- 172.21.15.5:0/501210085 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff19415bc30 con 0x7ff19415aac0 2024-03-24T10:56:28.479 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.480+0000 7ff19a2d4640 1 -- 172.21.15.5:0/501210085 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1940681d0 con 0x7ff19415aac0 2024-03-24T10:56:28.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.784+0000 7ff190ff9640 1 -- 172.21.15.5:0/501210085 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff1880417a0 con 0x7ff19415aac0 2024-03-24T10:56:28.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.784+0000 7ff190ff9640 1 --2- 172.21.15.5:0/501210085 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff18006b860 0x7ff18006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:28.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.784+0000 7ff193fff640 1 --2- 172.21.15.5:0/501210085 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff18006b860 0x7ff18006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:28.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.784+0000 7ff193fff640 1 --2- 172.21.15.5:0/501210085 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff18006b860 0x7ff18006dd20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff184001780 tx=0x7ff184009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:28.787 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.787+0000 7ff190ff9640 1 -- 172.21.15.5:0/501210085 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7ff18803c070 con 0x7ff19415aac0 2024-03-24T10:56:28.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:28.800+0000 7ff190ff9640 1 -- 172.21.15.5:0/501210085 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff1880784f0 con 0x7ff19415aac0 2024-03-24T10:56:29.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:29.024+0000 7ff19a2d4640 1 -- 172.21.15.5:0/501210085 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1940cf770 con 0x7ff19415aac0 2024-03-24T10:56:29.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:29.047+0000 7ff190ff9640 1 -- 172.21.15.5:0/501210085 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7ff18807bbb0 con 0x7ff19415aac0 2024-03-24T10:56:29.047 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:29.047 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:29.048 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:29.049 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:29.049+0000 7ff19a2d4640 1 -- 172.21.15.5:0/501210085 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff18006b860 msgr2=0x7ff18006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:29.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:29.049+0000 7ff19a2d4640 1 --2- 172.21.15.5:0/501210085 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff18006b860 0x7ff18006dd20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff184001780 tx=0x7ff184009040 comp rx=0 tx=0).stop 2024-03-24T10:56:29.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:29.049+0000 7ff19a2d4640 1 -- 172.21.15.5:0/501210085 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff19415aac0 msgr2=0x7ff1941f0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:29.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:29.049+0000 7ff19a2d4640 1 --2- 172.21.15.5:0/501210085 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff19415aac0 0x7ff1941f0140 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff188000c00 tx=0x7ff188004730 comp rx=0 tx=0).stop 2024-03-24T10:56:29.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:29.049+0000 7ff19a2d4640 1 -- 172.21.15.5:0/501210085 shutdown_connections 2024-03-24T10:56:29.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:29.049+0000 7ff19a2d4640 1 --2- 172.21.15.5:0/501210085 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff18006b860 0x7ff18006dd20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:29.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:29.049+0000 7ff19a2d4640 1 --2- 172.21.15.5:0/501210085 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff19415aea0 0x7ff19414fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:29.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:29.049+0000 7ff19a2d4640 1 --2- 172.21.15.5:0/501210085 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff19415aac0 0x7ff1941f0140 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:29.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:29.049+0000 7ff19a2d4640 1 --2- 172.21.15.5:0/501210085 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff1940cf100 0x7ff1941e39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:29.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:29.049+0000 7ff19a2d4640 1 -- 172.21.15.5:0/501210085 >> 172.21.15.5:0/501210085 conn(0x7ff1940cfb90 msgr2=0x7ff1940c8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:29.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:29.049+0000 7ff19a2d4640 1 -- 172.21.15.5:0/501210085 shutdown_connections 2024-03-24T10:56:29.050 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:29.049+0000 7ff19a2d4640 1 -- 172.21.15.5:0/501210085 wait complete. 2024-03-24T10:56:29.665 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:29.666 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:30.666 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:30.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.772+0000 7fc6147f4640 1 Processor -- start 2024-03-24T10:56:30.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.772+0000 7fc6147f4640 1 -- start start 2024-03-24T10:56:30.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.773+0000 7fc6147f4640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc60c1548e0 0x7fc60c06b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:30.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.773+0000 7fc6147f4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc60c0732e0 0x7fc60c15e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:30.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.773+0000 7fc6147f4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc60c0c4d70 con 0x7fc60c154500 2024-03-24T10:56:30.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.773+0000 7fc6147f4640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fc60c0ca160 con 0x7fc60c1548e0 2024-03-24T10:56:30.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.773+0000 7fc6147f4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc60c0c9550 con 0x7fc60c0732e0 2024-03-24T10:56:30.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.773+0000 7fc611d8f640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc60c1548e0 0x7fc60c06b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:30.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.774+0000 7fc611d8f640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc60c1548e0 0x7fc60c06b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52240/0 (socket says 172.21.15.5:52240) 2024-03-24T10:56:30.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.774+0000 7fc611d8f640 1 -- 172.21.15.5:0/2883186384 learned_addr learned my addr 172.21.15.5:0/2883186384 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:30.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.776+0000 7fc611d8f640 1 -- 172.21.15.5:0/2883186384 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc60c0732e0 msgr2=0x7fc60c15e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:30.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.776+0000 7fc611d8f640 1 --2- 172.21.15.5:0/2883186384 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc60c0732e0 0x7fc60c15e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:30.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.776+0000 7fc611d8f640 1 -- 172.21.15.5:0/2883186384 >> v1:172.21.15.57:6789/0 conn(0x7fc60c154500 legacy=0x7fc60c0cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:30.776 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.776+0000 7fc611d8f640 1 -- 172.21.15.5:0/2883186384 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc60c0c9830 con 0x7fc60c1548e0 2024-03-24T10:56:30.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.778+0000 7fc611d8f640 1 --2- 172.21.15.5:0/2883186384 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc60c1548e0 0x7fc60c06b8c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc009870 tx=0x7fc5fc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9b810c05fcb2b6e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:30.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.782+0000 7fc61158e640 1 -- 172.21.15.5:0/2883186384 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc031640 con 0x7fc60c1548e0 2024-03-24T10:56:30.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.782+0000 7fc61158e640 1 -- 172.21.15.5:0/2883186384 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5fc0317a0 con 0x7fc60c1548e0 2024-03-24T10:56:30.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.782+0000 7fc61158e640 1 -- 172.21.15.5:0/2883186384 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc031b30 con 0x7fc60c1548e0 2024-03-24T10:56:30.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.782+0000 7fc6147f4640 1 -- 172.21.15.5:0/2883186384 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc60c1548e0 msgr2=0x7fc60c06b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:30.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.782+0000 7fc6147f4640 1 --2- 172.21.15.5:0/2883186384 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc60c1548e0 0x7fc60c06b8c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc009870 tx=0x7fc5fc02b8f0 comp rx=0 tx=0).stop 2024-03-24T10:56:30.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.782+0000 7fc6147f4640 1 -- 172.21.15.5:0/2883186384 shutdown_connections 2024-03-24T10:56:30.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.782+0000 7fc6147f4640 1 --2- 172.21.15.5:0/2883186384 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc60c0732e0 0x7fc60c15e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:30.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.782+0000 7fc6147f4640 1 --2- 172.21.15.5:0/2883186384 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc60c1548e0 0x7fc60c06b8c0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:30.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.782+0000 7fc6147f4640 1 -- 172.21.15.5:0/2883186384 >> 172.21.15.5:0/2883186384 conn(0x7fc60c06c240 msgr2=0x7fc60c06c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:30.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.782+0000 7fc6147f4640 1 -- 172.21.15.5:0/2883186384 shutdown_connections 2024-03-24T10:56:30.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.782+0000 7fc6147f4640 1 -- 172.21.15.5:0/2883186384 wait complete. 2024-03-24T10:56:30.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.784+0000 7fc6147f4640 1 Processor -- start 2024-03-24T10:56:30.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.784+0000 7fc6147f4640 1 -- start start 2024-03-24T10:56:30.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.784+0000 7fc6147f4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc60c0732e0 0x7fc60c151560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:30.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.784+0000 7fc6147f4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc60c154500 0x7fc60c1f1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:30.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.784+0000 7fc6147f4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc60c1548e0 0x7fc60c14f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:30.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.784+0000 7fc6147f4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fc60c1f1760 con 0x7fc60c154500 2024-03-24T10:56:30.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.784+0000 7fc6147f4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc60c15a930 con 0x7fc60c0732e0 2024-03-24T10:56:30.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.784+0000 7fc6147f4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc60c1f11b0 con 0x7fc60c1548e0 2024-03-24T10:56:30.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.785+0000 7fc611d8f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc60c154500 0x7fc60c1f1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:30.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.785+0000 7fc611d8f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc60c154500 0x7fc60c1f1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52242/0 (socket says 172.21.15.5:52242) 2024-03-24T10:56:30.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.785+0000 7fc611d8f640 1 -- 172.21.15.5:0/2083851863 learned_addr learned my addr 172.21.15.5:0/2083851863 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:30.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.789+0000 7fc611d8f640 1 -- 172.21.15.5:0/2083851863 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc60c1548e0 msgr2=0x7fc60c14f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:30.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.789+0000 7fc611d8f640 1 --2- 172.21.15.5:0/2083851863 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc60c1548e0 0x7fc60c14f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:30.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.789+0000 7fc611d8f640 1 -- 172.21.15.5:0/2083851863 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc60c0732e0 msgr2=0x7fc60c151560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:30.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.789+0000 7fc611d8f640 1 --2- 172.21.15.5:0/2083851863 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc60c0732e0 0x7fc60c151560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:30.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.789+0000 7fc611d8f640 1 -- 172.21.15.5:0/2083851863 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5fc0094d0 con 0x7fc60c154500 2024-03-24T10:56:30.791 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.791+0000 7fc611d8f640 1 --2- 172.21.15.5:0/2083851863 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc60c154500 0x7fc60c1f1bd0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc009870 tx=0x7fc5fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:30.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.795+0000 7fc602ffd640 1 -- 172.21.15.5:0/2083851863 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc03ac60 con 0x7fc60c154500 2024-03-24T10:56:30.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.795+0000 7fc602ffd640 1 -- 172.21.15.5:0/2083851863 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5fc03adc0 con 0x7fc60c154500 2024-03-24T10:56:30.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.795+0000 7fc6147f4640 1 -- 172.21.15.5:0/2083851863 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc60c1f1480 con 0x7fc60c154500 2024-03-24T10:56:30.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.795+0000 7fc6147f4640 1 -- 172.21.15.5:0/2083851863 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc60c158ad0 con 0x7fc60c154500 2024-03-24T10:56:30.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.796+0000 7fc602ffd640 1 -- 172.21.15.5:0/2083851863 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc044630 con 0x7fc60c154500 2024-03-24T10:56:30.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.796+0000 7fc6147f4640 1 -- 172.21.15.5:0/2083851863 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc60c072b90 con 0x7fc60c154500 2024-03-24T10:56:30.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.821+0000 7fc602ffd640 1 -- 172.21.15.5:0/2083851863 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc5fc02f050 con 0x7fc60c154500 2024-03-24T10:56:30.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.822+0000 7fc602ffd640 1 --2- 172.21.15.5:0/2083851863 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc5f006b890 0x7fc5f006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:30.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.822+0000 7fc612590640 1 --2- 172.21.15.5:0/2083851863 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc5f006b890 0x7fc5f006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:30.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:30.822+0000 7fc612590640 1 --2- 172.21.15.5:0/2083851863 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc5f006b890 0x7fc5f006dd50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc608001560 tx=0x7fc608007990 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:31.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:31.087+0000 7fc602ffd640 1 -- 172.21.15.5:0/2083851863 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fc5fc07b4b0 con 0x7fc60c154500 2024-03-24T10:56:31.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:31.087+0000 7fc602ffd640 1 -- 172.21.15.5:0/2083851863 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc5fc077050 con 0x7fc60c154500 2024-03-24T10:56:31.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:31.310+0000 7fc6147f4640 1 -- 172.21.15.5:0/2083851863 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc60c04f540 con 0x7fc60c154500 2024-03-24T10:56:31.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:31.331+0000 7fc602ffd640 1 -- 172.21.15.5:0/2083851863 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7fc5fc031e60 con 0x7fc60c154500 2024-03-24T10:56:31.332 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:31.332 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:31.332 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:31.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:31.334+0000 7fc6147f4640 1 -- 172.21.15.5:0/2083851863 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc5f006b890 msgr2=0x7fc5f006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:31.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:31.334+0000 7fc6147f4640 1 --2- 172.21.15.5:0/2083851863 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc5f006b890 0x7fc5f006dd50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc608001560 tx=0x7fc608007990 comp rx=0 tx=0).stop 2024-03-24T10:56:31.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:31.334+0000 7fc6147f4640 1 -- 172.21.15.5:0/2083851863 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc60c154500 msgr2=0x7fc60c1f1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:31.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:31.334+0000 7fc6147f4640 1 --2- 172.21.15.5:0/2083851863 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc60c154500 0x7fc60c1f1bd0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc009870 tx=0x7fc5fc0046c0 comp rx=0 tx=0).stop 2024-03-24T10:56:31.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:31.334+0000 7fc6147f4640 1 -- 172.21.15.5:0/2083851863 shutdown_connections 2024-03-24T10:56:31.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:31.334+0000 7fc6147f4640 1 --2- 172.21.15.5:0/2083851863 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc5f006b890 0x7fc5f006dd50 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:31.335 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:31.334+0000 7fc6147f4640 1 --2- 172.21.15.5:0/2083851863 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc60c154500 0x7fc60c1f1bd0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:31.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:31.334+0000 7fc6147f4640 1 --2- 172.21.15.5:0/2083851863 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc60c1548e0 0x7fc60c14f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:31.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:31.334+0000 7fc6147f4640 1 --2- 172.21.15.5:0/2083851863 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc60c0732e0 0x7fc60c151560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:31.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:31.334+0000 7fc6147f4640 1 -- 172.21.15.5:0/2083851863 >> 172.21.15.5:0/2083851863 conn(0x7fc60c06c240 msgr2=0x7fc60c0c52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:31.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:31.334+0000 7fc6147f4640 1 -- 172.21.15.5:0/2083851863 shutdown_connections 2024-03-24T10:56:31.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:31.334+0000 7fc6147f4640 1 -- 172.21.15.5:0/2083851863 wait complete. 2024-03-24T10:56:31.949 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:31.949 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:32.747 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:56:32.749 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:56:32.950 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:33.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.053+0000 7f1af1c84640 1 Processor -- start 2024-03-24T10:56:33.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.053+0000 7f1af1c84640 1 -- start start 2024-03-24T10:56:33.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.053+0000 7f1af1c84640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1aec15aea0 0x7f1aec0cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:33.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.053+0000 7f1af1c84640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1aec06cd50 con 0x7f1aec15aac0 2024-03-24T10:56:33.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.053+0000 7f1af1c84640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1aec0cdb10 con 0x7f1aec0cf100 2024-03-24T10:56:33.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.053+0000 7f1af1c84640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f1aec0ce790 con 0x7f1aec15aea0 2024-03-24T10:56:33.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.054+0000 7f1aebfff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f1aec0cf100 0x7f1aec0ca5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:36012/0 (socket says 172.21.15.5:36012) 2024-03-24T10:56:33.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.054+0000 7f1aebfff640 1 -- 172.21.15.5:0/4106243988 learned_addr learned my addr 172.21.15.5:0/4106243988 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:56:33.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.054+0000 7f1aeaffd640 1 --2- 172.21.15.5:0/4106243988 >> v2:172.21.15.5:3300/0 conn(0x7f1aec15aea0 0x7f1aec0cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:33.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.058+0000 7f1aea7fc640 1 -- 172.21.15.5:0/4106243988 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 801681801 0 0) 0x7f1aec0cdb10 con 0x7f1aec0cf100 2024-03-24T10:56:33.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.058+0000 7f1aea7fc640 1 -- 172.21.15.5:0/4106243988 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ad4003520 con 0x7f1aec0cf100 2024-03-24T10:56:33.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.058+0000 7f1aeaffd640 1 -- 172.21.15.5:0/4106243988 >> v1:172.21.15.57:6789/0 conn(0x7f1aec0cf100 legacy=0x7f1aec0ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:33.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.058+0000 7f1aeaffd640 1 -- 172.21.15.5:0/4106243988 >> v1:172.21.15.5:6789/0 conn(0x7f1aec15aac0 legacy=0x7f1aec06c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:33.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.058+0000 7f1aeaffd640 1 -- 172.21.15.5:0/4106243988 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aec0cddf0 con 0x7f1aec15aea0 2024-03-24T10:56:33.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.062+0000 7f1aeaffd640 1 --2- 172.21.15.5:0/4106243988 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1aec15aea0 0x7f1aec0cebc0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f1ae00097a0 tx=0x7f1ae002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b70b14ef98b3ea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:33.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.068+0000 7f1aea7fc640 1 -- 172.21.15.5:0/4106243988 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae002f810 con 0x7f1aec15aea0 2024-03-24T10:56:33.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.068+0000 7f1aea7fc640 1 -- 172.21.15.5:0/4106243988 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ae002f970 con 0x7f1aec15aea0 2024-03-24T10:56:33.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.068+0000 7f1aea7fc640 1 -- 172.21.15.5:0/4106243988 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae002fd00 con 0x7f1aec15aea0 2024-03-24T10:56:33.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.068+0000 7f1af1c84640 1 -- 172.21.15.5:0/4106243988 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1aec15aea0 msgr2=0x7f1aec0cebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:33.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.068+0000 7f1af1c84640 1 --2- 172.21.15.5:0/4106243988 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1aec15aea0 0x7f1aec0cebc0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f1ae00097a0 tx=0x7f1ae002b9f0 comp rx=0 tx=0).stop 2024-03-24T10:56:33.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.068+0000 7f1af1c84640 1 -- 172.21.15.5:0/4106243988 shutdown_connections 2024-03-24T10:56:33.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.068+0000 7f1af1c84640 1 --2- 172.21.15.5:0/4106243988 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1aec15aea0 0x7f1aec0cebc0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:33.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.068+0000 7f1af1c84640 1 -- 172.21.15.5:0/4106243988 >> 172.21.15.5:0/4106243988 conn(0x7f1aec0cfb90 msgr2=0x7f1aec0cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:33.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.068+0000 7f1af1c84640 1 -- 172.21.15.5:0/4106243988 shutdown_connections 2024-03-24T10:56:33.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.069+0000 7f1af1c84640 1 -- 172.21.15.5:0/4106243988 wait complete. 2024-03-24T10:56:33.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.070+0000 7f1af1c84640 1 Processor -- start 2024-03-24T10:56:33.070 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.070+0000 7f1af1c84640 1 -- start start 2024-03-24T10:56:33.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.071+0000 7f1af1c84640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1aec0cf100 0x7f1aec1f0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:33.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.071+0000 7f1af1c84640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1aec15aac0 0x7f1aec0aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:33.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.071+0000 7f1af1c84640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1aec15aea0 0x7f1aec153320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:33.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.071+0000 7f1af1c84640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f1aec0afb20 con 0x7f1aec0cf100 2024-03-24T10:56:33.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.071+0000 7f1af1c84640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1aec157e30 con 0x7f1aec15aea0 2024-03-24T10:56:33.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.071+0000 7f1af1c84640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1aec1f1040 con 0x7f1aec15aac0 2024-03-24T10:56:33.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.071+0000 7f1aeb7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1aec0cf100 0x7f1aec1f0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:33.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.071+0000 7f1aebfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1aec15aea0 0x7f1aec153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:33.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.071+0000 7f1aeaffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1aec15aac0 0x7f1aec0aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:33.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.071+0000 7f1aeb7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1aec0cf100 0x7f1aec1f0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39158/0 (socket says 172.21.15.5:39158) 2024-03-24T10:56:33.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.071+0000 7f1aeb7fe640 1 -- 172.21.15.5:0/2689722437 learned_addr learned my addr 172.21.15.5:0/2689722437 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:33.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.075+0000 7f1aeb7fe640 1 -- 172.21.15.5:0/2689722437 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1aec15aac0 msgr2=0x7f1aec0aff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:33.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.075+0000 7f1aeb7fe640 1 --2- 172.21.15.5:0/2689722437 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1aec15aac0 0x7f1aec0aff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:33.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.075+0000 7f1aeb7fe640 1 -- 172.21.15.5:0/2689722437 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1aec15aea0 msgr2=0x7f1aec153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:33.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.075+0000 7f1aeb7fe640 1 --2- 172.21.15.5:0/2689722437 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1aec15aea0 0x7f1aec153320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:33.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.075+0000 7f1aeb7fe640 1 -- 172.21.15.5:0/2689722437 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae0009400 con 0x7f1aec0cf100 2024-03-24T10:56:33.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.077+0000 7f1aeb7fe640 1 --2- 172.21.15.5:0/2689722437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1aec0cf100 0x7f1aec1f0b00 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f1adc00c400 tx=0x7f1adc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:33.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.082+0000 7f1ad3fff640 1 -- 172.21.15.5:0/2689722437 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1adc0052d0 con 0x7f1aec0cf100 2024-03-24T10:56:33.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.082+0000 7f1ad3fff640 1 -- 172.21.15.5:0/2689722437 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1adc005450 con 0x7f1aec0cf100 2024-03-24T10:56:33.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.082+0000 7f1af1c84640 1 -- 172.21.15.5:0/2689722437 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aec0af4f0 con 0x7f1aec0cf100 2024-03-24T10:56:33.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.082+0000 7f1ad3fff640 1 -- 172.21.15.5:0/2689722437 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1adc007260 con 0x7f1aec0cf100 2024-03-24T10:56:33.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.082+0000 7f1af1c84640 1 -- 172.21.15.5:0/2689722437 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aec0b18c0 con 0x7f1aec0cf100 2024-03-24T10:56:33.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.083+0000 7f1af1c84640 1 -- 172.21.15.5:0/2689722437 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aec0afc60 con 0x7f1aec0cf100 2024-03-24T10:56:33.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.107+0000 7f1ad3fff640 1 -- 172.21.15.5:0/2689722437 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1adc0073c0 con 0x7f1aec0cf100 2024-03-24T10:56:33.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.108+0000 7f1ad3fff640 1 --2- 172.21.15.5:0/2689722437 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1ad806b810 0x7f1ad806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:33.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.108+0000 7f1aeaffd640 1 --2- 172.21.15.5:0/2689722437 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1ad806b810 0x7f1ad806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:33.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.108+0000 7f1aeaffd640 1 --2- 172.21.15.5:0/2689722437 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1ad806b810 0x7f1ad806dcd0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0036000 tx=0x7f1ae0008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:33.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.111+0000 7f1ad3fff640 1 -- 172.21.15.5:0/2689722437 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f1adc0580d0 con 0x7f1aec0cf100 2024-03-24T10:56:33.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.404+0000 7f1ad3fff640 1 -- 172.21.15.5:0/2689722437 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1adc054370 con 0x7f1aec0cf100 2024-03-24T10:56:33.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.641+0000 7f1af1c84640 1 -- 172.21.15.5:0/2689722437 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1aec0afc60 con 0x7f1aec0cf100 2024-03-24T10:56:33.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.655+0000 7f1ad3fff640 1 -- 172.21.15.5:0/2689722437 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f1aec0afc60 con 0x7f1aec0cf100 2024-03-24T10:56:33.655 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:33.656 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:33.656 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:33.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.658+0000 7f1af1c84640 1 -- 172.21.15.5:0/2689722437 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1ad806b810 msgr2=0x7f1ad806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:33.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.658+0000 7f1af1c84640 1 --2- 172.21.15.5:0/2689722437 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1ad806b810 0x7f1ad806dcd0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0036000 tx=0x7f1ae0008030 comp rx=0 tx=0).stop 2024-03-24T10:56:33.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.658+0000 7f1af1c84640 1 -- 172.21.15.5:0/2689722437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1aec0cf100 msgr2=0x7f1aec1f0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:33.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.658+0000 7f1af1c84640 1 --2- 172.21.15.5:0/2689722437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1aec0cf100 0x7f1aec1f0b00 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f1adc00c400 tx=0x7f1adc00c8c0 comp rx=0 tx=0).stop 2024-03-24T10:56:33.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.658+0000 7f1af1c84640 1 -- 172.21.15.5:0/2689722437 shutdown_connections 2024-03-24T10:56:33.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.658+0000 7f1af1c84640 1 --2- 172.21.15.5:0/2689722437 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1ad806b810 0x7f1ad806dcd0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:33.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.658+0000 7f1af1c84640 1 --2- 172.21.15.5:0/2689722437 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1aec15aea0 0x7f1aec153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:33.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.658+0000 7f1af1c84640 1 --2- 172.21.15.5:0/2689722437 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1aec15aac0 0x7f1aec0aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:33.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.658+0000 7f1af1c84640 1 --2- 172.21.15.5:0/2689722437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1aec0cf100 0x7f1aec1f0b00 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:33.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.658+0000 7f1af1c84640 1 -- 172.21.15.5:0/2689722437 >> 172.21.15.5:0/2689722437 conn(0x7f1aec0cfb90 msgr2=0x7f1aec0c6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:33.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.659+0000 7f1af1c84640 1 -- 172.21.15.5:0/2689722437 shutdown_connections 2024-03-24T10:56:33.660 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:33.659+0000 7f1af1c84640 1 -- 172.21.15.5:0/2689722437 wait complete. 2024-03-24T10:56:34.277 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:34.277 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:35.278 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:35.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.382+0000 7f200394f640 1 Processor -- start 2024-03-24T10:56:35.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.382+0000 7f200394f640 1 -- start start 2024-03-24T10:56:35.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.383+0000 7f200394f640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1ffc1543a0 0x7f1ffc15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:35.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.383+0000 7f200394f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1ffc06d4f0 0x7f1ffc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:35.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.383+0000 7f200394f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1ffc0ccaa0 0x7f1ffc06bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:35.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.384+0000 7f200394f640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f1ffc0c6750 con 0x7f1ffc1543a0 2024-03-24T10:56:35.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.384+0000 7f200394f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f1ffc0cceb0 con 0x7f1ffc06d4f0 2024-03-24T10:56:35.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.384+0000 7f200394f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1ffc0cc380 con 0x7f1ffc0ccaa0 2024-03-24T10:56:35.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.384+0000 7f2001eec640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1ffc0ccaa0 0x7f1ffc06bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:35.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.384+0000 7f2000eea640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1ffc06d4f0 0x7f1ffc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:35.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.385+0000 7f2001eec640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1ffc0ccaa0 0x7f1ffc06bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:44444/0 (socket says 172.21.15.5:44444) 2024-03-24T10:56:35.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.385+0000 7f2001eec640 1 -- 172.21.15.5:0/926169748 learned_addr learned my addr 172.21.15.5:0/926169748 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:35.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.387+0000 7f2001eec640 1 -- 172.21.15.5:0/926169748 >> v2:172.21.15.57:3300/0 conn(0x7f1ffc06d4f0 msgr2=0x7f1ffc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:35.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.387+0000 7f2001eec640 1 --2- 172.21.15.5:0/926169748 >> v2:172.21.15.57:3300/0 conn(0x7f1ffc06d4f0 0x7f1ffc0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:35.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.387+0000 7f2001eec640 1 -- 172.21.15.5:0/926169748 >> v2:172.21.15.5:3300/0 conn(0x7f1ffc1543a0 msgr2=0x7f1ffc15ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:35.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.387+0000 7f2001eec640 1 --2- 172.21.15.5:0/926169748 >> v2:172.21.15.5:3300/0 conn(0x7f1ffc1543a0 0x7f1ffc15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:35.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.387+0000 7f2001eec640 1 -- 172.21.15.5:0/926169748 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ffc0cb2f0 con 0x7f1ffc0ccaa0 2024-03-24T10:56:35.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.389+0000 7f2001eec640 1 --2- 172.21.15.5:0/926169748 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1ffc0ccaa0 0x7f1ffc06bf50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc0c6990 tx=0x7f1ff402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a33cd6095121bec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:35.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.395+0000 7f1ff3fff640 1 -- 172.21.15.5:0/926169748 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff402cb00 con 0x7f1ffc0ccaa0 2024-03-24T10:56:35.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.395+0000 7f1ff3fff640 1 -- 172.21.15.5:0/926169748 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ff402cc60 con 0x7f1ffc0ccaa0 2024-03-24T10:56:35.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.395+0000 7f1ff3fff640 1 -- 172.21.15.5:0/926169748 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff40343f0 con 0x7f1ffc0ccaa0 2024-03-24T10:56:35.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.395+0000 7f200394f640 1 -- 172.21.15.5:0/926169748 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1ffc0ccaa0 msgr2=0x7f1ffc06bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:35.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.395+0000 7f200394f640 1 --2- 172.21.15.5:0/926169748 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1ffc0ccaa0 0x7f1ffc06bf50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc0c6990 tx=0x7f1ff402bf60 comp rx=0 tx=0).stop 2024-03-24T10:56:35.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.396+0000 7f200394f640 1 -- 172.21.15.5:0/926169748 shutdown_connections 2024-03-24T10:56:35.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.396+0000 7f200394f640 1 --2- 172.21.15.5:0/926169748 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1ffc0ccaa0 0x7f1ffc06bf50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:35.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.396+0000 7f200394f640 1 --2- 172.21.15.5:0/926169748 >> v2:172.21.15.57:3300/0 conn(0x7f1ffc06d4f0 0x7f1ffc0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:35.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.396+0000 7f200394f640 1 --2- 172.21.15.5:0/926169748 >> v2:172.21.15.5:3300/0 conn(0x7f1ffc1543a0 0x7f1ffc15ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:35.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.396+0000 7f200394f640 1 -- 172.21.15.5:0/926169748 >> 172.21.15.5:0/926169748 conn(0x7f1ffc072ef0 msgr2=0x7f1ffc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:35.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.396+0000 7f200394f640 1 -- 172.21.15.5:0/926169748 shutdown_connections 2024-03-24T10:56:35.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.396+0000 7f200394f640 1 -- 172.21.15.5:0/926169748 wait complete. 2024-03-24T10:56:35.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.398+0000 7f200394f640 1 Processor -- start 2024-03-24T10:56:35.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.398+0000 7f200394f640 1 -- start start 2024-03-24T10:56:35.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.399+0000 7f200394f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ffc06d4f0 0x7f1ffc0cea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:35.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.399+0000 7f200394f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ffc0ccaa0 0x7f1ffc149a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:35.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.399+0000 7f200394f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1ffc1543a0 0x7f1ffc143b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:35.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.399+0000 7f200394f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc0cf1c0 con 0x7f1ffc06d4f0 2024-03-24T10:56:35.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.399+0000 7f200394f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1ffc155ff0 con 0x7f1ffc0ccaa0 2024-03-24T10:56:35.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.399+0000 7f200394f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1ffc0cef80 con 0x7f1ffc1543a0 2024-03-24T10:56:35.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.399+0000 7f2001eec640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1ffc1543a0 0x7f1ffc143b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:35.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.399+0000 7f2000eea640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ffc0ccaa0 0x7f1ffc149a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:35.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.400+0000 7f20016eb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ffc06d4f0 0x7f1ffc0cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:35.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.400+0000 7f2001eec640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1ffc1543a0 0x7f1ffc143b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:44452/0 (socket says 172.21.15.5:44452) 2024-03-24T10:56:35.400 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.400+0000 7f2001eec640 1 -- 172.21.15.5:0/2863931634 learned_addr learned my addr 172.21.15.5:0/2863931634 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:35.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.403+0000 7f2001eec640 1 -- 172.21.15.5:0/2863931634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ffc0ccaa0 msgr2=0x7f1ffc149a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:35.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.403+0000 7f2001eec640 1 --2- 172.21.15.5:0/2863931634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ffc0ccaa0 0x7f1ffc149a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:35.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.403+0000 7f2001eec640 1 -- 172.21.15.5:0/2863931634 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ffc06d4f0 msgr2=0x7f1ffc0cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:35.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.403+0000 7f2001eec640 1 --2- 172.21.15.5:0/2863931634 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ffc06d4f0 0x7f1ffc0cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:35.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.403+0000 7f2001eec640 1 -- 172.21.15.5:0/2863931634 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ff4009b30 con 0x7f1ffc1543a0 2024-03-24T10:56:35.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.406+0000 7f2001eec640 1 --2- 172.21.15.5:0/2863931634 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1ffc1543a0 0x7f1ffc143b00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1ff402c570 tx=0x7f1ff4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:35.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.409+0000 7f1ff1ffb640 1 -- 172.21.15.5:0/2863931634 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4009510 con 0x7f1ffc1543a0 2024-03-24T10:56:35.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.409+0000 7f200394f640 1 -- 172.21.15.5:0/2863931634 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ffc156590 con 0x7f1ffc1543a0 2024-03-24T10:56:35.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.409+0000 7f200394f640 1 -- 172.21.15.5:0/2863931634 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ffc0ce760 con 0x7f1ffc1543a0 2024-03-24T10:56:35.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.410+0000 7f1ff1ffb640 1 -- 172.21.15.5:0/2863931634 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ff403e450 con 0x7f1ffc1543a0 2024-03-24T10:56:35.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.410+0000 7f1ff1ffb640 1 -- 172.21.15.5:0/2863931634 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ff403e7c0 con 0x7f1ffc1543a0 2024-03-24T10:56:35.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.411+0000 7f200394f640 1 -- 172.21.15.5:0/2863931634 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ffc156720 con 0x7f1ffc1543a0 2024-03-24T10:56:35.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.430+0000 7f1ff1ffb640 1 -- 172.21.15.5:0/2863931634 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1ff403e920 con 0x7f1ffc1543a0 2024-03-24T10:56:35.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.431+0000 7f1ff1ffb640 1 --2- 172.21.15.5:0/2863931634 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1fe006b840 0x7f1fe006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:35.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.431+0000 7f20016eb640 1 --2- 172.21.15.5:0/2863931634 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1fe006b840 0x7f1fe006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:35.431 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.431+0000 7f20016eb640 1 --2- 172.21.15.5:0/2863931634 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1fe006b840 0x7f1fe006dd00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1ff80098e0 tx=0x7f1ff8009060 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:35.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.434+0000 7f1ff1ffb640 1 -- 172.21.15.5:0/2863931634 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f1ff407bde0 con 0x7f1ffc1543a0 2024-03-24T10:56:35.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.699+0000 7f1ff1ffb640 1 -- 172.21.15.5:0/2863931634 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1ff4078080 con 0x7f1ffc1543a0 2024-03-24T10:56:35.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:35.928+0000 7f200394f640 1 -- 172.21.15.5:0/2863931634 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ffc0681d0 con 0x7f1ffc1543a0 2024-03-24T10:56:36.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:36.030+0000 7f1ff1ffb640 1 -- 172.21.15.5:0/2863931634 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f1ff407b740 con 0x7f1ffc1543a0 2024-03-24T10:56:36.030 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:36.031 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:36.031 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:36.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:36.032+0000 7f200394f640 1 -- 172.21.15.5:0/2863931634 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1fe006b840 msgr2=0x7f1fe006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:36.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:36.032+0000 7f200394f640 1 --2- 172.21.15.5:0/2863931634 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1fe006b840 0x7f1fe006dd00 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1ff80098e0 tx=0x7f1ff8009060 comp rx=0 tx=0).stop 2024-03-24T10:56:36.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:36.032+0000 7f200394f640 1 -- 172.21.15.5:0/2863931634 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1ffc1543a0 msgr2=0x7f1ffc143b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:36.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:36.032+0000 7f200394f640 1 --2- 172.21.15.5:0/2863931634 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1ffc1543a0 0x7f1ffc143b00 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1ff402c570 tx=0x7f1ff4004a40 comp rx=0 tx=0).stop 2024-03-24T10:56:36.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:36.032+0000 7f200394f640 1 -- 172.21.15.5:0/2863931634 shutdown_connections 2024-03-24T10:56:36.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:36.032+0000 7f200394f640 1 --2- 172.21.15.5:0/2863931634 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1fe006b840 0x7f1fe006dd00 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:36.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:36.032+0000 7f200394f640 1 --2- 172.21.15.5:0/2863931634 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1ffc1543a0 0x7f1ffc143b00 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:36.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:36.032+0000 7f200394f640 1 --2- 172.21.15.5:0/2863931634 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1ffc0ccaa0 0x7f1ffc149a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:36.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:36.032+0000 7f200394f640 1 --2- 172.21.15.5:0/2863931634 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1ffc06d4f0 0x7f1ffc0cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:36.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:36.032+0000 7f200394f640 1 -- 172.21.15.5:0/2863931634 >> 172.21.15.5:0/2863931634 conn(0x7f1ffc072ef0 msgr2=0x7f1ffc0c4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:36.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:36.032+0000 7f200394f640 1 -- 172.21.15.5:0/2863931634 shutdown_connections 2024-03-24T10:56:36.033 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:36.032+0000 7f200394f640 1 -- 172.21.15.5:0/2863931634 wait complete. 2024-03-24T10:56:36.648 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:36.648 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:37.649 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:37.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.753+0000 7faa54cdf640 1 Processor -- start 2024-03-24T10:56:37.754 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.753+0000 7faa54cdf640 1 -- start start 2024-03-24T10:56:37.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.754+0000 7faa54cdf640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faa500cb270 0x7faa500cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:37.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.754+0000 7faa54cdf640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa500ccba0 con 0x7faa500cd960 2024-03-24T10:56:37.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.754+0000 7faa54cdf640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa500cce40 con 0x7faa500cae90 2024-03-24T10:56:37.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.754+0000 7faa54cdf640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7faa500cc9b0 con 0x7faa500cb270 2024-03-24T10:56:37.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.754+0000 7faa4dd9b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faa500cb270 0x7faa500cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:37.755 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.755+0000 7faa4e59c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7faa500cae90 0x7faa500cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:36022/0 (socket says 172.21.15.5:36022) 2024-03-24T10:56:37.756 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.755+0000 7faa4e59c640 1 -- 172.21.15.5:0/3832457173 learned_addr learned my addr 172.21.15.5:0/3832457173 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:56:37.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.758+0000 7faa4dd9b640 1 -- 172.21.15.5:0/3832457173 >> v1:172.21.15.57:6789/0 conn(0x7faa500cae90 legacy=0x7faa500cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:37.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.758+0000 7faa4dd9b640 1 -- 172.21.15.5:0/3832457173 >> v1:172.21.15.5:6789/0 conn(0x7faa500cd960 legacy=0x7faa500cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:37.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.758+0000 7faa4dd9b640 1 -- 172.21.15.5:0/3832457173 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa5015a590 con 0x7faa500cb270 2024-03-24T10:56:37.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.762+0000 7faa4dd9b640 1 --2- 172.21.15.5:0/3832457173 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa500cb270 0x7faa500cf530 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7faa440097a0 tx=0x7faa4402b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f115dce9e0616312 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:37.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.768+0000 7faa4d59a640 1 -- 172.21.15.5:0/3832457173 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4402f810 con 0x7faa500cb270 2024-03-24T10:56:37.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.768+0000 7faa4d59a640 1 -- 172.21.15.5:0/3832457173 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa4402f970 con 0x7faa500cb270 2024-03-24T10:56:37.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.768+0000 7faa4d59a640 1 -- 172.21.15.5:0/3832457173 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4402fd00 con 0x7faa500cb270 2024-03-24T10:56:37.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.769+0000 7faa54cdf640 1 -- 172.21.15.5:0/3832457173 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa500cb270 msgr2=0x7faa500cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:37.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.769+0000 7faa54cdf640 1 --2- 172.21.15.5:0/3832457173 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa500cb270 0x7faa500cf530 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7faa440097a0 tx=0x7faa4402b9f0 comp rx=0 tx=0).stop 2024-03-24T10:56:37.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.769+0000 7faa54cdf640 1 -- 172.21.15.5:0/3832457173 shutdown_connections 2024-03-24T10:56:37.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.769+0000 7faa54cdf640 1 --2- 172.21.15.5:0/3832457173 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa500cb270 0x7faa500cf530 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:37.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.769+0000 7faa54cdf640 1 -- 172.21.15.5:0/3832457173 >> 172.21.15.5:0/3832457173 conn(0x7faa500cfb80 msgr2=0x7faa500cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:37.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.769+0000 7faa54cdf640 1 -- 172.21.15.5:0/3832457173 shutdown_connections 2024-03-24T10:56:37.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.769+0000 7faa54cdf640 1 -- 172.21.15.5:0/3832457173 wait complete. 2024-03-24T10:56:37.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.771+0000 7faa54cdf640 1 Processor -- start 2024-03-24T10:56:37.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.771+0000 7faa54cdf640 1 -- start start 2024-03-24T10:56:37.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.772+0000 7faa54cdf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa500cae90 0x7faa500b5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:37.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.772+0000 7faa54cdf640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa500cb270 0x7faa500b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:37.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.772+0000 7faa54cdf640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faa500cd960 0x7faa500b4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:37.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.772+0000 7faa54cdf640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7faa500b3200 con 0x7faa500cb270 2024-03-24T10:56:37.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.772+0000 7faa54cdf640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7faa500b5940 con 0x7faa500cae90 2024-03-24T10:56:37.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.772+0000 7faa54cdf640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7faa500c65c0 con 0x7faa500cd960 2024-03-24T10:56:37.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.772+0000 7faa4dd9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa500cb270 0x7faa500b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:37.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.772+0000 7faa4e59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa500cae90 0x7faa500b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:37.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.772+0000 7faa4ed9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faa500cd960 0x7faa500b4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:37.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.772+0000 7faa4dd9b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa500cb270 0x7faa500b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39212/0 (socket says 172.21.15.5:39212) 2024-03-24T10:56:37.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.773+0000 7faa4dd9b640 1 -- 172.21.15.5:0/920396326 learned_addr learned my addr 172.21.15.5:0/920396326 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:37.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.776+0000 7faa4dd9b640 1 -- 172.21.15.5:0/920396326 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faa500cd960 msgr2=0x7faa500b4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:37.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.776+0000 7faa4dd9b640 1 --2- 172.21.15.5:0/920396326 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faa500cd960 0x7faa500b4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:37.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.776+0000 7faa4dd9b640 1 -- 172.21.15.5:0/920396326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa500cae90 msgr2=0x7faa500b5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:37.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.776+0000 7faa4dd9b640 1 --2- 172.21.15.5:0/920396326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa500cae90 0x7faa500b5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:37.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.776+0000 7faa4dd9b640 1 -- 172.21.15.5:0/920396326 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa44009400 con 0x7faa500cb270 2024-03-24T10:56:37.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.778+0000 7faa4dd9b640 1 --2- 172.21.15.5:0/920396326 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa500cb270 0x7faa500b3f30 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7faa4402c000 tx=0x7faa44004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:37.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.783+0000 7faa3effd640 1 -- 172.21.15.5:0/920396326 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4403c040 con 0x7faa500cb270 2024-03-24T10:56:37.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.783+0000 7faa3effd640 1 -- 172.21.15.5:0/920396326 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa440379f0 con 0x7faa500cb270 2024-03-24T10:56:37.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.783+0000 7faa54cdf640 1 -- 172.21.15.5:0/920396326 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa501f0420 con 0x7faa500cb270 2024-03-24T10:56:37.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.783+0000 7faa54cdf640 1 -- 172.21.15.5:0/920396326 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa500b2bb0 con 0x7faa500cb270 2024-03-24T10:56:37.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.783+0000 7faa3effd640 1 -- 172.21.15.5:0/920396326 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa44037da0 con 0x7faa500cb270 2024-03-24T10:56:37.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.784+0000 7faa54cdf640 1 -- 172.21.15.5:0/920396326 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa500681d0 con 0x7faa500cb270 2024-03-24T10:56:37.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.808+0000 7faa3effd640 1 -- 172.21.15.5:0/920396326 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7faa4404b3b0 con 0x7faa500cb270 2024-03-24T10:56:37.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.809+0000 7faa3effd640 1 --2- 172.21.15.5:0/920396326 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faa3006b860 0x7faa3006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:37.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.809+0000 7faa4e59c640 1 --2- 172.21.15.5:0/920396326 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faa3006b860 0x7faa3006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:37.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.809+0000 7faa4e59c640 1 --2- 172.21.15.5:0/920396326 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faa3006b860 0x7faa3006dd20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7faa40005880 tx=0x7faa40009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:37.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:37.811+0000 7faa3effd640 1 -- 172.21.15.5:0/920396326 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7faa4407b020 con 0x7faa500cb270 2024-03-24T10:56:38.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:38.112+0000 7faa3effd640 1 -- 172.21.15.5:0/920396326 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7faa440772c0 con 0x7faa500cb270 2024-03-24T10:56:38.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:38.336+0000 7faa54cdf640 1 -- 172.21.15.5:0/920396326 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa500d0000 con 0x7faa500cb270 2024-03-24T10:56:38.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:38.363+0000 7faa3effd640 1 -- 172.21.15.5:0/920396326 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7faa44047090 con 0x7faa500cb270 2024-03-24T10:56:38.364 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:38.364 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:38.364 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:38.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:38.365+0000 7faa54cdf640 1 -- 172.21.15.5:0/920396326 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faa3006b860 msgr2=0x7faa3006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:38.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:38.365+0000 7faa54cdf640 1 --2- 172.21.15.5:0/920396326 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faa3006b860 0x7faa3006dd20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7faa40005880 tx=0x7faa40009040 comp rx=0 tx=0).stop 2024-03-24T10:56:38.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:38.365+0000 7faa54cdf640 1 -- 172.21.15.5:0/920396326 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa500cb270 msgr2=0x7faa500b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:38.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:38.365+0000 7faa54cdf640 1 --2- 172.21.15.5:0/920396326 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa500cb270 0x7faa500b3f30 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7faa4402c000 tx=0x7faa44004730 comp rx=0 tx=0).stop 2024-03-24T10:56:38.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:38.365+0000 7faa54cdf640 1 -- 172.21.15.5:0/920396326 shutdown_connections 2024-03-24T10:56:38.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:38.365+0000 7faa54cdf640 1 --2- 172.21.15.5:0/920396326 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faa3006b860 0x7faa3006dd20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:38.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:38.365+0000 7faa54cdf640 1 --2- 172.21.15.5:0/920396326 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faa500cb270 0x7faa500b3f30 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:38.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:38.365+0000 7faa54cdf640 1 --2- 172.21.15.5:0/920396326 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faa500cd960 0x7faa500b4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:38.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:38.365+0000 7faa54cdf640 1 --2- 172.21.15.5:0/920396326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faa500cae90 0x7faa500b5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:38.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:38.365+0000 7faa54cdf640 1 -- 172.21.15.5:0/920396326 >> 172.21.15.5:0/920396326 conn(0x7faa500cfb80 msgr2=0x7faa500cddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:38.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:38.365+0000 7faa54cdf640 1 -- 172.21.15.5:0/920396326 shutdown_connections 2024-03-24T10:56:38.367 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:38.365+0000 7faa54cdf640 1 -- 172.21.15.5:0/920396326 wait complete. 2024-03-24T10:56:38.982 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:38.982 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:39.984 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:40.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.086+0000 7faafe513640 1 Processor -- start 2024-03-24T10:56:40.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.086+0000 7faafe513640 1 -- start start 2024-03-24T10:56:40.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.087+0000 7faafe513640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faaf81543a0 0x7faaf815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:40.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.087+0000 7faafe513640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faaf806d4f0 0x7faaf80cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:40.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.087+0000 7faafe513640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7faaf80ccaa0 0x7faaf806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:40.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.087+0000 7faafe513640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7faaf80c6750 con 0x7faaf806d4f0 2024-03-24T10:56:40.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.087+0000 7faafe513640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7faaf80cc380 con 0x7faaf80ccaa0 2024-03-24T10:56:40.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.087+0000 7faafe513640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7faaf80cb2f0 con 0x7faaf81543a0 2024-03-24T10:56:40.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.087+0000 7faaf77fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faaf806d4f0 0x7faaf80cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:40.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.088+0000 7faafcab0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7faaf80ccaa0 0x7faaf806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:40.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.088+0000 7faaf7fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faaf81543a0 0x7faaf815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:40.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.088+0000 7faaf77fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7faaf806d4f0 0x7faaf80cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39226/0 (socket says 172.21.15.5:39226) 2024-03-24T10:56:40.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.088+0000 7faaf77fe640 1 -- 172.21.15.5:0/2389605612 learned_addr learned my addr 172.21.15.5:0/2389605612 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:40.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.090+0000 7faaf77fe640 1 -- 172.21.15.5:0/2389605612 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faaf81543a0 msgr2=0x7faaf815ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:40.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.090+0000 7faaf77fe640 1 --2- 172.21.15.5:0/2389605612 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faaf81543a0 0x7faaf815ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:40.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.090+0000 7faaf77fe640 1 -- 172.21.15.5:0/2389605612 >> v2:172.21.15.57:3300/0 conn(0x7faaf80ccaa0 msgr2=0x7faaf806bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:40.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.090+0000 7faaf77fe640 1 --2- 172.21.15.5:0/2389605612 >> v2:172.21.15.57:3300/0 conn(0x7faaf80ccaa0 0x7faaf806bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:40.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.090+0000 7faaf77fe640 1 -- 172.21.15.5:0/2389605612 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf80ca7e0 con 0x7faaf806d4f0 2024-03-24T10:56:40.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.092+0000 7faaf77fe640 1 --2- 172.21.15.5:0/2389605612 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaf806d4f0 0x7faaf80cc560 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7faaf806de60 tx=0x7faaec02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b564c57e80339952 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:40.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.095+0000 7faaf6ffd640 1 -- 172.21.15.5:0/2389605612 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaec030860 con 0x7faaf806d4f0 2024-03-24T10:56:40.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.096+0000 7faaf6ffd640 1 -- 172.21.15.5:0/2389605612 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaec0309c0 con 0x7faaf806d4f0 2024-03-24T10:56:40.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.096+0000 7faaf6ffd640 1 -- 172.21.15.5:0/2389605612 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaec030d50 con 0x7faaf806d4f0 2024-03-24T10:56:40.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.096+0000 7faafe513640 1 -- 172.21.15.5:0/2389605612 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaf806d4f0 msgr2=0x7faaf80cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:40.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.096+0000 7faafe513640 1 --2- 172.21.15.5:0/2389605612 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaf806d4f0 0x7faaf80cc560 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7faaf806de60 tx=0x7faaec02b9a0 comp rx=0 tx=0).stop 2024-03-24T10:56:40.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.097+0000 7faafe513640 1 -- 172.21.15.5:0/2389605612 shutdown_connections 2024-03-24T10:56:40.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.097+0000 7faafe513640 1 --2- 172.21.15.5:0/2389605612 >> v2:172.21.15.57:3300/0 conn(0x7faaf80ccaa0 0x7faaf806bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:40.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.097+0000 7faafe513640 1 --2- 172.21.15.5:0/2389605612 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaf806d4f0 0x7faaf80cc560 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:40.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.097+0000 7faafe513640 1 --2- 172.21.15.5:0/2389605612 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faaf81543a0 0x7faaf815ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:40.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.097+0000 7faafe513640 1 -- 172.21.15.5:0/2389605612 >> 172.21.15.5:0/2389605612 conn(0x7faaf8072ef0 msgr2=0x7faaf8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:40.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.097+0000 7faafe513640 1 -- 172.21.15.5:0/2389605612 shutdown_connections 2024-03-24T10:56:40.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.097+0000 7faafe513640 1 -- 172.21.15.5:0/2389605612 wait complete. 2024-03-24T10:56:40.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.098+0000 7faafe513640 1 Processor -- start 2024-03-24T10:56:40.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.099+0000 7faafe513640 1 -- start start 2024-03-24T10:56:40.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.099+0000 7faafe513640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faaf806d4f0 0x7faaf81576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:40.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.099+0000 7faafe513640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faaf80ccaa0 0x7faaf8156910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:40.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.099+0000 7faafe513640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaf81543a0 0x7faaf81f0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:40.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.099+0000 7faafe513640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7faaf8154a60 con 0x7faaf81543a0 2024-03-24T10:56:40.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.099+0000 7faafe513640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7faaf80c7720 con 0x7faaf806d4f0 2024-03-24T10:56:40.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.099+0000 7faafe513640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7faaf80c7a00 con 0x7faaf80ccaa0 2024-03-24T10:56:40.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.100+0000 7faaf7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faaf806d4f0 0x7faaf81576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:40.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.100+0000 7faafcab0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaf81543a0 0x7faaf81f0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:40.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.100+0000 7faaf77fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faaf80ccaa0 0x7faaf8156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:40.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.100+0000 7faaf7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faaf806d4f0 0x7faaf81576e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36420/0 (socket says 172.21.15.5:36420) 2024-03-24T10:56:40.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.100+0000 7faaf7fff640 1 -- 172.21.15.5:0/1757213586 learned_addr learned my addr 172.21.15.5:0/1757213586 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:40.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.104+0000 7faaf7fff640 1 -- 172.21.15.5:0/1757213586 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faaf80ccaa0 msgr2=0x7faaf8156910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:40.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.104+0000 7faaf7fff640 1 --2- 172.21.15.5:0/1757213586 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faaf80ccaa0 0x7faaf8156910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:40.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.104+0000 7faaf7fff640 1 -- 172.21.15.5:0/1757213586 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaf81543a0 msgr2=0x7faaf81f0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:40.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.104+0000 7faaf7fff640 1 --2- 172.21.15.5:0/1757213586 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaf81543a0 0x7faaf81f0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:40.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.104+0000 7faaf7fff640 1 -- 172.21.15.5:0/1757213586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaec0094d0 con 0x7faaf806d4f0 2024-03-24T10:56:40.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.107+0000 7faaf7fff640 1 --2- 172.21.15.5:0/1757213586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faaf806d4f0 0x7faaf81576e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7faae800e870 tx=0x7faae800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:40.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.111+0000 7faaf4ff9640 1 -- 172.21.15.5:0/1757213586 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae80094f0 con 0x7faaf806d4f0 2024-03-24T10:56:40.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.111+0000 7faaf4ff9640 1 -- 172.21.15.5:0/1757213586 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faae800c940 con 0x7faaf806d4f0 2024-03-24T10:56:40.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.111+0000 7faafe513640 1 -- 172.21.15.5:0/1757213586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaf8154cb0 con 0x7faaf806d4f0 2024-03-24T10:56:40.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.111+0000 7faaf4ff9640 1 -- 172.21.15.5:0/1757213586 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae800ccf0 con 0x7faaf806d4f0 2024-03-24T10:56:40.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.111+0000 7faafe513640 1 -- 172.21.15.5:0/1757213586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaf80c7410 con 0x7faaf806d4f0 2024-03-24T10:56:40.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.112+0000 7faafe513640 1 -- 172.21.15.5:0/1757213586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaf80c7bc0 con 0x7faaf806d4f0 2024-03-24T10:56:40.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.133+0000 7faaf4ff9640 1 -- 172.21.15.5:0/1757213586 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7faae8013470 con 0x7faaf806d4f0 2024-03-24T10:56:40.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.133+0000 7faaf4ff9640 1 --2- 172.21.15.5:0/1757213586 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faae406b860 0x7faae406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:40.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.133+0000 7faaf77fe640 1 --2- 172.21.15.5:0/1757213586 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faae406b860 0x7faae406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:40.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.133+0000 7faaf77fe640 1 --2- 172.21.15.5:0/1757213586 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faae406b860 0x7faae406dd20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faaec036000 tx=0x7faaec008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:40.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.386+0000 7faaf4ff9640 1 -- 172.21.15.5:0/1757213586 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7faae8058fc0 con 0x7faaf806d4f0 2024-03-24T10:56:40.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.397+0000 7faaf4ff9640 1 -- 172.21.15.5:0/1757213586 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7faae8055260 con 0x7faaf806d4f0 2024-03-24T10:56:40.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.624+0000 7faafe513640 1 -- 172.21.15.5:0/1757213586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faaf80c7bc0 con 0x7faaf806d4f0 2024-03-24T10:56:40.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.663+0000 7faaf4ff9640 1 -- 172.21.15.5:0/1757213586 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7faaf80c7bc0 con 0x7faaf806d4f0 2024-03-24T10:56:40.664 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:40.665 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:40.665 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:40.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.665+0000 7faafe513640 1 -- 172.21.15.5:0/1757213586 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faae406b860 msgr2=0x7faae406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:40.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.665+0000 7faafe513640 1 --2- 172.21.15.5:0/1757213586 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faae406b860 0x7faae406dd20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faaec036000 tx=0x7faaec008030 comp rx=0 tx=0).stop 2024-03-24T10:56:40.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.665+0000 7faafe513640 1 -- 172.21.15.5:0/1757213586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faaf806d4f0 msgr2=0x7faaf81576e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:40.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.665+0000 7faafe513640 1 --2- 172.21.15.5:0/1757213586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faaf806d4f0 0x7faaf81576e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7faae800e870 tx=0x7faae800ed30 comp rx=0 tx=0).stop 2024-03-24T10:56:40.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.665+0000 7faafe513640 1 -- 172.21.15.5:0/1757213586 shutdown_connections 2024-03-24T10:56:40.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.665+0000 7faafe513640 1 --2- 172.21.15.5:0/1757213586 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7faae406b860 0x7faae406dd20 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:40.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.665+0000 7faafe513640 1 --2- 172.21.15.5:0/1757213586 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7faaf81543a0 0x7faaf81f0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:40.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.665+0000 7faafe513640 1 --2- 172.21.15.5:0/1757213586 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7faaf80ccaa0 0x7faaf8156910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:40.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.665+0000 7faafe513640 1 --2- 172.21.15.5:0/1757213586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7faaf806d4f0 0x7faaf81576e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:40.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.665+0000 7faafe513640 1 -- 172.21.15.5:0/1757213586 >> 172.21.15.5:0/1757213586 conn(0x7faaf8072ef0 msgr2=0x7faaf806ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:40.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.665+0000 7faafe513640 1 -- 172.21.15.5:0/1757213586 shutdown_connections 2024-03-24T10:56:40.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:40.665+0000 7faafe513640 1 -- 172.21.15.5:0/1757213586 wait complete. 2024-03-24T10:56:41.282 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:41.283 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:42.286 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:42.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.387+0000 7f0e07cd4640 1 Processor -- start 2024-03-24T10:56:42.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.387+0000 7f0e07cd4640 1 -- start start 2024-03-24T10:56:42.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.388+0000 7f0e07cd4640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0e0006bf50 0x7f0e0006c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:42.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.388+0000 7f0e07cd4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e000cb2f0 con 0x7f0e0015ac00 2024-03-24T10:56:42.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.388+0000 7f0e07cd4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e0015b070 con 0x7f0e001543a0 2024-03-24T10:56:42.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.388+0000 7f0e07cd4640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f0e000c6750 con 0x7f0e0006bf50 2024-03-24T10:56:42.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.389+0000 7f0e05a70640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f0e001543a0 0x7f0e0006d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:36032/0 (socket says 172.21.15.5:36032) 2024-03-24T10:56:42.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.389+0000 7f0e05a70640 1 -- 172.21.15.5:0/712038521 learned_addr learned my addr 172.21.15.5:0/712038521 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:56:42.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.389+0000 7f0e06271640 1 --2- 172.21.15.5:0/712038521 >> v2:172.21.15.5:3300/0 conn(0x7f0e0006bf50 0x7f0e0006c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:42.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.392+0000 7f0e04a6e640 1 -- 172.21.15.5:0/712038521 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2683414485 0 0) 0x7f0e0015b070 con 0x7f0e001543a0 2024-03-24T10:56:42.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.392+0000 7f0e04a6e640 1 -- 172.21.15.5:0/712038521 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dec003520 con 0x7f0e001543a0 2024-03-24T10:56:42.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.395+0000 7f0e06271640 1 -- 172.21.15.5:0/712038521 >> v1:172.21.15.57:6789/0 conn(0x7f0e001543a0 legacy=0x7f0e0006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:42.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.395+0000 7f0e06271640 1 -- 172.21.15.5:0/712038521 >> v1:172.21.15.5:6789/0 conn(0x7f0e0015ac00 legacy=0x7f0e000cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:42.396 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.395+0000 7f0e06271640 1 -- 172.21.15.5:0/712038521 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e000ca7e0 con 0x7f0e0006bf50 2024-03-24T10:56:42.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.398+0000 7f0e06271640 1 --2- 172.21.15.5:0/712038521 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e0006bf50 0x7f0e0006c330 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009d90 tx=0x7f0df802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad8b7f0c8c58de51 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:42.401 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.400+0000 7f0e04a6e640 1 -- 172.21.15.5:0/712038521 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df802cd00 con 0x7f0e0006bf50 2024-03-24T10:56:42.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.402+0000 7f0e04a6e640 1 -- 172.21.15.5:0/712038521 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0df802ce60 con 0x7f0e0006bf50 2024-03-24T10:56:42.403 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.402+0000 7f0e04a6e640 1 -- 172.21.15.5:0/712038521 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df8034640 con 0x7f0e0006bf50 2024-03-24T10:56:42.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.403+0000 7f0e07cd4640 1 -- 172.21.15.5:0/712038521 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e0006bf50 msgr2=0x7f0e0006c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:42.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.403+0000 7f0e07cd4640 1 --2- 172.21.15.5:0/712038521 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e0006bf50 0x7f0e0006c330 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009d90 tx=0x7f0df802bf50 comp rx=0 tx=0).stop 2024-03-24T10:56:42.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.403+0000 7f0e07cd4640 1 -- 172.21.15.5:0/712038521 shutdown_connections 2024-03-24T10:56:42.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.403+0000 7f0e07cd4640 1 --2- 172.21.15.5:0/712038521 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e0006bf50 0x7f0e0006c330 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:42.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.403+0000 7f0e07cd4640 1 -- 172.21.15.5:0/712038521 >> 172.21.15.5:0/712038521 conn(0x7f0e00072ef0 msgr2=0x7f0e00061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:42.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.403+0000 7f0e07cd4640 1 -- 172.21.15.5:0/712038521 shutdown_connections 2024-03-24T10:56:42.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.403+0000 7f0e07cd4640 1 -- 172.21.15.5:0/712038521 wait complete. 2024-03-24T10:56:42.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.405+0000 7f0e07cd4640 1 Processor -- start 2024-03-24T10:56:42.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.405+0000 7f0e07cd4640 1 -- start start 2024-03-24T10:56:42.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.405+0000 7f0e07cd4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e0006bf50 0x7f0e000b4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:42.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.405+0000 7f0e07cd4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e001543a0 0x7f0e001f2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:42.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.405+0000 7f0e07cd4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0e0015ac00 0x7f0e00144510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:42.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.405+0000 7f0e07cd4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0014c650 con 0x7f0e0006bf50 2024-03-24T10:56:42.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.405+0000 7f0e07cd4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0e000cf7b0 con 0x7f0e001543a0 2024-03-24T10:56:42.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.405+0000 7f0e07cd4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0e000cecd0 con 0x7f0e0015ac00 2024-03-24T10:56:42.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.406+0000 7f0e05a70640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e0006bf50 0x7f0e000b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:42.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.406+0000 7f0e0526f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e001543a0 0x7f0e001f2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:42.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.406+0000 7f0e05a70640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e0006bf50 0x7f0e000b4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39242/0 (socket says 172.21.15.5:39242) 2024-03-24T10:56:42.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.406+0000 7f0e05a70640 1 -- 172.21.15.5:0/2786259797 learned_addr learned my addr 172.21.15.5:0/2786259797 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:42.408 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.406+0000 7f0e06271640 1 --2- 172.21.15.5:0/2786259797 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0e0015ac00 0x7f0e00144510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:42.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.410+0000 7f0e05a70640 1 -- 172.21.15.5:0/2786259797 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0e0015ac00 msgr2=0x7f0e00144510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:42.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.410+0000 7f0e05a70640 1 --2- 172.21.15.5:0/2786259797 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0e0015ac00 0x7f0e00144510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:42.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.410+0000 7f0e05a70640 1 -- 172.21.15.5:0/2786259797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e001543a0 msgr2=0x7f0e001f2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:42.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.410+0000 7f0e05a70640 1 --2- 172.21.15.5:0/2786259797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e001543a0 0x7f0e001f2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:42.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.410+0000 7f0e05a70640 1 -- 172.21.15.5:0/2786259797 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0df8009a70 con 0x7f0e0006bf50 2024-03-24T10:56:42.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.412+0000 7f0e05a70640 1 --2- 172.21.15.5:0/2786259797 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e0006bf50 0x7f0e000b4980 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00c500 tx=0x7f0dfc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:42.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.416+0000 7f0df67fc640 1 -- 172.21.15.5:0/2786259797 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc005ce0 con 0x7f0e0006bf50 2024-03-24T10:56:42.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.416+0000 7f0df67fc640 1 -- 172.21.15.5:0/2786259797 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dfc005e40 con 0x7f0e0006bf50 2024-03-24T10:56:42.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.416+0000 7f0e07cd4640 1 -- 172.21.15.5:0/2786259797 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e00158df0 con 0x7f0e0006bf50 2024-03-24T10:56:42.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.416+0000 7f0df67fc640 1 -- 172.21.15.5:0/2786259797 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc007330 con 0x7f0e0006bf50 2024-03-24T10:56:42.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.416+0000 7f0e07cd4640 1 -- 172.21.15.5:0/2786259797 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e000cf270 con 0x7f0e0006bf50 2024-03-24T10:56:42.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.418+0000 7f0e07cd4640 1 -- 172.21.15.5:0/2786259797 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e000681d0 con 0x7f0e0006bf50 2024-03-24T10:56:42.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.686+0000 7f0df67fc640 1 -- 172.21.15.5:0/2786259797 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0dfc007490 con 0x7f0e0006bf50 2024-03-24T10:56:42.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.687+0000 7f0df67fc640 1 --2- 172.21.15.5:0/2786259797 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0de406b890 0x7f0de406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:42.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.687+0000 7f0e0526f640 1 --2- 172.21.15.5:0/2786259797 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0de406b890 0x7f0de406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:42.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.687+0000 7f0e0526f640 1 --2- 172.21.15.5:0/2786259797 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0de406b890 0x7f0de406dd50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0e0014be50 tx=0x7f0df0009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:42.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.703+0000 7f0df67fc640 1 -- 172.21.15.5:0/2786259797 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f0dfc007740 con 0x7f0e0006bf50 2024-03-24T10:56:42.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.703+0000 7f0df67fc640 1 -- 172.21.15.5:0/2786259797 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0dfc054790 con 0x7f0e0006bf50 2024-03-24T10:56:42.926 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.925+0000 7f0e07cd4640 1 -- 172.21.15.5:0/2786259797 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e00151310 con 0x7f0e0006bf50 2024-03-24T10:56:42.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.939+0000 7f0df67fc640 1 -- 172.21.15.5:0/2786259797 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f0dfc017090 con 0x7f0e0006bf50 2024-03-24T10:56:42.941 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:42.941 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:42.941 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:42.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.941+0000 7f0e07cd4640 1 -- 172.21.15.5:0/2786259797 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0de406b890 msgr2=0x7f0de406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:42.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.941+0000 7f0e07cd4640 1 --2- 172.21.15.5:0/2786259797 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0de406b890 0x7f0de406dd50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0e0014be50 tx=0x7f0df0009040 comp rx=0 tx=0).stop 2024-03-24T10:56:42.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.941+0000 7f0e07cd4640 1 -- 172.21.15.5:0/2786259797 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e0006bf50 msgr2=0x7f0e000b4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:42.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.941+0000 7f0e07cd4640 1 --2- 172.21.15.5:0/2786259797 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e0006bf50 0x7f0e000b4980 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00c500 tx=0x7f0dfc00c9c0 comp rx=0 tx=0).stop 2024-03-24T10:56:42.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.941+0000 7f0e07cd4640 1 -- 172.21.15.5:0/2786259797 shutdown_connections 2024-03-24T10:56:42.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.941+0000 7f0e07cd4640 1 --2- 172.21.15.5:0/2786259797 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0de406b890 0x7f0de406dd50 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:42.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.941+0000 7f0e07cd4640 1 --2- 172.21.15.5:0/2786259797 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0e0015ac00 0x7f0e00144510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:42.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.941+0000 7f0e07cd4640 1 --2- 172.21.15.5:0/2786259797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0e001543a0 0x7f0e001f2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:42.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.941+0000 7f0e07cd4640 1 --2- 172.21.15.5:0/2786259797 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0e0006bf50 0x7f0e000b4980 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:42.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.941+0000 7f0e07cd4640 1 -- 172.21.15.5:0/2786259797 >> 172.21.15.5:0/2786259797 conn(0x7f0e00072ef0 msgr2=0x7f0e001505a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:42.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.942+0000 7f0e07cd4640 1 -- 172.21.15.5:0/2786259797 shutdown_connections 2024-03-24T10:56:42.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:42.942+0000 7f0e07cd4640 1 -- 172.21.15.5:0/2786259797 wait complete. 2024-03-24T10:56:43.559 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:43.559 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:44.560 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:44.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.663+0000 7f66a3543640 1 Processor -- start 2024-03-24T10:56:44.663 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.663+0000 7f66a3543640 1 -- start start 2024-03-24T10:56:44.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.664+0000 7f66a3543640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f669c06fc60 0x7f669c0c4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:44.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.664+0000 7f66a3543640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f669c15b530 con 0x7f669c152140 2024-03-24T10:56:44.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.664+0000 7f66a3543640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f669c15b810 con 0x7f669c157f80 2024-03-24T10:56:44.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.664+0000 7f66a3543640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f669c159430 con 0x7f669c06fc60 2024-03-24T10:56:44.664 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.664+0000 7f66a1ae0640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f669c157f80 0x7f669c1628c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:43680/0 (socket says 172.21.15.5:43680) 2024-03-24T10:56:44.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.664+0000 7f66a1ae0640 1 -- 172.21.15.5:0/1464318062 learned_addr learned my addr 172.21.15.5:0/1464318062 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:56:44.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.665+0000 7f66a0ade640 1 --2- 172.21.15.5:0/1464318062 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f669c06fc60 0x7f669c0c4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:44.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.667+0000 7f66a0ade640 1 -- 172.21.15.5:0/1464318062 >> v1:172.21.15.57:6789/0 conn(0x7f669c157f80 legacy=0x7f669c1628c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:44.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.667+0000 7f66a0ade640 1 -- 172.21.15.5:0/1464318062 >> v1:172.21.15.5:6789/0 conn(0x7f669c152140 legacy=0x7f669c15bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:44.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.667+0000 7f6693fff640 1 -- 172.21.15.5:0/1464318062 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2634225973 0 0) 0x7f669c15b530 con 0x7f669c152140 2024-03-24T10:56:44.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.667+0000 7f66a0ade640 1 -- 172.21.15.5:0/1464318062 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f669c1584c0 con 0x7f669c06fc60 2024-03-24T10:56:44.669 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.670+0000 7f66a0ade640 1 --2- 172.21.15.5:0/1464318062 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f669c06fc60 0x7f669c0c4e40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f668c0097d0 tx=0x7f668c02b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec9da38610429b94 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:44.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.674+0000 7f6693fff640 1 -- 172.21.15.5:0/1464318062 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c030890 con 0x7f669c06fc60 2024-03-24T10:56:44.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.674+0000 7f6693fff640 1 -- 172.21.15.5:0/1464318062 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f668c0309f0 con 0x7f669c06fc60 2024-03-24T10:56:44.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.674+0000 7f6693fff640 1 -- 172.21.15.5:0/1464318062 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c030d80 con 0x7f669c06fc60 2024-03-24T10:56:44.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.675+0000 7f66a3543640 1 -- 172.21.15.5:0/1464318062 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f669c06fc60 msgr2=0x7f669c0c4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:44.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.675+0000 7f66a3543640 1 --2- 172.21.15.5:0/1464318062 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f669c06fc60 0x7f669c0c4e40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f668c0097d0 tx=0x7f668c02b9d0 comp rx=0 tx=0).stop 2024-03-24T10:56:44.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.675+0000 7f66a3543640 1 -- 172.21.15.5:0/1464318062 shutdown_connections 2024-03-24T10:56:44.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.675+0000 7f66a3543640 1 --2- 172.21.15.5:0/1464318062 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f669c06fc60 0x7f669c0c4e40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:44.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.675+0000 7f66a3543640 1 -- 172.21.15.5:0/1464318062 >> 172.21.15.5:0/1464318062 conn(0x7f669c06b130 msgr2=0x7f669c06b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:44.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.675+0000 7f66a3543640 1 -- 172.21.15.5:0/1464318062 shutdown_connections 2024-03-24T10:56:44.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.675+0000 7f66a3543640 1 -- 172.21.15.5:0/1464318062 wait complete. 2024-03-24T10:56:44.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.676+0000 7f66a3543640 1 Processor -- start 2024-03-24T10:56:44.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.676+0000 7f66a3543640 1 -- start start 2024-03-24T10:56:44.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.677+0000 7f66a3543640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f669c06fc60 0x7f669c14a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:44.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.677+0000 7f66a3543640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f669c152140 0x7f669c1ef9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:44.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.677+0000 7f66a3543640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f669c157f80 0x7f669c1eff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:44.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.677+0000 7f66a3543640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f669c0c34c0 con 0x7f669c152140 2024-03-24T10:56:44.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.677+0000 7f66a3543640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f669c1f3e80 con 0x7f669c157f80 2024-03-24T10:56:44.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.677+0000 7f66a3543640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f669c14a800 con 0x7f669c06fc60 2024-03-24T10:56:44.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.678+0000 7f66a0ade640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f669c152140 0x7f669c1ef9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:44.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.678+0000 7f66a0ade640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f669c152140 0x7f669c1ef9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:47482/0 (socket says 172.21.15.5:47482) 2024-03-24T10:56:44.678 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.678+0000 7f66a0ade640 1 -- 172.21.15.5:0/3220977894 learned_addr learned my addr 172.21.15.5:0/3220977894 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:44.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.679+0000 7f66a1ae0640 1 --2- 172.21.15.5:0/3220977894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f669c157f80 0x7f669c1eff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:44.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.679+0000 7f66a12df640 1 --2- 172.21.15.5:0/3220977894 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f669c06fc60 0x7f669c14a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:44.682 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.683+0000 7f66a0ade640 1 -- 172.21.15.5:0/3220977894 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f669c06fc60 msgr2=0x7f669c14a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:44.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.683+0000 7f66a0ade640 1 --2- 172.21.15.5:0/3220977894 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f669c06fc60 0x7f669c14a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:44.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.683+0000 7f66a0ade640 1 -- 172.21.15.5:0/3220977894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f669c157f80 msgr2=0x7f669c1eff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:44.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.683+0000 7f66a0ade640 1 --2- 172.21.15.5:0/3220977894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f669c157f80 0x7f669c1eff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:44.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.683+0000 7f66a0ade640 1 -- 172.21.15.5:0/3220977894 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f668c009400 con 0x7f669c152140 2024-03-24T10:56:44.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.688+0000 7f66a0ade640 1 --2- 172.21.15.5:0/3220977894 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f669c152140 0x7f669c1ef9e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f668c0098c0 tx=0x7f668c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:44.690 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.690+0000 7f6691ffb640 1 -- 172.21.15.5:0/3220977894 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c03e040 con 0x7f669c152140 2024-03-24T10:56:44.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.691+0000 7f66a3543640 1 -- 172.21.15.5:0/3220977894 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f669c0c31e0 con 0x7f669c152140 2024-03-24T10:56:44.691 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.691+0000 7f66a3543640 1 -- 172.21.15.5:0/3220977894 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f669c1f33a0 con 0x7f669c152140 2024-03-24T10:56:44.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.692+0000 7f66a3543640 1 -- 172.21.15.5:0/3220977894 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f669c062a00 con 0x7f669c152140 2024-03-24T10:56:44.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.694+0000 7f6691ffb640 1 -- 172.21.15.5:0/3220977894 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f668c04d440 con 0x7f669c152140 2024-03-24T10:56:44.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.694+0000 7f6691ffb640 1 -- 172.21.15.5:0/3220977894 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668c04d7d0 con 0x7f669c152140 2024-03-24T10:56:44.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.974+0000 7f6691ffb640 1 -- 172.21.15.5:0/3220977894 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f668c043440 con 0x7f669c152140 2024-03-24T10:56:44.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.975+0000 7f6691ffb640 1 --2- 172.21.15.5:0/3220977894 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f668006b840 0x7f668006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:44.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.975+0000 7f66a12df640 1 --2- 172.21.15.5:0/3220977894 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f668006b840 0x7f668006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:44.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.975+0000 7f66a12df640 1 --2- 172.21.15.5:0/3220977894 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f668006b840 0x7f668006dd00 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6698000f60 tx=0x7f6698009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:44.988 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.989+0000 7f6691ffb640 1 -- 172.21.15.5:0/3220977894 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f668c07bba0 con 0x7f669c152140 2024-03-24T10:56:44.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:44.989+0000 7f6691ffb640 1 -- 172.21.15.5:0/3220977894 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f668c077e80 con 0x7f669c152140 2024-03-24T10:56:45.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:45.208+0000 7f66a3543640 1 -- 172.21.15.5:0/3220977894 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f669c0681d0 con 0x7f669c152140 2024-03-24T10:56:45.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:45.222+0000 7f6691ffb640 1 -- 172.21.15.5:0/3220977894 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f668c07b500 con 0x7f669c152140 2024-03-24T10:56:45.222 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:45.222 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:45.222 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:45.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:45.225+0000 7f66a3543640 1 -- 172.21.15.5:0/3220977894 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f668006b840 msgr2=0x7f668006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:45.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:45.225+0000 7f66a3543640 1 --2- 172.21.15.5:0/3220977894 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f668006b840 0x7f668006dd00 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6698000f60 tx=0x7f6698009040 comp rx=0 tx=0).stop 2024-03-24T10:56:45.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:45.225+0000 7f66a3543640 1 -- 172.21.15.5:0/3220977894 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f669c152140 msgr2=0x7f669c1ef9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:45.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:45.225+0000 7f66a3543640 1 --2- 172.21.15.5:0/3220977894 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f669c152140 0x7f669c1ef9e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f668c0098c0 tx=0x7f668c004730 comp rx=0 tx=0).stop 2024-03-24T10:56:45.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:45.225+0000 7f66a3543640 1 -- 172.21.15.5:0/3220977894 shutdown_connections 2024-03-24T10:56:45.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:45.225+0000 7f66a3543640 1 --2- 172.21.15.5:0/3220977894 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f668006b840 0x7f668006dd00 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:45.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:45.225+0000 7f66a3543640 1 --2- 172.21.15.5:0/3220977894 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f669c152140 0x7f669c1ef9e0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:45.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:45.225+0000 7f66a3543640 1 --2- 172.21.15.5:0/3220977894 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f669c157f80 0x7f669c1eff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:45.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:45.225+0000 7f66a3543640 1 --2- 172.21.15.5:0/3220977894 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f669c06fc60 0x7f669c14a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:45.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:45.225+0000 7f66a3543640 1 -- 172.21.15.5:0/3220977894 >> 172.21.15.5:0/3220977894 conn(0x7f669c06b130 msgr2=0x7f669c0c5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:45.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:45.225+0000 7f66a3543640 1 -- 172.21.15.5:0/3220977894 shutdown_connections 2024-03-24T10:56:45.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:45.225+0000 7f66a3543640 1 -- 172.21.15.5:0/3220977894 wait complete. 2024-03-24T10:56:45.839 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:45.839 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:46.841 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:46.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.943+0000 7fa41ca8f640 1 Processor -- start 2024-03-24T10:56:46.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.943+0000 7fa41ca8f640 1 -- start start 2024-03-24T10:56:46.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.944+0000 7fa41ca8f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa4181543a0 0x7fa41815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:46.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.944+0000 7fa41ca8f640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa41806bf50 0x7fa41806c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:46.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.944+0000 7fa41ca8f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4180cb2f0 con 0x7fa41806d4f0 2024-03-24T10:56:46.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.944+0000 7fa41ca8f640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fa4180c6750 con 0x7fa41806bf50 2024-03-24T10:56:46.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.944+0000 7fa41ca8f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fa4180cc380 con 0x7fa4181543a0 2024-03-24T10:56:46.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.945+0000 7fa416d9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa41806bf50 0x7fa41806c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:46.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.945+0000 7fa41659c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa4181543a0 0x7fa41815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:46.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.945+0000 7fa416d9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa41806bf50 0x7fa41806c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:47488/0 (socket says 172.21.15.5:47488) 2024-03-24T10:56:46.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.945+0000 7fa416d9d640 1 -- 172.21.15.5:0/255273028 learned_addr learned my addr 172.21.15.5:0/255273028 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:46.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.949+0000 7fa416d9d640 1 -- 172.21.15.5:0/255273028 >> v2:172.21.15.57:3300/0 conn(0x7fa4181543a0 msgr2=0x7fa41815ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:46.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.949+0000 7fa416d9d640 1 --2- 172.21.15.5:0/255273028 >> v2:172.21.15.57:3300/0 conn(0x7fa4181543a0 0x7fa41815ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:46.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.949+0000 7fa416d9d640 1 -- 172.21.15.5:0/255273028 >> v1:172.21.15.57:6789/0 conn(0x7fa41806d4f0 legacy=0x7fa4180cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:46.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.949+0000 7fa416d9d640 1 -- 172.21.15.5:0/255273028 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4180ca7e0 con 0x7fa41806bf50 2024-03-24T10:56:46.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.951+0000 7fa416d9d640 1 --2- 172.21.15.5:0/255273028 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa41806bf50 0x7fa41806c330 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa4180cd600 tx=0x7fa41002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e9ebbdb3404116c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:46.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.956+0000 7fa41559a640 1 -- 172.21.15.5:0/255273028 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41002cbb0 con 0x7fa41806bf50 2024-03-24T10:56:46.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.956+0000 7fa41559a640 1 -- 172.21.15.5:0/255273028 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa41002cd10 con 0x7fa41806bf50 2024-03-24T10:56:46.956 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.956+0000 7fa41559a640 1 -- 172.21.15.5:0/255273028 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa410034610 con 0x7fa41806bf50 2024-03-24T10:56:46.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.956+0000 7fa41ca8f640 1 -- 172.21.15.5:0/255273028 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa41806bf50 msgr2=0x7fa41806c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:46.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.956+0000 7fa41ca8f640 1 --2- 172.21.15.5:0/255273028 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa41806bf50 0x7fa41806c330 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa4180cd600 tx=0x7fa41002c010 comp rx=0 tx=0).stop 2024-03-24T10:56:46.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.957+0000 7fa41ca8f640 1 -- 172.21.15.5:0/255273028 shutdown_connections 2024-03-24T10:56:46.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.957+0000 7fa41ca8f640 1 --2- 172.21.15.5:0/255273028 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa41806bf50 0x7fa41806c330 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:46.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.957+0000 7fa41ca8f640 1 --2- 172.21.15.5:0/255273028 >> v2:172.21.15.57:3300/0 conn(0x7fa4181543a0 0x7fa41815ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:46.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.957+0000 7fa41ca8f640 1 -- 172.21.15.5:0/255273028 >> 172.21.15.5:0/255273028 conn(0x7fa418072ef0 msgr2=0x7fa418061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:46.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.957+0000 7fa41ca8f640 1 -- 172.21.15.5:0/255273028 shutdown_connections 2024-03-24T10:56:46.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.957+0000 7fa41ca8f640 1 -- 172.21.15.5:0/255273028 wait complete. 2024-03-24T10:56:46.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.958+0000 7fa41ca8f640 1 Processor -- start 2024-03-24T10:56:46.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.958+0000 7fa41ca8f640 1 -- start start 2024-03-24T10:56:46.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.959+0000 7fa41ca8f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa41806bf50 0x7fa4180cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:46.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.959+0000 7fa41ca8f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa41806d4f0 0x7fa4180b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:46.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.959+0000 7fa41ca8f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4181543a0 0x7fa41814ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:46.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.959+0000 7fa41ca8f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa4181f2ab0 con 0x7fa41806bf50 2024-03-24T10:56:46.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.959+0000 7fa41ca8f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa4181f2ce0 con 0x7fa4181543a0 2024-03-24T10:56:46.959 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.959+0000 7fa41ca8f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fa4181f2f80 con 0x7fa41806d4f0 2024-03-24T10:56:46.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.959+0000 7fa41659c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa41806bf50 0x7fa4180cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:46.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.959+0000 7fa415d9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa41806d4f0 0x7fa4180b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:46.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.960+0000 7fa415d9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa41806d4f0 0x7fa4180b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:54378/0 (socket says 172.21.15.5:54378) 2024-03-24T10:56:46.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.960+0000 7fa415d9b640 1 -- 172.21.15.5:0/3751165493 learned_addr learned my addr 172.21.15.5:0/3751165493 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:46.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.960+0000 7fa416d9d640 1 --2- 172.21.15.5:0/3751165493 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4181543a0 0x7fa41814ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:46.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.964+0000 7fa415d9b640 1 -- 172.21.15.5:0/3751165493 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4181543a0 msgr2=0x7fa41814ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:46.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.964+0000 7fa415d9b640 1 --2- 172.21.15.5:0/3751165493 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4181543a0 0x7fa41814ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:46.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.964+0000 7fa415d9b640 1 -- 172.21.15.5:0/3751165493 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa41806bf50 msgr2=0x7fa4180cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:46.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.964+0000 7fa415d9b640 1 --2- 172.21.15.5:0/3751165493 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa41806bf50 0x7fa4180cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:46.964 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.964+0000 7fa415d9b640 1 -- 172.21.15.5:0/3751165493 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa410009b30 con 0x7fa41806d4f0 2024-03-24T10:56:46.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.966+0000 7fa415d9b640 1 --2- 172.21.15.5:0/3751165493 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa41806d4f0 0x7fa4180b41c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa40c00c430 tx=0x7fa40c00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:46.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.970+0000 7fa406ffd640 1 -- 172.21.15.5:0/3751165493 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40c0052d0 con 0x7fa41806d4f0 2024-03-24T10:56:46.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.971+0000 7fa406ffd640 1 -- 172.21.15.5:0/3751165493 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa40c005450 con 0x7fa41806d4f0 2024-03-24T10:56:46.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.971+0000 7fa41ca8f640 1 -- 172.21.15.5:0/3751165493 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4181f3260 con 0x7fa41806d4f0 2024-03-24T10:56:46.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.971+0000 7fa406ffd640 1 -- 172.21.15.5:0/3751165493 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa40c007260 con 0x7fa41806d4f0 2024-03-24T10:56:46.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.971+0000 7fa41ca8f640 1 -- 172.21.15.5:0/3751165493 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4181f3820 con 0x7fa41806d4f0 2024-03-24T10:56:46.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.972+0000 7fa41ca8f640 1 -- 172.21.15.5:0/3751165493 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4180681d0 con 0x7fa41806d4f0 2024-03-24T10:56:46.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.996+0000 7fa406ffd640 1 -- 172.21.15.5:0/3751165493 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa40c0073c0 con 0x7fa41806d4f0 2024-03-24T10:56:46.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.996+0000 7fa406ffd640 1 --2- 172.21.15.5:0/3751165493 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa3f806b860 0x7fa3f806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:46.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.997+0000 7fa41659c640 1 --2- 172.21.15.5:0/3751165493 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa3f806b860 0x7fa3f806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:46.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:46.997+0000 7fa41659c640 1 --2- 172.21.15.5:0/3751165493 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa3f806b860 0x7fa3f806dd20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa408009aa0 tx=0x7fa408009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:47.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:47.278+0000 7fa406ffd640 1 -- 172.21.15.5:0/3751165493 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fa40c007670 con 0x7fa41806d4f0 2024-03-24T10:56:47.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:47.289+0000 7fa406ffd640 1 -- 172.21.15.5:0/3751165493 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa40c054550 con 0x7fa41806d4f0 2024-03-24T10:56:47.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:47.510+0000 7fa41ca8f640 1 -- 172.21.15.5:0/3751165493 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa418061ed0 con 0x7fa41806d4f0 2024-03-24T10:56:47.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:47.535+0000 7fa406ffd640 1 -- 172.21.15.5:0/3751165493 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7fa40c057c10 con 0x7fa41806d4f0 2024-03-24T10:56:47.535 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:47.535 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:47.535 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:47.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:47.537+0000 7fa41ca8f640 1 -- 172.21.15.5:0/3751165493 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa3f806b860 msgr2=0x7fa3f806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:47.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:47.537+0000 7fa41ca8f640 1 --2- 172.21.15.5:0/3751165493 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa3f806b860 0x7fa3f806dd20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa408009aa0 tx=0x7fa408009190 comp rx=0 tx=0).stop 2024-03-24T10:56:47.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:47.537+0000 7fa41ca8f640 1 -- 172.21.15.5:0/3751165493 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa41806d4f0 msgr2=0x7fa4180b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:47.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:47.537+0000 7fa41ca8f640 1 --2- 172.21.15.5:0/3751165493 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa41806d4f0 0x7fa4180b41c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa40c00c430 tx=0x7fa40c00c8f0 comp rx=0 tx=0).stop 2024-03-24T10:56:47.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:47.537+0000 7fa41ca8f640 1 -- 172.21.15.5:0/3751165493 shutdown_connections 2024-03-24T10:56:47.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:47.537+0000 7fa41ca8f640 1 --2- 172.21.15.5:0/3751165493 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa3f806b860 0x7fa3f806dd20 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:47.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:47.537+0000 7fa41ca8f640 1 --2- 172.21.15.5:0/3751165493 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa4181543a0 0x7fa41814ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:47.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:47.537+0000 7fa41ca8f640 1 --2- 172.21.15.5:0/3751165493 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa41806d4f0 0x7fa4180b41c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:47.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:47.537+0000 7fa41ca8f640 1 --2- 172.21.15.5:0/3751165493 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa41806bf50 0x7fa4180cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:47.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:47.537+0000 7fa41ca8f640 1 -- 172.21.15.5:0/3751165493 >> 172.21.15.5:0/3751165493 conn(0x7fa418072ef0 msgr2=0x7fa418151bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:47.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:47.537+0000 7fa41ca8f640 1 -- 172.21.15.5:0/3751165493 shutdown_connections 2024-03-24T10:56:47.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:47.537+0000 7fa41ca8f640 1 -- 172.21.15.5:0/3751165493 wait complete. 2024-03-24T10:56:48.154 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:48.154 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:49.155 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:49.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.258+0000 7f01eb255640 1 Processor -- start 2024-03-24T10:56:49.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.259+0000 7f01eb255640 1 -- start start 2024-03-24T10:56:49.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.259+0000 7f01eb255640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f01e40cae90 0x7f01e40cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:49.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.259+0000 7f01eb255640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f01e40cd960 0x7f01e40cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:49.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.259+0000 7f01eb255640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e40ccba0 con 0x7f01e40cb270 2024-03-24T10:56:49.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.259+0000 7f01eb255640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f01e40cc9b0 con 0x7f01e40cd960 2024-03-24T10:56:49.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.259+0000 7f01eb255640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f01e40cce40 con 0x7f01e40cae90 2024-03-24T10:56:49.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.260+0000 7f01e8ff1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f01e40cae90 0x7f01e40cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:49.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.260+0000 7f01e97f2640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f01e40cd960 0x7f01e40cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:49.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.261+0000 7f01e8ff1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f01e40cae90 0x7f01e40cf530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:54382/0 (socket says 172.21.15.5:54382) 2024-03-24T10:56:49.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.261+0000 7f01e8ff1640 1 -- 172.21.15.5:0/2523765848 learned_addr learned my addr 172.21.15.5:0/2523765848 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:49.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.264+0000 7f01e97f2640 1 -- 172.21.15.5:0/2523765848 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f01e40cae90 msgr2=0x7f01e40cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:49.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.264+0000 7f01e97f2640 1 --2- 172.21.15.5:0/2523765848 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f01e40cae90 0x7f01e40cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:49.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.264+0000 7f01e97f2640 1 -- 172.21.15.5:0/2523765848 >> v1:172.21.15.57:6789/0 conn(0x7f01e40cb270 legacy=0x7f01e40cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:49.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.264+0000 7f01e97f2640 1 -- 172.21.15.5:0/2523765848 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e415a590 con 0x7f01e40cd960 2024-03-24T10:56:49.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.264+0000 7f01e8ff1640 1 --2- 172.21.15.5:0/2523765848 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f01e40cae90 0x7f01e40cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:56:49.269 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.270+0000 7f01e97f2640 1 --2- 172.21.15.5:0/2523765848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e40cd960 0x7f01e40cb6f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f01dc009d90 tx=0x7f01dc02bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90547e6e75217708 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:49.272 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.272+0000 7f01e37fe640 1 -- 172.21.15.5:0/2523765848 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01dc02cce0 con 0x7f01e40cd960 2024-03-24T10:56:49.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.275+0000 7f01e37fe640 1 -- 172.21.15.5:0/2523765848 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01dc02ce40 con 0x7f01e40cd960 2024-03-24T10:56:49.275 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.275+0000 7f01e37fe640 1 -- 172.21.15.5:0/2523765848 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01dc034660 con 0x7f01e40cd960 2024-03-24T10:56:49.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.275+0000 7f01eb255640 1 -- 172.21.15.5:0/2523765848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e40cd960 msgr2=0x7f01e40cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:49.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.275+0000 7f01eb255640 1 --2- 172.21.15.5:0/2523765848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e40cd960 0x7f01e40cb6f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f01dc009d90 tx=0x7f01dc02bf30 comp rx=0 tx=0).stop 2024-03-24T10:56:49.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.275+0000 7f01eb255640 1 -- 172.21.15.5:0/2523765848 shutdown_connections 2024-03-24T10:56:49.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.275+0000 7f01eb255640 1 --2- 172.21.15.5:0/2523765848 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e40cd960 0x7f01e40cb6f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:49.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.275+0000 7f01eb255640 1 --2- 172.21.15.5:0/2523765848 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f01e40cae90 0x7f01e40cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:49.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.275+0000 7f01eb255640 1 -- 172.21.15.5:0/2523765848 >> 172.21.15.5:0/2523765848 conn(0x7f01e40cfb80 msgr2=0x7f01e40cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:49.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.276+0000 7f01eb255640 1 -- 172.21.15.5:0/2523765848 shutdown_connections 2024-03-24T10:56:49.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.276+0000 7f01eb255640 1 -- 172.21.15.5:0/2523765848 wait complete. 2024-03-24T10:56:49.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.277+0000 7f01eb255640 1 Processor -- start 2024-03-24T10:56:49.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.277+0000 7f01eb255640 1 -- start start 2024-03-24T10:56:49.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.278+0000 7f01eb255640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01e40cae90 0x7f01e40ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:49.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.278+0000 7f01eb255640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f01e40cb270 0x7f01e41580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:49.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.278+0000 7f01eb255640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e40cd960 0x7f01e415b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:49.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.278+0000 7f01eb255640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f01e415c480 con 0x7f01e40cae90 2024-03-24T10:56:49.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.278+0000 7f01eb255640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f01e4158600 con 0x7f01e40cd960 2024-03-24T10:56:49.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.278+0000 7f01eb255640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f01e4155450 con 0x7f01e40cb270 2024-03-24T10:56:49.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.278+0000 7f01e8ff1640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01e40cae90 0x7f01e40ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:49.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.278+0000 7f01e3fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f01e40cb270 0x7f01e41580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:49.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.278+0000 7f01e97f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e40cd960 0x7f01e415b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:49.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.279+0000 7f01e97f2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e40cd960 0x7f01e415b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36278/0 (socket says 172.21.15.5:36278) 2024-03-24T10:56:49.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.279+0000 7f01e97f2640 1 -- 172.21.15.5:0/310068180 learned_addr learned my addr 172.21.15.5:0/310068180 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:49.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.283+0000 7f01e97f2640 1 -- 172.21.15.5:0/310068180 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f01e40cb270 msgr2=0x7f01e41580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:49.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.283+0000 7f01e97f2640 1 --2- 172.21.15.5:0/310068180 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f01e40cb270 0x7f01e41580c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:49.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.283+0000 7f01e97f2640 1 -- 172.21.15.5:0/310068180 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01e40cae90 msgr2=0x7f01e40ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:49.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.283+0000 7f01e97f2640 1 --2- 172.21.15.5:0/310068180 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01e40cae90 0x7f01e40ce230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:49.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.283+0000 7f01e97f2640 1 -- 172.21.15.5:0/310068180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d80094d0 con 0x7f01e40cd960 2024-03-24T10:56:49.286 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.286+0000 7f01e97f2640 1 --2- 172.21.15.5:0/310068180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e40cd960 0x7f01e415b5e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f01dc009eb0 tx=0x7f01dc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:49.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.291+0000 7f01e17fa640 1 -- 172.21.15.5:0/310068180 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01dc03d600 con 0x7f01e40cd960 2024-03-24T10:56:49.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.291+0000 7f01e17fa640 1 -- 172.21.15.5:0/310068180 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01dc03d780 con 0x7f01e40cd960 2024-03-24T10:56:49.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.291+0000 7f01e17fa640 1 -- 172.21.15.5:0/310068180 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01dc03db30 con 0x7f01e40cd960 2024-03-24T10:56:49.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.291+0000 7f01eb255640 1 -- 172.21.15.5:0/310068180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01dc009a70 con 0x7f01e40cd960 2024-03-24T10:56:49.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.291+0000 7f01eb255640 1 -- 172.21.15.5:0/310068180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e4155170 con 0x7f01e40cd960 2024-03-24T10:56:49.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.292+0000 7f01eb255640 1 -- 172.21.15.5:0/310068180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01e4159250 con 0x7f01e40cd960 2024-03-24T10:56:49.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.572+0000 7f01e17fa640 1 -- 172.21.15.5:0/310068180 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f01dc03b040 con 0x7f01e40cd960 2024-03-24T10:56:49.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.573+0000 7f01e17fa640 1 --2- 172.21.15.5:0/310068180 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f01c806b890 0x7f01c806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:49.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.573+0000 7f01e8ff1640 1 --2- 172.21.15.5:0/310068180 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f01c806b890 0x7f01c806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:49.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.574+0000 7f01e8ff1640 1 --2- 172.21.15.5:0/310068180 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f01c806b890 0x7f01c806dd50 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f01d8006ca0 tx=0x7f01d8009240 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:49.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.579+0000 7f01e17fa640 1 -- 172.21.15.5:0/310068180 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f01dc07b500 con 0x7f01e40cd960 2024-03-24T10:56:49.595 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.595+0000 7f01e17fa640 1 -- 172.21.15.5:0/310068180 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f01dc0777a0 con 0x7f01e40cd960 2024-03-24T10:56:49.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.820+0000 7f01eb255640 1 -- 172.21.15.5:0/310068180 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01e415a190 con 0x7f01e40cd960 2024-03-24T10:56:49.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.972+0000 7f01e17fa640 1 -- 172.21.15.5:0/310068180 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f01dc04e020 con 0x7f01e40cd960 2024-03-24T10:56:49.972 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:49.972 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:49.972 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:49.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.974+0000 7f01eb255640 1 -- 172.21.15.5:0/310068180 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f01c806b890 msgr2=0x7f01c806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:49.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.974+0000 7f01eb255640 1 --2- 172.21.15.5:0/310068180 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f01c806b890 0x7f01c806dd50 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f01d8006ca0 tx=0x7f01d8009240 comp rx=0 tx=0).stop 2024-03-24T10:56:49.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.974+0000 7f01eb255640 1 -- 172.21.15.5:0/310068180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e40cd960 msgr2=0x7f01e415b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:49.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.974+0000 7f01eb255640 1 --2- 172.21.15.5:0/310068180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e40cd960 0x7f01e415b5e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f01dc009eb0 tx=0x7f01dc004a40 comp rx=0 tx=0).stop 2024-03-24T10:56:49.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.974+0000 7f01eb255640 1 -- 172.21.15.5:0/310068180 shutdown_connections 2024-03-24T10:56:49.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.974+0000 7f01eb255640 1 --2- 172.21.15.5:0/310068180 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f01c806b890 0x7f01c806dd50 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:49.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.974+0000 7f01eb255640 1 --2- 172.21.15.5:0/310068180 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f01e40cd960 0x7f01e415b5e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:49.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.974+0000 7f01eb255640 1 --2- 172.21.15.5:0/310068180 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f01e40cb270 0x7f01e41580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:49.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.974+0000 7f01eb255640 1 --2- 172.21.15.5:0/310068180 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f01e40cae90 0x7f01e40ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:49.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.974+0000 7f01eb255640 1 -- 172.21.15.5:0/310068180 >> 172.21.15.5:0/310068180 conn(0x7f01e40cfb80 msgr2=0x7f01e40b5390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:49.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.974+0000 7f01eb255640 1 -- 172.21.15.5:0/310068180 shutdown_connections 2024-03-24T10:56:49.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:49.974+0000 7f01eb255640 1 -- 172.21.15.5:0/310068180 wait complete. 2024-03-24T10:56:50.593 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:50.594 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:51.595 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:51.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.700+0000 7f21668f4640 1 Processor -- start 2024-03-24T10:56:51.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.700+0000 7f21668f4640 1 -- start start 2024-03-24T10:56:51.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.701+0000 7f21668f4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f216015ac00 0x7f21600cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:51.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.701+0000 7f21668f4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21600cb2f0 con 0x7f21600ccaa0 2024-03-24T10:56:51.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.701+0000 7f21668f4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f216015b070 con 0x7f21601543a0 2024-03-24T10:56:51.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.701+0000 7f21668f4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f21600c6750 con 0x7f216015ac00 2024-03-24T10:56:51.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.702+0000 7f215f7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f216015ac00 0x7f21600cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:51.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.702+0000 7f215f7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f216015ac00 0x7f21600cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:54396/0 (socket says 172.21.15.5:54396) 2024-03-24T10:56:51.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.702+0000 7f215f7fe640 1 -- 172.21.15.5:0/3514049540 learned_addr learned my addr 172.21.15.5:0/3514049540 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:51.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.702+0000 7f215ffff640 1 --1- 172.21.15.5:0/3514049540 >> v1:172.21.15.57:6789/0 conn(0x7f21601543a0 0x7f216006d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:43694/0 (socket says 172.21.15.5:43694) 2024-03-24T10:56:51.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.705+0000 7f215f7fe640 1 -- 172.21.15.5:0/3514049540 >> v1:172.21.15.57:6789/0 conn(0x7f21601543a0 legacy=0x7f216006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:51.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.705+0000 7f215f7fe640 1 -- 172.21.15.5:0/3514049540 >> v1:172.21.15.5:6789/0 conn(0x7f21600ccaa0 legacy=0x7f216006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:51.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.705+0000 7f215f7fe640 1 -- 172.21.15.5:0/3514049540 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21600ca7e0 con 0x7f216015ac00 2024-03-24T10:56:51.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.707+0000 7f215f7fe640 1 --2- 172.21.15.5:0/3514049540 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f216015ac00 0x7f21600cc560 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f21540098a0 tx=0x7f215402ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e8affdb751ba8ee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:51.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.712+0000 7f215effd640 1 -- 172.21.15.5:0/3514049540 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215402f830 con 0x7f216015ac00 2024-03-24T10:56:51.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.712+0000 7f215effd640 1 -- 172.21.15.5:0/3514049540 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f215402f990 con 0x7f216015ac00 2024-03-24T10:56:51.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.712+0000 7f215effd640 1 -- 172.21.15.5:0/3514049540 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215402fd20 con 0x7f216015ac00 2024-03-24T10:56:51.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.713+0000 7f21668f4640 1 -- 172.21.15.5:0/3514049540 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f216015ac00 msgr2=0x7f21600cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:51.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.713+0000 7f21668f4640 1 --2- 172.21.15.5:0/3514049540 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f216015ac00 0x7f21600cc560 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f21540098a0 tx=0x7f215402ba10 comp rx=0 tx=0).stop 2024-03-24T10:56:51.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.713+0000 7f21668f4640 1 -- 172.21.15.5:0/3514049540 shutdown_connections 2024-03-24T10:56:51.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.713+0000 7f21668f4640 1 --2- 172.21.15.5:0/3514049540 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f216015ac00 0x7f21600cc560 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:51.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.713+0000 7f21668f4640 1 -- 172.21.15.5:0/3514049540 >> 172.21.15.5:0/3514049540 conn(0x7f2160072ef0 msgr2=0x7f2160061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:51.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.713+0000 7f21668f4640 1 -- 172.21.15.5:0/3514049540 shutdown_connections 2024-03-24T10:56:51.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.713+0000 7f21668f4640 1 -- 172.21.15.5:0/3514049540 wait complete. 2024-03-24T10:56:51.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.714+0000 7f21668f4640 1 Processor -- start 2024-03-24T10:56:51.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.714+0000 7f21668f4640 1 -- start start 2024-03-24T10:56:51.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.715+0000 7f21668f4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21600ccaa0 0x7f21600b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:51.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.715+0000 7f21668f4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21601543a0 0x7f216014f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:51.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.715+0000 7f21668f4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f216015ac00 0x7f21601f06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:51.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.715+0000 7f21668f4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f2160144fc0 con 0x7f21600ccaa0 2024-03-24T10:56:51.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.715+0000 7f21668f4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f21601f0210 con 0x7f216015ac00 2024-03-24T10:56:51.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.715+0000 7f21668f4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f21600b4040 con 0x7f21601543a0 2024-03-24T10:56:51.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.715+0000 7f215f7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21601543a0 0x7f216014f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:51.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.715+0000 7f215ffff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21600ccaa0 0x7f21600b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:51.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.716+0000 7f215f7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21601543a0 0x7f216014f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:54410/0 (socket says 172.21.15.5:54410) 2024-03-24T10:56:51.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.716+0000 7f215f7fe640 1 -- 172.21.15.5:0/3846837177 learned_addr learned my addr 172.21.15.5:0/3846837177 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:51.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.716+0000 7f2164e91640 1 --2- 172.21.15.5:0/3846837177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f216015ac00 0x7f21601f06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:51.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.721+0000 7f215f7fe640 1 -- 172.21.15.5:0/3846837177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f216015ac00 msgr2=0x7f21601f06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:51.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.721+0000 7f215f7fe640 1 --2- 172.21.15.5:0/3846837177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f216015ac00 0x7f21601f06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:51.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.721+0000 7f215f7fe640 1 -- 172.21.15.5:0/3846837177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21600ccaa0 msgr2=0x7f21600b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:51.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.721+0000 7f215f7fe640 1 --2- 172.21.15.5:0/3846837177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21600ccaa0 0x7f21600b3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:51.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.721+0000 7f215f7fe640 1 -- 172.21.15.5:0/3846837177 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21540094d0 con 0x7f21601543a0 2024-03-24T10:56:51.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.729+0000 7f215f7fe640 1 --2- 172.21.15.5:0/3846837177 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21601543a0 0x7f216014f360 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f21540090b0 tx=0x7f2154004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:51.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.737+0000 7f215cff9640 1 -- 172.21.15.5:0/3846837177 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215403c040 con 0x7f21601543a0 2024-03-24T10:56:51.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.737+0000 7f215cff9640 1 -- 172.21.15.5:0/3846837177 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f215402f7e0 con 0x7f21601543a0 2024-03-24T10:56:51.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.737+0000 7f215cff9640 1 -- 172.21.15.5:0/3846837177 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2154042630 con 0x7f21601543a0 2024-03-24T10:56:51.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.737+0000 7f21668f4640 1 -- 172.21.15.5:0/3846837177 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f216015a2a0 con 0x7f21601543a0 2024-03-24T10:56:51.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.737+0000 7f21668f4640 1 -- 172.21.15.5:0/3846837177 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f21601f35e0 con 0x7f21601543a0 2024-03-24T10:56:51.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.738+0000 7f21668f4640 1 -- 172.21.15.5:0/3846837177 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21600681d0 con 0x7f21601543a0 2024-03-24T10:56:51.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.770+0000 7f215cff9640 1 -- 172.21.15.5:0/3846837177 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f215402d3c0 con 0x7f21601543a0 2024-03-24T10:56:51.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.770+0000 7f215cff9640 1 --2- 172.21.15.5:0/3846837177 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f214c06b810 0x7f214c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:51.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.771+0000 7f215ffff640 1 --2- 172.21.15.5:0/3846837177 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f214c06b810 0x7f214c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:51.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:51.771+0000 7f215ffff640 1 --2- 172.21.15.5:0/3846837177 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f214c06b810 0x7f214c06dcd0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2150000f60 tx=0x7f2150000ed0 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:52.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:52.042+0000 7f215cff9640 1 -- 172.21.15.5:0/3846837177 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f215407b300 con 0x7f21601543a0 2024-03-24T10:56:52.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:52.062+0000 7f215cff9640 1 -- 172.21.15.5:0/3846837177 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f21540775a0 con 0x7f21601543a0 2024-03-24T10:56:52.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:52.290+0000 7f21668f4640 1 -- 172.21.15.5:0/3846837177 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21601f3b70 con 0x7f21601543a0 2024-03-24T10:56:52.338 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:52.338+0000 7f215cff9640 1 -- 172.21.15.5:0/3846837177 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f2154047020 con 0x7f21601543a0 2024-03-24T10:56:52.339 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:52.339 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:52.339 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:52.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:52.340+0000 7f21668f4640 1 -- 172.21.15.5:0/3846837177 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f214c06b810 msgr2=0x7f214c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:52.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:52.340+0000 7f21668f4640 1 --2- 172.21.15.5:0/3846837177 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f214c06b810 0x7f214c06dcd0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2150000f60 tx=0x7f2150000ed0 comp rx=0 tx=0).stop 2024-03-24T10:56:52.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:52.340+0000 7f21668f4640 1 -- 172.21.15.5:0/3846837177 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21601543a0 msgr2=0x7f216014f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:52.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:52.340+0000 7f21668f4640 1 --2- 172.21.15.5:0/3846837177 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21601543a0 0x7f216014f360 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f21540090b0 tx=0x7f2154004730 comp rx=0 tx=0).stop 2024-03-24T10:56:52.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:52.340+0000 7f21668f4640 1 -- 172.21.15.5:0/3846837177 shutdown_connections 2024-03-24T10:56:52.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:52.340+0000 7f21668f4640 1 --2- 172.21.15.5:0/3846837177 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f214c06b810 0x7f214c06dcd0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:52.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:52.340+0000 7f21668f4640 1 --2- 172.21.15.5:0/3846837177 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f216015ac00 0x7f21601f06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:52.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:52.340+0000 7f21668f4640 1 --2- 172.21.15.5:0/3846837177 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21601543a0 0x7f216014f360 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:52.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:52.340+0000 7f21668f4640 1 --2- 172.21.15.5:0/3846837177 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21600ccaa0 0x7f21600b3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:52.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:52.340+0000 7f21668f4640 1 -- 172.21.15.5:0/3846837177 >> 172.21.15.5:0/3846837177 conn(0x7f2160072ef0 msgr2=0x7f216006be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:52.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:52.340+0000 7f21668f4640 1 -- 172.21.15.5:0/3846837177 shutdown_connections 2024-03-24T10:56:52.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:52.340+0000 7f21668f4640 1 -- 172.21.15.5:0/3846837177 wait complete. 2024-03-24T10:56:52.957 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:52.957 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:53.958 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:54.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.061+0000 7f9f3e09c640 1 Processor -- start 2024-03-24T10:56:54.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.061+0000 7f9f3e09c640 1 -- start start 2024-03-24T10:56:54.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.063+0000 7f9f3e09c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9f380cc8f0 0x7f9f3815b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:54.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.063+0000 7f9f3e09c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9f3806c580 0x7f9f381538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:54.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.063+0000 7f9f3e09c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9f380cf110 0x7f9f38153df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:54.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.063+0000 7f9f3e09c640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f9f380cd890 con 0x7f9f3806c580 2024-03-24T10:56:54.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.063+0000 7f9f3e09c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f9f3806cae0 con 0x7f9f380cf110 2024-03-24T10:56:54.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.063+0000 7f9f3e09c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f9f380cdb50 con 0x7f9f380cc8f0 2024-03-24T10:56:54.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.063+0000 7f9f36ffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9f3806c580 0x7f9f381538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:54.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.063+0000 7f9f37fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9f380cf110 0x7f9f38153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:54.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.063+0000 7f9f377fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9f380cc8f0 0x7f9f3815b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:54.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.064+0000 7f9f36ffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9f3806c580 0x7f9f381538b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43272/0 (socket says 172.21.15.5:43272) 2024-03-24T10:56:54.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.064+0000 7f9f36ffd640 1 -- 172.21.15.5:0/1225632773 learned_addr learned my addr 172.21.15.5:0/1225632773 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:54.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.066+0000 7f9f36ffd640 1 -- 172.21.15.5:0/1225632773 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9f380cc8f0 msgr2=0x7f9f3815b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:54.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.066+0000 7f9f36ffd640 1 --2- 172.21.15.5:0/1225632773 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9f380cc8f0 0x7f9f3815b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:54.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.066+0000 7f9f36ffd640 1 -- 172.21.15.5:0/1225632773 >> v2:172.21.15.57:3300/0 conn(0x7f9f380cf110 msgr2=0x7f9f38153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:54.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.066+0000 7f9f36ffd640 1 --2- 172.21.15.5:0/1225632773 >> v2:172.21.15.57:3300/0 conn(0x7f9f380cf110 0x7f9f38153df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:54.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.066+0000 7f9f36ffd640 1 -- 172.21.15.5:0/1225632773 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f380cde30 con 0x7f9f3806c580 2024-03-24T10:56:54.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.068+0000 7f9f36ffd640 1 --2- 172.21.15.5:0/1225632773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9f3806c580 0x7f9f381538b0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9f380cfa60 tx=0x7f9f2c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a816297b6da4e9ce server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:54.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.072+0000 7f9f367fc640 1 -- 172.21.15.5:0/1225632773 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c031870 con 0x7f9f3806c580 2024-03-24T10:56:54.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.072+0000 7f9f367fc640 1 -- 172.21.15.5:0/1225632773 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f2c0319d0 con 0x7f9f3806c580 2024-03-24T10:56:54.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.072+0000 7f9f367fc640 1 -- 172.21.15.5:0/1225632773 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c031d60 con 0x7f9f3806c580 2024-03-24T10:56:54.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.073+0000 7f9f3e09c640 1 -- 172.21.15.5:0/1225632773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9f3806c580 msgr2=0x7f9f381538b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:54.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.073+0000 7f9f3e09c640 1 --2- 172.21.15.5:0/1225632773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9f3806c580 0x7f9f381538b0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f9f380cfa60 tx=0x7f9f2c02b910 comp rx=0 tx=0).stop 2024-03-24T10:56:54.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.073+0000 7f9f3e09c640 1 -- 172.21.15.5:0/1225632773 shutdown_connections 2024-03-24T10:56:54.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.073+0000 7f9f3e09c640 1 --2- 172.21.15.5:0/1225632773 >> v2:172.21.15.57:3300/0 conn(0x7f9f380cf110 0x7f9f38153df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:54.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.073+0000 7f9f3e09c640 1 --2- 172.21.15.5:0/1225632773 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9f3806c580 0x7f9f381538b0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:54.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.073+0000 7f9f3e09c640 1 --2- 172.21.15.5:0/1225632773 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9f380cc8f0 0x7f9f3815b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:54.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.073+0000 7f9f3e09c640 1 -- 172.21.15.5:0/1225632773 >> 172.21.15.5:0/1225632773 conn(0x7f9f380b1820 msgr2=0x7f9f380b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:54.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.073+0000 7f9f3e09c640 1 -- 172.21.15.5:0/1225632773 shutdown_connections 2024-03-24T10:56:54.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.073+0000 7f9f3e09c640 1 -- 172.21.15.5:0/1225632773 wait complete. 2024-03-24T10:56:54.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.074+0000 7f9f3e09c640 1 Processor -- start 2024-03-24T10:56:54.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.075+0000 7f9f3e09c640 1 -- start start 2024-03-24T10:56:54.075 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.075+0000 7f9f3e09c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9f3806c580 0x7f9f381f0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:54.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.075+0000 7f9f3e09c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f380cc8f0 0x7f9f3815ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:54.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.075+0000 7f9f3e09c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9f380cf110 0x7f9f381e65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:54.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.075+0000 7f9f3e09c640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f9f381588f0 con 0x7f9f380cf110 2024-03-24T10:56:54.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.075+0000 7f9f3e09c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9f38158bd0 con 0x7f9f380cc8f0 2024-03-24T10:56:54.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.075+0000 7f9f3e09c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f9f38158eb0 con 0x7f9f3806c580 2024-03-24T10:56:54.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.076+0000 7f9f37fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9f380cf110 0x7f9f381e65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:54.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.076+0000 7f9f36ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f380cc8f0 0x7f9f3815ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:54.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.076+0000 7f9f377fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9f3806c580 0x7f9f381f0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:54.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.076+0000 7f9f37fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9f380cf110 0x7f9f381e65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43274/0 (socket says 172.21.15.5:43274) 2024-03-24T10:56:54.077 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.076+0000 7f9f37fff640 1 -- 172.21.15.5:0/2294889195 learned_addr learned my addr 172.21.15.5:0/2294889195 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:54.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.080+0000 7f9f37fff640 1 -- 172.21.15.5:0/2294889195 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9f3806c580 msgr2=0x7f9f381f0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:54.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.080+0000 7f9f37fff640 1 --2- 172.21.15.5:0/2294889195 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9f3806c580 0x7f9f381f0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:54.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.080+0000 7f9f37fff640 1 -- 172.21.15.5:0/2294889195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f380cc8f0 msgr2=0x7f9f3815ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:54.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.080+0000 7f9f37fff640 1 --2- 172.21.15.5:0/2294889195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f380cc8f0 0x7f9f3815ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:54.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.080+0000 7f9f37fff640 1 -- 172.21.15.5:0/2294889195 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f2c0094d0 con 0x7f9f380cf110 2024-03-24T10:56:54.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.083+0000 7f9f37fff640 1 --2- 172.21.15.5:0/2294889195 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9f380cf110 0x7f9f381e65c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f9f3000efc0 tx=0x7f9f3000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:54.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.088+0000 7f9f1ffff640 1 -- 172.21.15.5:0/2294889195 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f30014480 con 0x7f9f380cf110 2024-03-24T10:56:54.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.088+0000 7f9f1ffff640 1 -- 172.21.15.5:0/2294889195 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f3000f040 con 0x7f9f380cf110 2024-03-24T10:56:54.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.088+0000 7f9f3e09c640 1 -- 172.21.15.5:0/2294889195 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f38159190 con 0x7f9f380cf110 2024-03-24T10:56:54.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.088+0000 7f9f1ffff640 1 -- 172.21.15.5:0/2294889195 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f30014820 con 0x7f9f380cf110 2024-03-24T10:56:54.088 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.088+0000 7f9f3e09c640 1 -- 172.21.15.5:0/2294889195 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f38154a70 con 0x7f9f380cf110 2024-03-24T10:56:54.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.089+0000 7f9f3e09c640 1 -- 172.21.15.5:0/2294889195 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f380b3070 con 0x7f9f380cf110 2024-03-24T10:56:54.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.120+0000 7f9f1ffff640 1 -- 172.21.15.5:0/2294889195 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9f30014980 con 0x7f9f380cf110 2024-03-24T10:56:54.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.120+0000 7f9f1ffff640 1 --2- 172.21.15.5:0/2294889195 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9f2406b860 0x7f9f2406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:54.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.121+0000 7f9f377fe640 1 --2- 172.21.15.5:0/2294889195 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9f2406b860 0x7f9f2406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:54.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.121+0000 7f9f377fe640 1 --2- 172.21.15.5:0/2294889195 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9f2406b860 0x7f9f2406dd20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9f28009b40 tx=0x7f9f28009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:54.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.126+0000 7f9f1ffff640 1 -- 172.21.15.5:0/2294889195 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f9f30059c40 con 0x7f9f380cf110 2024-03-24T10:56:54.394 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.394+0000 7f9f1ffff640 1 -- 172.21.15.5:0/2294889195 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9f30055ee0 con 0x7f9f380cf110 2024-03-24T10:56:54.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.651+0000 7f9f3e09c640 1 -- 172.21.15.5:0/2294889195 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f38154c60 con 0x7f9f380cf110 2024-03-24T10:56:54.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.665+0000 7f9f1ffff640 1 -- 172.21.15.5:0/2294889195 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f9f30018020 con 0x7f9f380cf110 2024-03-24T10:56:54.665 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:54.665 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:54.665 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:54.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.667+0000 7f9f3e09c640 1 -- 172.21.15.5:0/2294889195 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9f2406b860 msgr2=0x7f9f2406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:54.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.667+0000 7f9f3e09c640 1 --2- 172.21.15.5:0/2294889195 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9f2406b860 0x7f9f2406dd20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9f28009b40 tx=0x7f9f28009190 comp rx=0 tx=0).stop 2024-03-24T10:56:54.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.667+0000 7f9f3e09c640 1 -- 172.21.15.5:0/2294889195 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9f380cf110 msgr2=0x7f9f381e65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:54.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.667+0000 7f9f3e09c640 1 --2- 172.21.15.5:0/2294889195 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9f380cf110 0x7f9f381e65c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f9f3000efc0 tx=0x7f9f3000c490 comp rx=0 tx=0).stop 2024-03-24T10:56:54.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.667+0000 7f9f3e09c640 1 -- 172.21.15.5:0/2294889195 shutdown_connections 2024-03-24T10:56:54.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.667+0000 7f9f3e09c640 1 --2- 172.21.15.5:0/2294889195 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9f2406b860 0x7f9f2406dd20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:54.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.667+0000 7f9f3e09c640 1 --2- 172.21.15.5:0/2294889195 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9f380cf110 0x7f9f381e65c0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:54.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.667+0000 7f9f3e09c640 1 --2- 172.21.15.5:0/2294889195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9f380cc8f0 0x7f9f3815ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:54.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.667+0000 7f9f3e09c640 1 --2- 172.21.15.5:0/2294889195 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9f3806c580 0x7f9f381f0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:54.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.667+0000 7f9f3e09c640 1 -- 172.21.15.5:0/2294889195 >> 172.21.15.5:0/2294889195 conn(0x7f9f380b1820 msgr2=0x7f9f380b2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:54.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.667+0000 7f9f3e09c640 1 -- 172.21.15.5:0/2294889195 shutdown_connections 2024-03-24T10:56:54.668 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:54.667+0000 7f9f3e09c640 1 -- 172.21.15.5:0/2294889195 wait complete. 2024-03-24T10:56:55.282 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:55.282 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:56.283 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:56.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.387+0000 7f692ec55640 1 Processor -- start 2024-03-24T10:56:56.387 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.387+0000 7f692ec55640 1 -- start start 2024-03-24T10:56:56.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.388+0000 7f692ec55640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f69280c4300 0x7f69280cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:56.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.388+0000 7f692ec55640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f69280cdfb0 0x7f692806c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:56.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.388+0000 7f692ec55640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69280c4720 con 0x7f6928154220 2024-03-24T10:56:56.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.388+0000 7f692ec55640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f69280c8bf0 con 0x7f69280c4300 2024-03-24T10:56:56.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.388+0000 7f692ec55640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f69280c9050 con 0x7f69280cdfb0 2024-03-24T10:56:56.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.388+0000 7f6927fff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f69280c4300 0x7f69280cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:56.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.388+0000 7f692d1f2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f69280cdfb0 0x7f692806c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:56.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.389+0000 7f692d1f2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f69280cdfb0 0x7f692806c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:37338/0 (socket says 172.21.15.5:37338) 2024-03-24T10:56:56.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.389+0000 7f692d1f2640 1 -- 172.21.15.5:0/1499249626 learned_addr learned my addr 172.21.15.5:0/1499249626 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:56.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.392+0000 7f692d1f2640 1 -- 172.21.15.5:0/1499249626 >> v2:172.21.15.5:3300/0 conn(0x7f69280c4300 msgr2=0x7f69280cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:56:56.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.392+0000 7f692d1f2640 1 --2- 172.21.15.5:0/1499249626 >> v2:172.21.15.5:3300/0 conn(0x7f69280c4300 0x7f69280cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:56.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.392+0000 7f692d1f2640 1 -- 172.21.15.5:0/1499249626 >> v1:172.21.15.5:6789/0 conn(0x7f6928154220 legacy=0x7f69280c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:56.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.392+0000 7f692d1f2640 1 -- 172.21.15.5:0/1499249626 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692815a5f0 con 0x7f69280cdfb0 2024-03-24T10:56:56.395 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.395+0000 7f692d1f2640 1 --2- 172.21.15.5:0/1499249626 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f69280cdfb0 0x7f692806c910 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6920009e50 tx=0x7f692002bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58399e677cb82183 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:56.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.398+0000 7f69277fe640 1 -- 172.21.15.5:0/1499249626 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f692002cb90 con 0x7f69280cdfb0 2024-03-24T10:56:56.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.401+0000 7f69277fe640 1 -- 172.21.15.5:0/1499249626 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f692002ccf0 con 0x7f69280cdfb0 2024-03-24T10:56:56.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.401+0000 7f69277fe640 1 -- 172.21.15.5:0/1499249626 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6920034610 con 0x7f69280cdfb0 2024-03-24T10:56:56.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.401+0000 7f692ec55640 1 -- 172.21.15.5:0/1499249626 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f69280cdfb0 msgr2=0x7f692806c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:56.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.401+0000 7f692ec55640 1 --2- 172.21.15.5:0/1499249626 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f69280cdfb0 0x7f692806c910 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6920009e50 tx=0x7f692002bff0 comp rx=0 tx=0).stop 2024-03-24T10:56:56.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.402+0000 7f692ec55640 1 -- 172.21.15.5:0/1499249626 shutdown_connections 2024-03-24T10:56:56.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.402+0000 7f692ec55640 1 --2- 172.21.15.5:0/1499249626 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f69280cdfb0 0x7f692806c910 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:56.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.402+0000 7f692ec55640 1 --2- 172.21.15.5:0/1499249626 >> v2:172.21.15.5:3300/0 conn(0x7f69280c4300 0x7f69280cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:56.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.402+0000 7f692ec55640 1 -- 172.21.15.5:0/1499249626 >> 172.21.15.5:0/1499249626 conn(0x7f692806f830 msgr2=0x7f692806fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:56.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.402+0000 7f692ec55640 1 -- 172.21.15.5:0/1499249626 shutdown_connections 2024-03-24T10:56:56.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.402+0000 7f692ec55640 1 -- 172.21.15.5:0/1499249626 wait complete. 2024-03-24T10:56:56.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.403+0000 7f692ec55640 1 Processor -- start 2024-03-24T10:56:56.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.404+0000 7f692ec55640 1 -- start start 2024-03-24T10:56:56.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.405+0000 7f692ec55640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69280c4300 0x7f69280ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:56.406 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.405+0000 7f692ec55640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69280cdfb0 0x7f692815b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:56.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.405+0000 7f692ec55640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6928154220 0x7f692815bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:56.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.405+0000 7f692ec55640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f69281f0c40 con 0x7f69280c4300 2024-03-24T10:56:56.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.405+0000 7f692ec55640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f692815e2f0 con 0x7f69280cdfb0 2024-03-24T10:56:56.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.405+0000 7f692ec55640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f69281f0310 con 0x7f6928154220 2024-03-24T10:56:56.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.405+0000 7f692c9f1640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69280c4300 0x7f69280ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:56.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.406+0000 7f6927fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69280cdfb0 0x7f692815b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:56.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.406+0000 7f692d1f2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6928154220 0x7f692815bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:56.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.406+0000 7f692c9f1640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69280c4300 0x7f69280ccba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43286/0 (socket says 172.21.15.5:43286) 2024-03-24T10:56:56.407 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.406+0000 7f692c9f1640 1 -- 172.21.15.5:0/939947363 learned_addr learned my addr 172.21.15.5:0/939947363 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:56.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.409+0000 7f692c9f1640 1 -- 172.21.15.5:0/939947363 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6928154220 msgr2=0x7f692815bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:56.409 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.409+0000 7f692c9f1640 1 --2- 172.21.15.5:0/939947363 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6928154220 0x7f692815bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:56.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.409+0000 7f692c9f1640 1 -- 172.21.15.5:0/939947363 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69280cdfb0 msgr2=0x7f692815b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:56.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.409+0000 7f692c9f1640 1 --2- 172.21.15.5:0/939947363 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69280cdfb0 0x7f692815b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:56.410 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.409+0000 7f692c9f1640 1 -- 172.21.15.5:0/939947363 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6920009b30 con 0x7f69280c4300 2024-03-24T10:56:56.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.412+0000 7f692c9f1640 1 --2- 172.21.15.5:0/939947363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69280c4300 0x7f69280ccba0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f691c00c3c0 tx=0x7f691c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:56.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.415+0000 7f69257fa640 1 -- 172.21.15.5:0/939947363 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f691c0052d0 con 0x7f69280c4300 2024-03-24T10:56:56.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.415+0000 7f69257fa640 1 -- 172.21.15.5:0/939947363 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f691c005450 con 0x7f69280c4300 2024-03-24T10:56:56.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.415+0000 7f69257fa640 1 -- 172.21.15.5:0/939947363 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f691c007260 con 0x7f69280c4300 2024-03-24T10:56:56.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.415+0000 7f692ec55640 1 -- 172.21.15.5:0/939947363 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69281f05f0 con 0x7f69280c4300 2024-03-24T10:56:56.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.415+0000 7f692ec55640 1 -- 172.21.15.5:0/939947363 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69281f10d0 con 0x7f69280c4300 2024-03-24T10:56:56.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.416+0000 7f692ec55640 1 -- 172.21.15.5:0/939947363 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69280681d0 con 0x7f69280c4300 2024-03-24T10:56:56.694 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.694+0000 7f69257fa640 1 -- 172.21.15.5:0/939947363 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f691c0073c0 con 0x7f69280c4300 2024-03-24T10:56:56.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.695+0000 7f69257fa640 1 --2- 172.21.15.5:0/939947363 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f690c06b810 0x7f690c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:56.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.695+0000 7f6927fff640 1 --2- 172.21.15.5:0/939947363 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f690c06b810 0x7f690c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:56.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.695+0000 7f6927fff640 1 --2- 172.21.15.5:0/939947363 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f690c06b810 0x7f690c06dcd0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6928159d90 tx=0x7f6914009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:56.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.714+0000 7f69257fa640 1 -- 172.21.15.5:0/939947363 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f691c058230 con 0x7f69280c4300 2024-03-24T10:56:56.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.714+0000 7f69257fa640 1 -- 172.21.15.5:0/939947363 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f691c0544d0 con 0x7f69280c4300 2024-03-24T10:56:56.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.939+0000 7f692ec55640 1 -- 172.21.15.5:0/939947363 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69280c4c80 con 0x7f69280c4300 2024-03-24T10:56:56.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.960+0000 7f69257fa640 1 -- 172.21.15.5:0/939947363 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f691c020020 con 0x7f69280c4300 2024-03-24T10:56:56.961 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:56.961 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:56.961 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:56.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.962+0000 7f692ec55640 1 -- 172.21.15.5:0/939947363 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f690c06b810 msgr2=0x7f690c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:56.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.962+0000 7f692ec55640 1 --2- 172.21.15.5:0/939947363 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f690c06b810 0x7f690c06dcd0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f6928159d90 tx=0x7f6914009190 comp rx=0 tx=0).stop 2024-03-24T10:56:56.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.962+0000 7f692ec55640 1 -- 172.21.15.5:0/939947363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69280c4300 msgr2=0x7f69280ccba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:56.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.962+0000 7f692ec55640 1 --2- 172.21.15.5:0/939947363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69280c4300 0x7f69280ccba0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f691c00c3c0 tx=0x7f691c00c880 comp rx=0 tx=0).stop 2024-03-24T10:56:56.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.963+0000 7f692ec55640 1 -- 172.21.15.5:0/939947363 shutdown_connections 2024-03-24T10:56:56.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.963+0000 7f692ec55640 1 --2- 172.21.15.5:0/939947363 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f690c06b810 0x7f690c06dcd0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:56.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.963+0000 7f692ec55640 1 --2- 172.21.15.5:0/939947363 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6928154220 0x7f692815bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:56.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.963+0000 7f692ec55640 1 --2- 172.21.15.5:0/939947363 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69280cdfb0 0x7f692815b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:56.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.963+0000 7f692ec55640 1 --2- 172.21.15.5:0/939947363 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69280c4300 0x7f69280ccba0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:56.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.963+0000 7f692ec55640 1 -- 172.21.15.5:0/939947363 >> 172.21.15.5:0/939947363 conn(0x7f692806f830 msgr2=0x7f6928152aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:56.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.963+0000 7f692ec55640 1 -- 172.21.15.5:0/939947363 shutdown_connections 2024-03-24T10:56:56.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:56.963+0000 7f692ec55640 1 -- 172.21.15.5:0/939947363 wait complete. 2024-03-24T10:56:57.579 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:57.579 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:56:58.580 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:56:58.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.682+0000 7ff6d59c7640 1 Processor -- start 2024-03-24T10:56:58.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.682+0000 7ff6d59c7640 1 -- start start 2024-03-24T10:56:58.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.683+0000 7ff6d59c7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff6d006bf50 0x7ff6d006c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:58.683 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.683+0000 7ff6d59c7640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6d00cb2f0 con 0x7ff6d01543a0 2024-03-24T10:56:58.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.683+0000 7ff6d59c7640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6d015b070 con 0x7ff6d015ac00 2024-03-24T10:56:58.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.683+0000 7ff6d59c7640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7ff6d00c6750 con 0x7ff6d006bf50 2024-03-24T10:56:58.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.684+0000 7ff6cf7fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff6d006bf50 0x7ff6d006c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:58.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.684+0000 7ff6cf7fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff6d006bf50 0x7ff6d006c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43298/0 (socket says 172.21.15.5:43298) 2024-03-24T10:56:58.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.684+0000 7ff6cf7fe640 1 -- 172.21.15.5:0/1054556455 learned_addr learned my addr 172.21.15.5:0/1054556455 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:58.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.687+0000 7ff6ce7fc640 1 -- 172.21.15.5:0/1054556455 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3794598822 0 0) 0x7ff6d015b070 con 0x7ff6d015ac00 2024-03-24T10:56:58.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.687+0000 7ff6ce7fc640 1 -- 172.21.15.5:0/1054556455 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6bc003520 con 0x7ff6d015ac00 2024-03-24T10:56:58.687 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.688+0000 7ff6cf7fe640 1 -- 172.21.15.5:0/1054556455 >> v1:172.21.15.57:6789/0 conn(0x7ff6d015ac00 legacy=0x7ff6d00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:58.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.688+0000 7ff6cf7fe640 1 -- 172.21.15.5:0/1054556455 >> v1:172.21.15.5:6789/0 conn(0x7ff6d01543a0 legacy=0x7ff6d006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:58.688 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.688+0000 7ff6cf7fe640 1 -- 172.21.15.5:0/1054556455 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d00ca7e0 con 0x7ff6d006bf50 2024-03-24T10:56:58.692 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.692+0000 7ff6cf7fe640 1 --2- 172.21.15.5:0/1054556455 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6d006bf50 0x7ff6d006c330 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8009e50 tx=0x7ff6c802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b745382465bfb13b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:58.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.698+0000 7ff6ce7fc640 1 -- 172.21.15.5:0/1054556455 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c802cbb0 con 0x7ff6d006bf50 2024-03-24T10:56:58.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.698+0000 7ff6ce7fc640 1 -- 172.21.15.5:0/1054556455 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6c802cd10 con 0x7ff6d006bf50 2024-03-24T10:56:58.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.698+0000 7ff6ce7fc640 1 -- 172.21.15.5:0/1054556455 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8034610 con 0x7ff6d006bf50 2024-03-24T10:56:58.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.698+0000 7ff6d59c7640 1 -- 172.21.15.5:0/1054556455 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6d006bf50 msgr2=0x7ff6d006c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:58.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.698+0000 7ff6d59c7640 1 --2- 172.21.15.5:0/1054556455 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6d006bf50 0x7ff6d006c330 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8009e50 tx=0x7ff6c802c010 comp rx=0 tx=0).stop 2024-03-24T10:56:58.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.698+0000 7ff6d59c7640 1 -- 172.21.15.5:0/1054556455 shutdown_connections 2024-03-24T10:56:58.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.698+0000 7ff6d59c7640 1 --2- 172.21.15.5:0/1054556455 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6d006bf50 0x7ff6d006c330 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:58.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.698+0000 7ff6d59c7640 1 -- 172.21.15.5:0/1054556455 >> 172.21.15.5:0/1054556455 conn(0x7ff6d0072ef0 msgr2=0x7ff6d0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:58.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.699+0000 7ff6d59c7640 1 -- 172.21.15.5:0/1054556455 shutdown_connections 2024-03-24T10:56:58.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.699+0000 7ff6d59c7640 1 -- 172.21.15.5:0/1054556455 wait complete. 2024-03-24T10:56:58.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.700+0000 7ff6d59c7640 1 Processor -- start 2024-03-24T10:56:58.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.700+0000 7ff6d59c7640 1 -- start start 2024-03-24T10:56:58.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.701+0000 7ff6d59c7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6d006bf50 0x7ff6d01446a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:58.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.701+0000 7ff6d59c7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff6d01543a0 0x7ff6d0156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:58.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.701+0000 7ff6d59c7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6d015ac00 0x7ff6d0156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:58.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.701+0000 7ff6d59c7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d0159da0 con 0x7ff6d015ac00 2024-03-24T10:56:58.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.701+0000 7ff6d59c7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d0154c00 con 0x7ff6d006bf50 2024-03-24T10:56:58.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.701+0000 7ff6d59c7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff6d0154ee0 con 0x7ff6d01543a0 2024-03-24T10:56:58.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.701+0000 7ff6cf7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6d015ac00 0x7ff6d0156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:58.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.701+0000 7ff6c7fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff6d01543a0 0x7ff6d0156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:58.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.701+0000 7ff6ceffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6d006bf50 0x7ff6d01446a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:58.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.702+0000 7ff6c7fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff6d01543a0 0x7ff6d0156790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:37352/0 (socket says 172.21.15.5:37352) 2024-03-24T10:56:58.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.702+0000 7ff6c7fff640 1 -- 172.21.15.5:0/900858935 learned_addr learned my addr 172.21.15.5:0/900858935 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:56:58.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.705+0000 7ff6cf7fe640 1 -- 172.21.15.5:0/900858935 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff6d01543a0 msgr2=0x7ff6d0156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:58.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.705+0000 7ff6cf7fe640 1 --2- 172.21.15.5:0/900858935 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff6d01543a0 0x7ff6d0156790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:58.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.705+0000 7ff6cf7fe640 1 -- 172.21.15.5:0/900858935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6d006bf50 msgr2=0x7ff6d01446a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:58.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.705+0000 7ff6cf7fe640 1 --2- 172.21.15.5:0/900858935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6d006bf50 0x7ff6d01446a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:58.706 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.705+0000 7ff6cf7fe640 1 -- 172.21.15.5:0/900858935 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c8009b30 con 0x7ff6d015ac00 2024-03-24T10:56:58.708 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.708+0000 7ff6cf7fe640 1 --2- 172.21.15.5:0/900858935 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6d015ac00 0x7ff6d0156cd0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8032000 tx=0x7ff6c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:58.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.713+0000 7ff6ccff9640 1 -- 172.21.15.5:0/900858935 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8009510 con 0x7ff6d015ac00 2024-03-24T10:56:58.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.713+0000 7ff6ccff9640 1 -- 172.21.15.5:0/900858935 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6c803e450 con 0x7ff6d015ac00 2024-03-24T10:56:58.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.713+0000 7ff6ccff9640 1 -- 172.21.15.5:0/900858935 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c803e820 con 0x7ff6d015ac00 2024-03-24T10:56:58.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.713+0000 7ff6d59c7640 1 -- 172.21.15.5:0/900858935 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6d015dd70 con 0x7ff6d015ac00 2024-03-24T10:56:58.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.713+0000 7ff6d59c7640 1 -- 172.21.15.5:0/900858935 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6d0144be0 con 0x7ff6d015ac00 2024-03-24T10:56:58.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.714+0000 7ff6d59c7640 1 -- 172.21.15.5:0/900858935 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6d0153730 con 0x7ff6d015ac00 2024-03-24T10:56:58.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.733+0000 7ff6ccff9640 1 -- 172.21.15.5:0/900858935 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff6c803e980 con 0x7ff6d015ac00 2024-03-24T10:56:58.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.734+0000 7ff6ccff9640 1 --2- 172.21.15.5:0/900858935 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff6b006b810 0x7ff6b006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:56:58.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.734+0000 7ff6ceffd640 1 --2- 172.21.15.5:0/900858935 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff6b006b810 0x7ff6b006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:56:58.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.734+0000 7ff6ceffd640 1 --2- 172.21.15.5:0/900858935 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff6b006b810 0x7ff6b006dcd0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0001780 tx=0x7ff6c0009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:56:58.739 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.739+0000 7ff6ccff9640 1 -- 172.21.15.5:0/900858935 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7ff6c807bcc0 con 0x7ff6d015ac00 2024-03-24T10:56:58.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:58.998+0000 7ff6ccff9640 1 -- 172.21.15.5:0/900858935 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff6c8077f60 con 0x7ff6d015ac00 2024-03-24T10:56:59.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:59.216+0000 7ff6d59c7640 1 -- 172.21.15.5:0/900858935 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6d015e230 con 0x7ff6d015ac00 2024-03-24T10:56:59.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:59.238+0000 7ff6ccff9640 1 -- 172.21.15.5:0/900858935 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7ff6c8043020 con 0x7ff6d015ac00 2024-03-24T10:56:59.238 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:56:59.238 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:56:59.239 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:56:59.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:59.240+0000 7ff6d59c7640 1 -- 172.21.15.5:0/900858935 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff6b006b810 msgr2=0x7ff6b006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:59.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:59.240+0000 7ff6d59c7640 1 --2- 172.21.15.5:0/900858935 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff6b006b810 0x7ff6b006dcd0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0001780 tx=0x7ff6c0009040 comp rx=0 tx=0).stop 2024-03-24T10:56:59.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:59.240+0000 7ff6d59c7640 1 -- 172.21.15.5:0/900858935 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6d015ac00 msgr2=0x7ff6d0156cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:56:59.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:59.240+0000 7ff6d59c7640 1 --2- 172.21.15.5:0/900858935 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6d015ac00 0x7ff6d0156cd0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8032000 tx=0x7ff6c8004a40 comp rx=0 tx=0).stop 2024-03-24T10:56:59.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:59.240+0000 7ff6d59c7640 1 -- 172.21.15.5:0/900858935 shutdown_connections 2024-03-24T10:56:59.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:59.240+0000 7ff6d59c7640 1 --2- 172.21.15.5:0/900858935 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff6b006b810 0x7ff6b006dcd0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:59.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:59.240+0000 7ff6d59c7640 1 --2- 172.21.15.5:0/900858935 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff6d015ac00 0x7ff6d0156cd0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:59.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:59.240+0000 7ff6d59c7640 1 --2- 172.21.15.5:0/900858935 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff6d01543a0 0x7ff6d0156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:59.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:59.240+0000 7ff6d59c7640 1 --2- 172.21.15.5:0/900858935 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff6d006bf50 0x7ff6d01446a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:56:59.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:59.240+0000 7ff6d59c7640 1 -- 172.21.15.5:0/900858935 >> 172.21.15.5:0/900858935 conn(0x7ff6d0072ef0 msgr2=0x7ff6d0152cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:56:59.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:59.240+0000 7ff6d59c7640 1 -- 172.21.15.5:0/900858935 shutdown_connections 2024-03-24T10:56:59.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:56:59.240+0000 7ff6d59c7640 1 -- 172.21.15.5:0/900858935 wait complete. 2024-03-24T10:56:59.855 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:56:59.855 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:00.856 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:00.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.958+0000 7fb5056d0640 1 Processor -- start 2024-03-24T10:57:00.958 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.958+0000 7fb5056d0640 1 -- start start 2024-03-24T10:57:00.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.960+0000 7fb5056d0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb5001543a0 0x7fb50015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:00.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.960+0000 7fb5056d0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb50006d4f0 0x7fb5000cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:00.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.960+0000 7fb5056d0640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5000cb2f0 con 0x7fb5000ccaa0 2024-03-24T10:57:00.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.960+0000 7fb5056d0640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fb5000c6750 con 0x7fb50006d4f0 2024-03-24T10:57:00.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.960+0000 7fb5056d0640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fb5000cc380 con 0x7fb5001543a0 2024-03-24T10:57:00.960 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.960+0000 7fb4fe7fc640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb50006d4f0 0x7fb5000cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:00.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.960+0000 7fb4feffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb5001543a0 0x7fb50015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:00.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.961+0000 7fb4fe7fc640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb50006d4f0 0x7fb5000cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43316/0 (socket says 172.21.15.5:43316) 2024-03-24T10:57:00.961 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.961+0000 7fb4fe7fc640 1 -- 172.21.15.5:0/1545440937 learned_addr learned my addr 172.21.15.5:0/1545440937 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:00.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.963+0000 7fb4fe7fc640 1 -- 172.21.15.5:0/1545440937 >> v2:172.21.15.57:3300/0 conn(0x7fb5001543a0 msgr2=0x7fb50015ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:00.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.963+0000 7fb4fe7fc640 1 --2- 172.21.15.5:0/1545440937 >> v2:172.21.15.57:3300/0 conn(0x7fb5001543a0 0x7fb50015ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:00.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.963+0000 7fb4fe7fc640 1 -- 172.21.15.5:0/1545440937 >> v1:172.21.15.57:6789/0 conn(0x7fb5000ccaa0 legacy=0x7fb50006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:00.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.963+0000 7fb4fe7fc640 1 -- 172.21.15.5:0/1545440937 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5000ca7e0 con 0x7fb50006d4f0 2024-03-24T10:57:00.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.965+0000 7fb4fe7fc640 1 --2- 172.21.15.5:0/1545440937 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb50006d4f0 0x7fb5000cc560 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb5000cd600 tx=0x7fb4f402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=871f6bdf234aa19b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:00.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.969+0000 7fb4fdffb640 1 -- 172.21.15.5:0/1545440937 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f4030860 con 0x7fb50006d4f0 2024-03-24T10:57:00.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.969+0000 7fb4fdffb640 1 -- 172.21.15.5:0/1545440937 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4f40309c0 con 0x7fb50006d4f0 2024-03-24T10:57:00.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.969+0000 7fb4fdffb640 1 -- 172.21.15.5:0/1545440937 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f4030d50 con 0x7fb50006d4f0 2024-03-24T10:57:00.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.970+0000 7fb5056d0640 1 -- 172.21.15.5:0/1545440937 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb50006d4f0 msgr2=0x7fb5000cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:00.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.970+0000 7fb5056d0640 1 --2- 172.21.15.5:0/1545440937 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb50006d4f0 0x7fb5000cc560 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb5000cd600 tx=0x7fb4f402b9a0 comp rx=0 tx=0).stop 2024-03-24T10:57:00.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.970+0000 7fb5056d0640 1 -- 172.21.15.5:0/1545440937 shutdown_connections 2024-03-24T10:57:00.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.970+0000 7fb5056d0640 1 --2- 172.21.15.5:0/1545440937 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb50006d4f0 0x7fb5000cc560 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:00.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.970+0000 7fb5056d0640 1 --2- 172.21.15.5:0/1545440937 >> v2:172.21.15.57:3300/0 conn(0x7fb5001543a0 0x7fb50015ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:00.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.970+0000 7fb5056d0640 1 -- 172.21.15.5:0/1545440937 >> 172.21.15.5:0/1545440937 conn(0x7fb500072ef0 msgr2=0x7fb500061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:00.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.970+0000 7fb5056d0640 1 -- 172.21.15.5:0/1545440937 shutdown_connections 2024-03-24T10:57:00.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.970+0000 7fb5056d0640 1 -- 172.21.15.5:0/1545440937 wait complete. 2024-03-24T10:57:00.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.972+0000 7fb5056d0640 1 Processor -- start 2024-03-24T10:57:00.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.972+0000 7fb5056d0640 1 -- start start 2024-03-24T10:57:00.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.973+0000 7fb5056d0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb50006d4f0 0x7fb5000cf060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:00.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.973+0000 7fb5056d0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5000ccaa0 0x7fb5001f3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:00.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.973+0000 7fb5056d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb5001543a0 0x7fb500148000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:00.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.973+0000 7fb5056d0640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fb500155800 con 0x7fb5000ccaa0 2024-03-24T10:57:00.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.973+0000 7fb5056d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb5001f4110 con 0x7fb5001543a0 2024-03-24T10:57:00.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.973+0000 7fb5056d0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fb500151220 con 0x7fb50006d4f0 2024-03-24T10:57:00.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.974+0000 7fb4feffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb50006d4f0 0x7fb5000cf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:00.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.974+0000 7fb4ff7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb5001543a0 0x7fb500148000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:00.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.974+0000 7fb4fe7fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5000ccaa0 0x7fb5001f3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:00.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.974+0000 7fb4feffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb50006d4f0 0x7fb5000cf060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:37368/0 (socket says 172.21.15.5:37368) 2024-03-24T10:57:00.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.974+0000 7fb4fe7fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5000ccaa0 0x7fb5001f3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:43318/0 (socket says 172.21.15.5:43318) 2024-03-24T10:57:00.974 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.974+0000 7fb4feffd640 1 -- 172.21.15.5:0/2914677414 learned_addr learned my addr 172.21.15.5:0/2914677414 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:00.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.977+0000 7fb4fe7fc640 1 -- 172.21.15.5:0/2914677414 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb50006d4f0 msgr2=0x7fb5000cf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:00.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.977+0000 7fb4fe7fc640 1 --2- 172.21.15.5:0/2914677414 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb50006d4f0 0x7fb5000cf060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:00.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.977+0000 7fb4fe7fc640 1 -- 172.21.15.5:0/2914677414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb5001543a0 msgr2=0x7fb500148000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:00.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.977+0000 7fb4fe7fc640 1 --2- 172.21.15.5:0/2914677414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb5001543a0 0x7fb500148000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:00.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.977+0000 7fb4fe7fc640 1 -- 172.21.15.5:0/2914677414 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f40094d0 con 0x7fb5000ccaa0 2024-03-24T10:57:00.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.980+0000 7fb4fe7fc640 1 --2- 172.21.15.5:0/2914677414 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5000ccaa0 0x7fb5001f3bd0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4033000 tx=0x7fb4f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:00.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.983+0000 7fb4e77fe640 1 -- 172.21.15.5:0/2914677414 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f4039c90 con 0x7fb5000ccaa0 2024-03-24T10:57:00.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.983+0000 7fb4e77fe640 1 -- 172.21.15.5:0/2914677414 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4f4039df0 con 0x7fb5000ccaa0 2024-03-24T10:57:00.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.983+0000 7fb4e77fe640 1 -- 172.21.15.5:0/2914677414 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f4043640 con 0x7fb5000ccaa0 2024-03-24T10:57:00.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.983+0000 7fb5056d0640 1 -- 172.21.15.5:0/2914677414 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb500150180 con 0x7fb5000ccaa0 2024-03-24T10:57:00.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.983+0000 7fb5056d0640 1 -- 172.21.15.5:0/2914677414 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb500150740 con 0x7fb5000ccaa0 2024-03-24T10:57:00.984 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:00.985+0000 7fb5056d0640 1 -- 172.21.15.5:0/2914677414 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5000681d0 con 0x7fb5000ccaa0 2024-03-24T10:57:01.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:01.267+0000 7fb4e77fe640 1 -- 172.21.15.5:0/2914677414 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb4f40437a0 con 0x7fb5000ccaa0 2024-03-24T10:57:01.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:01.268+0000 7fb4e77fe640 1 --2- 172.21.15.5:0/2914677414 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb4ec06b890 0x7fb4ec06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:01.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:01.268+0000 7fb4feffd640 1 --2- 172.21.15.5:0/2914677414 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb4ec06b890 0x7fb4ec06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:01.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:01.268+0000 7fb4feffd640 1 --2- 172.21.15.5:0/2914677414 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb4ec06b890 0x7fb4ec06dd50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009b60 tx=0x7fb4f0009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:01.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:01.283+0000 7fb4e77fe640 1 -- 172.21.15.5:0/2914677414 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fb4f403e070 con 0x7fb5000ccaa0 2024-03-24T10:57:01.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:01.283+0000 7fb4e77fe640 1 -- 172.21.15.5:0/2914677414 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb4f40d9e10 con 0x7fb5000ccaa0 2024-03-24T10:57:01.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:01.525+0000 7fb5056d0640 1 -- 172.21.15.5:0/2914677414 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb500061f60 con 0x7fb5000ccaa0 2024-03-24T10:57:01.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:01.573+0000 7fb4e77fe640 1 -- 172.21.15.5:0/2914677414 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7fb4f4034070 con 0x7fb5000ccaa0 2024-03-24T10:57:01.574 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:01.574 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:57:01.574 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:57:01.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:01.576+0000 7fb5056d0640 1 -- 172.21.15.5:0/2914677414 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb4ec06b890 msgr2=0x7fb4ec06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:01.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:01.576+0000 7fb5056d0640 1 --2- 172.21.15.5:0/2914677414 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb4ec06b890 0x7fb4ec06dd50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009b60 tx=0x7fb4f0009190 comp rx=0 tx=0).stop 2024-03-24T10:57:01.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:01.576+0000 7fb5056d0640 1 -- 172.21.15.5:0/2914677414 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5000ccaa0 msgr2=0x7fb5001f3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:01.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:01.576+0000 7fb5056d0640 1 --2- 172.21.15.5:0/2914677414 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5000ccaa0 0x7fb5001f3bd0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fb4f4033000 tx=0x7fb4f4004730 comp rx=0 tx=0).stop 2024-03-24T10:57:01.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:01.576+0000 7fb5056d0640 1 -- 172.21.15.5:0/2914677414 shutdown_connections 2024-03-24T10:57:01.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:01.576+0000 7fb5056d0640 1 --2- 172.21.15.5:0/2914677414 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb4ec06b890 0x7fb4ec06dd50 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:01.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:01.576+0000 7fb5056d0640 1 --2- 172.21.15.5:0/2914677414 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb5000ccaa0 0x7fb5001f3bd0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:01.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:01.576+0000 7fb5056d0640 1 --2- 172.21.15.5:0/2914677414 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb5001543a0 0x7fb500148000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:01.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:01.576+0000 7fb5056d0640 1 --2- 172.21.15.5:0/2914677414 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb50006d4f0 0x7fb5000cf060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:01.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:01.576+0000 7fb5056d0640 1 -- 172.21.15.5:0/2914677414 >> 172.21.15.5:0/2914677414 conn(0x7fb500072ef0 msgr2=0x7fb500153170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:01.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:01.576+0000 7fb5056d0640 1 -- 172.21.15.5:0/2914677414 shutdown_connections 2024-03-24T10:57:01.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:01.576+0000 7fb5056d0640 1 -- 172.21.15.5:0/2914677414 wait complete. 2024-03-24T10:57:02.196 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:57:02.196 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:02.814 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:57:02.817 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:57:03.197 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:03.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.300+0000 7f1156047640 1 Processor -- start 2024-03-24T10:57:03.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.300+0000 7f1156047640 1 -- start start 2024-03-24T10:57:03.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.301+0000 7f1156047640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f11500cb270 0x7f11500cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:03.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.301+0000 7f1156047640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11500ccba0 con 0x7f11500cae90 2024-03-24T10:57:03.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.301+0000 7f1156047640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11500cce40 con 0x7f11500cd960 2024-03-24T10:57:03.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.301+0000 7f1156047640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f11500cc9b0 con 0x7f11500cb270 2024-03-24T10:57:03.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.301+0000 7f114effd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f11500cb270 0x7f11500cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:03.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.302+0000 7f114f7fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f11500cae90 0x7f11500cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:33048/0 (socket says 172.21.15.5:33048) 2024-03-24T10:57:03.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.302+0000 7f114f7fe640 1 -- 172.21.15.5:0/774351710 learned_addr learned my addr 172.21.15.5:0/774351710 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:57:03.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.305+0000 7f114e7fc640 1 -- 172.21.15.5:0/774351710 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1665731258 0 0) 0x7f11500ccba0 con 0x7f11500cae90 2024-03-24T10:57:03.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.305+0000 7f114e7fc640 1 -- 172.21.15.5:0/774351710 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1138003520 con 0x7f11500cae90 2024-03-24T10:57:03.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.305+0000 7f114e7fc640 1 -- 172.21.15.5:0/774351710 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2577574269 0 0) 0x7f11500cce40 con 0x7f11500cd960 2024-03-24T10:57:03.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.305+0000 7f114e7fc640 1 -- 172.21.15.5:0/774351710 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11500ccba0 con 0x7f11500cd960 2024-03-24T10:57:03.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.306+0000 7f114effd640 1 -- 172.21.15.5:0/774351710 >> v1:172.21.15.57:6789/0 conn(0x7f11500cd960 legacy=0x7f11500cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:03.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.306+0000 7f114effd640 1 -- 172.21.15.5:0/774351710 >> v1:172.21.15.5:6789/0 conn(0x7f11500cae90 legacy=0x7f11500cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:03.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.306+0000 7f114effd640 1 -- 172.21.15.5:0/774351710 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f115015a590 con 0x7f11500cb270 2024-03-24T10:57:03.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.309+0000 7f114effd640 1 --2- 172.21.15.5:0/774351710 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f11500cb270 0x7f11500cf530 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f11440097d0 tx=0x7f114402b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea3ca81a3f91a7eb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:03.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.314+0000 7f114e7fc640 1 -- 172.21.15.5:0/774351710 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1144030880 con 0x7f11500cb270 2024-03-24T10:57:03.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.314+0000 7f114e7fc640 1 -- 172.21.15.5:0/774351710 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11440309e0 con 0x7f11500cb270 2024-03-24T10:57:03.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.314+0000 7f114e7fc640 1 -- 172.21.15.5:0/774351710 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1144030d70 con 0x7f11500cb270 2024-03-24T10:57:03.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.314+0000 7f1156047640 1 -- 172.21.15.5:0/774351710 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f11500cb270 msgr2=0x7f11500cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:03.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.314+0000 7f1156047640 1 --2- 172.21.15.5:0/774351710 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f11500cb270 0x7f11500cf530 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f11440097d0 tx=0x7f114402b9c0 comp rx=0 tx=0).stop 2024-03-24T10:57:03.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.314+0000 7f1156047640 1 -- 172.21.15.5:0/774351710 shutdown_connections 2024-03-24T10:57:03.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.314+0000 7f1156047640 1 --2- 172.21.15.5:0/774351710 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f11500cb270 0x7f11500cf530 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:03.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.315+0000 7f1156047640 1 -- 172.21.15.5:0/774351710 >> 172.21.15.5:0/774351710 conn(0x7f11500cfb80 msgr2=0x7f11500cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:03.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.315+0000 7f1156047640 1 -- 172.21.15.5:0/774351710 shutdown_connections 2024-03-24T10:57:03.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.315+0000 7f1156047640 1 -- 172.21.15.5:0/774351710 wait complete. 2024-03-24T10:57:03.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.316+0000 7f1156047640 1 Processor -- start 2024-03-24T10:57:03.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.316+0000 7f1156047640 1 -- start start 2024-03-24T10:57:03.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.317+0000 7f1156047640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11500cae90 0x7f11500c6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:03.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.317+0000 7f1156047640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11500cb270 0x7f11500b0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:03.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.317+0000 7f1156047640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f11500cd960 0x7f115015d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:03.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.317+0000 7f1156047640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f11500b2750 con 0x7f11500cb270 2024-03-24T10:57:03.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.317+0000 7f1156047640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f11501553b0 con 0x7f11500cae90 2024-03-24T10:57:03.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.317+0000 7f1156047640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f115015cdf0 con 0x7f11500cd960 2024-03-24T10:57:03.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.317+0000 7f114effd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11500cb270 0x7f11500b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:03.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.317+0000 7f114ffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f11500cd960 0x7f115015d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:03.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.317+0000 7f114f7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11500cae90 0x7f11500c6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:03.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.318+0000 7f114effd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11500cb270 0x7f11500b0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36142/0 (socket says 172.21.15.5:36142) 2024-03-24T10:57:03.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.318+0000 7f114effd640 1 -- 172.21.15.5:0/1751518358 learned_addr learned my addr 172.21.15.5:0/1751518358 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:03.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.321+0000 7f114effd640 1 -- 172.21.15.5:0/1751518358 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f11500cd960 msgr2=0x7f115015d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:03.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.321+0000 7f114effd640 1 --2- 172.21.15.5:0/1751518358 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f11500cd960 0x7f115015d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:03.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.321+0000 7f114effd640 1 -- 172.21.15.5:0/1751518358 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11500cae90 msgr2=0x7f11500c6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:03.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.321+0000 7f114effd640 1 --2- 172.21.15.5:0/1751518358 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11500cae90 0x7f11500c6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:03.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.321+0000 7f114effd640 1 -- 172.21.15.5:0/1751518358 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1144009400 con 0x7f11500cb270 2024-03-24T10:57:03.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.324+0000 7f114effd640 1 --2- 172.21.15.5:0/1751518358 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11500cb270 0x7f11500b0860 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f114402bfd0 tx=0x7f1144004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:03.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.328+0000 7f1137fff640 1 -- 172.21.15.5:0/1751518358 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114403e040 con 0x7f11500cb270 2024-03-24T10:57:03.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.328+0000 7f1137fff640 1 -- 172.21.15.5:0/1751518358 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1144039cd0 con 0x7f11500cb270 2024-03-24T10:57:03.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.328+0000 7f1156047640 1 -- 172.21.15.5:0/1751518358 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11501f30a0 con 0x7f11500cb270 2024-03-24T10:57:03.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.328+0000 7f1137fff640 1 -- 172.21.15.5:0/1751518358 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1144043680 con 0x7f11500cb270 2024-03-24T10:57:03.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.328+0000 7f1156047640 1 -- 172.21.15.5:0/1751518358 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11501f38e0 con 0x7f11500cb270 2024-03-24T10:57:03.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.329+0000 7f1156047640 1 -- 172.21.15.5:0/1751518358 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11500681d0 con 0x7f11500cb270 2024-03-24T10:57:03.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.350+0000 7f1137fff640 1 -- 172.21.15.5:0/1751518358 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f11440437e0 con 0x7f11500cb270 2024-03-24T10:57:03.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.350+0000 7f1137fff640 1 --2- 172.21.15.5:0/1751518358 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f113c06b810 0x7f113c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:03.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.350+0000 7f114f7fe640 1 --2- 172.21.15.5:0/1751518358 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f113c06b810 0x7f113c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:03.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.351+0000 7f114f7fe640 1 --2- 172.21.15.5:0/1751518358 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f113c06b810 0x7f113c06dcd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f11400012e0 tx=0x7f11400019f0 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:03.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.353+0000 7f1137fff640 1 -- 172.21.15.5:0/1751518358 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f114407b720 con 0x7f11500cb270 2024-03-24T10:57:03.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.612+0000 7f1137fff640 1 -- 172.21.15.5:0/1751518358 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f11440779c0 con 0x7f11500cb270 2024-03-24T10:57:03.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.862+0000 7f1156047640 1 -- 172.21.15.5:0/1751518358 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11500c8f60 con 0x7f11500cb270 2024-03-24T10:57:03.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.875+0000 7f1137fff640 1 -- 172.21.15.5:0/1751518358 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f1144034090 con 0x7f11500cb270 2024-03-24T10:57:03.876 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:03.876 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:57:03.876 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:57:03.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.877+0000 7f1156047640 1 -- 172.21.15.5:0/1751518358 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f113c06b810 msgr2=0x7f113c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:03.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.877+0000 7f1156047640 1 --2- 172.21.15.5:0/1751518358 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f113c06b810 0x7f113c06dcd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f11400012e0 tx=0x7f11400019f0 comp rx=0 tx=0).stop 2024-03-24T10:57:03.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.877+0000 7f1156047640 1 -- 172.21.15.5:0/1751518358 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11500cb270 msgr2=0x7f11500b0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:03.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.877+0000 7f1156047640 1 --2- 172.21.15.5:0/1751518358 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11500cb270 0x7f11500b0860 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f114402bfd0 tx=0x7f1144004730 comp rx=0 tx=0).stop 2024-03-24T10:57:03.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.877+0000 7f1156047640 1 -- 172.21.15.5:0/1751518358 shutdown_connections 2024-03-24T10:57:03.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.877+0000 7f1156047640 1 --2- 172.21.15.5:0/1751518358 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f113c06b810 0x7f113c06dcd0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:03.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.877+0000 7f1156047640 1 --2- 172.21.15.5:0/1751518358 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f11500cb270 0x7f11500b0860 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:03.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.877+0000 7f1156047640 1 --2- 172.21.15.5:0/1751518358 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f11500cd960 0x7f115015d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:03.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.877+0000 7f1156047640 1 --2- 172.21.15.5:0/1751518358 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11500cae90 0x7f11500c6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:03.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.877+0000 7f1156047640 1 -- 172.21.15.5:0/1751518358 >> 172.21.15.5:0/1751518358 conn(0x7f11500cfb80 msgr2=0x7f115006bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:03.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.878+0000 7f1156047640 1 -- 172.21.15.5:0/1751518358 shutdown_connections 2024-03-24T10:57:03.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:03.878+0000 7f1156047640 1 -- 172.21.15.5:0/1751518358 wait complete. 2024-03-24T10:57:04.493 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:57:04.494 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:05.494 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:05.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.598+0000 7fd7c19f4640 1 Processor -- start 2024-03-24T10:57:05.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.598+0000 7fd7c19f4640 1 -- start start 2024-03-24T10:57:05.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.599+0000 7fd7c19f4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd7bc15ac00 0x7fd7bc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:05.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.599+0000 7fd7c19f4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7bc0cb2f0 con 0x7fd7bc0ccaa0 2024-03-24T10:57:05.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.599+0000 7fd7c19f4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7bc15b070 con 0x7fd7bc1543a0 2024-03-24T10:57:05.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.599+0000 7fd7c19f4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fd7bc0c6750 con 0x7fd7bc15ac00 2024-03-24T10:57:05.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.600+0000 7fd7bb7fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fd7bc0ccaa0 0x7fd7bc06bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:33054/0 (socket says 172.21.15.5:33054) 2024-03-24T10:57:05.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.600+0000 7fd7bb7fe640 1 -- 172.21.15.5:0/4191166703 learned_addr learned my addr 172.21.15.5:0/4191166703 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:57:05.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.600+0000 7fd7b3fff640 1 --2- 172.21.15.5:0/4191166703 >> v2:172.21.15.57:3300/0 conn(0x7fd7bc15ac00 0x7fd7bc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:05.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.603+0000 7fd7ba7fc640 1 -- 172.21.15.5:0/4191166703 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1520092608 0 0) 0x7fd7bc0cb2f0 con 0x7fd7bc0ccaa0 2024-03-24T10:57:05.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.603+0000 7fd7ba7fc640 1 -- 172.21.15.5:0/4191166703 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7a8003520 con 0x7fd7bc0ccaa0 2024-03-24T10:57:05.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.605+0000 7fd7b3fff640 1 -- 172.21.15.5:0/4191166703 >> v1:172.21.15.57:6789/0 conn(0x7fd7bc1543a0 legacy=0x7fd7bc06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:05.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.605+0000 7fd7b3fff640 1 -- 172.21.15.5:0/4191166703 >> v1:172.21.15.5:6789/0 conn(0x7fd7bc0ccaa0 legacy=0x7fd7bc06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:05.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.605+0000 7fd7b3fff640 1 -- 172.21.15.5:0/4191166703 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7bc0ca7e0 con 0x7fd7bc15ac00 2024-03-24T10:57:05.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.610+0000 7fd7b3fff640 1 --2- 172.21.15.5:0/4191166703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7bc15ac00 0x7fd7bc0cc560 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40097a0 tx=0x7fd7a402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc6cd8c6aa674cfe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:05.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.615+0000 7fd7ba7fc640 1 -- 172.21.15.5:0/4191166703 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a4030610 con 0x7fd7bc15ac00 2024-03-24T10:57:05.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.615+0000 7fd7ba7fc640 1 -- 172.21.15.5:0/4191166703 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7a4030770 con 0x7fd7bc15ac00 2024-03-24T10:57:05.615 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.615+0000 7fd7ba7fc640 1 -- 172.21.15.5:0/4191166703 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a4030b00 con 0x7fd7bc15ac00 2024-03-24T10:57:05.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.616+0000 7fd7c19f4640 1 -- 172.21.15.5:0/4191166703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7bc15ac00 msgr2=0x7fd7bc0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:05.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.616+0000 7fd7c19f4640 1 --2- 172.21.15.5:0/4191166703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7bc15ac00 0x7fd7bc0cc560 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40097a0 tx=0x7fd7a402b960 comp rx=0 tx=0).stop 2024-03-24T10:57:05.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.616+0000 7fd7c19f4640 1 -- 172.21.15.5:0/4191166703 shutdown_connections 2024-03-24T10:57:05.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.616+0000 7fd7c19f4640 1 --2- 172.21.15.5:0/4191166703 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7bc15ac00 0x7fd7bc0cc560 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:05.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.616+0000 7fd7c19f4640 1 -- 172.21.15.5:0/4191166703 >> 172.21.15.5:0/4191166703 conn(0x7fd7bc072ef0 msgr2=0x7fd7bc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:05.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.616+0000 7fd7c19f4640 1 -- 172.21.15.5:0/4191166703 shutdown_connections 2024-03-24T10:57:05.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.616+0000 7fd7c19f4640 1 -- 172.21.15.5:0/4191166703 wait complete. 2024-03-24T10:57:05.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.617+0000 7fd7c19f4640 1 Processor -- start 2024-03-24T10:57:05.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.617+0000 7fd7c19f4640 1 -- start start 2024-03-24T10:57:05.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.618+0000 7fd7c19f4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7bc0ccaa0 0x7fd7bc1ef730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:05.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.618+0000 7fd7c19f4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7bc1543a0 0x7fd7bc15cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:05.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.618+0000 7fd7c19f4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd7bc15ac00 0x7fd7bc0b4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:05.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.618+0000 7fd7c19f4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd7bc15caf0 con 0x7fd7bc0ccaa0 2024-03-24T10:57:05.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.618+0000 7fd7c19f4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd7bc15c1c0 con 0x7fd7bc1543a0 2024-03-24T10:57:05.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.618+0000 7fd7c19f4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd7bc15b920 con 0x7fd7bc15ac00 2024-03-24T10:57:05.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.618+0000 7fd7baffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7bc0ccaa0 0x7fd7bc1ef730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:05.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.618+0000 7fd7b3fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7bc1543a0 0x7fd7bc15cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:05.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.619+0000 7fd7bb7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd7bc15ac00 0x7fd7bc0b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:05.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.619+0000 7fd7baffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7bc0ccaa0 0x7fd7bc1ef730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:36156/0 (socket says 172.21.15.5:36156) 2024-03-24T10:57:05.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.619+0000 7fd7baffd640 1 -- 172.21.15.5:0/3611534048 learned_addr learned my addr 172.21.15.5:0/3611534048 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:05.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.622+0000 7fd7baffd640 1 -- 172.21.15.5:0/3611534048 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd7bc15ac00 msgr2=0x7fd7bc0b4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:05.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.622+0000 7fd7baffd640 1 --2- 172.21.15.5:0/3611534048 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd7bc15ac00 0x7fd7bc0b4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:05.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.622+0000 7fd7baffd640 1 -- 172.21.15.5:0/3611534048 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7bc1543a0 msgr2=0x7fd7bc15cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:05.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.622+0000 7fd7baffd640 1 --2- 172.21.15.5:0/3611534048 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7bc1543a0 0x7fd7bc15cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:05.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.622+0000 7fd7baffd640 1 -- 172.21.15.5:0/3611534048 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a4009400 con 0x7fd7bc0ccaa0 2024-03-24T10:57:05.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.622+0000 7fd7b3fff640 1 --2- 172.21.15.5:0/3611534048 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7bc1543a0 0x7fd7bc15cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:57:05.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.625+0000 7fd7baffd640 1 --2- 172.21.15.5:0/3611534048 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7bc0ccaa0 0x7fd7bc1ef730 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00c430 tx=0x7fd7ac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:05.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.630+0000 7fd7b8ff9640 1 -- 172.21.15.5:0/3611534048 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac0052d0 con 0x7fd7bc0ccaa0 2024-03-24T10:57:05.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.630+0000 7fd7b8ff9640 1 -- 172.21.15.5:0/3611534048 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7ac005450 con 0x7fd7bc0ccaa0 2024-03-24T10:57:05.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.630+0000 7fd7b8ff9640 1 -- 172.21.15.5:0/3611534048 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac007260 con 0x7fd7bc0ccaa0 2024-03-24T10:57:05.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.630+0000 7fd7c19f4640 1 -- 172.21.15.5:0/3611534048 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7bc15bc00 con 0x7fd7bc0ccaa0 2024-03-24T10:57:05.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.630+0000 7fd7c19f4640 1 -- 172.21.15.5:0/3611534048 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7bc15c710 con 0x7fd7bc0ccaa0 2024-03-24T10:57:05.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.631+0000 7fd7c19f4640 1 -- 172.21.15.5:0/3611534048 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7bc0681d0 con 0x7fd7bc0ccaa0 2024-03-24T10:57:05.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.661+0000 7fd7b8ff9640 1 -- 172.21.15.5:0/3611534048 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd7ac0073c0 con 0x7fd7bc0ccaa0 2024-03-24T10:57:05.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.661+0000 7fd7b8ff9640 1 --2- 172.21.15.5:0/3611534048 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd79c06b860 0x7fd79c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:05.661 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.662+0000 7fd7b3fff640 1 --2- 172.21.15.5:0/3611534048 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd79c06b860 0x7fd79c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:05.662 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.662+0000 7fd7b3fff640 1 --2- 172.21.15.5:0/3611534048 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd79c06b860 0x7fd79c06dd20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4036000 tx=0x7fd7a403b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:05.674 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.674+0000 7fd7b8ff9640 1 -- 172.21.15.5:0/3611534048 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fd7ac007670 con 0x7fd7bc0ccaa0 2024-03-24T10:57:05.955 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:05.955+0000 7fd7b8ff9640 1 -- 172.21.15.5:0/3611534048 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd7ac054550 con 0x7fd7bc0ccaa0 2024-03-24T10:57:06.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:06.184+0000 7fd7c19f4640 1 -- 172.21.15.5:0/3611534048 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7bc151b30 con 0x7fd7bc0ccaa0 2024-03-24T10:57:06.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:06.197+0000 7fd7b8ff9640 1 -- 172.21.15.5:0/3611534048 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7fd7ac057c10 con 0x7fd7bc0ccaa0 2024-03-24T10:57:06.197 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:06.198 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:57:06.198 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:57:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:06.199+0000 7fd7c19f4640 1 -- 172.21.15.5:0/3611534048 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd79c06b860 msgr2=0x7fd79c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:06.199+0000 7fd7c19f4640 1 --2- 172.21.15.5:0/3611534048 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd79c06b860 0x7fd79c06dd20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4036000 tx=0x7fd7a403b040 comp rx=0 tx=0).stop 2024-03-24T10:57:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:06.199+0000 7fd7c19f4640 1 -- 172.21.15.5:0/3611534048 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7bc0ccaa0 msgr2=0x7fd7bc1ef730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:06.199+0000 7fd7c19f4640 1 --2- 172.21.15.5:0/3611534048 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7bc0ccaa0 0x7fd7bc1ef730 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac00c430 tx=0x7fd7ac00c8f0 comp rx=0 tx=0).stop 2024-03-24T10:57:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:06.200+0000 7fd7c19f4640 1 -- 172.21.15.5:0/3611534048 shutdown_connections 2024-03-24T10:57:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:06.200+0000 7fd7c19f4640 1 --2- 172.21.15.5:0/3611534048 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd79c06b860 0x7fd79c06dd20 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:06.200+0000 7fd7c19f4640 1 --2- 172.21.15.5:0/3611534048 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd7bc15ac00 0x7fd7bc0b4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:06.200+0000 7fd7c19f4640 1 --2- 172.21.15.5:0/3611534048 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd7bc1543a0 0x7fd7bc15cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:06.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:06.200+0000 7fd7c19f4640 1 --2- 172.21.15.5:0/3611534048 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd7bc0ccaa0 0x7fd7bc1ef730 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:06.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:06.200+0000 7fd7c19f4640 1 -- 172.21.15.5:0/3611534048 >> 172.21.15.5:0/3611534048 conn(0x7fd7bc072ef0 msgr2=0x7fd7bc14e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:06.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:06.200+0000 7fd7c19f4640 1 -- 172.21.15.5:0/3611534048 shutdown_connections 2024-03-24T10:57:06.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:06.200+0000 7fd7c19f4640 1 -- 172.21.15.5:0/3611534048 wait complete. 2024-03-24T10:57:06.815 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:57:06.815 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:07.816 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:07.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.918+0000 7f0ae582e640 1 Processor -- start 2024-03-24T10:57:07.918 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.918+0000 7f0ae582e640 1 -- start start 2024-03-24T10:57:07.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.919+0000 7f0ae582e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0ae015ac00 0x7f0ae00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:07.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.919+0000 7f0ae582e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae00cb2f0 con 0x7f0ae00ccaa0 2024-03-24T10:57:07.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.919+0000 7f0ae582e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae015b070 con 0x7f0ae01543a0 2024-03-24T10:57:07.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.919+0000 7f0ae582e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f0ae00c6750 con 0x7f0ae015ac00 2024-03-24T10:57:07.919 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.919+0000 7f0ad7fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f0ae015ac00 0x7f0ae00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:07.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.920+0000 7f0adf7fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f0ae00ccaa0 0x7f0ae006bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:33070/0 (socket says 172.21.15.5:33070) 2024-03-24T10:57:07.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.920+0000 7f0adf7fe640 1 -- 172.21.15.5:0/3910828994 learned_addr learned my addr 172.21.15.5:0/3910828994 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:57:07.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.922+0000 7f0ade7fc640 1 -- 172.21.15.5:0/3910828994 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 145268966 0 0) 0x7f0ae00cb2f0 con 0x7f0ae00ccaa0 2024-03-24T10:57:07.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.922+0000 7f0ade7fc640 1 -- 172.21.15.5:0/3910828994 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0acc003520 con 0x7f0ae00ccaa0 2024-03-24T10:57:07.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.923+0000 7f0ad7fff640 1 -- 172.21.15.5:0/3910828994 >> v1:172.21.15.57:6789/0 conn(0x7f0ae01543a0 legacy=0x7f0ae006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:07.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.923+0000 7f0ad7fff640 1 -- 172.21.15.5:0/3910828994 >> v1:172.21.15.5:6789/0 conn(0x7f0ae00ccaa0 legacy=0x7f0ae006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:07.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.923+0000 7f0ad7fff640 1 -- 172.21.15.5:0/3910828994 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae00ca7e0 con 0x7f0ae015ac00 2024-03-24T10:57:07.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.929+0000 7f0ad7fff640 1 --2- 172.21.15.5:0/3910828994 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ae015ac00 0x7f0ae00cc560 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0ac80097a0 tx=0x7f0ac802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c24657c22740fbd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:07.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.935+0000 7f0ade7fc640 1 -- 172.21.15.5:0/3910828994 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8031830 con 0x7f0ae015ac00 2024-03-24T10:57:07.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.935+0000 7f0ade7fc640 1 -- 172.21.15.5:0/3910828994 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ac8031990 con 0x7f0ae015ac00 2024-03-24T10:57:07.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.935+0000 7f0ade7fc640 1 -- 172.21.15.5:0/3910828994 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8031d20 con 0x7f0ae015ac00 2024-03-24T10:57:07.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.935+0000 7f0ae582e640 1 -- 172.21.15.5:0/3910828994 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ae015ac00 msgr2=0x7f0ae00cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:07.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.935+0000 7f0ae582e640 1 --2- 172.21.15.5:0/3910828994 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ae015ac00 0x7f0ae00cc560 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0ac80097a0 tx=0x7f0ac802b8d0 comp rx=0 tx=0).stop 2024-03-24T10:57:07.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.935+0000 7f0ae582e640 1 -- 172.21.15.5:0/3910828994 shutdown_connections 2024-03-24T10:57:07.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.935+0000 7f0ae582e640 1 --2- 172.21.15.5:0/3910828994 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ae015ac00 0x7f0ae00cc560 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:07.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.935+0000 7f0ae582e640 1 -- 172.21.15.5:0/3910828994 >> 172.21.15.5:0/3910828994 conn(0x7f0ae0072ef0 msgr2=0x7f0ae0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:07.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.935+0000 7f0ae582e640 1 -- 172.21.15.5:0/3910828994 shutdown_connections 2024-03-24T10:57:07.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.936+0000 7f0ae582e640 1 -- 172.21.15.5:0/3910828994 wait complete. 2024-03-24T10:57:07.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.937+0000 7f0ae582e640 1 Processor -- start 2024-03-24T10:57:07.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.937+0000 7f0ae582e640 1 -- start start 2024-03-24T10:57:07.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.938+0000 7f0ae582e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ae00ccaa0 0x7f0ae014a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:07.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.938+0000 7f0ae582e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ae01543a0 0x7f0ae01eff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:07.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.938+0000 7f0ae582e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0ae015ac00 0x7f0ae01f0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:07.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.938+0000 7f0ae582e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae014df50 con 0x7f0ae00ccaa0 2024-03-24T10:57:07.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.938+0000 7f0ae582e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae015bdf0 con 0x7f0ae01543a0 2024-03-24T10:57:07.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.938+0000 7f0ae582e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae014dcb0 con 0x7f0ae015ac00 2024-03-24T10:57:07.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.938+0000 7f0adeffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ae00ccaa0 0x7f0ae014a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:07.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.938+0000 7f0adf7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0ae015ac00 0x7f0ae01f0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:07.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.938+0000 7f0ad7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ae01543a0 0x7f0ae01eff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:07.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.939+0000 7f0adf7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0ae015ac00 0x7f0ae01f0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:48386/0 (socket says 172.21.15.5:48386) 2024-03-24T10:57:07.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.939+0000 7f0adf7fe640 1 -- 172.21.15.5:0/4281489626 learned_addr learned my addr 172.21.15.5:0/4281489626 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:07.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.942+0000 7f0adf7fe640 1 -- 172.21.15.5:0/4281489626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ae01543a0 msgr2=0x7f0ae01eff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:07.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.942+0000 7f0adf7fe640 1 --2- 172.21.15.5:0/4281489626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ae01543a0 0x7f0ae01eff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:07.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.942+0000 7f0adf7fe640 1 -- 172.21.15.5:0/4281489626 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ae00ccaa0 msgr2=0x7f0ae014a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:07.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.942+0000 7f0adf7fe640 1 --2- 172.21.15.5:0/4281489626 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ae00ccaa0 0x7f0ae014a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:07.943 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.942+0000 7f0adf7fe640 1 -- 172.21.15.5:0/4281489626 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac8009400 con 0x7f0ae015ac00 2024-03-24T10:57:07.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.945+0000 7f0adf7fe640 1 --2- 172.21.15.5:0/4281489626 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0ae015ac00 0x7f0ae01f0470 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0ad800cb70 tx=0x7f0ad800a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:07.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.950+0000 7f0adcff9640 1 -- 172.21.15.5:0/4281489626 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad800ad40 con 0x7f0ae015ac00 2024-03-24T10:57:07.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.950+0000 7f0adcff9640 1 -- 172.21.15.5:0/4281489626 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ad8012400 con 0x7f0ae015ac00 2024-03-24T10:57:07.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.950+0000 7f0adcff9640 1 -- 172.21.15.5:0/4281489626 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad800aea0 con 0x7f0ae015ac00 2024-03-24T10:57:07.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.950+0000 7f0ae582e640 1 -- 172.21.15.5:0/4281489626 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae015b450 con 0x7f0ae015ac00 2024-03-24T10:57:07.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.950+0000 7f0ae582e640 1 -- 172.21.15.5:0/4281489626 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae015ba10 con 0x7f0ae015ac00 2024-03-24T10:57:07.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.951+0000 7f0ae582e640 1 -- 172.21.15.5:0/4281489626 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ae00681d0 con 0x7f0ae015ac00 2024-03-24T10:57:07.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.975+0000 7f0adcff9640 1 -- 172.21.15.5:0/4281489626 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0ad8007070 con 0x7f0ae015ac00 2024-03-24T10:57:07.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.976+0000 7f0adcff9640 1 --2- 172.21.15.5:0/4281489626 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0ac006b860 0x7f0ac006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:07.976 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.976+0000 7f0adeffd640 1 --2- 172.21.15.5:0/4281489626 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0ac006b860 0x7f0ac006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:07.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.977+0000 7f0adeffd640 1 --2- 172.21.15.5:0/4281489626 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0ac006b860 0x7f0ac006dd20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0004f60 tx=0x7f0ad0007e30 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:07.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:07.979+0000 7f0adcff9640 1 -- 172.21.15.5:0/4281489626 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f0ad8058e60 con 0x7f0ae015ac00 2024-03-24T10:57:08.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:08.268+0000 7f0adcff9640 1 -- 172.21.15.5:0/4281489626 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0ad8055100 con 0x7f0ae015ac00 2024-03-24T10:57:08.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:08.495+0000 7f0ae582e640 1 -- 172.21.15.5:0/4281489626 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ae014caa0 con 0x7f0ae015ac00 2024-03-24T10:57:08.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:08.516+0000 7f0adcff9640 1 -- 172.21.15.5:0/4281489626 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f0ad80587c0 con 0x7f0ae015ac00 2024-03-24T10:57:08.517 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:08.517 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:57:08.517 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:57:08.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:08.519+0000 7f0ae582e640 1 -- 172.21.15.5:0/4281489626 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0ac006b860 msgr2=0x7f0ac006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:08.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:08.519+0000 7f0ae582e640 1 --2- 172.21.15.5:0/4281489626 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0ac006b860 0x7f0ac006dd20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0004f60 tx=0x7f0ad0007e30 comp rx=0 tx=0).stop 2024-03-24T10:57:08.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:08.520+0000 7f0ae582e640 1 -- 172.21.15.5:0/4281489626 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0ae015ac00 msgr2=0x7f0ae01f0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:08.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:08.520+0000 7f0ae582e640 1 --2- 172.21.15.5:0/4281489626 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0ae015ac00 0x7f0ae01f0470 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0ad800cb70 tx=0x7f0ad800a430 comp rx=0 tx=0).stop 2024-03-24T10:57:08.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:08.520+0000 7f0ae582e640 1 -- 172.21.15.5:0/4281489626 shutdown_connections 2024-03-24T10:57:08.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:08.520+0000 7f0ae582e640 1 --2- 172.21.15.5:0/4281489626 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0ac006b860 0x7f0ac006dd20 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:08.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:08.520+0000 7f0ae582e640 1 --2- 172.21.15.5:0/4281489626 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0ae015ac00 0x7f0ae01f0470 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:08.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:08.520+0000 7f0ae582e640 1 --2- 172.21.15.5:0/4281489626 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0ae01543a0 0x7f0ae01eff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:08.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:08.520+0000 7f0ae582e640 1 --2- 172.21.15.5:0/4281489626 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0ae00ccaa0 0x7f0ae014a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:08.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:08.520+0000 7f0ae582e640 1 -- 172.21.15.5:0/4281489626 >> 172.21.15.5:0/4281489626 conn(0x7f0ae0072ef0 msgr2=0x7f0ae0153480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:08.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:08.520+0000 7f0ae582e640 1 -- 172.21.15.5:0/4281489626 shutdown_connections 2024-03-24T10:57:08.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:08.520+0000 7f0ae582e640 1 -- 172.21.15.5:0/4281489626 wait complete. 2024-03-24T10:57:09.136 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:57:09.136 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:10.137 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:10.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.239+0000 7fd38c89d640 1 Processor -- start 2024-03-24T10:57:10.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.239+0000 7fd38c89d640 1 -- start start 2024-03-24T10:57:10.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.240+0000 7fd38c89d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd3881543a0 0x7fd38815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:10.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.240+0000 7fd38c89d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd38806bf50 0x7fd38806c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:10.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.240+0000 7fd38c89d640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3880cb2f0 con 0x7fd38806d4f0 2024-03-24T10:57:10.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.240+0000 7fd38c89d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fd3880c6750 con 0x7fd38806bf50 2024-03-24T10:57:10.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.240+0000 7fd38c89d640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd3880cc380 con 0x7fd3881543a0 2024-03-24T10:57:10.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.241+0000 7fd38659c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd3881543a0 0x7fd38815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:10.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.241+0000 7fd385d9b640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fd38806d4f0 0x7fd3880cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:33086/0 (socket says 172.21.15.5:33086) 2024-03-24T10:57:10.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.241+0000 7fd385d9b640 1 -- 172.21.15.5:0/122341310 learned_addr learned my addr 172.21.15.5:0/122341310 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:57:10.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.241+0000 7fd386d9d640 1 --2- 172.21.15.5:0/122341310 >> v2:172.21.15.57:3300/0 conn(0x7fd38806bf50 0x7fd38806c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:10.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.244+0000 7fd38659c640 1 -- 172.21.15.5:0/122341310 >> v2:172.21.15.57:3300/0 conn(0x7fd38806bf50 msgr2=0x7fd38806c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:10.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.244+0000 7fd38659c640 1 --2- 172.21.15.5:0/122341310 >> v2:172.21.15.57:3300/0 conn(0x7fd38806bf50 0x7fd38806c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:10.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.244+0000 7fd38659c640 1 -- 172.21.15.5:0/122341310 >> v1:172.21.15.5:6789/0 conn(0x7fd38806d4f0 legacy=0x7fd3880cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:10.244 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.244+0000 7fd38659c640 1 -- 172.21.15.5:0/122341310 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3880ca7e0 con 0x7fd3881543a0 2024-03-24T10:57:10.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.246+0000 7fd38659c640 1 --2- 172.21.15.5:0/122341310 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd3881543a0 0x7fd38815ac00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd3780097a0 tx=0x7fd37802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ab9c127478dc51 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:10.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.250+0000 7fd38559a640 1 -- 172.21.15.5:0/122341310 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd378030610 con 0x7fd3881543a0 2024-03-24T10:57:10.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.250+0000 7fd38559a640 1 -- 172.21.15.5:0/122341310 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd378030770 con 0x7fd3881543a0 2024-03-24T10:57:10.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.250+0000 7fd38559a640 1 -- 172.21.15.5:0/122341310 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd378030b00 con 0x7fd3881543a0 2024-03-24T10:57:10.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.250+0000 7fd38c89d640 1 -- 172.21.15.5:0/122341310 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd3881543a0 msgr2=0x7fd38815ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:10.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.250+0000 7fd38c89d640 1 --2- 172.21.15.5:0/122341310 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd3881543a0 0x7fd38815ac00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd3780097a0 tx=0x7fd37802b960 comp rx=0 tx=0).stop 2024-03-24T10:57:10.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.251+0000 7fd38c89d640 1 -- 172.21.15.5:0/122341310 shutdown_connections 2024-03-24T10:57:10.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.251+0000 7fd38c89d640 1 --2- 172.21.15.5:0/122341310 >> v2:172.21.15.57:3300/0 conn(0x7fd38806bf50 0x7fd38806c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:10.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.251+0000 7fd38c89d640 1 --2- 172.21.15.5:0/122341310 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd3881543a0 0x7fd38815ac00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:10.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.251+0000 7fd38c89d640 1 -- 172.21.15.5:0/122341310 >> 172.21.15.5:0/122341310 conn(0x7fd388072ef0 msgr2=0x7fd388061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:10.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.251+0000 7fd38c89d640 1 -- 172.21.15.5:0/122341310 shutdown_connections 2024-03-24T10:57:10.251 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.251+0000 7fd38c89d640 1 -- 172.21.15.5:0/122341310 wait complete. 2024-03-24T10:57:10.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.252+0000 7fd38c89d640 1 Processor -- start 2024-03-24T10:57:10.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.252+0000 7fd38c89d640 1 -- start start 2024-03-24T10:57:10.252 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.253+0000 7fd38c89d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd38806bf50 0x7fd38814b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:10.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.253+0000 7fd38c89d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd38806d4f0 0x7fd38814fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:10.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.253+0000 7fd38c89d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd3881543a0 0x7fd38814a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:10.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.253+0000 7fd38c89d640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd38814bf40 con 0x7fd38806bf50 2024-03-24T10:57:10.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.253+0000 7fd38c89d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd38814c990 con 0x7fd38806d4f0 2024-03-24T10:57:10.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.253+0000 7fd38c89d640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd3880ce990 con 0x7fd3881543a0 2024-03-24T10:57:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.253+0000 7fd386d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd3881543a0 0x7fd38814a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.253+0000 7fd38659c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd38806bf50 0x7fd38814b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.253+0000 7fd385d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd38806d4f0 0x7fd38814fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.253+0000 7fd386d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd3881543a0 0x7fd38814a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:48398/0 (socket says 172.21.15.5:48398) 2024-03-24T10:57:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.253+0000 7fd386d9d640 1 -- 172.21.15.5:0/2490790141 learned_addr learned my addr 172.21.15.5:0/2490790141 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:10.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.259+0000 7fd38659c640 1 -- 172.21.15.5:0/2490790141 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd3881543a0 msgr2=0x7fd38814a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:10.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.259+0000 7fd38659c640 1 --2- 172.21.15.5:0/2490790141 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd3881543a0 0x7fd38814a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:10.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.259+0000 7fd38659c640 1 -- 172.21.15.5:0/2490790141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd38806d4f0 msgr2=0x7fd38814fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:10.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.259+0000 7fd38659c640 1 --2- 172.21.15.5:0/2490790141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd38806d4f0 0x7fd38814fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:10.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.259+0000 7fd38659c640 1 -- 172.21.15.5:0/2490790141 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd378009400 con 0x7fd38806bf50 2024-03-24T10:57:10.261 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.262+0000 7fd38659c640 1 --2- 172.21.15.5:0/2490790141 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd38806bf50 0x7fd38814b430 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd378033000 tx=0x7fd378004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:10.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.267+0000 7fd376ffd640 1 -- 172.21.15.5:0/2490790141 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37803e040 con 0x7fd38806bf50 2024-03-24T10:57:10.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.267+0000 7fd376ffd640 1 -- 172.21.15.5:0/2490790141 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd378039a40 con 0x7fd38806bf50 2024-03-24T10:57:10.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.267+0000 7fd38c89d640 1 -- 172.21.15.5:0/2490790141 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd388157500 con 0x7fd38806bf50 2024-03-24T10:57:10.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.267+0000 7fd376ffd640 1 -- 172.21.15.5:0/2490790141 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd378039df0 con 0x7fd38806bf50 2024-03-24T10:57:10.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.267+0000 7fd38c89d640 1 -- 172.21.15.5:0/2490790141 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3880b47c0 con 0x7fd38806bf50 2024-03-24T10:57:10.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.268+0000 7fd38c89d640 1 -- 172.21.15.5:0/2490790141 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3880b4620 con 0x7fd38806bf50 2024-03-24T10:57:10.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.547+0000 7fd376ffd640 1 -- 172.21.15.5:0/2490790141 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd3780443f0 con 0x7fd38806bf50 2024-03-24T10:57:10.547 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.547+0000 7fd376ffd640 1 --2- 172.21.15.5:0/2490790141 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd36806b860 0x7fd36806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:10.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.547+0000 7fd385d9b640 1 --2- 172.21.15.5:0/2490790141 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd36806b860 0x7fd36806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:10.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.548+0000 7fd385d9b640 1 --2- 172.21.15.5:0/2490790141 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd36806b860 0x7fd36806dd20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd3880ccfd0 tx=0x7fd37c007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:10.577 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.577+0000 7fd376ffd640 1 -- 172.21.15.5:0/2490790141 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fd37807ae90 con 0x7fd38806bf50 2024-03-24T10:57:10.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.578+0000 7fd376ffd640 1 -- 172.21.15.5:0/2490790141 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd378077130 con 0x7fd38806bf50 2024-03-24T10:57:10.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.806+0000 7fd38c89d640 1 -- 172.21.15.5:0/2490790141 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3880b4620 con 0x7fd38806bf50 2024-03-24T10:57:10.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.821+0000 7fd376ffd640 1 -- 172.21.15.5:0/2490790141 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7fd3880b4620 con 0x7fd38806bf50 2024-03-24T10:57:10.821 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:10.821 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:57:10.822 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:57:10.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.823+0000 7fd38c89d640 1 -- 172.21.15.5:0/2490790141 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd36806b860 msgr2=0x7fd36806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.823+0000 7fd38c89d640 1 --2- 172.21.15.5:0/2490790141 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd36806b860 0x7fd36806dd20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd3880ccfd0 tx=0x7fd37c007e80 comp rx=0 tx=0).stop 2024-03-24T10:57:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.823+0000 7fd38c89d640 1 -- 172.21.15.5:0/2490790141 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd38806bf50 msgr2=0x7fd38814b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.823+0000 7fd38c89d640 1 --2- 172.21.15.5:0/2490790141 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd38806bf50 0x7fd38814b430 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd378033000 tx=0x7fd378004730 comp rx=0 tx=0).stop 2024-03-24T10:57:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.823+0000 7fd38c89d640 1 -- 172.21.15.5:0/2490790141 shutdown_connections 2024-03-24T10:57:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.823+0000 7fd38c89d640 1 --2- 172.21.15.5:0/2490790141 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd36806b860 0x7fd36806dd20 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.823+0000 7fd38c89d640 1 --2- 172.21.15.5:0/2490790141 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd3881543a0 0x7fd38814a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.823+0000 7fd38c89d640 1 --2- 172.21.15.5:0/2490790141 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd38806d4f0 0x7fd38814fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.823+0000 7fd38c89d640 1 --2- 172.21.15.5:0/2490790141 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd38806bf50 0x7fd38814b430 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.823+0000 7fd38c89d640 1 -- 172.21.15.5:0/2490790141 >> 172.21.15.5:0/2490790141 conn(0x7fd388072ef0 msgr2=0x7fd388151b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.823+0000 7fd38c89d640 1 -- 172.21.15.5:0/2490790141 shutdown_connections 2024-03-24T10:57:10.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:10.823+0000 7fd38c89d640 1 -- 172.21.15.5:0/2490790141 wait complete. 2024-03-24T10:57:11.439 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:57:11.440 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:11.934 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:13:03.435 36946== Warning: unimplemented fcntl command: 1036 2024-03-24T10:57:11.944 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:13:03.408 36957== Warning: unimplemented fcntl command: 1036 2024-03-24T10:57:11.945 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:13:03.410 36957== Warning: unimplemented fcntl command: 1036 2024-03-24T10:57:11.951 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:13:03.415 36958== Warning: unimplemented fcntl command: 1036 2024-03-24T10:57:11.953 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:13:03.418 36958== Warning: unimplemented fcntl command: 1036 2024-03-24T10:57:11.985 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:13:03.486 36946== Warning: unimplemented fcntl command: 1036 2024-03-24T10:57:12.073 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:13:03.574 36946== Warning: unimplemented fcntl command: 1036 2024-03-24T10:57:12.088 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:13:03.552 36958== Warning: unimplemented fcntl command: 1036 2024-03-24T10:57:12.152 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:13:03.616 36957== Warning: unimplemented fcntl command: 1036 2024-03-24T10:57:12.441 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:12.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.546+0000 7f1d3f864640 1 Processor -- start 2024-03-24T10:57:12.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.546+0000 7f1d3f864640 1 -- start start 2024-03-24T10:57:12.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.547+0000 7f1d3f864640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1d3815ac00 0x7f1d380cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:12.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.547+0000 7f1d3f864640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1d380ccaa0 0x7f1d3806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:12.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.547+0000 7f1d3f864640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d380cb2f0 con 0x7f1d381543a0 2024-03-24T10:57:12.548 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.547+0000 7f1d3f864640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f1d380c6750 con 0x7f1d3815ac00 2024-03-24T10:57:12.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.547+0000 7f1d3f864640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1d380cceb0 con 0x7f1d380ccaa0 2024-03-24T10:57:12.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.547+0000 7f1d3cdff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1d3815ac00 0x7f1d380cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:12.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.547+0000 7f1d3de01640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1d380ccaa0 0x7f1d3806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:12.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.548+0000 7f1d3cdff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1d3815ac00 0x7f1d380cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51938/0 (socket says 172.21.15.5:51938) 2024-03-24T10:57:12.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.548+0000 7f1d3cdff640 1 -- 172.21.15.5:0/1075865854 learned_addr learned my addr 172.21.15.5:0/1075865854 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:12.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.550+0000 7f1d3cdff640 1 -- 172.21.15.5:0/1075865854 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1d380ccaa0 msgr2=0x7f1d3806bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:12.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.550+0000 7f1d3cdff640 1 --2- 172.21.15.5:0/1075865854 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1d380ccaa0 0x7f1d3806bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:12.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.550+0000 7f1d3cdff640 1 -- 172.21.15.5:0/1075865854 >> v1:172.21.15.5:6789/0 conn(0x7f1d381543a0 legacy=0x7f1d3806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:12.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.550+0000 7f1d3cdff640 1 -- 172.21.15.5:0/1075865854 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d3815b070 con 0x7f1d3815ac00 2024-03-24T10:57:12.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.552+0000 7f1d3cdff640 1 --2- 172.21.15.5:0/1075865854 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d3815ac00 0x7f1d380cc560 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f1d28009870 tx=0x7f1d2802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55dd36a684ea5e4d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:12.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.557+0000 7f1d2ffff640 1 -- 172.21.15.5:0/1075865854 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d28031640 con 0x7f1d3815ac00 2024-03-24T10:57:12.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.557+0000 7f1d2ffff640 1 -- 172.21.15.5:0/1075865854 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d280317a0 con 0x7f1d3815ac00 2024-03-24T10:57:12.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.557+0000 7f1d2ffff640 1 -- 172.21.15.5:0/1075865854 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d28031b30 con 0x7f1d3815ac00 2024-03-24T10:57:12.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.558+0000 7f1d3f864640 1 -- 172.21.15.5:0/1075865854 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d3815ac00 msgr2=0x7f1d380cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:12.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.558+0000 7f1d3f864640 1 --2- 172.21.15.5:0/1075865854 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d3815ac00 0x7f1d380cc560 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f1d28009870 tx=0x7f1d2802b8f0 comp rx=0 tx=0).stop 2024-03-24T10:57:12.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.558+0000 7f1d3f864640 1 -- 172.21.15.5:0/1075865854 shutdown_connections 2024-03-24T10:57:12.559 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.558+0000 7f1d3f864640 1 --2- 172.21.15.5:0/1075865854 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1d380ccaa0 0x7f1d3806bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:12.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.558+0000 7f1d3f864640 1 --2- 172.21.15.5:0/1075865854 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d3815ac00 0x7f1d380cc560 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:12.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.558+0000 7f1d3f864640 1 -- 172.21.15.5:0/1075865854 >> 172.21.15.5:0/1075865854 conn(0x7f1d38072ef0 msgr2=0x7f1d38061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:12.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.558+0000 7f1d3f864640 1 -- 172.21.15.5:0/1075865854 shutdown_connections 2024-03-24T10:57:12.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.558+0000 7f1d3f864640 1 -- 172.21.15.5:0/1075865854 wait complete. 2024-03-24T10:57:12.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.559+0000 7f1d3f864640 1 Processor -- start 2024-03-24T10:57:12.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.560+0000 7f1d3f864640 1 -- start start 2024-03-24T10:57:12.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.560+0000 7f1d3f864640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1d380ccaa0 0x7f1d38159d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:12.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.560+0000 7f1d3f864640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d381543a0 0x7f1d3815db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:12.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.560+0000 7f1d3f864640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d3815ac00 0x7f1d380ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:12.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.560+0000 7f1d3f864640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3814a4b0 con 0x7f1d381543a0 2024-03-24T10:57:12.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.560+0000 7f1d3f864640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1d38159140 con 0x7f1d3815ac00 2024-03-24T10:57:12.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.560+0000 7f1d3f864640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1d380b4b70 con 0x7f1d380ccaa0 2024-03-24T10:57:12.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.561+0000 7f1d3cdff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d381543a0 0x7f1d3815db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:12.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.561+0000 7f1d3de01640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d3815ac00 0x7f1d380ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:12.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.561+0000 7f1d3cdff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d381543a0 0x7f1d3815db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51954/0 (socket says 172.21.15.5:51954) 2024-03-24T10:57:12.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.561+0000 7f1d3cdff640 1 -- 172.21.15.5:0/3734675790 learned_addr learned my addr 172.21.15.5:0/3734675790 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:12.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.561+0000 7f1d3d600640 1 --2- 172.21.15.5:0/3734675790 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1d380ccaa0 0x7f1d38159d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:12.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.565+0000 7f1d3cdff640 1 -- 172.21.15.5:0/3734675790 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1d380ccaa0 msgr2=0x7f1d38159d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:12.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.565+0000 7f1d3cdff640 1 --2- 172.21.15.5:0/3734675790 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1d380ccaa0 0x7f1d38159d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:12.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.565+0000 7f1d3cdff640 1 -- 172.21.15.5:0/3734675790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d3815ac00 msgr2=0x7f1d380ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:12.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.565+0000 7f1d3cdff640 1 --2- 172.21.15.5:0/3734675790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d3815ac00 0x7f1d380ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:12.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.565+0000 7f1d3cdff640 1 -- 172.21.15.5:0/3734675790 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d280094d0 con 0x7f1d381543a0 2024-03-24T10:57:12.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.568+0000 7f1d3cdff640 1 --2- 172.21.15.5:0/3734675790 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d381543a0 0x7f1d3815db80 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1d28000c00 tx=0x7f1d280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:12.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.572+0000 7f1d2dffb640 1 -- 172.21.15.5:0/3734675790 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2803a930 con 0x7f1d381543a0 2024-03-24T10:57:12.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.572+0000 7f1d2dffb640 1 -- 172.21.15.5:0/3734675790 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d2803aa90 con 0x7f1d381543a0 2024-03-24T10:57:12.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.572+0000 7f1d2dffb640 1 -- 172.21.15.5:0/3734675790 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2803ae40 con 0x7f1d381543a0 2024-03-24T10:57:12.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.572+0000 7f1d3f864640 1 -- 172.21.15.5:0/3734675790 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d380cdb80 con 0x7f1d381543a0 2024-03-24T10:57:12.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.572+0000 7f1d3f864640 1 -- 172.21.15.5:0/3734675790 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d38158af0 con 0x7f1d381543a0 2024-03-24T10:57:12.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.573+0000 7f1d3f864640 1 -- 172.21.15.5:0/3734675790 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d380ce470 con 0x7f1d381543a0 2024-03-24T10:57:12.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.845+0000 7f1d2dffb640 1 -- 172.21.15.5:0/3734675790 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1d2802f050 con 0x7f1d381543a0 2024-03-24T10:57:12.845 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.845+0000 7f1d2dffb640 1 --2- 172.21.15.5:0/3734675790 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1d1c06b890 0x7f1d1c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:12.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.845+0000 7f1d3d600640 1 --2- 172.21.15.5:0/3734675790 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1d1c06b890 0x7f1d1c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:12.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.846+0000 7f1d3d600640 1 --2- 172.21.15.5:0/3734675790 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1d1c06b890 0x7f1d1c06dd50 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1d340017f0 tx=0x7f1d34009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:12.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.861+0000 7f1d2dffb640 1 -- 172.21.15.5:0/3734675790 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f1d2807ab00 con 0x7f1d381543a0 2024-03-24T10:57:12.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:12.861+0000 7f1d2dffb640 1 -- 172.21.15.5:0/3734675790 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1d28076da0 con 0x7f1d381543a0 2024-03-24T10:57:13.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:13.107+0000 7f1d3f864640 1 -- 172.21.15.5:0/3734675790 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d380681d0 con 0x7f1d381543a0 2024-03-24T10:57:13.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:13.120+0000 7f1d2dffb640 1 -- 172.21.15.5:0/3734675790 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f1d2807a460 con 0x7f1d381543a0 2024-03-24T10:57:13.120 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:13.121 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:57:13.121 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:57:13.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:13.122+0000 7f1d3f864640 1 -- 172.21.15.5:0/3734675790 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1d1c06b890 msgr2=0x7f1d1c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:13.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:13.122+0000 7f1d3f864640 1 --2- 172.21.15.5:0/3734675790 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1d1c06b890 0x7f1d1c06dd50 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1d340017f0 tx=0x7f1d34009040 comp rx=0 tx=0).stop 2024-03-24T10:57:13.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:13.122+0000 7f1d3f864640 1 -- 172.21.15.5:0/3734675790 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d381543a0 msgr2=0x7f1d3815db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:13.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:13.122+0000 7f1d3f864640 1 --2- 172.21.15.5:0/3734675790 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d381543a0 0x7f1d3815db80 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1d28000c00 tx=0x7f1d280046c0 comp rx=0 tx=0).stop 2024-03-24T10:57:13.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:13.122+0000 7f1d3f864640 1 -- 172.21.15.5:0/3734675790 shutdown_connections 2024-03-24T10:57:13.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:13.122+0000 7f1d3f864640 1 --2- 172.21.15.5:0/3734675790 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1d1c06b890 0x7f1d1c06dd50 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:13.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:13.122+0000 7f1d3f864640 1 --2- 172.21.15.5:0/3734675790 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1d381543a0 0x7f1d3815db80 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:13.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:13.122+0000 7f1d3f864640 1 --2- 172.21.15.5:0/3734675790 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1d3815ac00 0x7f1d380ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:13.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:13.122+0000 7f1d3f864640 1 --2- 172.21.15.5:0/3734675790 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1d380ccaa0 0x7f1d38159d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:13.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:13.122+0000 7f1d3f864640 1 -- 172.21.15.5:0/3734675790 >> 172.21.15.5:0/3734675790 conn(0x7f1d38072ef0 msgr2=0x7f1d38150730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:13.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:13.123+0000 7f1d3f864640 1 -- 172.21.15.5:0/3734675790 shutdown_connections 2024-03-24T10:57:13.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:13.123+0000 7f1d3f864640 1 -- 172.21.15.5:0/3734675790 wait complete. 2024-03-24T10:57:13.738 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:57:13.738 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:14.740 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:14.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.842+0000 7f021bc8e640 1 Processor -- start 2024-03-24T10:57:14.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.842+0000 7f021bc8e640 1 -- start start 2024-03-24T10:57:14.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.843+0000 7f021bc8e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f021406c910 0x7f021406cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:14.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.843+0000 7f021bc8e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02140c4720 con 0x7f02140c4300 2024-03-24T10:57:14.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.843+0000 7f021bc8e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02140c9050 con 0x7f0214154220 2024-03-24T10:57:14.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.843+0000 7f021bc8e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f02140c8bf0 con 0x7f021406c910 2024-03-24T10:57:14.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.844+0000 7f0219229640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f02140c4300 0x7f02140cda70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:52582/0 (socket says 172.21.15.5:52582) 2024-03-24T10:57:14.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.844+0000 7f0219229640 1 -- 172.21.15.5:0/2300797506 learned_addr learned my addr 172.21.15.5:0/2300797506 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:57:14.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.844+0000 7f021a22b640 1 --2- 172.21.15.5:0/2300797506 >> v2:172.21.15.57:3300/0 conn(0x7f021406c910 0x7f021406cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:14.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.846+0000 7f0218a28640 1 -- 172.21.15.5:0/2300797506 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 404137014 0 0) 0x7f02140c4720 con 0x7f02140c4300 2024-03-24T10:57:14.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.846+0000 7f0218a28640 1 -- 172.21.15.5:0/2300797506 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0200003520 con 0x7f02140c4300 2024-03-24T10:57:14.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.848+0000 7f021a22b640 1 -- 172.21.15.5:0/2300797506 >> v1:172.21.15.57:6789/0 conn(0x7f0214154220 legacy=0x7f02140c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:14.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.848+0000 7f021a22b640 1 -- 172.21.15.5:0/2300797506 >> v1:172.21.15.5:6789/0 conn(0x7f02140c4300 legacy=0x7f02140cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:14.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.848+0000 7f021a22b640 1 -- 172.21.15.5:0/2300797506 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f021406d280 con 0x7f021406c910 2024-03-24T10:57:14.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.853+0000 7f021a22b640 1 --2- 172.21.15.5:0/2300797506 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f021406c910 0x7f021406cd40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f020c009d90 tx=0x7f020c02bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6bf0c98b6af09ef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:14.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.858+0000 7f0218a28640 1 -- 172.21.15.5:0/2300797506 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020c02cd80 con 0x7f021406c910 2024-03-24T10:57:14.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.858+0000 7f0218a28640 1 -- 172.21.15.5:0/2300797506 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f020c034440 con 0x7f021406c910 2024-03-24T10:57:14.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.858+0000 7f0218a28640 1 -- 172.21.15.5:0/2300797506 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f020c0347d0 con 0x7f021406c910 2024-03-24T10:57:14.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.858+0000 7f021bc8e640 1 -- 172.21.15.5:0/2300797506 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f021406c910 msgr2=0x7f021406cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:14.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.858+0000 7f021bc8e640 1 --2- 172.21.15.5:0/2300797506 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f021406c910 0x7f021406cd40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f020c009d90 tx=0x7f020c02bfd0 comp rx=0 tx=0).stop 2024-03-24T10:57:14.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.858+0000 7f021bc8e640 1 -- 172.21.15.5:0/2300797506 shutdown_connections 2024-03-24T10:57:14.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.858+0000 7f021bc8e640 1 --2- 172.21.15.5:0/2300797506 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f021406c910 0x7f021406cd40 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:14.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.858+0000 7f021bc8e640 1 -- 172.21.15.5:0/2300797506 >> 172.21.15.5:0/2300797506 conn(0x7f021406f830 msgr2=0x7f021406fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:14.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.859+0000 7f021bc8e640 1 -- 172.21.15.5:0/2300797506 shutdown_connections 2024-03-24T10:57:14.859 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.859+0000 7f021bc8e640 1 -- 172.21.15.5:0/2300797506 wait complete. 2024-03-24T10:57:14.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.860+0000 7f021bc8e640 1 Processor -- start 2024-03-24T10:57:14.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.860+0000 7f021bc8e640 1 -- start start 2024-03-24T10:57:14.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.860+0000 7f021bc8e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f021406c910 0x7f02140c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:14.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.860+0000 7f021bc8e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02140c4300 0x7f02141f0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:14.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.860+0000 7f021bc8e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0214154220 0x7f02140c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:14.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.860+0000 7f021bc8e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f02141e4470 con 0x7f021406c910 2024-03-24T10:57:14.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.860+0000 7f021bc8e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f02140c7420 con 0x7f0214154220 2024-03-24T10:57:14.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.860+0000 7f021bc8e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f02141f0dd0 con 0x7f02140c4300 2024-03-24T10:57:14.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.861+0000 7f0219a2a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f021406c910 0x7f02140c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:14.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.861+0000 7f0219229640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02140c4300 0x7f02141f0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:14.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.861+0000 7f021a22b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0214154220 0x7f02140c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:14.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.861+0000 7f0219a2a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f021406c910 0x7f02140c6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51956/0 (socket says 172.21.15.5:51956) 2024-03-24T10:57:14.862 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.861+0000 7f0219a2a640 1 -- 172.21.15.5:0/629270614 learned_addr learned my addr 172.21.15.5:0/629270614 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.865+0000 7f0219a2a640 1 -- 172.21.15.5:0/629270614 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02140c4300 msgr2=0x7f02141f0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.865+0000 7f0219a2a640 1 --2- 172.21.15.5:0/629270614 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02140c4300 0x7f02141f0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.865+0000 7f0219a2a640 1 -- 172.21.15.5:0/629270614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0214154220 msgr2=0x7f02140c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.865+0000 7f0219a2a640 1 --2- 172.21.15.5:0/629270614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0214154220 0x7f02140c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:14.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.865+0000 7f0219a2a640 1 -- 172.21.15.5:0/629270614 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f020c009a70 con 0x7f021406c910 2024-03-24T10:57:14.867 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.867+0000 7f0219a2a640 1 --2- 172.21.15.5:0/629270614 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f021406c910 0x7f02140c6ee0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f021000c3c0 tx=0x7f021000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.871+0000 7f020a7fc640 1 -- 172.21.15.5:0/629270614 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02100052d0 con 0x7f021406c910 2024-03-24T10:57:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.871+0000 7f020a7fc640 1 -- 172.21.15.5:0/629270614 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0210005450 con 0x7f021406c910 2024-03-24T10:57:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.871+0000 7f020a7fc640 1 -- 172.21.15.5:0/629270614 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0210007260 con 0x7f021406c910 2024-03-24T10:57:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.871+0000 7f021bc8e640 1 -- 172.21.15.5:0/629270614 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02140c6a50 con 0x7f021406c910 2024-03-24T10:57:14.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.871+0000 7f021bc8e640 1 -- 172.21.15.5:0/629270614 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02141e39d0 con 0x7f021406c910 2024-03-24T10:57:14.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.872+0000 7f021bc8e640 1 -- 172.21.15.5:0/629270614 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02140681d0 con 0x7f021406c910 2024-03-24T10:57:14.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.892+0000 7f020a7fc640 1 -- 172.21.15.5:0/629270614 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f02100073c0 con 0x7f021406c910 2024-03-24T10:57:14.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.893+0000 7f020a7fc640 1 --2- 172.21.15.5:0/629270614 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f01f806b840 0x7f01f806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:14.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.893+0000 7f0219229640 1 --2- 172.21.15.5:0/629270614 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f01f806b840 0x7f01f806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:14.893 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:14.893+0000 7f0219229640 1 --2- 172.21.15.5:0/629270614 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f01f806b840 0x7f01f806dd00 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f02140cd020 tx=0x7f0204007d30 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:15.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:15.151+0000 7f020a7fc640 1 -- 172.21.15.5:0/629270614 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f0210007670 con 0x7f021406c910 2024-03-24T10:57:15.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:15.151+0000 7f020a7fc640 1 -- 172.21.15.5:0/629270614 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0210054860 con 0x7f021406c910 2024-03-24T10:57:15.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:15.373+0000 7f021bc8e640 1 -- 172.21.15.5:0/629270614 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f021406c240 con 0x7f021406c910 2024-03-24T10:57:15.385 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:15.385+0000 7f020a7fc640 1 -- 172.21.15.5:0/629270614 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f0210020020 con 0x7f021406c910 2024-03-24T10:57:15.385 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:15.385 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:57:15.386 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:57:15.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:15.388+0000 7f021bc8e640 1 -- 172.21.15.5:0/629270614 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f01f806b840 msgr2=0x7f01f806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:15.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:15.388+0000 7f021bc8e640 1 --2- 172.21.15.5:0/629270614 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f01f806b840 0x7f01f806dd00 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f02140cd020 tx=0x7f0204007d30 comp rx=0 tx=0).stop 2024-03-24T10:57:15.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:15.388+0000 7f021bc8e640 1 -- 172.21.15.5:0/629270614 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f021406c910 msgr2=0x7f02140c6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:15.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:15.388+0000 7f021bc8e640 1 --2- 172.21.15.5:0/629270614 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f021406c910 0x7f02140c6ee0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f021000c3c0 tx=0x7f021000c880 comp rx=0 tx=0).stop 2024-03-24T10:57:15.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:15.388+0000 7f021bc8e640 1 -- 172.21.15.5:0/629270614 shutdown_connections 2024-03-24T10:57:15.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:15.388+0000 7f021bc8e640 1 --2- 172.21.15.5:0/629270614 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f01f806b840 0x7f01f806dd00 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:15.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:15.388+0000 7f021bc8e640 1 --2- 172.21.15.5:0/629270614 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0214154220 0x7f02140c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:15.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:15.388+0000 7f021bc8e640 1 --2- 172.21.15.5:0/629270614 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02140c4300 0x7f02141f0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:15.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:15.388+0000 7f021bc8e640 1 --2- 172.21.15.5:0/629270614 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f021406c910 0x7f02140c6ee0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:15.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:15.388+0000 7f021bc8e640 1 -- 172.21.15.5:0/629270614 >> 172.21.15.5:0/629270614 conn(0x7f021406f830 msgr2=0x7f02141534b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:15.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:15.388+0000 7f021bc8e640 1 -- 172.21.15.5:0/629270614 shutdown_connections 2024-03-24T10:57:15.389 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:15.388+0000 7f021bc8e640 1 -- 172.21.15.5:0/629270614 wait complete. 2024-03-24T10:57:16.003 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:57:16.003 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:17.004 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:17.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.106+0000 7f35d2d02640 1 Processor -- start 2024-03-24T10:57:17.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.107+0000 7f35d2d02640 1 -- start start 2024-03-24T10:57:17.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.107+0000 7f35d2d02640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35cc0cace0 0x7f35cc1432d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:17.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.107+0000 7f35d2d02640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f35cc15a310 0x7f35cc143810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:17.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.107+0000 7f35d2d02640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35cc0cb0c0 con 0x7f35cc06c240 2024-03-24T10:57:17.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.107+0000 7f35d2d02640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f35cc06f110 con 0x7f35cc15a310 2024-03-24T10:57:17.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.107+0000 7f35d2d02640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f35cc15b320 con 0x7f35cc0cace0 2024-03-24T10:57:17.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.108+0000 7f35d0a9e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35cc0cace0 0x7f35cc1432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:17.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.108+0000 7f35cbfff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f35cc15a310 0x7f35cc143810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:17.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.109+0000 7f35d0a9e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35cc0cace0 0x7f35cc1432d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:49526/0 (socket says 172.21.15.5:49526) 2024-03-24T10:57:17.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.109+0000 7f35d0a9e640 1 -- 172.21.15.5:0/1235030796 learned_addr learned my addr 172.21.15.5:0/1235030796 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:17.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.111+0000 7f35d0a9e640 1 -- 172.21.15.5:0/1235030796 >> v2:172.21.15.57:3300/0 conn(0x7f35cc15a310 msgr2=0x7f35cc143810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:17.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.111+0000 7f35d0a9e640 1 --2- 172.21.15.5:0/1235030796 >> v2:172.21.15.57:3300/0 conn(0x7f35cc15a310 0x7f35cc143810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:17.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.111+0000 7f35d0a9e640 1 -- 172.21.15.5:0/1235030796 >> v1:172.21.15.57:6789/0 conn(0x7f35cc06c240 legacy=0x7f35cc06f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:17.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.111+0000 7f35d0a9e640 1 -- 172.21.15.5:0/1235030796 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35cc06cf70 con 0x7f35cc0cace0 2024-03-24T10:57:17.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.113+0000 7f35d0a9e640 1 --2- 172.21.15.5:0/1235030796 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35cc0cace0 0x7f35cc1432d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f35c0009870 tx=0x7f35c002ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbc2b74daa195875 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:17.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.118+0000 7f35cb7fe640 1 -- 172.21.15.5:0/1235030796 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c002f840 con 0x7f35cc0cace0 2024-03-24T10:57:17.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.118+0000 7f35cb7fe640 1 -- 172.21.15.5:0/1235030796 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35c002f9a0 con 0x7f35cc0cace0 2024-03-24T10:57:17.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.118+0000 7f35cb7fe640 1 -- 172.21.15.5:0/1235030796 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c002fd30 con 0x7f35cc0cace0 2024-03-24T10:57:17.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.119+0000 7f35d2d02640 1 -- 172.21.15.5:0/1235030796 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35cc0cace0 msgr2=0x7f35cc1432d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:17.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.119+0000 7f35d2d02640 1 --2- 172.21.15.5:0/1235030796 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35cc0cace0 0x7f35cc1432d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f35c0009870 tx=0x7f35c002ba20 comp rx=0 tx=0).stop 2024-03-24T10:57:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.119+0000 7f35d2d02640 1 -- 172.21.15.5:0/1235030796 shutdown_connections 2024-03-24T10:57:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.119+0000 7f35d2d02640 1 --2- 172.21.15.5:0/1235030796 >> v2:172.21.15.57:3300/0 conn(0x7f35cc15a310 0x7f35cc143810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.119+0000 7f35d2d02640 1 --2- 172.21.15.5:0/1235030796 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35cc0cace0 0x7f35cc1432d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.119+0000 7f35d2d02640 1 -- 172.21.15.5:0/1235030796 >> 172.21.15.5:0/1235030796 conn(0x7f35cc071240 msgr2=0x7f35cc071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.119+0000 7f35d2d02640 1 -- 172.21.15.5:0/1235030796 shutdown_connections 2024-03-24T10:57:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.119+0000 7f35d2d02640 1 -- 172.21.15.5:0/1235030796 wait complete. 2024-03-24T10:57:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.120+0000 7f35d2d02640 1 Processor -- start 2024-03-24T10:57:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.120+0000 7f35d2d02640 1 -- start start 2024-03-24T10:57:17.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.121+0000 7f35d2d02640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35cc06c240 0x7f35cc140e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.121+0000 7f35d2d02640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35cc0cace0 0x7f35cc0af6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.121+0000 7f35d2d02640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35cc140a20 0x7f35cc1f0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.121+0000 7f35d2d02640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc13f0a0 con 0x7f35cc0cace0 2024-03-24T10:57:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.121+0000 7f35d2d02640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f35cc1f0060 con 0x7f35cc140a20 2024-03-24T10:57:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.121+0000 7f35d2d02640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f35cc1f0300 con 0x7f35cc06c240 2024-03-24T10:57:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.122+0000 7f35cbfff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35cc0cace0 0x7f35cc0af6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.122+0000 7f35d0a9e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35cc06c240 0x7f35cc140e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.122+0000 7f35cbfff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35cc0cace0 0x7f35cc0af6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51960/0 (socket says 172.21.15.5:51960) 2024-03-24T10:57:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.122+0000 7f35cbfff640 1 -- 172.21.15.5:0/473970708 learned_addr learned my addr 172.21.15.5:0/473970708 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:17.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.122+0000 7f35d129f640 1 --2- 172.21.15.5:0/473970708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35cc140a20 0x7f35cc1f0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:17.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.126+0000 7f35cbfff640 1 -- 172.21.15.5:0/473970708 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35cc06c240 msgr2=0x7f35cc140e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:17.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.126+0000 7f35cbfff640 1 --2- 172.21.15.5:0/473970708 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35cc06c240 0x7f35cc140e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:17.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.126+0000 7f35cbfff640 1 -- 172.21.15.5:0/473970708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35cc140a20 msgr2=0x7f35cc1f0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:17.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.126+0000 7f35cbfff640 1 --2- 172.21.15.5:0/473970708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35cc140a20 0x7f35cc1f0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:17.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.126+0000 7f35cbfff640 1 -- 172.21.15.5:0/473970708 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c00094d0 con 0x7f35cc0cace0 2024-03-24T10:57:17.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.128+0000 7f35cbfff640 1 --2- 172.21.15.5:0/473970708 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35cc0cace0 0x7f35cc0af6b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f35b800e9b0 tx=0x7f35b800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:17.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.130+0000 7f35c97fa640 1 -- 172.21.15.5:0/473970708 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b8009670 con 0x7f35cc0cace0 2024-03-24T10:57:17.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.131+0000 7f35d2d02640 1 -- 172.21.15.5:0/473970708 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35cc141a70 con 0x7f35cc0cace0 2024-03-24T10:57:17.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.131+0000 7f35d2d02640 1 -- 172.21.15.5:0/473970708 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35cc142030 con 0x7f35cc0cace0 2024-03-24T10:57:17.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.132+0000 7f35d2d02640 1 -- 172.21.15.5:0/473970708 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35cc06fdc0 con 0x7f35cc0cace0 2024-03-24T10:57:17.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.132+0000 7f35c97fa640 1 -- 172.21.15.5:0/473970708 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35b800c9a0 con 0x7f35cc0cace0 2024-03-24T10:57:17.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.132+0000 7f35c97fa640 1 -- 172.21.15.5:0/473970708 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b800cd10 con 0x7f35cc0cace0 2024-03-24T10:57:17.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.405+0000 7f35c97fa640 1 -- 172.21.15.5:0/473970708 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f35b8013470 con 0x7f35cc0cace0 2024-03-24T10:57:17.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.405+0000 7f35c97fa640 1 --2- 172.21.15.5:0/473970708 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f35b006b810 0x7f35b006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:17.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.405+0000 7f35d0a9e640 1 --2- 172.21.15.5:0/473970708 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f35b006b810 0x7f35b006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:17.405 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.406+0000 7f35d0a9e640 1 --2- 172.21.15.5:0/473970708 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f35b006b810 0x7f35b006dcd0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f35c0036000 tx=0x7f35c0008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.418+0000 7f35c97fa640 1 -- 172.21.15.5:0/473970708 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f35b8059350 con 0x7f35cc0cace0 2024-03-24T10:57:17.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.418+0000 7f35c97fa640 1 -- 172.21.15.5:0/473970708 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f35b80555f0 con 0x7f35cc0cace0 2024-03-24T10:57:17.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.700+0000 7f35d2d02640 1 -- 172.21.15.5:0/473970708 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35cc0681d0 con 0x7f35cc0cace0 2024-03-24T10:57:17.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.720+0000 7f35c97fa640 1 -- 172.21.15.5:0/473970708 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f35b8021090 con 0x7f35cc0cace0 2024-03-24T10:57:17.720 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:17.721 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:57:17.721 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:57:17.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.722+0000 7f35d2d02640 1 -- 172.21.15.5:0/473970708 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f35b006b810 msgr2=0x7f35b006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:17.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.722+0000 7f35d2d02640 1 --2- 172.21.15.5:0/473970708 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f35b006b810 0x7f35b006dcd0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f35c0036000 tx=0x7f35c0008030 comp rx=0 tx=0).stop 2024-03-24T10:57:17.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.722+0000 7f35d2d02640 1 -- 172.21.15.5:0/473970708 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35cc0cace0 msgr2=0x7f35cc0af6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:17.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.722+0000 7f35d2d02640 1 --2- 172.21.15.5:0/473970708 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35cc0cace0 0x7f35cc0af6b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f35b800e9b0 tx=0x7f35b800ee70 comp rx=0 tx=0).stop 2024-03-24T10:57:17.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.723+0000 7f35d2d02640 1 -- 172.21.15.5:0/473970708 shutdown_connections 2024-03-24T10:57:17.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.723+0000 7f35d2d02640 1 --2- 172.21.15.5:0/473970708 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f35b006b810 0x7f35b006dcd0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:17.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.723+0000 7f35d2d02640 1 --2- 172.21.15.5:0/473970708 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35cc0cace0 0x7f35cc0af6b0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:17.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.723+0000 7f35d2d02640 1 --2- 172.21.15.5:0/473970708 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35cc140a20 0x7f35cc1f0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:17.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.723+0000 7f35d2d02640 1 --2- 172.21.15.5:0/473970708 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35cc06c240 0x7f35cc140e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:17.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.723+0000 7f35d2d02640 1 -- 172.21.15.5:0/473970708 >> 172.21.15.5:0/473970708 conn(0x7f35cc071240 msgr2=0x7f35cc159630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:17.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.723+0000 7f35d2d02640 1 -- 172.21.15.5:0/473970708 shutdown_connections 2024-03-24T10:57:17.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:17.723+0000 7f35d2d02640 1 -- 172.21.15.5:0/473970708 wait complete. 2024-03-24T10:57:18.339 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:57:18.339 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:19.340 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:19.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.442+0000 7f36013f7640 1 Processor -- start 2024-03-24T10:57:19.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.442+0000 7f36013f7640 1 -- start start 2024-03-24T10:57:19.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.443+0000 7f36013f7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35fc06bf50 0x7f35fc06c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:19.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.443+0000 7f36013f7640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35fc0cb2f0 con 0x7f35fc1543a0 2024-03-24T10:57:19.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.443+0000 7f36013f7640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35fc15b070 con 0x7f35fc15ac00 2024-03-24T10:57:19.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.443+0000 7f36013f7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f35fc0c6750 con 0x7f35fc06bf50 2024-03-24T10:57:19.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.444+0000 7f35faffd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f35fc1543a0 0x7f35fc06d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:52590/0 (socket says 172.21.15.5:52590) 2024-03-24T10:57:19.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.444+0000 7f35faffd640 1 -- 172.21.15.5:0/2611621265 learned_addr learned my addr 172.21.15.5:0/2611621265 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:57:19.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.444+0000 7f35fb7fe640 1 --2- 172.21.15.5:0/2611621265 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35fc06bf50 0x7f35fc06c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:19.446 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.447+0000 7f35fb7fe640 1 -- 172.21.15.5:0/2611621265 >> v1:172.21.15.57:6789/0 conn(0x7f35fc15ac00 legacy=0x7f35fc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:19.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.447+0000 7f35fb7fe640 1 -- 172.21.15.5:0/2611621265 >> v1:172.21.15.5:6789/0 conn(0x7f35fc1543a0 legacy=0x7f35fc06d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:19.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.447+0000 7f35fb7fe640 1 -- 172.21.15.5:0/2611621265 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35fc0ca7e0 con 0x7f35fc06bf50 2024-03-24T10:57:19.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.449+0000 7f35fb7fe640 1 --2- 172.21.15.5:0/2611621265 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35fc06bf50 0x7f35fc06c360 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f35ec0097d0 tx=0x7f35ec02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20d0868c3122b76 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:19.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.453+0000 7f35f9ffb640 1 -- 172.21.15.5:0/2611621265 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ec031890 con 0x7f35fc06bf50 2024-03-24T10:57:19.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.453+0000 7f35f9ffb640 1 -- 172.21.15.5:0/2611621265 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35ec0319f0 con 0x7f35fc06bf50 2024-03-24T10:57:19.453 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.453+0000 7f35f9ffb640 1 -- 172.21.15.5:0/2611621265 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ec031d80 con 0x7f35fc06bf50 2024-03-24T10:57:19.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.454+0000 7f36013f7640 1 -- 172.21.15.5:0/2611621265 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35fc06bf50 msgr2=0x7f35fc06c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:19.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.454+0000 7f36013f7640 1 --2- 172.21.15.5:0/2611621265 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35fc06bf50 0x7f35fc06c360 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f35ec0097d0 tx=0x7f35ec02b930 comp rx=0 tx=0).stop 2024-03-24T10:57:19.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.454+0000 7f36013f7640 1 -- 172.21.15.5:0/2611621265 shutdown_connections 2024-03-24T10:57:19.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.454+0000 7f36013f7640 1 --2- 172.21.15.5:0/2611621265 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35fc06bf50 0x7f35fc06c360 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:19.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.454+0000 7f36013f7640 1 -- 172.21.15.5:0/2611621265 >> 172.21.15.5:0/2611621265 conn(0x7f35fc072ef0 msgr2=0x7f35fc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:19.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.454+0000 7f36013f7640 1 -- 172.21.15.5:0/2611621265 shutdown_connections 2024-03-24T10:57:19.454 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.454+0000 7f36013f7640 1 -- 172.21.15.5:0/2611621265 wait complete. 2024-03-24T10:57:19.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.455+0000 7f36013f7640 1 Processor -- start 2024-03-24T10:57:19.455 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.455+0000 7f36013f7640 1 -- start start 2024-03-24T10:57:19.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.456+0000 7f36013f7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35fc06bf50 0x7f35fc14d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:19.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.456+0000 7f36013f7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35fc1543a0 0x7f35fc1f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:19.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.456+0000 7f36013f7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35fc15ac00 0x7f35fc159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:19.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.456+0000 7f36013f7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc0cf850 con 0x7f35fc1543a0 2024-03-24T10:57:19.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.456+0000 7f36013f7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f35fc14da80 con 0x7f35fc06bf50 2024-03-24T10:57:19.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.456+0000 7f36013f7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f35fc1f0c00 con 0x7f35fc15ac00 2024-03-24T10:57:19.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.457+0000 7f35fa7fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35fc1543a0 0x7f35fc1f3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:19.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.457+0000 7f35faffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35fc06bf50 0x7f35fc14d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:19.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.457+0000 7f35fb7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35fc15ac00 0x7f35fc159570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:19.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.457+0000 7f35fa7fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35fc1543a0 0x7f35fc1f3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51974/0 (socket says 172.21.15.5:51974) 2024-03-24T10:57:19.457 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.457+0000 7f35fa7fc640 1 -- 172.21.15.5:0/123792600 learned_addr learned my addr 172.21.15.5:0/123792600 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:19.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.460+0000 7f35fa7fc640 1 -- 172.21.15.5:0/123792600 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35fc15ac00 msgr2=0x7f35fc159570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:19.460 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.460+0000 7f35fa7fc640 1 --2- 172.21.15.5:0/123792600 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35fc15ac00 0x7f35fc159570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:19.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.460+0000 7f35fa7fc640 1 -- 172.21.15.5:0/123792600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35fc06bf50 msgr2=0x7f35fc14d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:19.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.460+0000 7f35fa7fc640 1 --2- 172.21.15.5:0/123792600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35fc06bf50 0x7f35fc14d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:19.461 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.460+0000 7f35fa7fc640 1 -- 172.21.15.5:0/123792600 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35ec009400 con 0x7f35fc1543a0 2024-03-24T10:57:19.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.463+0000 7f35fa7fc640 1 --2- 172.21.15.5:0/123792600 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35fc1543a0 0x7f35fc1f3ab0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f35e400c3c0 tx=0x7f35e400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:19.467 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.467+0000 7f35f37fe640 1 -- 172.21.15.5:0/123792600 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e40052d0 con 0x7f35fc1543a0 2024-03-24T10:57:19.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.467+0000 7f35f37fe640 1 -- 172.21.15.5:0/123792600 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35e4005450 con 0x7f35fc1543a0 2024-03-24T10:57:19.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.467+0000 7f36013f7640 1 -- 172.21.15.5:0/123792600 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35fc0cf570 con 0x7f35fc1543a0 2024-03-24T10:57:19.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.467+0000 7f35f37fe640 1 -- 172.21.15.5:0/123792600 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e4007260 con 0x7f35fc1543a0 2024-03-24T10:57:19.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.467+0000 7f36013f7640 1 -- 172.21.15.5:0/123792600 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35fc1f0120 con 0x7f35fc1543a0 2024-03-24T10:57:19.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.469+0000 7f36013f7640 1 -- 172.21.15.5:0/123792600 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35fc0681d0 con 0x7f35fc1543a0 2024-03-24T10:57:19.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.493+0000 7f35f37fe640 1 -- 172.21.15.5:0/123792600 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f35e40073c0 con 0x7f35fc1543a0 2024-03-24T10:57:19.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.493+0000 7f35f37fe640 1 --2- 172.21.15.5:0/123792600 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f35dc06b860 0x7f35dc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:19.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.493+0000 7f35faffd640 1 --2- 172.21.15.5:0/123792600 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f35dc06b860 0x7f35dc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:19.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.494+0000 7f35faffd640 1 --2- 172.21.15.5:0/123792600 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f35dc06b860 0x7f35dc06dd20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f35f40012e0 tx=0x7f35f4001a30 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:19.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.789+0000 7f35f37fe640 1 -- 172.21.15.5:0/123792600 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f35e4058230 con 0x7f35fc1543a0 2024-03-24T10:57:19.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:19.789+0000 7f35f37fe640 1 -- 172.21.15.5:0/123792600 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f35e40544d0 con 0x7f35fc1543a0 2024-03-24T10:57:20.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:20.072+0000 7f36013f7640 1 -- 172.21.15.5:0/123792600 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35fc150ed0 con 0x7f35fc1543a0 2024-03-24T10:57:20.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:20.100+0000 7f35f37fe640 1 -- 172.21.15.5:0/123792600 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6211 (secure 0 0 0) 0x7f35e4020070 con 0x7f35fc1543a0 2024-03-24T10:57:20.101 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:20.101 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:55:40.322525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4258,4254,4252,4251]},"id":1}]} 2024-03-24T10:57:20.101 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 23 2024-03-24T10:57:20.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:20.103+0000 7f36013f7640 1 -- 172.21.15.5:0/123792600 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f35dc06b860 msgr2=0x7f35dc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:20.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:20.103+0000 7f36013f7640 1 --2- 172.21.15.5:0/123792600 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f35dc06b860 0x7f35dc06dd20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f35f40012e0 tx=0x7f35f4001a30 comp rx=0 tx=0).stop 2024-03-24T10:57:20.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:20.103+0000 7f36013f7640 1 -- 172.21.15.5:0/123792600 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35fc1543a0 msgr2=0x7f35fc1f3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:20.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:20.103+0000 7f36013f7640 1 --2- 172.21.15.5:0/123792600 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35fc1543a0 0x7f35fc1f3ab0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f35e400c3c0 tx=0x7f35e400c880 comp rx=0 tx=0).stop 2024-03-24T10:57:20.103 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:20.103+0000 7f36013f7640 1 -- 172.21.15.5:0/123792600 shutdown_connections 2024-03-24T10:57:20.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:20.103+0000 7f36013f7640 1 --2- 172.21.15.5:0/123792600 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f35dc06b860 0x7f35dc06dd20 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:20.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:20.103+0000 7f36013f7640 1 --2- 172.21.15.5:0/123792600 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f35fc1543a0 0x7f35fc1f3ab0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:20.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:20.103+0000 7f36013f7640 1 --2- 172.21.15.5:0/123792600 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f35fc15ac00 0x7f35fc159570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:20.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:20.103+0000 7f36013f7640 1 --2- 172.21.15.5:0/123792600 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35fc06bf50 0x7f35fc14d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:20.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:20.103+0000 7f36013f7640 1 -- 172.21.15.5:0/123792600 >> 172.21.15.5:0/123792600 conn(0x7f35fc072ef0 msgr2=0x7f35fc151bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:20.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:20.103+0000 7f36013f7640 1 -- 172.21.15.5:0/123792600 shutdown_connections 2024-03-24T10:57:20.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:20.103+0000 7f36013f7640 1 -- 172.21.15.5:0/123792600 wait complete. 2024-03-24T10:57:20.718 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:55:40.322525+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4258, 4254, 4252, 4251]} 2024-03-24T10:57:20.718 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:21.718 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:21.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.822+0000 7f88edfc7640 1 Processor -- start 2024-03-24T10:57:21.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.822+0000 7f88edfc7640 1 -- start start 2024-03-24T10:57:21.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.823+0000 7f88edfc7640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f88e8154220 0x7f88e80c4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:21.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.823+0000 7f88edfc7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f88e80c84e0 0x7f88e80cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:21.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.823+0000 7f88edfc7640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88e80c9050 con 0x7f88e80c8900 2024-03-24T10:57:21.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.823+0000 7f88edfc7640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f88e815a5f0 con 0x7f88e80c84e0 2024-03-24T10:57:21.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.823+0000 7f88edfc7640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f88e815a8d0 con 0x7f88e8154220 2024-03-24T10:57:21.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.823+0000 7f88e6ffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f88e80c84e0 0x7f88e80cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:21.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.823+0000 7f88e77fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f88e8154220 0x7f88e80c4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:21.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.824+0000 7f88e6ffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f88e80c84e0 0x7f88e80cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:51988/0 (socket says 172.21.15.5:51988) 2024-03-24T10:57:21.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.824+0000 7f88e6ffd640 1 -- 172.21.15.5:0/1007532214 learned_addr learned my addr 172.21.15.5:0/1007532214 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:21.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.826+0000 7f88e6ffd640 1 -- 172.21.15.5:0/1007532214 >> v2:172.21.15.57:3300/0 conn(0x7f88e8154220 msgr2=0x7f88e80c4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:21.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.826+0000 7f88e6ffd640 1 --2- 172.21.15.5:0/1007532214 >> v2:172.21.15.57:3300/0 conn(0x7f88e8154220 0x7f88e80c4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:21.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.826+0000 7f88e6ffd640 1 -- 172.21.15.5:0/1007532214 >> v1:172.21.15.57:6789/0 conn(0x7f88e80c8900 legacy=0x7f88e806c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:21.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.826+0000 7f88e6ffd640 1 -- 172.21.15.5:0/1007532214 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e815abb0 con 0x7f88e80c84e0 2024-03-24T10:57:21.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.828+0000 7f88e6ffd640 1 --2- 172.21.15.5:0/1007532214 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88e80c84e0 0x7f88e80cda70 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f88e80b0a50 tx=0x7f88dc02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31d28deda57e458e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:21.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.832+0000 7f88e67fc640 1 -- 172.21.15.5:0/1007532214 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88dc02f850 con 0x7f88e80c84e0 2024-03-24T10:57:21.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.832+0000 7f88e67fc640 1 -- 172.21.15.5:0/1007532214 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88dc02f9b0 con 0x7f88e80c84e0 2024-03-24T10:57:21.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.832+0000 7f88e67fc640 1 -- 172.21.15.5:0/1007532214 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88dc02fd40 con 0x7f88e80c84e0 2024-03-24T10:57:21.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.833+0000 7f88edfc7640 1 -- 172.21.15.5:0/1007532214 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88e80c84e0 msgr2=0x7f88e80cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:21.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.833+0000 7f88edfc7640 1 --2- 172.21.15.5:0/1007532214 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88e80c84e0 0x7f88e80cda70 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f88e80b0a50 tx=0x7f88dc02ba30 comp rx=0 tx=0).stop 2024-03-24T10:57:21.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.833+0000 7f88edfc7640 1 -- 172.21.15.5:0/1007532214 shutdown_connections 2024-03-24T10:57:21.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.833+0000 7f88edfc7640 1 --2- 172.21.15.5:0/1007532214 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88e80c84e0 0x7f88e80cda70 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:21.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.833+0000 7f88edfc7640 1 --2- 172.21.15.5:0/1007532214 >> v2:172.21.15.57:3300/0 conn(0x7f88e8154220 0x7f88e80c4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:21.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.833+0000 7f88edfc7640 1 -- 172.21.15.5:0/1007532214 >> 172.21.15.5:0/1007532214 conn(0x7f88e806f830 msgr2=0x7f88e806fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:21.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.833+0000 7f88edfc7640 1 -- 172.21.15.5:0/1007532214 shutdown_connections 2024-03-24T10:57:21.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.833+0000 7f88edfc7640 1 -- 172.21.15.5:0/1007532214 wait complete. 2024-03-24T10:57:21.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.834+0000 7f88edfc7640 1 Processor -- start 2024-03-24T10:57:21.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.834+0000 7f88edfc7640 1 -- start start 2024-03-24T10:57:21.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.835+0000 7f88edfc7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88e80c84e0 0x7f88e80ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:21.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.835+0000 7f88edfc7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f88e80c8900 0x7f88e8152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:21.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.835+0000 7f88edfc7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88e8154220 0x7f88e81f3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:21.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.835+0000 7f88edfc7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f88e80ca7b0 con 0x7f88e80c84e0 2024-03-24T10:57:21.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.835+0000 7f88edfc7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f88e814ff60 con 0x7f88e8154220 2024-03-24T10:57:21.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.835+0000 7f88edfc7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f88e80c9640 con 0x7f88e80c8900 2024-03-24T10:57:21.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.836+0000 7f88e77fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88e80c84e0 0x7f88e80ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:21.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.836+0000 7f88e7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88e8154220 0x7f88e81f3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:21.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.836+0000 7f88e6ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f88e80c8900 0x7f88e8152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:21.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.836+0000 7f88e7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88e8154220 0x7f88e81f3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:37324/0 (socket says 172.21.15.5:37324) 2024-03-24T10:57:21.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.836+0000 7f88e7fff640 1 -- 172.21.15.5:0/1915787159 learned_addr learned my addr 172.21.15.5:0/1915787159 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:21.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.840+0000 7f88e77fe640 1 -- 172.21.15.5:0/1915787159 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f88e80c8900 msgr2=0x7f88e8152710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:21.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.840+0000 7f88e77fe640 1 --2- 172.21.15.5:0/1915787159 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f88e80c8900 0x7f88e8152710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:21.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.840+0000 7f88e77fe640 1 -- 172.21.15.5:0/1915787159 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88e8154220 msgr2=0x7f88e81f3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:21.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.840+0000 7f88e77fe640 1 --2- 172.21.15.5:0/1915787159 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88e8154220 0x7f88e81f3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:21.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.840+0000 7f88e77fe640 1 -- 172.21.15.5:0/1915787159 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88dc0094d0 con 0x7f88e80c84e0 2024-03-24T10:57:21.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.842+0000 7f88e77fe640 1 --2- 172.21.15.5:0/1915787159 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88e80c84e0 0x7f88e80ca270 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f88d800e850 tx=0x7f88d800ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:21.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.845+0000 7f88cffff640 1 -- 172.21.15.5:0/1915787159 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d80095b0 con 0x7f88e80c84e0 2024-03-24T10:57:21.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.845+0000 7f88cffff640 1 -- 172.21.15.5:0/1915787159 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88d800c940 con 0x7f88e80c84e0 2024-03-24T10:57:21.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.845+0000 7f88edfc7640 1 -- 172.21.15.5:0/1915787159 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88e80cace0 con 0x7f88e80c84e0 2024-03-24T10:57:21.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.845+0000 7f88cffff640 1 -- 172.21.15.5:0/1915787159 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d800ccd0 con 0x7f88e80c84e0 2024-03-24T10:57:21.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.845+0000 7f88edfc7640 1 -- 172.21.15.5:0/1915787159 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88e814f910 con 0x7f88e80c84e0 2024-03-24T10:57:21.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.847+0000 7f88edfc7640 1 -- 172.21.15.5:0/1915787159 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88e80681d0 con 0x7f88e80c84e0 2024-03-24T10:57:21.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.865+0000 7f88cffff640 1 -- 172.21.15.5:0/1915787159 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f88d8013470 con 0x7f88e80c84e0 2024-03-24T10:57:21.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.866+0000 7f88cffff640 1 --2- 172.21.15.5:0/1915787159 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f88d406b860 0x7f88d406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:21.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.866+0000 7f88e6ffd640 1 --2- 172.21.15.5:0/1915787159 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f88d406b860 0x7f88d406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:21.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.867+0000 7f88e6ffd640 1 --2- 172.21.15.5:0/1915787159 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f88d406b860 0x7f88d406dd20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f88dc036000 tx=0x7f88dc008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:21.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:21.870+0000 7f88cffff640 1 -- 172.21.15.5:0/1915787159 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f88d800ce30 con 0x7f88e80c84e0 2024-03-24T10:57:22.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:22.128+0000 7f88cffff640 1 -- 172.21.15.5:0/1915787159 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f88d80555f0 con 0x7f88e80c84e0 2024-03-24T10:57:22.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:22.377+0000 7f88edfc7640 1 -- 172.21.15.5:0/1915787159 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88e8157f10 con 0x7f88e80c84e0 2024-03-24T10:57:22.390 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:22.390+0000 7f88cffff640 1 -- 172.21.15.5:0/1915787159 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f88d8021090 con 0x7f88e80c84e0 2024-03-24T10:57:22.390 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:22.390 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:57:22.391 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:57:22.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:22.392+0000 7f88edfc7640 1 -- 172.21.15.5:0/1915787159 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f88d406b860 msgr2=0x7f88d406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:22.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:22.392+0000 7f88edfc7640 1 --2- 172.21.15.5:0/1915787159 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f88d406b860 0x7f88d406dd20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f88dc036000 tx=0x7f88dc008030 comp rx=0 tx=0).stop 2024-03-24T10:57:22.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:22.392+0000 7f88edfc7640 1 -- 172.21.15.5:0/1915787159 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88e80c84e0 msgr2=0x7f88e80ca270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:22.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:22.392+0000 7f88edfc7640 1 --2- 172.21.15.5:0/1915787159 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88e80c84e0 0x7f88e80ca270 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f88d800e850 tx=0x7f88d800ed10 comp rx=0 tx=0).stop 2024-03-24T10:57:22.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:22.392+0000 7f88edfc7640 1 -- 172.21.15.5:0/1915787159 shutdown_connections 2024-03-24T10:57:22.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:22.392+0000 7f88edfc7640 1 --2- 172.21.15.5:0/1915787159 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f88d406b860 0x7f88d406dd20 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:22.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:22.392+0000 7f88edfc7640 1 --2- 172.21.15.5:0/1915787159 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f88e8154220 0x7f88e81f3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:22.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:22.392+0000 7f88edfc7640 1 --2- 172.21.15.5:0/1915787159 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f88e80c8900 0x7f88e8152710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:22.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:22.392+0000 7f88edfc7640 1 --2- 172.21.15.5:0/1915787159 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f88e80c84e0 0x7f88e80ca270 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:22.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:22.392+0000 7f88edfc7640 1 -- 172.21.15.5:0/1915787159 >> 172.21.15.5:0/1915787159 conn(0x7f88e806f830 msgr2=0x7f88e806fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:22.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:22.392+0000 7f88edfc7640 1 -- 172.21.15.5:0/1915787159 shutdown_connections 2024-03-24T10:57:22.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:22.392+0000 7f88edfc7640 1 -- 172.21.15.5:0/1915787159 wait complete. 2024-03-24T10:57:23.007 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:57:23.007 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:24.008 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:24.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.110+0000 7f9101274640 1 Processor -- start 2024-03-24T10:57:24.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.110+0000 7f9101274640 1 -- start start 2024-03-24T10:57:24.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.111+0000 7f9101274640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f90fc1543a0 0x7f90fc15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:24.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.111+0000 7f9101274640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f90fc06d4f0 0x7f90fc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:24.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.111+0000 7f9101274640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90fc0cb2f0 con 0x7f90fc0ccaa0 2024-03-24T10:57:24.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.111+0000 7f9101274640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f90fc0c6750 con 0x7f90fc1543a0 2024-03-24T10:57:24.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.111+0000 7f9101274640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f90fc0cc380 con 0x7f90fc06d4f0 2024-03-24T10:57:24.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.111+0000 7f90faffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f90fc1543a0 0x7f90fc15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:24.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.111+0000 7f90fa7fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f90fc06d4f0 0x7f90fc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:24.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.112+0000 7f90faffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f90fc1543a0 0x7f90fc15ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38972/0 (socket says 172.21.15.5:38972) 2024-03-24T10:57:24.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.112+0000 7f90faffd640 1 -- 172.21.15.5:0/1103519775 learned_addr learned my addr 172.21.15.5:0/1103519775 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:24.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.114+0000 7f90faffd640 1 -- 172.21.15.5:0/1103519775 >> v2:172.21.15.57:3300/0 conn(0x7f90fc06d4f0 msgr2=0x7f90fc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:24.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.114+0000 7f90faffd640 1 --2- 172.21.15.5:0/1103519775 >> v2:172.21.15.57:3300/0 conn(0x7f90fc06d4f0 0x7f90fc0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:24.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.114+0000 7f90faffd640 1 -- 172.21.15.5:0/1103519775 >> v1:172.21.15.57:6789/0 conn(0x7f90fc0ccaa0 legacy=0x7f90fc06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:24.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.114+0000 7f90faffd640 1 -- 172.21.15.5:0/1103519775 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90fc0ca7e0 con 0x7f90fc1543a0 2024-03-24T10:57:24.116 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.116+0000 7f90faffd640 1 --2- 172.21.15.5:0/1103519775 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90fc1543a0 0x7f90fc15ac00 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0c6990 tx=0x7f90ec02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=551cee09682d0926 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:24.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.120+0000 7f90f9ffb640 1 -- 172.21.15.5:0/1103519775 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ec02f850 con 0x7f90fc1543a0 2024-03-24T10:57:24.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.120+0000 7f90f9ffb640 1 -- 172.21.15.5:0/1103519775 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90ec02f9b0 con 0x7f90fc1543a0 2024-03-24T10:57:24.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.120+0000 7f90f9ffb640 1 -- 172.21.15.5:0/1103519775 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ec02fd40 con 0x7f90fc1543a0 2024-03-24T10:57:24.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.121+0000 7f9101274640 1 -- 172.21.15.5:0/1103519775 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90fc1543a0 msgr2=0x7f90fc15ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:24.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.121+0000 7f9101274640 1 --2- 172.21.15.5:0/1103519775 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90fc1543a0 0x7f90fc15ac00 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0c6990 tx=0x7f90ec02ba30 comp rx=0 tx=0).stop 2024-03-24T10:57:24.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.121+0000 7f9101274640 1 -- 172.21.15.5:0/1103519775 shutdown_connections 2024-03-24T10:57:24.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.121+0000 7f9101274640 1 --2- 172.21.15.5:0/1103519775 >> v2:172.21.15.57:3300/0 conn(0x7f90fc06d4f0 0x7f90fc0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:24.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.121+0000 7f9101274640 1 --2- 172.21.15.5:0/1103519775 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90fc1543a0 0x7f90fc15ac00 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:24.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.121+0000 7f9101274640 1 -- 172.21.15.5:0/1103519775 >> 172.21.15.5:0/1103519775 conn(0x7f90fc072ef0 msgr2=0x7f90fc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:24.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.121+0000 7f9101274640 1 -- 172.21.15.5:0/1103519775 shutdown_connections 2024-03-24T10:57:24.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.121+0000 7f9101274640 1 -- 172.21.15.5:0/1103519775 wait complete. 2024-03-24T10:57:24.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.122+0000 7f9101274640 1 Processor -- start 2024-03-24T10:57:24.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.122+0000 7f9101274640 1 -- start start 2024-03-24T10:57:24.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.123+0000 7f9101274640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90fc06d4f0 0x7f90fc15d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:24.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.123+0000 7f9101274640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90fc0ccaa0 0x7f90fc1f06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:24.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.123+0000 7f9101274640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90fc1543a0 0x7f90fc1f0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:24.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.123+0000 7f9101274640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f90fc0cece0 con 0x7f90fc0ccaa0 2024-03-24T10:57:24.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.123+0000 7f9101274640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f90fc1f0250 con 0x7f90fc06d4f0 2024-03-24T10:57:24.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.123+0000 7f9101274640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f90fc15e320 con 0x7f90fc1543a0 2024-03-24T10:57:24.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.123+0000 7f90fa7fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90fc0ccaa0 0x7f90fc1f06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:24.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.124+0000 7f90faffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90fc06d4f0 0x7f90fc15d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:24.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.124+0000 7f90fb7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90fc1543a0 0x7f90fc1f0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:24.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.124+0000 7f90fa7fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90fc0ccaa0 0x7f90fc1f06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38978/0 (socket says 172.21.15.5:38978) 2024-03-24T10:57:24.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.124+0000 7f90fa7fc640 1 -- 172.21.15.5:0/2967522299 learned_addr learned my addr 172.21.15.5:0/2967522299 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:24.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.128+0000 7f90faffd640 1 -- 172.21.15.5:0/2967522299 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90fc1543a0 msgr2=0x7f90fc1f0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:24.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.128+0000 7f90faffd640 1 --2- 172.21.15.5:0/2967522299 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90fc1543a0 0x7f90fc1f0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:24.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.128+0000 7f90faffd640 1 -- 172.21.15.5:0/2967522299 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90fc0ccaa0 msgr2=0x7f90fc1f06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:24.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.128+0000 7f90faffd640 1 --2- 172.21.15.5:0/2967522299 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90fc0ccaa0 0x7f90fc1f06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:24.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.128+0000 7f90faffd640 1 -- 172.21.15.5:0/2967522299 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90ec0094d0 con 0x7f90fc06d4f0 2024-03-24T10:57:24.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.131+0000 7f90faffd640 1 --2- 172.21.15.5:0/2967522299 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90fc06d4f0 0x7f90fc15d630 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f90ec033000 tx=0x7f90ec004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:24.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.136+0000 7f90e37fe640 1 -- 172.21.15.5:0/2967522299 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ec037c40 con 0x7f90fc06d4f0 2024-03-24T10:57:24.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.136+0000 7f90e37fe640 1 -- 172.21.15.5:0/2967522299 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90ec037dc0 con 0x7f90fc06d4f0 2024-03-24T10:57:24.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.136+0000 7f9101274640 1 -- 172.21.15.5:0/2967522299 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90fc0cef30 con 0x7f90fc06d4f0 2024-03-24T10:57:24.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.136+0000 7f90e37fe640 1 -- 172.21.15.5:0/2967522299 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ec041640 con 0x7f90fc06d4f0 2024-03-24T10:57:24.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.136+0000 7f9101274640 1 -- 172.21.15.5:0/2967522299 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90fc0cf4f0 con 0x7f90fc06d4f0 2024-03-24T10:57:24.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.137+0000 7f9101274640 1 -- 172.21.15.5:0/2967522299 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90fc0681d0 con 0x7f90fc06d4f0 2024-03-24T10:57:24.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.161+0000 7f90e37fe640 1 -- 172.21.15.5:0/2967522299 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f90ec0417a0 con 0x7f90fc06d4f0 2024-03-24T10:57:24.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.162+0000 7f90e37fe640 1 --2- 172.21.15.5:0/2967522299 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f90e806b810 0x7f90e806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:24.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.162+0000 7f90fa7fc640 1 --2- 172.21.15.5:0/2967522299 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f90e806b810 0x7f90e806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:24.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.162+0000 7f90fa7fc640 1 --2- 172.21.15.5:0/2967522299 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f90e806b810 0x7f90e806dcd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f90fc1edc50 tx=0x7f90f0009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:24.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.416+0000 7f90e37fe640 1 -- 172.21.15.5:0/2967522299 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f90ec03c070 con 0x7f90fc06d4f0 2024-03-24T10:57:24.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.427+0000 7f90e37fe640 1 -- 172.21.15.5:0/2967522299 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f90ec0783f0 con 0x7f90fc06d4f0 2024-03-24T10:57:24.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.684+0000 7f9101274640 1 -- 172.21.15.5:0/2967522299 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90fc1f0020 con 0x7f90fc06d4f0 2024-03-24T10:57:24.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.700+0000 7f90e37fe640 1 -- 172.21.15.5:0/2967522299 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f90ec07bab0 con 0x7f90fc06d4f0 2024-03-24T10:57:24.700 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:24.700 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:57:24.701 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:57:24.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.702+0000 7f9101274640 1 -- 172.21.15.5:0/2967522299 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f90e806b810 msgr2=0x7f90e806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:24.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.702+0000 7f9101274640 1 --2- 172.21.15.5:0/2967522299 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f90e806b810 0x7f90e806dcd0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f90fc1edc50 tx=0x7f90f0009190 comp rx=0 tx=0).stop 2024-03-24T10:57:24.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.702+0000 7f9101274640 1 -- 172.21.15.5:0/2967522299 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90fc06d4f0 msgr2=0x7f90fc15d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:24.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.702+0000 7f9101274640 1 --2- 172.21.15.5:0/2967522299 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90fc06d4f0 0x7f90fc15d630 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f90ec033000 tx=0x7f90ec004730 comp rx=0 tx=0).stop 2024-03-24T10:57:24.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.702+0000 7f9101274640 1 -- 172.21.15.5:0/2967522299 shutdown_connections 2024-03-24T10:57:24.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.702+0000 7f9101274640 1 --2- 172.21.15.5:0/2967522299 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f90e806b810 0x7f90e806dcd0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:24.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.702+0000 7f9101274640 1 --2- 172.21.15.5:0/2967522299 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f90fc0ccaa0 0x7f90fc1f06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:24.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.702+0000 7f9101274640 1 --2- 172.21.15.5:0/2967522299 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f90fc1543a0 0x7f90fc1f0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:24.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.702+0000 7f9101274640 1 --2- 172.21.15.5:0/2967522299 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90fc06d4f0 0x7f90fc15d630 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:24.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.702+0000 7f9101274640 1 -- 172.21.15.5:0/2967522299 >> 172.21.15.5:0/2967522299 conn(0x7f90fc072ef0 msgr2=0x7f90fc06c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:24.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.702+0000 7f9101274640 1 -- 172.21.15.5:0/2967522299 shutdown_connections 2024-03-24T10:57:24.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:24.702+0000 7f9101274640 1 -- 172.21.15.5:0/2967522299 wait complete. 2024-03-24T10:57:25.319 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:57:25.319 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:26.320 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:26.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.423+0000 7fbdcdd13640 1 Processor -- start 2024-03-24T10:57:26.423 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.423+0000 7fbdcdd13640 1 -- start start 2024-03-24T10:57:26.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.424+0000 7fbdcdd13640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbdc81543a0 0x7fbdc815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:26.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.424+0000 7fbdcdd13640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbdc806d4f0 0x7fbdc80cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:26.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.424+0000 7fbdcdd13640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdc80cb2f0 con 0x7fbdc80ccaa0 2024-03-24T10:57:26.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.424+0000 7fbdcdd13640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fbdc80c6750 con 0x7fbdc806d4f0 2024-03-24T10:57:26.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.424+0000 7fbdcdd13640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fbdc80cc380 con 0x7fbdc81543a0 2024-03-24T10:57:26.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.425+0000 7fbdc6ffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbdc806d4f0 0x7fbdc80cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:26.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.425+0000 7fbdc77fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fbdc81543a0 0x7fbdc815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:26.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.425+0000 7fbdc6ffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fbdc806d4f0 0x7fbdc80cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38982/0 (socket says 172.21.15.5:38982) 2024-03-24T10:57:26.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.425+0000 7fbdc6ffd640 1 -- 172.21.15.5:0/4165225766 learned_addr learned my addr 172.21.15.5:0/4165225766 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:26.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.428+0000 7fbdc6ffd640 1 -- 172.21.15.5:0/4165225766 >> v2:172.21.15.57:3300/0 conn(0x7fbdc81543a0 msgr2=0x7fbdc815ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:26.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.428+0000 7fbdc6ffd640 1 --2- 172.21.15.5:0/4165225766 >> v2:172.21.15.57:3300/0 conn(0x7fbdc81543a0 0x7fbdc815ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:26.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.428+0000 7fbdc6ffd640 1 -- 172.21.15.5:0/4165225766 >> v1:172.21.15.57:6789/0 conn(0x7fbdc80ccaa0 legacy=0x7fbdc806bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:26.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.428+0000 7fbdc6ffd640 1 -- 172.21.15.5:0/4165225766 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc80ca7e0 con 0x7fbdc806d4f0 2024-03-24T10:57:26.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.430+0000 7fbdc6ffd640 1 --2- 172.21.15.5:0/4165225766 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbdc806d4f0 0x7fbdc80cc560 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fbdc80cd600 tx=0x7fbdbc02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cdb88558e3b853e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:26.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.434+0000 7fbdc67fc640 1 -- 172.21.15.5:0/4165225766 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc02f850 con 0x7fbdc806d4f0 2024-03-24T10:57:26.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.434+0000 7fbdc67fc640 1 -- 172.21.15.5:0/4165225766 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdbc02f9b0 con 0x7fbdc806d4f0 2024-03-24T10:57:26.434 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.434+0000 7fbdc67fc640 1 -- 172.21.15.5:0/4165225766 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc02fd40 con 0x7fbdc806d4f0 2024-03-24T10:57:26.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.435+0000 7fbdcdd13640 1 -- 172.21.15.5:0/4165225766 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbdc806d4f0 msgr2=0x7fbdc80cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:26.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.435+0000 7fbdcdd13640 1 --2- 172.21.15.5:0/4165225766 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbdc806d4f0 0x7fbdc80cc560 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fbdc80cd600 tx=0x7fbdbc02ba30 comp rx=0 tx=0).stop 2024-03-24T10:57:26.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.435+0000 7fbdcdd13640 1 -- 172.21.15.5:0/4165225766 shutdown_connections 2024-03-24T10:57:26.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.435+0000 7fbdcdd13640 1 --2- 172.21.15.5:0/4165225766 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbdc806d4f0 0x7fbdc80cc560 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:26.435 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.435+0000 7fbdcdd13640 1 --2- 172.21.15.5:0/4165225766 >> v2:172.21.15.57:3300/0 conn(0x7fbdc81543a0 0x7fbdc815ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:26.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.435+0000 7fbdcdd13640 1 -- 172.21.15.5:0/4165225766 >> 172.21.15.5:0/4165225766 conn(0x7fbdc8072ef0 msgr2=0x7fbdc8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:26.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.435+0000 7fbdcdd13640 1 -- 172.21.15.5:0/4165225766 shutdown_connections 2024-03-24T10:57:26.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.435+0000 7fbdcdd13640 1 -- 172.21.15.5:0/4165225766 wait complete. 2024-03-24T10:57:26.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.437+0000 7fbdcdd13640 1 Processor -- start 2024-03-24T10:57:26.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.437+0000 7fbdcdd13640 1 -- start start 2024-03-24T10:57:26.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.438+0000 7fbdcdd13640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdc806d4f0 0x7fbdc80c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:26.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.438+0000 7fbdcdd13640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbdc80ccaa0 0x7fbdc81f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:26.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.438+0000 7fbdcdd13640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbdc81543a0 0x7fbdc8159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:26.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.438+0000 7fbdcdd13640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc81f40f0 con 0x7fbdc81543a0 2024-03-24T10:57:26.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.438+0000 7fbdcdd13640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc8151690 con 0x7fbdc806d4f0 2024-03-24T10:57:26.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.438+0000 7fbdcdd13640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fbdc8150a90 con 0x7fbdc80ccaa0 2024-03-24T10:57:26.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.439+0000 7fbdc7fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbdc81543a0 0x7fbdc8159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:26.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.439+0000 7fbdc77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdc806d4f0 0x7fbdc80c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:26.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.439+0000 7fbdc6ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbdc80ccaa0 0x7fbdc81f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:26.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.439+0000 7fbdc7fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbdc81543a0 0x7fbdc8159350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38996/0 (socket says 172.21.15.5:38996) 2024-03-24T10:57:26.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.439+0000 7fbdc7fff640 1 -- 172.21.15.5:0/3434694461 learned_addr learned my addr 172.21.15.5:0/3434694461 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:26.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.443+0000 7fbdc7fff640 1 -- 172.21.15.5:0/3434694461 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbdc80ccaa0 msgr2=0x7fbdc81f3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:26.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.443+0000 7fbdc7fff640 1 --2- 172.21.15.5:0/3434694461 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbdc80ccaa0 0x7fbdc81f3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:26.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.443+0000 7fbdc7fff640 1 -- 172.21.15.5:0/3434694461 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdc806d4f0 msgr2=0x7fbdc80c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:26.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.443+0000 7fbdc7fff640 1 --2- 172.21.15.5:0/3434694461 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdc806d4f0 0x7fbdc80c9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:26.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.443+0000 7fbdc7fff640 1 -- 172.21.15.5:0/3434694461 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdbc0094d0 con 0x7fbdc81543a0 2024-03-24T10:57:26.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.445+0000 7fbdc7fff640 1 --2- 172.21.15.5:0/3434694461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbdc81543a0 0x7fbdc8159350 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0001eb0 tx=0x7fbdc000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:26.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.449+0000 7fbdaffff640 1 -- 172.21.15.5:0/3434694461 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc00052d0 con 0x7fbdc81543a0 2024-03-24T10:57:26.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.449+0000 7fbdaffff640 1 -- 172.21.15.5:0/3434694461 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdc0005430 con 0x7fbdc81543a0 2024-03-24T10:57:26.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.449+0000 7fbdaffff640 1 -- 172.21.15.5:0/3434694461 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0003810 con 0x7fbdc81543a0 2024-03-24T10:57:26.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.449+0000 7fbdcdd13640 1 -- 172.21.15.5:0/3434694461 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdc8150d30 con 0x7fbdc81543a0 2024-03-24T10:57:26.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.450+0000 7fbdcdd13640 1 -- 172.21.15.5:0/3434694461 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdc81efdd0 con 0x7fbdc81543a0 2024-03-24T10:57:26.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.450+0000 7fbdcdd13640 1 -- 172.21.15.5:0/3434694461 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdc80c9ff0 con 0x7fbdc81543a0 2024-03-24T10:57:26.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.475+0000 7fbdaffff640 1 -- 172.21.15.5:0/3434694461 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbdc0003970 con 0x7fbdc81543a0 2024-03-24T10:57:26.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.475+0000 7fbdaffff640 1 --2- 172.21.15.5:0/3434694461 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbdb406b860 0x7fbdb406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:26.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.476+0000 7fbdc77fe640 1 --2- 172.21.15.5:0/3434694461 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbdb406b860 0x7fbdb406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:26.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.476+0000 7fbdc77fe640 1 --2- 172.21.15.5:0/3434694461 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbdb406b860 0x7fbdb406dd20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8011340 tx=0x7fbdb800b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:26.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.789+0000 7fbdaffff640 1 -- 172.21.15.5:0/3434694461 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fbdc00583b0 con 0x7fbdc81543a0 2024-03-24T10:57:26.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:26.789+0000 7fbdaffff640 1 -- 172.21.15.5:0/3434694461 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbdc0054650 con 0x7fbdc81543a0 2024-03-24T10:57:27.010 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:27.010+0000 7fbdcdd13640 1 -- 172.21.15.5:0/3434694461 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbdc81efc30 con 0x7fbdc81543a0 2024-03-24T10:57:27.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:27.025+0000 7fbdaffff640 1 -- 172.21.15.5:0/3434694461 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7fbdc81efc30 con 0x7fbdc81543a0 2024-03-24T10:57:27.025 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:27.025 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:57:27.026 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:57:27.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:27.027+0000 7fbdcdd13640 1 -- 172.21.15.5:0/3434694461 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbdb406b860 msgr2=0x7fbdb406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:27.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:27.027+0000 7fbdcdd13640 1 --2- 172.21.15.5:0/3434694461 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbdb406b860 0x7fbdb406dd20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fbdb8011340 tx=0x7fbdb800b040 comp rx=0 tx=0).stop 2024-03-24T10:57:27.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:27.027+0000 7fbdcdd13640 1 -- 172.21.15.5:0/3434694461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbdc81543a0 msgr2=0x7fbdc8159350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:27.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:27.027+0000 7fbdcdd13640 1 --2- 172.21.15.5:0/3434694461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbdc81543a0 0x7fbdc8159350 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0001eb0 tx=0x7fbdc000c890 comp rx=0 tx=0).stop 2024-03-24T10:57:27.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:27.027+0000 7fbdcdd13640 1 -- 172.21.15.5:0/3434694461 shutdown_connections 2024-03-24T10:57:27.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:27.027+0000 7fbdcdd13640 1 --2- 172.21.15.5:0/3434694461 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fbdb406b860 0x7fbdb406dd20 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:27.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:27.027+0000 7fbdcdd13640 1 --2- 172.21.15.5:0/3434694461 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fbdc81543a0 0x7fbdc8159350 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:27.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:27.027+0000 7fbdcdd13640 1 --2- 172.21.15.5:0/3434694461 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fbdc80ccaa0 0x7fbdc81f3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:27.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:27.027+0000 7fbdcdd13640 1 --2- 172.21.15.5:0/3434694461 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fbdc806d4f0 0x7fbdc80c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:27.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:27.027+0000 7fbdcdd13640 1 -- 172.21.15.5:0/3434694461 >> 172.21.15.5:0/3434694461 conn(0x7fbdc8072ef0 msgr2=0x7fbdc806be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:27.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:27.027+0000 7fbdcdd13640 1 -- 172.21.15.5:0/3434694461 shutdown_connections 2024-03-24T10:57:27.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:27.027+0000 7fbdcdd13640 1 -- 172.21.15.5:0/3434694461 wait complete. 2024-03-24T10:57:27.642 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:57:27.642 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:28.643 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:28.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.745+0000 7fcc14c0e640 1 Processor -- start 2024-03-24T10:57:28.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.745+0000 7fcc14c0e640 1 -- start start 2024-03-24T10:57:28.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.747+0000 7fcc14c0e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcc1015ac00 0x7fcc100cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:28.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.747+0000 7fcc14c0e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc100ccaa0 0x7fcc1006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:28.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.747+0000 7fcc14c0e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc100cb2f0 con 0x7fcc101543a0 2024-03-24T10:57:28.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.747+0000 7fcc14c0e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fcc100c6750 con 0x7fcc1015ac00 2024-03-24T10:57:28.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.747+0000 7fcc14c0e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fcc100cceb0 con 0x7fcc100ccaa0 2024-03-24T10:57:28.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.747+0000 7fcc07fff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcc1015ac00 0x7fcc100cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:28.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.747+0000 7fcc0ed9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc100ccaa0 0x7fcc1006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:28.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.748+0000 7fcc07fff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fcc1015ac00 0x7fcc100cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39010/0 (socket says 172.21.15.5:39010) 2024-03-24T10:57:28.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.748+0000 7fcc07fff640 1 -- 172.21.15.5:0/4272240437 learned_addr learned my addr 172.21.15.5:0/4272240437 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:28.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.750+0000 7fcc07fff640 1 -- 172.21.15.5:0/4272240437 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc100ccaa0 msgr2=0x7fcc1006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:28.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.750+0000 7fcc07fff640 1 --2- 172.21.15.5:0/4272240437 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc100ccaa0 0x7fcc1006bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:28.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.750+0000 7fcc07fff640 1 -- 172.21.15.5:0/4272240437 >> v1:172.21.15.5:6789/0 conn(0x7fcc101543a0 legacy=0x7fcc1006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:28.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.750+0000 7fcc07fff640 1 -- 172.21.15.5:0/4272240437 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1015b070 con 0x7fcc1015ac00 2024-03-24T10:57:28.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.752+0000 7fcc07fff640 1 --2- 172.21.15.5:0/4272240437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc1015ac00 0x7fcc100cc560 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009870 tx=0x7fcbf802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82496b6bbb3360c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:28.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.757+0000 7fcc0dd9b640 1 -- 172.21.15.5:0/4272240437 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8031640 con 0x7fcc1015ac00 2024-03-24T10:57:28.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.757+0000 7fcc0dd9b640 1 -- 172.21.15.5:0/4272240437 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbf80317a0 con 0x7fcc1015ac00 2024-03-24T10:57:28.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.757+0000 7fcc0dd9b640 1 -- 172.21.15.5:0/4272240437 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8031b30 con 0x7fcc1015ac00 2024-03-24T10:57:28.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.758+0000 7fcc14c0e640 1 -- 172.21.15.5:0/4272240437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc1015ac00 msgr2=0x7fcc100cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:28.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.758+0000 7fcc14c0e640 1 --2- 172.21.15.5:0/4272240437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc1015ac00 0x7fcc100cc560 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009870 tx=0x7fcbf802b8f0 comp rx=0 tx=0).stop 2024-03-24T10:57:28.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.758+0000 7fcc14c0e640 1 -- 172.21.15.5:0/4272240437 shutdown_connections 2024-03-24T10:57:28.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.758+0000 7fcc14c0e640 1 --2- 172.21.15.5:0/4272240437 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc100ccaa0 0x7fcc1006bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:28.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.758+0000 7fcc14c0e640 1 --2- 172.21.15.5:0/4272240437 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc1015ac00 0x7fcc100cc560 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:28.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.758+0000 7fcc14c0e640 1 -- 172.21.15.5:0/4272240437 >> 172.21.15.5:0/4272240437 conn(0x7fcc10072ef0 msgr2=0x7fcc10061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:28.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.758+0000 7fcc14c0e640 1 -- 172.21.15.5:0/4272240437 shutdown_connections 2024-03-24T10:57:28.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.758+0000 7fcc14c0e640 1 -- 172.21.15.5:0/4272240437 wait complete. 2024-03-24T10:57:28.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.760+0000 7fcc14c0e640 1 Processor -- start 2024-03-24T10:57:28.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.760+0000 7fcc14c0e640 1 -- start start 2024-03-24T10:57:28.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.762+0000 7fcc14c0e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc100ccaa0 0x7fcc100b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:28.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.762+0000 7fcc14c0e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc101543a0 0x7fcc100b3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:28.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.762+0000 7fcc14c0e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc1015ac00 0x7fcc10159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:28.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.762+0000 7fcc14c0e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fcc10152880 con 0x7fcc100ccaa0 2024-03-24T10:57:28.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.762+0000 7fcc14c0e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fcc101f08c0 con 0x7fcc1015ac00 2024-03-24T10:57:28.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.762+0000 7fcc14c0e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fcc10158bb0 con 0x7fcc101543a0 2024-03-24T10:57:28.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.762+0000 7fcc0e59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc100ccaa0 0x7fcc100b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:28.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.762+0000 7fcc07fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc101543a0 0x7fcc100b3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:28.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.762+0000 7fcc0ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc1015ac00 0x7fcc10159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:28.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.762+0000 7fcc0e59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc100ccaa0 0x7fcc100b4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39014/0 (socket says 172.21.15.5:39014) 2024-03-24T10:57:28.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.762+0000 7fcc0e59c640 1 -- 172.21.15.5:0/3839360885 learned_addr learned my addr 172.21.15.5:0/3839360885 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:28.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.766+0000 7fcc0e59c640 1 -- 172.21.15.5:0/3839360885 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc101543a0 msgr2=0x7fcc100b3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:28.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.766+0000 7fcc0e59c640 1 --2- 172.21.15.5:0/3839360885 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc101543a0 0x7fcc100b3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:28.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.766+0000 7fcc0e59c640 1 -- 172.21.15.5:0/3839360885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc1015ac00 msgr2=0x7fcc10159040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:28.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.766+0000 7fcc0e59c640 1 --2- 172.21.15.5:0/3839360885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc1015ac00 0x7fcc10159040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:28.767 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.766+0000 7fcc0e59c640 1 -- 172.21.15.5:0/3839360885 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf80094d0 con 0x7fcc100ccaa0 2024-03-24T10:57:28.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.768+0000 7fcc0e59c640 1 --2- 172.21.15.5:0/3839360885 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc100ccaa0 0x7fcc100b4400 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000c410 tx=0x7fcc0000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:28.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.772+0000 7fcc06ffd640 1 -- 172.21.15.5:0/3839360885 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc000035f0 con 0x7fcc100ccaa0 2024-03-24T10:57:28.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.772+0000 7fcc06ffd640 1 -- 172.21.15.5:0/3839360885 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc00003750 con 0x7fcc100ccaa0 2024-03-24T10:57:28.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.772+0000 7fcc06ffd640 1 -- 172.21.15.5:0/3839360885 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc00003b40 con 0x7fcc100ccaa0 2024-03-24T10:57:28.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.772+0000 7fcc14c0e640 1 -- 172.21.15.5:0/3839360885 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc101f0dd0 con 0x7fcc100ccaa0 2024-03-24T10:57:28.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.772+0000 7fcc14c0e640 1 -- 172.21.15.5:0/3839360885 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc10158280 con 0x7fcc100ccaa0 2024-03-24T10:57:28.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:28.773+0000 7fcc14c0e640 1 -- 172.21.15.5:0/3839360885 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc100b5e70 con 0x7fcc100ccaa0 2024-03-24T10:57:29.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:29.041+0000 7fcc06ffd640 1 -- 172.21.15.5:0/3839360885 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcc00003ca0 con 0x7fcc100ccaa0 2024-03-24T10:57:29.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:29.041+0000 7fcc06ffd640 1 --2- 172.21.15.5:0/3839360885 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcbf006b890 0x7fcbf006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:29.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:29.041+0000 7fcc07fff640 1 --2- 172.21.15.5:0/3839360885 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcbf006b890 0x7fcbf006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:29.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:29.042+0000 7fcc07fff640 1 --2- 172.21.15.5:0/3839360885 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcbf006b890 0x7fcbf006dd50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8036000 tx=0x7fcbf803c040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:29.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:29.056+0000 7fcc06ffd640 1 -- 172.21.15.5:0/3839360885 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fcc000010c0 con 0x7fcc100ccaa0 2024-03-24T10:57:29.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:29.056+0000 7fcc06ffd640 1 -- 172.21.15.5:0/3839360885 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcc00054740 con 0x7fcc100ccaa0 2024-03-24T10:57:29.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:29.296+0000 7fcc14c0e640 1 -- 172.21.15.5:0/3839360885 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc100681d0 con 0x7fcc100ccaa0 2024-03-24T10:57:29.319 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:29.318+0000 7fcc06ffd640 1 -- 172.21.15.5:0/3839360885 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7fcc00017020 con 0x7fcc100ccaa0 2024-03-24T10:57:29.319 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:29.319 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:57:29.319 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:57:29.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:29.320+0000 7fcc14c0e640 1 -- 172.21.15.5:0/3839360885 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcbf006b890 msgr2=0x7fcbf006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:29.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:29.320+0000 7fcc14c0e640 1 --2- 172.21.15.5:0/3839360885 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcbf006b890 0x7fcbf006dd50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8036000 tx=0x7fcbf803c040 comp rx=0 tx=0).stop 2024-03-24T10:57:29.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:29.320+0000 7fcc14c0e640 1 -- 172.21.15.5:0/3839360885 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc100ccaa0 msgr2=0x7fcc100b4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:29.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:29.320+0000 7fcc14c0e640 1 --2- 172.21.15.5:0/3839360885 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc100ccaa0 0x7fcc100b4400 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000c410 tx=0x7fcc0000c8d0 comp rx=0 tx=0).stop 2024-03-24T10:57:29.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:29.320+0000 7fcc14c0e640 1 -- 172.21.15.5:0/3839360885 shutdown_connections 2024-03-24T10:57:29.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:29.320+0000 7fcc14c0e640 1 --2- 172.21.15.5:0/3839360885 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fcbf006b890 0x7fcbf006dd50 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:29.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:29.320+0000 7fcc14c0e640 1 --2- 172.21.15.5:0/3839360885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcc1015ac00 0x7fcc10159040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:29.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:29.320+0000 7fcc14c0e640 1 --2- 172.21.15.5:0/3839360885 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fcc101543a0 0x7fcc100b3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:29.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:29.320+0000 7fcc14c0e640 1 --2- 172.21.15.5:0/3839360885 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fcc100ccaa0 0x7fcc100b4400 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:29.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:29.320+0000 7fcc14c0e640 1 -- 172.21.15.5:0/3839360885 >> 172.21.15.5:0/3839360885 conn(0x7fcc10072ef0 msgr2=0x7fcc101507f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:29.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:29.320+0000 7fcc14c0e640 1 -- 172.21.15.5:0/3839360885 shutdown_connections 2024-03-24T10:57:29.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:29.320+0000 7fcc14c0e640 1 -- 172.21.15.5:0/3839360885 wait complete. 2024-03-24T10:57:29.936 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:57:29.936 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:30.937 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:31.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.042+0000 7ff0df577640 1 Processor -- start 2024-03-24T10:57:31.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.043+0000 7ff0df577640 1 -- start start 2024-03-24T10:57:31.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.043+0000 7ff0df577640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff0d80c58c0 0x7ff0d80caab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:31.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.043+0000 7ff0df577640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0d8156fc0 con 0x7ff0d80c9610 2024-03-24T10:57:31.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.043+0000 7ff0df577640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0d81572a0 con 0x7ff0d80c2ea0 2024-03-24T10:57:31.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.043+0000 7ff0df577640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7ff0d80cb210 con 0x7ff0d80c58c0 2024-03-24T10:57:31.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.044+0000 7ff0ddb14640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7ff0d80c9610 0x7ff0d8150840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:36800/0 (socket says 172.21.15.5:36800) 2024-03-24T10:57:31.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.044+0000 7ff0ddb14640 1 -- 172.21.15.5:0/3905083771 learned_addr learned my addr 172.21.15.5:0/3905083771 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:57:31.045 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.044+0000 7ff0dcb12640 1 --2- 172.21.15.5:0/3905083771 >> v2:172.21.15.57:3300/0 conn(0x7ff0d80c58c0 0x7ff0d80caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:31.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.047+0000 7ff0cffff640 1 -- 172.21.15.5:0/3905083771 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4017414847 0 0) 0x7ff0d8156fc0 con 0x7ff0d80c9610 2024-03-24T10:57:31.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.047+0000 7ff0cffff640 1 -- 172.21.15.5:0/3905083771 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0c4003520 con 0x7ff0d80c9610 2024-03-24T10:57:31.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.047+0000 7ff0dcb12640 1 -- 172.21.15.5:0/3905083771 >> v1:172.21.15.57:6789/0 conn(0x7ff0d80c2ea0 legacy=0x7ff0d80c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:31.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.047+0000 7ff0dcb12640 1 -- 172.21.15.5:0/3905083771 >> v1:172.21.15.5:6789/0 conn(0x7ff0d80c9610 legacy=0x7ff0d8150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:31.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.047+0000 7ff0dcb12640 1 -- 172.21.15.5:0/3905083771 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d8157580 con 0x7ff0d80c58c0 2024-03-24T10:57:31.054 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.054+0000 7ff0dcb12640 1 --2- 172.21.15.5:0/3905083771 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0d80c58c0 0x7ff0d80caab0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff0c80097a0 tx=0x7ff0c802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a4065ec4ac74182 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:31.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.056+0000 7ff0cffff640 1 -- 172.21.15.5:0/3905083771 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c802f810 con 0x7ff0d80c58c0 2024-03-24T10:57:31.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.059+0000 7ff0cffff640 1 -- 172.21.15.5:0/3905083771 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0c802f970 con 0x7ff0d80c58c0 2024-03-24T10:57:31.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.059+0000 7ff0cffff640 1 -- 172.21.15.5:0/3905083771 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c802fce0 con 0x7ff0d80c58c0 2024-03-24T10:57:31.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.059+0000 7ff0df577640 1 -- 172.21.15.5:0/3905083771 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0d80c58c0 msgr2=0x7ff0d80caab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:31.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.059+0000 7ff0df577640 1 --2- 172.21.15.5:0/3905083771 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0d80c58c0 0x7ff0d80caab0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff0c80097a0 tx=0x7ff0c802b9f0 comp rx=0 tx=0).stop 2024-03-24T10:57:31.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.059+0000 7ff0df577640 1 -- 172.21.15.5:0/3905083771 shutdown_connections 2024-03-24T10:57:31.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.059+0000 7ff0df577640 1 --2- 172.21.15.5:0/3905083771 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0d80c58c0 0x7ff0d80caab0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:31.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.059+0000 7ff0df577640 1 -- 172.21.15.5:0/3905083771 >> 172.21.15.5:0/3905083771 conn(0x7ff0d806b130 msgr2=0x7ff0d806b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:31.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.060+0000 7ff0df577640 1 -- 172.21.15.5:0/3905083771 shutdown_connections 2024-03-24T10:57:31.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.060+0000 7ff0df577640 1 -- 172.21.15.5:0/3905083771 wait complete. 2024-03-24T10:57:31.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.061+0000 7ff0df577640 1 Processor -- start 2024-03-24T10:57:31.062 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.061+0000 7ff0df577640 1 -- start start 2024-03-24T10:57:31.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.061+0000 7ff0df577640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0d80c2ea0 0x7ff0d8156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:31.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.061+0000 7ff0df577640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff0d80c58c0 0x7ff0d815abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:31.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.061+0000 7ff0df577640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0d80c9610 0x7ff0d81ebd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:31.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.061+0000 7ff0df577640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d8155410 con 0x7ff0d80c9610 2024-03-24T10:57:31.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.061+0000 7ff0df577640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d81556f0 con 0x7ff0d80c2ea0 2024-03-24T10:57:31.063 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.062+0000 7ff0df577640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff0d81559d0 con 0x7ff0d80c58c0 2024-03-24T10:57:31.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.062+0000 7ff0ddb14640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0d80c9610 0x7ff0d81ebd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:31.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.062+0000 7ff0dd313640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0d80c2ea0 0x7ff0d8156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:31.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.062+0000 7ff0dcb12640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff0d80c58c0 0x7ff0d815abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:31.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.062+0000 7ff0ddb14640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0d80c9610 0x7ff0d81ebd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39020/0 (socket says 172.21.15.5:39020) 2024-03-24T10:57:31.064 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.062+0000 7ff0ddb14640 1 -- 172.21.15.5:0/3020598625 learned_addr learned my addr 172.21.15.5:0/3020598625 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:31.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.066+0000 7ff0ddb14640 1 -- 172.21.15.5:0/3020598625 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff0d80c58c0 msgr2=0x7ff0d815abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:31.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.066+0000 7ff0ddb14640 1 --2- 172.21.15.5:0/3020598625 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff0d80c58c0 0x7ff0d815abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:31.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.066+0000 7ff0ddb14640 1 -- 172.21.15.5:0/3020598625 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0d80c2ea0 msgr2=0x7ff0d8156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:31.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.066+0000 7ff0ddb14640 1 --2- 172.21.15.5:0/3020598625 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0d80c2ea0 0x7ff0d8156790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:31.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.066+0000 7ff0ddb14640 1 -- 172.21.15.5:0/3020598625 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c8009400 con 0x7ff0d80c9610 2024-03-24T10:57:31.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.069+0000 7ff0ddb14640 1 --2- 172.21.15.5:0/3020598625 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0d80c9610 0x7ff0d81ebd50 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff0d000caa0 tx=0x7ff0d000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:31.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.072+0000 7ff0cdffb640 1 -- 172.21.15.5:0/3020598625 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d000ac00 con 0x7ff0d80c9610 2024-03-24T10:57:31.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.072+0000 7ff0cdffb640 1 -- 172.21.15.5:0/3020598625 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0d000ad60 con 0x7ff0d80c9610 2024-03-24T10:57:31.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.072+0000 7ff0df577640 1 -- 172.21.15.5:0/3020598625 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0d8155cb0 con 0x7ff0d80c9610 2024-03-24T10:57:31.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.072+0000 7ff0cdffb640 1 -- 172.21.15.5:0/3020598625 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0012620 con 0x7ff0d80c9610 2024-03-24T10:57:31.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.073+0000 7ff0df577640 1 -- 172.21.15.5:0/3020598625 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0d8156270 con 0x7ff0d80c9610 2024-03-24T10:57:31.074 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.073+0000 7ff0df577640 1 -- 172.21.15.5:0/3020598625 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0d8062a00 con 0x7ff0d80c9610 2024-03-24T10:57:31.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.093+0000 7ff0cdffb640 1 -- 172.21.15.5:0/3020598625 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff0d0007070 con 0x7ff0d80c9610 2024-03-24T10:57:31.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.094+0000 7ff0cdffb640 1 --2- 172.21.15.5:0/3020598625 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff0bc06b890 0x7ff0bc06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:31.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.094+0000 7ff0dd313640 1 --2- 172.21.15.5:0/3020598625 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff0bc06b890 0x7ff0bc06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:31.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.095+0000 7ff0dd313640 1 --2- 172.21.15.5:0/3020598625 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff0bc06b890 0x7ff0bc06dd50 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4001780 tx=0x7ff0d4009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:31.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.245+0000 7ff0cdffb640 1 -- 172.21.15.5:0/3020598625 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7ff0d0058d50 con 0x7ff0d80c9610 2024-03-24T10:57:31.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.354+0000 7ff0cdffb640 1 -- 172.21.15.5:0/3020598625 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff0d0054ff0 con 0x7ff0d80c9610 2024-03-24T10:57:31.581 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.580+0000 7ff0df577640 1 -- 172.21.15.5:0/3020598625 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0d8156460 con 0x7ff0d80c9610 2024-03-24T10:57:31.593 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.593+0000 7ff0cdffb640 1 -- 172.21.15.5:0/3020598625 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7ff0d0017090 con 0x7ff0d80c9610 2024-03-24T10:57:31.594 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:31.594 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:57:31.594 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:57:31.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.595+0000 7ff0df577640 1 -- 172.21.15.5:0/3020598625 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff0bc06b890 msgr2=0x7ff0bc06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:31.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.595+0000 7ff0df577640 1 --2- 172.21.15.5:0/3020598625 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff0bc06b890 0x7ff0bc06dd50 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff0d4001780 tx=0x7ff0d4009040 comp rx=0 tx=0).stop 2024-03-24T10:57:31.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.595+0000 7ff0df577640 1 -- 172.21.15.5:0/3020598625 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0d80c9610 msgr2=0x7ff0d81ebd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:31.596 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.595+0000 7ff0df577640 1 --2- 172.21.15.5:0/3020598625 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0d80c9610 0x7ff0d81ebd50 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff0d000caa0 tx=0x7ff0d000cf60 comp rx=0 tx=0).stop 2024-03-24T10:57:31.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.595+0000 7ff0df577640 1 -- 172.21.15.5:0/3020598625 shutdown_connections 2024-03-24T10:57:31.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.595+0000 7ff0df577640 1 --2- 172.21.15.5:0/3020598625 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff0bc06b890 0x7ff0bc06dd50 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:31.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.595+0000 7ff0df577640 1 --2- 172.21.15.5:0/3020598625 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff0d80c9610 0x7ff0d81ebd50 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:31.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.595+0000 7ff0df577640 1 --2- 172.21.15.5:0/3020598625 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff0d80c58c0 0x7ff0d815abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:31.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.595+0000 7ff0df577640 1 --2- 172.21.15.5:0/3020598625 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff0d80c2ea0 0x7ff0d8156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:31.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.595+0000 7ff0df577640 1 -- 172.21.15.5:0/3020598625 >> 172.21.15.5:0/3020598625 conn(0x7ff0d806b130 msgr2=0x7ff0d814eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:31.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.595+0000 7ff0df577640 1 -- 172.21.15.5:0/3020598625 shutdown_connections 2024-03-24T10:57:31.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:31.595+0000 7ff0df577640 1 -- 172.21.15.5:0/3020598625 wait complete. 2024-03-24T10:57:32.212 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:57:32.213 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:32.878 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:57:32.880 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:57:33.214 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:33.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.315+0000 7f09eead0640 1 Processor -- start 2024-03-24T10:57:33.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.315+0000 7f09eead0640 1 -- start start 2024-03-24T10:57:33.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.316+0000 7f09eead0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f09e806f7b0 0x7f09e815af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:33.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.316+0000 7f09eead0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f09e80c6790 0x7f09e80c86e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:33.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.316+0000 7f09eead0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f09e806fb90 0x7f09e806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:33.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.316+0000 7f09eead0640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f09e815ab70 con 0x7f09e806f7b0 2024-03-24T10:57:33.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.316+0000 7f09eead0640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f09e80c6c60 con 0x7f09e80c6790 2024-03-24T10:57:33.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.316+0000 7f09eead0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f09e8070020 con 0x7f09e806fb90 2024-03-24T10:57:33.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.316+0000 7f09ec86c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f09e806f7b0 0x7f09e815af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:33.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.316+0000 7f09e7fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f09e80c6790 0x7f09e80c86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:33.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.316+0000 7f09ed06d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f09e806fb90 0x7f09e806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:33.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.317+0000 7f09ec86c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f09e806f7b0 0x7f09e815af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:33050/0 (socket says 172.21.15.5:33050) 2024-03-24T10:57:33.318 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.317+0000 7f09ec86c640 1 -- 172.21.15.5:0/3452838742 learned_addr learned my addr 172.21.15.5:0/3452838742 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:33.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.319+0000 7f09ec86c640 1 -- 172.21.15.5:0/3452838742 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f09e806fb90 msgr2=0x7f09e806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:33.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.319+0000 7f09ec86c640 1 --2- 172.21.15.5:0/3452838742 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f09e806fb90 0x7f09e806bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:33.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.319+0000 7f09ec86c640 1 -- 172.21.15.5:0/3452838742 >> v2:172.21.15.57:3300/0 conn(0x7f09e80c6790 msgr2=0x7f09e80c86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:33.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.319+0000 7f09ec86c640 1 --2- 172.21.15.5:0/3452838742 >> v2:172.21.15.57:3300/0 conn(0x7f09e80c6790 0x7f09e80c86e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:33.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.319+0000 7f09ec86c640 1 -- 172.21.15.5:0/3452838742 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e80c6f20 con 0x7f09e806f7b0 2024-03-24T10:57:33.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.321+0000 7f09ec86c640 1 --2- 172.21.15.5:0/3452838742 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09e806f7b0 0x7f09e815af90 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f09e8073910 tx=0x7f09dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25b0731f2feacdeb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:33.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.327+0000 7f09e77fe640 1 -- 172.21.15.5:0/3452838742 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09dc031870 con 0x7f09e806f7b0 2024-03-24T10:57:33.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.327+0000 7f09e77fe640 1 -- 172.21.15.5:0/3452838742 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09dc0319d0 con 0x7f09e806f7b0 2024-03-24T10:57:33.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.327+0000 7f09e77fe640 1 -- 172.21.15.5:0/3452838742 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09dc031d60 con 0x7f09e806f7b0 2024-03-24T10:57:33.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.327+0000 7f09eead0640 1 -- 172.21.15.5:0/3452838742 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09e806f7b0 msgr2=0x7f09e815af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:33.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.327+0000 7f09eead0640 1 --2- 172.21.15.5:0/3452838742 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09e806f7b0 0x7f09e815af90 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f09e8073910 tx=0x7f09dc02b910 comp rx=0 tx=0).stop 2024-03-24T10:57:33.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.328+0000 7f09eead0640 1 -- 172.21.15.5:0/3452838742 shutdown_connections 2024-03-24T10:57:33.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.328+0000 7f09eead0640 1 --2- 172.21.15.5:0/3452838742 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f09e806fb90 0x7f09e806bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:33.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.328+0000 7f09eead0640 1 --2- 172.21.15.5:0/3452838742 >> v2:172.21.15.57:3300/0 conn(0x7f09e80c6790 0x7f09e80c86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:33.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.328+0000 7f09eead0640 1 --2- 172.21.15.5:0/3452838742 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09e806f7b0 0x7f09e815af90 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:33.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.328+0000 7f09eead0640 1 -- 172.21.15.5:0/3452838742 >> 172.21.15.5:0/3452838742 conn(0x7f09e806d790 msgr2=0x7f09e806dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:33.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.328+0000 7f09eead0640 1 -- 172.21.15.5:0/3452838742 shutdown_connections 2024-03-24T10:57:33.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.328+0000 7f09eead0640 1 -- 172.21.15.5:0/3452838742 wait complete. 2024-03-24T10:57:33.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.329+0000 7f09eead0640 1 Processor -- start 2024-03-24T10:57:33.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.329+0000 7f09eead0640 1 -- start start 2024-03-24T10:57:33.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.330+0000 7f09eead0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f09e806f7b0 0x7f09e80c3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:33.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.330+0000 7f09eead0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09e806fb90 0x7f09e815dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:33.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.330+0000 7f09eead0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09e80c6790 0x7f09e815e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:33.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.330+0000 7f09eead0640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f09e806b2f0 con 0x7f09e80c6790 2024-03-24T10:57:33.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.330+0000 7f09eead0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f09e815d580 con 0x7f09e806fb90 2024-03-24T10:57:33.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.330+0000 7f09eead0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f09e80c38a0 con 0x7f09e806f7b0 2024-03-24T10:57:33.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.331+0000 7f09ed06d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09e80c6790 0x7f09e815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:33.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.331+0000 7f09ec86c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f09e806f7b0 0x7f09e80c3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:33.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.331+0000 7f09ed06d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09e80c6790 0x7f09e815e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:33064/0 (socket says 172.21.15.5:33064) 2024-03-24T10:57:33.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.331+0000 7f09e7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09e806fb90 0x7f09e815dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:33.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.331+0000 7f09ed06d640 1 -- 172.21.15.5:0/3870459983 learned_addr learned my addr 172.21.15.5:0/3870459983 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:33.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.334+0000 7f09ed06d640 1 -- 172.21.15.5:0/3870459983 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f09e806f7b0 msgr2=0x7f09e80c3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:33.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.334+0000 7f09ed06d640 1 --2- 172.21.15.5:0/3870459983 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f09e806f7b0 0x7f09e80c3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:33.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.334+0000 7f09ed06d640 1 -- 172.21.15.5:0/3870459983 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09e806fb90 msgr2=0x7f09e815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:33.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.334+0000 7f09ed06d640 1 --2- 172.21.15.5:0/3870459983 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09e806fb90 0x7f09e815dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:33.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.334+0000 7f09ed06d640 1 -- 172.21.15.5:0/3870459983 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09dc0094d0 con 0x7f09e80c6790 2024-03-24T10:57:33.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.337+0000 7f09ed06d640 1 --2- 172.21.15.5:0/3870459983 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09e80c6790 0x7f09e815e010 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f09e000ef00 tx=0x7f09e000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:33.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.341+0000 7f09e57fa640 1 -- 172.21.15.5:0/3870459983 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0010040 con 0x7f09e80c6790 2024-03-24T10:57:33.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.341+0000 7f09e57fa640 1 -- 172.21.15.5:0/3870459983 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09e000bcc0 con 0x7f09e80c6790 2024-03-24T10:57:33.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.342+0000 7f09e57fa640 1 -- 172.21.15.5:0/3870459983 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0014640 con 0x7f09e80c6790 2024-03-24T10:57:33.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.342+0000 7f09eead0640 1 -- 172.21.15.5:0/3870459983 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09e815b700 con 0x7f09e80c6790 2024-03-24T10:57:33.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.342+0000 7f09eead0640 1 -- 172.21.15.5:0/3870459983 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09e815bcc0 con 0x7f09e80c6790 2024-03-24T10:57:33.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.343+0000 7f09eead0640 1 -- 172.21.15.5:0/3870459983 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09e80c74c0 con 0x7f09e80c6790 2024-03-24T10:57:33.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.367+0000 7f09e57fa640 1 -- 172.21.15.5:0/3870459983 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f09e00147a0 con 0x7f09e80c6790 2024-03-24T10:57:33.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.367+0000 7f09e57fa640 1 --2- 172.21.15.5:0/3870459983 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f09cc06b890 0x7f09cc06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:33.368 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.368+0000 7f09ec86c640 1 --2- 172.21.15.5:0/3870459983 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f09cc06b890 0x7f09cc06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:33.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.368+0000 7f09ec86c640 1 --2- 172.21.15.5:0/3870459983 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f09cc06b890 0x7f09cc06dd50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f09dc036000 tx=0x7f09dc03c040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:33.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.630+0000 7f09e57fa640 1 -- 172.21.15.5:0/3870459983 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f09e000be30 con 0x7f09e80c6790 2024-03-24T10:57:33.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.630+0000 7f09e57fa640 1 -- 172.21.15.5:0/3870459983 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f09e00558e0 con 0x7f09e80c6790 2024-03-24T10:57:33.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.875+0000 7f09eead0640 1 -- 172.21.15.5:0/3870459983 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09e80c74c0 con 0x7f09e80c6790 2024-03-24T10:57:33.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.898+0000 7f09e57fa640 1 -- 172.21.15.5:0/3870459983 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f09e80c74c0 con 0x7f09e80c6790 2024-03-24T10:57:33.899 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:33.899 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:57:33.900 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:57:33.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.900+0000 7f09eead0640 1 -- 172.21.15.5:0/3870459983 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f09cc06b890 msgr2=0x7f09cc06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:33.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.900+0000 7f09eead0640 1 --2- 172.21.15.5:0/3870459983 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f09cc06b890 0x7f09cc06dd50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f09dc036000 tx=0x7f09dc03c040 comp rx=0 tx=0).stop 2024-03-24T10:57:33.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.900+0000 7f09eead0640 1 -- 172.21.15.5:0/3870459983 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09e80c6790 msgr2=0x7f09e815e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:33.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.900+0000 7f09eead0640 1 --2- 172.21.15.5:0/3870459983 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09e80c6790 0x7f09e815e010 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f09e000ef00 tx=0x7f09e000b480 comp rx=0 tx=0).stop 2024-03-24T10:57:33.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.900+0000 7f09eead0640 1 -- 172.21.15.5:0/3870459983 shutdown_connections 2024-03-24T10:57:33.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.900+0000 7f09eead0640 1 --2- 172.21.15.5:0/3870459983 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f09cc06b890 0x7f09cc06dd50 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:33.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.900+0000 7f09eead0640 1 --2- 172.21.15.5:0/3870459983 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f09e80c6790 0x7f09e815e010 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:33.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.900+0000 7f09eead0640 1 --2- 172.21.15.5:0/3870459983 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f09e806fb90 0x7f09e815dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:33.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.900+0000 7f09eead0640 1 --2- 172.21.15.5:0/3870459983 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f09e806f7b0 0x7f09e80c3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:33.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.900+0000 7f09eead0640 1 -- 172.21.15.5:0/3870459983 >> 172.21.15.5:0/3870459983 conn(0x7f09e806d790 msgr2=0x7f09e80726e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:33.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.900+0000 7f09eead0640 1 -- 172.21.15.5:0/3870459983 shutdown_connections 2024-03-24T10:57:33.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:33.900+0000 7f09eead0640 1 -- 172.21.15.5:0/3870459983 wait complete. 2024-03-24T10:57:34.517 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:57:34.517 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:35.518 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:35.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.622+0000 7f06982cd640 1 Processor -- start 2024-03-24T10:57:35.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.622+0000 7f06982cd640 1 -- start start 2024-03-24T10:57:35.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.623+0000 7f06982cd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f069015ac00 0x7f06900cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:35.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.623+0000 7f06982cd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06900ccaa0 0x7f069006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:35.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.623+0000 7f06982cd640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06900cb2f0 con 0x7f06901543a0 2024-03-24T10:57:35.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.623+0000 7f06982cd640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f06900c6750 con 0x7f069015ac00 2024-03-24T10:57:35.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.623+0000 7f06982cd640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f06900cceb0 con 0x7f06900ccaa0 2024-03-24T10:57:35.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.623+0000 7f069686a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06900ccaa0 0x7f069006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:35.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.624+0000 7f0695868640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f069015ac00 0x7f06900cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:35.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.624+0000 7f069686a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06900ccaa0 0x7f069006bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:46404/0 (socket says 172.21.15.5:46404) 2024-03-24T10:57:35.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.624+0000 7f069686a640 1 -- 172.21.15.5:0/2915098799 learned_addr learned my addr 172.21.15.5:0/2915098799 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:35.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.627+0000 7f0695868640 1 -- 172.21.15.5:0/2915098799 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06900ccaa0 msgr2=0x7f069006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:35.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.627+0000 7f0695868640 1 --2- 172.21.15.5:0/2915098799 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06900ccaa0 0x7f069006bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:35.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.627+0000 7f0695868640 1 -- 172.21.15.5:0/2915098799 >> v1:172.21.15.5:6789/0 conn(0x7f06901543a0 legacy=0x7f069006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:35.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.627+0000 7f0695868640 1 -- 172.21.15.5:0/2915098799 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f069015b070 con 0x7f069015ac00 2024-03-24T10:57:35.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.630+0000 7f0695868640 1 --2- 172.21.15.5:0/2915098799 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f069015ac00 0x7f06900cc560 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f06800097a0 tx=0x7f068002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26b72b096bf8f842 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:35.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.635+0000 7f0695067640 1 -- 172.21.15.5:0/2915098799 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0680031830 con 0x7f069015ac00 2024-03-24T10:57:35.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.635+0000 7f0695067640 1 -- 172.21.15.5:0/2915098799 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0680031990 con 0x7f069015ac00 2024-03-24T10:57:35.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.635+0000 7f0695067640 1 -- 172.21.15.5:0/2915098799 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0680031d20 con 0x7f069015ac00 2024-03-24T10:57:35.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.636+0000 7f06982cd640 1 -- 172.21.15.5:0/2915098799 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f069015ac00 msgr2=0x7f06900cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:35.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.636+0000 7f06982cd640 1 --2- 172.21.15.5:0/2915098799 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f069015ac00 0x7f06900cc560 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f06800097a0 tx=0x7f068002b8d0 comp rx=0 tx=0).stop 2024-03-24T10:57:35.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.636+0000 7f06982cd640 1 -- 172.21.15.5:0/2915098799 shutdown_connections 2024-03-24T10:57:35.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.636+0000 7f06982cd640 1 --2- 172.21.15.5:0/2915098799 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06900ccaa0 0x7f069006bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:35.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.636+0000 7f06982cd640 1 --2- 172.21.15.5:0/2915098799 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f069015ac00 0x7f06900cc560 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:35.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.636+0000 7f06982cd640 1 -- 172.21.15.5:0/2915098799 >> 172.21.15.5:0/2915098799 conn(0x7f0690072ef0 msgr2=0x7f0690061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:35.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.636+0000 7f06982cd640 1 -- 172.21.15.5:0/2915098799 shutdown_connections 2024-03-24T10:57:35.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.636+0000 7f06982cd640 1 -- 172.21.15.5:0/2915098799 wait complete. 2024-03-24T10:57:35.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.637+0000 7f06982cd640 1 Processor -- start 2024-03-24T10:57:35.637 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.637+0000 7f06982cd640 1 -- start start 2024-03-24T10:57:35.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.638+0000 7f06982cd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06900ccaa0 0x7f06900b4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:35.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.638+0000 7f06982cd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06901543a0 0x7f0690159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:35.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.638+0000 7f06982cd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f069015ac00 0x7f0690149a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:35.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.638+0000 7f06982cd640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f0690158fb0 con 0x7f069015ac00 2024-03-24T10:57:35.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.638+0000 7f06982cd640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f06901f06e0 con 0x7f06901543a0 2024-03-24T10:57:35.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.638+0000 7f06982cd640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f06900b2510 con 0x7f06900ccaa0 2024-03-24T10:57:35.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.639+0000 7f069686a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f069015ac00 0x7f0690149a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:35.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.639+0000 7f0695868640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06901543a0 0x7f0690159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:35.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.639+0000 7f069686a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f069015ac00 0x7f0690149a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:33080/0 (socket says 172.21.15.5:33080) 2024-03-24T10:57:35.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.639+0000 7f069686a640 1 -- 172.21.15.5:0/4256169659 learned_addr learned my addr 172.21.15.5:0/4256169659 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:35.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.642+0000 7f069686a640 1 -- 172.21.15.5:0/4256169659 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06900ccaa0 msgr2=0x7f06900b4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:35.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.642+0000 7f069686a640 1 --2- 172.21.15.5:0/4256169659 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06900ccaa0 0x7f06900b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:35.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.642+0000 7f069686a640 1 -- 172.21.15.5:0/4256169659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06901543a0 msgr2=0x7f0690159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:35.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.642+0000 7f069686a640 1 --2- 172.21.15.5:0/4256169659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06901543a0 0x7f0690159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:35.643 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.642+0000 7f069686a640 1 -- 172.21.15.5:0/4256169659 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0680009400 con 0x7f069015ac00 2024-03-24T10:57:35.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.645+0000 7f069686a640 1 --2- 172.21.15.5:0/4256169659 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f069015ac00 0x7f0690149a40 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f068800efc0 tx=0x7f068800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:35.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.649+0000 7f0686ffd640 1 -- 172.21.15.5:0/4256169659 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f068800cd20 con 0x7f069015ac00 2024-03-24T10:57:35.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.649+0000 7f0686ffd640 1 -- 172.21.15.5:0/4256169659 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f068800f040 con 0x7f069015ac00 2024-03-24T10:57:35.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.649+0000 7f06982cd640 1 -- 172.21.15.5:0/4256169659 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0690155ef0 con 0x7f069015ac00 2024-03-24T10:57:35.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.650+0000 7f0686ffd640 1 -- 172.21.15.5:0/4256169659 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0688003bd0 con 0x7f069015ac00 2024-03-24T10:57:35.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.650+0000 7f06982cd640 1 -- 172.21.15.5:0/4256169659 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0690159200 con 0x7f069015ac00 2024-03-24T10:57:35.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.651+0000 7f06982cd640 1 -- 172.21.15.5:0/4256169659 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06900cb110 con 0x7f069015ac00 2024-03-24T10:57:35.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.675+0000 7f0686ffd640 1 -- 172.21.15.5:0/4256169659 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0688003d30 con 0x7f069015ac00 2024-03-24T10:57:35.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.675+0000 7f0686ffd640 1 --2- 172.21.15.5:0/4256169659 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f067406b840 0x7f067406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:35.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.675+0000 7f0696069640 1 --2- 172.21.15.5:0/4256169659 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f067406b840 0x7f067406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:35.675 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.676+0000 7f0696069640 1 --2- 172.21.15.5:0/4256169659 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f067406b840 0x7f067406dd00 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f068c000f60 tx=0x7f068c0079d0 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:35.679 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.679+0000 7f0686ffd640 1 -- 172.21.15.5:0/4256169659 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f068805a430 con 0x7f069015ac00 2024-03-24T10:57:35.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:35.972+0000 7f0686ffd640 1 -- 172.21.15.5:0/4256169659 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0688055fd0 con 0x7f069015ac00 2024-03-24T10:57:36.219 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:36.219+0000 7f06982cd640 1 -- 172.21.15.5:0/4256169659 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f06900681d0 con 0x7f069015ac00 2024-03-24T10:57:36.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:36.281+0000 7f0686ffd640 1 -- 172.21.15.5:0/4256169659 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f0688059690 con 0x7f069015ac00 2024-03-24T10:57:36.281 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:36.281 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:57:36.281 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:57:36.283 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:36.283+0000 7f06982cd640 1 -- 172.21.15.5:0/4256169659 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f067406b840 msgr2=0x7f067406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:36.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:36.283+0000 7f06982cd640 1 --2- 172.21.15.5:0/4256169659 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f067406b840 0x7f067406dd00 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f068c000f60 tx=0x7f068c0079d0 comp rx=0 tx=0).stop 2024-03-24T10:57:36.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:36.283+0000 7f06982cd640 1 -- 172.21.15.5:0/4256169659 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f069015ac00 msgr2=0x7f0690149a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:36.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:36.283+0000 7f06982cd640 1 --2- 172.21.15.5:0/4256169659 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f069015ac00 0x7f0690149a40 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f068800efc0 tx=0x7f068800c490 comp rx=0 tx=0).stop 2024-03-24T10:57:36.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:36.283+0000 7f06982cd640 1 -- 172.21.15.5:0/4256169659 shutdown_connections 2024-03-24T10:57:36.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:36.283+0000 7f06982cd640 1 --2- 172.21.15.5:0/4256169659 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f067406b840 0x7f067406dd00 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:36.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:36.283+0000 7f06982cd640 1 --2- 172.21.15.5:0/4256169659 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f069015ac00 0x7f0690149a40 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:36.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:36.283+0000 7f06982cd640 1 --2- 172.21.15.5:0/4256169659 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06901543a0 0x7f0690159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:36.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:36.283+0000 7f06982cd640 1 --2- 172.21.15.5:0/4256169659 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06900ccaa0 0x7f06900b4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:36.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:36.283+0000 7f06982cd640 1 -- 172.21.15.5:0/4256169659 >> 172.21.15.5:0/4256169659 conn(0x7f0690072ef0 msgr2=0x7f0690151bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:36.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:36.283+0000 7f06982cd640 1 -- 172.21.15.5:0/4256169659 shutdown_connections 2024-03-24T10:57:36.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:36.283+0000 7f06982cd640 1 -- 172.21.15.5:0/4256169659 wait complete. 2024-03-24T10:57:36.897 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:57:36.897 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:37.899 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:38.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.002+0000 7f8dc568f640 1 Processor -- start 2024-03-24T10:57:38.002 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.002+0000 7f8dc568f640 1 -- start start 2024-03-24T10:57:38.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.003+0000 7f8dc568f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8dc01543a0 0x7f8dc015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:38.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.003+0000 7f8dc568f640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dc00cb2f0 con 0x7f8dc006d4f0 2024-03-24T10:57:38.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.003+0000 7f8dc568f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dc00ca7e0 con 0x7f8dc006bf50 2024-03-24T10:57:38.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.003+0000 7f8dc568f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f8dc00c6750 con 0x7f8dc01543a0 2024-03-24T10:57:38.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.004+0000 7f8dbeffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8dc01543a0 0x7f8dc015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:38.003 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.004+0000 7f8dbe7fc640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f8dc006d4f0 0x7f8dc00cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:53580/0 (socket says 172.21.15.5:53580) 2024-03-24T10:57:38.004 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.004+0000 7f8dbe7fc640 1 -- 172.21.15.5:0/3140622805 learned_addr learned my addr 172.21.15.5:0/3140622805 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:57:38.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.006+0000 7f8dbdffb640 1 -- 172.21.15.5:0/3140622805 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1849631534 0 0) 0x7f8dc00ca7e0 con 0x7f8dc006bf50 2024-03-24T10:57:38.006 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.006+0000 7f8dbdffb640 1 -- 172.21.15.5:0/3140622805 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8da8003520 con 0x7f8dc006bf50 2024-03-24T10:57:38.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.007+0000 7f8dbdffb640 1 -- 172.21.15.5:0/3140622805 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3123806366 0 0) 0x7f8dc00cb2f0 con 0x7f8dc006d4f0 2024-03-24T10:57:38.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.007+0000 7f8dbdffb640 1 -- 172.21.15.5:0/3140622805 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dc00ca7e0 con 0x7f8dc006d4f0 2024-03-24T10:57:38.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.007+0000 7f8dbeffd640 1 -- 172.21.15.5:0/3140622805 >> v1:172.21.15.57:6789/0 conn(0x7f8dc006bf50 legacy=0x7f8dc006c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:38.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.007+0000 7f8dbeffd640 1 -- 172.21.15.5:0/3140622805 >> v1:172.21.15.5:6789/0 conn(0x7f8dc006d4f0 legacy=0x7f8dc00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:38.007 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.007+0000 7f8dbeffd640 1 -- 172.21.15.5:0/3140622805 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dc00cb520 con 0x7f8dc01543a0 2024-03-24T10:57:38.011 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.011+0000 7f8dbeffd640 1 --2- 172.21.15.5:0/3140622805 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8dc01543a0 0x7f8dc015ac00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8db00097d0 tx=0x7f8db002b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39c4351a99e4e95c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:38.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.016+0000 7f8dbdffb640 1 -- 172.21.15.5:0/3140622805 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0030880 con 0x7f8dc01543a0 2024-03-24T10:57:38.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.016+0000 7f8dbdffb640 1 -- 172.21.15.5:0/3140622805 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8db00309e0 con 0x7f8dc01543a0 2024-03-24T10:57:38.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.016+0000 7f8dbdffb640 1 -- 172.21.15.5:0/3140622805 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0030d70 con 0x7f8dc01543a0 2024-03-24T10:57:38.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.016+0000 7f8dc568f640 1 -- 172.21.15.5:0/3140622805 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8dc01543a0 msgr2=0x7f8dc015ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:38.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.016+0000 7f8dc568f640 1 --2- 172.21.15.5:0/3140622805 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8dc01543a0 0x7f8dc015ac00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8db00097d0 tx=0x7f8db002b9c0 comp rx=0 tx=0).stop 2024-03-24T10:57:38.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.016+0000 7f8dc568f640 1 -- 172.21.15.5:0/3140622805 shutdown_connections 2024-03-24T10:57:38.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.016+0000 7f8dc568f640 1 --2- 172.21.15.5:0/3140622805 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8dc01543a0 0x7f8dc015ac00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:38.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.016+0000 7f8dc568f640 1 -- 172.21.15.5:0/3140622805 >> 172.21.15.5:0/3140622805 conn(0x7f8dc0072ef0 msgr2=0x7f8dc0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:38.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.016+0000 7f8dc568f640 1 -- 172.21.15.5:0/3140622805 shutdown_connections 2024-03-24T10:57:38.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.016+0000 7f8dc568f640 1 -- 172.21.15.5:0/3140622805 wait complete. 2024-03-24T10:57:38.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.018+0000 7f8dc568f640 1 Processor -- start 2024-03-24T10:57:38.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.018+0000 7f8dc568f640 1 -- start start 2024-03-24T10:57:38.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.018+0000 7f8dc568f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8dc006bf50 0x7f8dc00ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:38.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.018+0000 7f8dc568f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8dc006d4f0 0x7f8dc00ce580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:38.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.018+0000 7f8dc568f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc01543a0 0x7f8dc00b3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:38.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.018+0000 7f8dc568f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc00b2fd0 con 0x7f8dc006bf50 2024-03-24T10:57:38.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.018+0000 7f8dc568f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc00cd8d0 con 0x7f8dc01543a0 2024-03-24T10:57:38.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.018+0000 7f8dc568f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f8dc00cdbb0 con 0x7f8dc006d4f0 2024-03-24T10:57:38.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.019+0000 7f8dbeffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8dc006bf50 0x7f8dc00ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:38.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.019+0000 7f8dbe7fc640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8dc006d4f0 0x7f8dc00ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:38.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.019+0000 7f8dbf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc01543a0 0x7f8dc00b3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:38.019 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.019+0000 7f8dbeffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8dc006bf50 0x7f8dc00ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:33090/0 (socket says 172.21.15.5:33090) 2024-03-24T10:57:38.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.019+0000 7f8dbeffd640 1 -- 172.21.15.5:0/902483092 learned_addr learned my addr 172.21.15.5:0/902483092 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:38.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.023+0000 7f8dbeffd640 1 -- 172.21.15.5:0/902483092 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8dc006d4f0 msgr2=0x7f8dc00ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:38.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.023+0000 7f8dbeffd640 1 --2- 172.21.15.5:0/902483092 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8dc006d4f0 0x7f8dc00ce580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:38.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.023+0000 7f8dbeffd640 1 -- 172.21.15.5:0/902483092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc01543a0 msgr2=0x7f8dc00b3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:38.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.023+0000 7f8dbeffd640 1 --2- 172.21.15.5:0/902483092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc01543a0 0x7f8dc00b3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:38.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.023+0000 7f8dbeffd640 1 -- 172.21.15.5:0/902483092 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db0009400 con 0x7f8dc006bf50 2024-03-24T10:57:38.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.025+0000 7f8dbeffd640 1 --2- 172.21.15.5:0/902483092 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8dc006bf50 0x7f8dc00ce040 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f8db002bfd0 tx=0x7f8db0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:38.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.030+0000 7f8da77fe640 1 -- 172.21.15.5:0/902483092 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db003e040 con 0x7f8dc006bf50 2024-03-24T10:57:38.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.030+0000 7f8da77fe640 1 -- 172.21.15.5:0/902483092 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8db0039cd0 con 0x7f8dc006bf50 2024-03-24T10:57:38.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.030+0000 7f8dc568f640 1 -- 172.21.15.5:0/902483092 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dc01f06e0 con 0x7f8dc006bf50 2024-03-24T10:57:38.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.030+0000 7f8da77fe640 1 -- 172.21.15.5:0/902483092 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0043680 con 0x7f8dc006bf50 2024-03-24T10:57:38.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.030+0000 7f8dc568f640 1 -- 172.21.15.5:0/902483092 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dc00b3500 con 0x7f8dc006bf50 2024-03-24T10:57:38.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.031+0000 7f8dc568f640 1 -- 172.21.15.5:0/902483092 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dc00681d0 con 0x7f8dc006bf50 2024-03-24T10:57:38.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.315+0000 7f8da77fe640 1 -- 172.21.15.5:0/902483092 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8db00423e0 con 0x7f8dc006bf50 2024-03-24T10:57:38.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.315+0000 7f8da77fe640 1 --2- 172.21.15.5:0/902483092 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8dac06b890 0x7f8dac06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:38.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.315+0000 7f8dbe7fc640 1 --2- 172.21.15.5:0/902483092 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8dac06b890 0x7f8dac06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:38.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.316+0000 7f8dbe7fc640 1 --2- 172.21.15.5:0/902483092 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8dac06b890 0x7f8dac06dd50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8dc014a270 tx=0x7f8db4009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:38.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.331+0000 7f8da77fe640 1 -- 172.21.15.5:0/902483092 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f8db007b1d0 con 0x7f8dc006bf50 2024-03-24T10:57:38.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.331+0000 7f8da77fe640 1 -- 172.21.15.5:0/902483092 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8db0077470 con 0x7f8dc006bf50 2024-03-24T10:57:38.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.556+0000 7f8dc568f640 1 -- 172.21.15.5:0/902483092 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8dc0149fc0 con 0x7f8dc006bf50 2024-03-24T10:57:38.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.568+0000 7f8da77fe640 1 -- 172.21.15.5:0/902483092 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f8db007ab30 con 0x7f8dc006bf50 2024-03-24T10:57:38.569 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:38.569 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:57:38.569 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:57:38.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.571+0000 7f8dc568f640 1 -- 172.21.15.5:0/902483092 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8dac06b890 msgr2=0x7f8dac06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:38.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.571+0000 7f8dc568f640 1 --2- 172.21.15.5:0/902483092 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8dac06b890 0x7f8dac06dd50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8dc014a270 tx=0x7f8db4009040 comp rx=0 tx=0).stop 2024-03-24T10:57:38.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.571+0000 7f8dc568f640 1 -- 172.21.15.5:0/902483092 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8dc006bf50 msgr2=0x7f8dc00ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:38.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.571+0000 7f8dc568f640 1 --2- 172.21.15.5:0/902483092 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8dc006bf50 0x7f8dc00ce040 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f8db002bfd0 tx=0x7f8db0004730 comp rx=0 tx=0).stop 2024-03-24T10:57:38.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.571+0000 7f8dc568f640 1 -- 172.21.15.5:0/902483092 shutdown_connections 2024-03-24T10:57:38.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.571+0000 7f8dc568f640 1 --2- 172.21.15.5:0/902483092 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8dac06b890 0x7f8dac06dd50 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:38.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.571+0000 7f8dc568f640 1 --2- 172.21.15.5:0/902483092 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8dc01543a0 0x7f8dc00b3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:38.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.571+0000 7f8dc568f640 1 --2- 172.21.15.5:0/902483092 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8dc006d4f0 0x7f8dc00ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:38.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.571+0000 7f8dc568f640 1 --2- 172.21.15.5:0/902483092 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8dc006bf50 0x7f8dc00ce040 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:38.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.571+0000 7f8dc568f640 1 -- 172.21.15.5:0/902483092 >> 172.21.15.5:0/902483092 conn(0x7f8dc0072ef0 msgr2=0x7f8dc00cc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:38.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.571+0000 7f8dc568f640 1 -- 172.21.15.5:0/902483092 shutdown_connections 2024-03-24T10:57:38.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:38.571+0000 7f8dc568f640 1 -- 172.21.15.5:0/902483092 wait complete. 2024-03-24T10:57:39.190 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:57:39.191 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:40.193 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:40.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.296+0000 7f81b6af4640 1 Processor -- start 2024-03-24T10:57:40.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.296+0000 7f81b6af4640 1 -- start start 2024-03-24T10:57:40.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.297+0000 7f81b6af4640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f81b01543a0 0x7f81b015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:40.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.297+0000 7f81b6af4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f81b006d4f0 0x7f81b00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:40.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.297+0000 7f81b6af4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81b00cb2f0 con 0x7f81b00ccaa0 2024-03-24T10:57:40.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.297+0000 7f81b6af4640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f81b00c6750 con 0x7f81b01543a0 2024-03-24T10:57:40.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.297+0000 7f81b6af4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f81b00cc380 con 0x7f81b006d4f0 2024-03-24T10:57:40.297 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.297+0000 7f81b4890640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f81b01543a0 0x7f81b015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:40.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.298+0000 7f81affff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f81b006d4f0 0x7f81b00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:40.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.298+0000 7f81b4890640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f81b01543a0 0x7f81b015ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:33092/0 (socket says 172.21.15.5:33092) 2024-03-24T10:57:40.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.298+0000 7f81b4890640 1 -- 172.21.15.5:0/1344848570 learned_addr learned my addr 172.21.15.5:0/1344848570 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:40.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.300+0000 7f81b4890640 1 -- 172.21.15.5:0/1344848570 >> v2:172.21.15.57:3300/0 conn(0x7f81b006d4f0 msgr2=0x7f81b00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:40.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.300+0000 7f81b4890640 1 --2- 172.21.15.5:0/1344848570 >> v2:172.21.15.57:3300/0 conn(0x7f81b006d4f0 0x7f81b00cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:40.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.300+0000 7f81b4890640 1 -- 172.21.15.5:0/1344848570 >> v1:172.21.15.5:6789/0 conn(0x7f81b00ccaa0 legacy=0x7f81b006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:40.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.300+0000 7f81b4890640 1 -- 172.21.15.5:0/1344848570 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81b00ca7e0 con 0x7f81b01543a0 2024-03-24T10:57:40.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.302+0000 7f81b4890640 1 --2- 172.21.15.5:0/1344848570 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81b01543a0 0x7f81b015ac00 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f81b00c6990 tx=0x7f81a402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79fb74f0ef88e608 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:40.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.307+0000 7f81af7fe640 1 -- 172.21.15.5:0/1344848570 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a4030860 con 0x7f81b01543a0 2024-03-24T10:57:40.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.307+0000 7f81af7fe640 1 -- 172.21.15.5:0/1344848570 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81a40309c0 con 0x7f81b01543a0 2024-03-24T10:57:40.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.307+0000 7f81af7fe640 1 -- 172.21.15.5:0/1344848570 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a4030d50 con 0x7f81b01543a0 2024-03-24T10:57:40.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.308+0000 7f81b6af4640 1 -- 172.21.15.5:0/1344848570 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81b01543a0 msgr2=0x7f81b015ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:40.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.308+0000 7f81b6af4640 1 --2- 172.21.15.5:0/1344848570 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81b01543a0 0x7f81b015ac00 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f81b00c6990 tx=0x7f81a402b9a0 comp rx=0 tx=0).stop 2024-03-24T10:57:40.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.308+0000 7f81b6af4640 1 -- 172.21.15.5:0/1344848570 shutdown_connections 2024-03-24T10:57:40.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.308+0000 7f81b6af4640 1 --2- 172.21.15.5:0/1344848570 >> v2:172.21.15.57:3300/0 conn(0x7f81b006d4f0 0x7f81b00cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:40.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.308+0000 7f81b6af4640 1 --2- 172.21.15.5:0/1344848570 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81b01543a0 0x7f81b015ac00 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:40.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.308+0000 7f81b6af4640 1 -- 172.21.15.5:0/1344848570 >> 172.21.15.5:0/1344848570 conn(0x7f81b0072ef0 msgr2=0x7f81b0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:40.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.308+0000 7f81b6af4640 1 -- 172.21.15.5:0/1344848570 shutdown_connections 2024-03-24T10:57:40.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.308+0000 7f81b6af4640 1 -- 172.21.15.5:0/1344848570 wait complete. 2024-03-24T10:57:40.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.310+0000 7f81b6af4640 1 Processor -- start 2024-03-24T10:57:40.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.310+0000 7f81b6af4640 1 -- start start 2024-03-24T10:57:40.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.310+0000 7f81b6af4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f81b006d4f0 0x7f81b01568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:40.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.310+0000 7f81b6af4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81b00ccaa0 0x7f81b0149910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:40.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.310+0000 7f81b6af4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81b01543a0 0x7f81b00cf140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:40.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.311+0000 7f81b6af4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f81b0159eb0 con 0x7f81b01543a0 2024-03-24T10:57:40.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.311+0000 7f81b6af4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f81b0156e10 con 0x7f81b00ccaa0 2024-03-24T10:57:40.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.311+0000 7f81b6af4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f81b006c0a0 con 0x7f81b006d4f0 2024-03-24T10:57:40.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.311+0000 7f81b5091640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81b01543a0 0x7f81b00cf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:40.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.311+0000 7f81b4890640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f81b006d4f0 0x7f81b01568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:40.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.311+0000 7f81affff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81b00ccaa0 0x7f81b0149910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:40.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.311+0000 7f81b5091640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81b01543a0 0x7f81b00cf140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:33104/0 (socket says 172.21.15.5:33104) 2024-03-24T10:57:40.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.311+0000 7f81b5091640 1 -- 172.21.15.5:0/3756254797 learned_addr learned my addr 172.21.15.5:0/3756254797 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:40.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.315+0000 7f81b5091640 1 -- 172.21.15.5:0/3756254797 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f81b006d4f0 msgr2=0x7f81b01568d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:40.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.315+0000 7f81b5091640 1 --2- 172.21.15.5:0/3756254797 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f81b006d4f0 0x7f81b01568d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:40.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.315+0000 7f81b5091640 1 -- 172.21.15.5:0/3756254797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81b00ccaa0 msgr2=0x7f81b0149910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:40.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.315+0000 7f81b5091640 1 --2- 172.21.15.5:0/3756254797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81b00ccaa0 0x7f81b0149910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:40.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.315+0000 7f81b5091640 1 -- 172.21.15.5:0/3756254797 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a40094d0 con 0x7f81b01543a0 2024-03-24T10:57:40.317 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.317+0000 7f81b5091640 1 --2- 172.21.15.5:0/3756254797 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81b01543a0 0x7f81b00cf140 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f81a800c780 tx=0x7f81a800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:40.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.321+0000 7f81ad7fa640 1 -- 172.21.15.5:0/3756254797 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a8003d80 con 0x7f81b01543a0 2024-03-24T10:57:40.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.321+0000 7f81ad7fa640 1 -- 172.21.15.5:0/3756254797 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81a800ce90 con 0x7f81b01543a0 2024-03-24T10:57:40.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.321+0000 7f81b6af4640 1 -- 172.21.15.5:0/3756254797 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81b00ceb90 con 0x7f81b01543a0 2024-03-24T10:57:40.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.321+0000 7f81ad7fa640 1 -- 172.21.15.5:0/3756254797 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a80052b0 con 0x7f81b01543a0 2024-03-24T10:57:40.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.321+0000 7f81b6af4640 1 -- 172.21.15.5:0/3756254797 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81b01598f0 con 0x7f81b01543a0 2024-03-24T10:57:40.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.322+0000 7f81b6af4640 1 -- 172.21.15.5:0/3756254797 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81b00681d0 con 0x7f81b01543a0 2024-03-24T10:57:40.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.589+0000 7f81ad7fa640 1 -- 172.21.15.5:0/3756254797 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f81a8005410 con 0x7f81b01543a0 2024-03-24T10:57:40.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.590+0000 7f81ad7fa640 1 --2- 172.21.15.5:0/3756254797 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f819406b810 0x7f819406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:40.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.590+0000 7f81b4890640 1 --2- 172.21.15.5:0/3756254797 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f819406b810 0x7f819406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:40.590 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.590+0000 7f81b4890640 1 --2- 172.21.15.5:0/3756254797 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f819406b810 0x7f819406dcd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f81a4036000 tx=0x7f81a4008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:40.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.604+0000 7f81ad7fa640 1 -- 172.21.15.5:0/3756254797 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f81a8058650 con 0x7f81b01543a0 2024-03-24T10:57:40.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.605+0000 7f81ad7fa640 1 -- 172.21.15.5:0/3756254797 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f81a80548f0 con 0x7f81b01543a0 2024-03-24T10:57:40.890 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.890+0000 7f81b6af4640 1 -- 172.21.15.5:0/3756254797 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81b014ade0 con 0x7f81b01543a0 2024-03-24T10:57:40.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.904+0000 7f81ad7fa640 1 -- 172.21.15.5:0/3756254797 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f81a8017070 con 0x7f81b01543a0 2024-03-24T10:57:40.904 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:40.904 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:57:40.905 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:57:40.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.906+0000 7f81b6af4640 1 -- 172.21.15.5:0/3756254797 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f819406b810 msgr2=0x7f819406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:40.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.906+0000 7f81b6af4640 1 --2- 172.21.15.5:0/3756254797 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f819406b810 0x7f819406dcd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f81a4036000 tx=0x7f81a4008030 comp rx=0 tx=0).stop 2024-03-24T10:57:40.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.906+0000 7f81b6af4640 1 -- 172.21.15.5:0/3756254797 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81b01543a0 msgr2=0x7f81b00cf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:40.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.906+0000 7f81b6af4640 1 --2- 172.21.15.5:0/3756254797 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81b01543a0 0x7f81b00cf140 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f81a800c780 tx=0x7f81a800cc40 comp rx=0 tx=0).stop 2024-03-24T10:57:40.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.906+0000 7f81b6af4640 1 -- 172.21.15.5:0/3756254797 shutdown_connections 2024-03-24T10:57:40.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.906+0000 7f81b6af4640 1 --2- 172.21.15.5:0/3756254797 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f819406b810 0x7f819406dcd0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:40.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.906+0000 7f81b6af4640 1 --2- 172.21.15.5:0/3756254797 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f81b01543a0 0x7f81b00cf140 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:40.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.906+0000 7f81b6af4640 1 --2- 172.21.15.5:0/3756254797 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f81b00ccaa0 0x7f81b0149910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:40.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.906+0000 7f81b6af4640 1 --2- 172.21.15.5:0/3756254797 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f81b006d4f0 0x7f81b01568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:40.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.906+0000 7f81b6af4640 1 -- 172.21.15.5:0/3756254797 >> 172.21.15.5:0/3756254797 conn(0x7f81b0072ef0 msgr2=0x7f81b01525f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:40.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.906+0000 7f81b6af4640 1 -- 172.21.15.5:0/3756254797 shutdown_connections 2024-03-24T10:57:40.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:40.906+0000 7f81b6af4640 1 -- 172.21.15.5:0/3756254797 wait complete. 2024-03-24T10:57:41.521 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:57:41.521 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:42.522 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:42.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.624+0000 7f98b4ceb640 1 Processor -- start 2024-03-24T10:57:42.624 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.624+0000 7f98b4ceb640 1 -- start start 2024-03-24T10:57:42.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.626+0000 7f98b4ceb640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f98b006bf50 0x7f98b006c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:42.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.626+0000 7f98b4ceb640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98b00cb2f0 con 0x7f98b015ac00 2024-03-24T10:57:42.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.626+0000 7f98b4ceb640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98b015b070 con 0x7f98b01543a0 2024-03-24T10:57:42.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.626+0000 7f98b4ceb640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f98b00c6750 con 0x7f98b006bf50 2024-03-24T10:57:42.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.626+0000 7f98a7fff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f98b015ac00 0x7f98b00cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:59898/0 (socket says 172.21.15.5:59898) 2024-03-24T10:57:42.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.626+0000 7f98a7fff640 1 -- 172.21.15.5:0/3320027188 learned_addr learned my addr 172.21.15.5:0/3320027188 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:57:42.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.626+0000 7f98aed9d640 1 --2- 172.21.15.5:0/3320027188 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f98b006bf50 0x7f98b006c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:42.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.629+0000 7f98add9b640 1 -- 172.21.15.5:0/3320027188 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1778995633 0 0) 0x7f98b00cb2f0 con 0x7f98b015ac00 2024-03-24T10:57:42.629 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.629+0000 7f98add9b640 1 -- 172.21.15.5:0/3320027188 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f989c003520 con 0x7f98b015ac00 2024-03-24T10:57:42.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.630+0000 7f98add9b640 1 -- 172.21.15.5:0/3320027188 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3764105774 0 0) 0x7f98b015b070 con 0x7f98b01543a0 2024-03-24T10:57:42.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.630+0000 7f98add9b640 1 -- 172.21.15.5:0/3320027188 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98b00cb2f0 con 0x7f98b01543a0 2024-03-24T10:57:42.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.631+0000 7f98aed9d640 1 -- 172.21.15.5:0/3320027188 >> v1:172.21.15.57:6789/0 conn(0x7f98b01543a0 legacy=0x7f98b006d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:42.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.631+0000 7f98aed9d640 1 -- 172.21.15.5:0/3320027188 >> v1:172.21.15.5:6789/0 conn(0x7f98b015ac00 legacy=0x7f98b00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:42.631 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.631+0000 7f98aed9d640 1 -- 172.21.15.5:0/3320027188 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b00ca7e0 con 0x7f98b006bf50 2024-03-24T10:57:42.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.634+0000 7f98aed9d640 1 --2- 172.21.15.5:0/3320027188 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f98b006bf50 0x7f98b006c360 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f98a8009dc0 tx=0x7f98a802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb7dfe942bb56888 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:42.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.639+0000 7f98add9b640 1 -- 172.21.15.5:0/3320027188 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a802ccd0 con 0x7f98b006bf50 2024-03-24T10:57:42.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.639+0000 7f98add9b640 1 -- 172.21.15.5:0/3320027188 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98a802ce30 con 0x7f98b006bf50 2024-03-24T10:57:42.639 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.639+0000 7f98add9b640 1 -- 172.21.15.5:0/3320027188 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98a8034660 con 0x7f98b006bf50 2024-03-24T10:57:42.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.640+0000 7f98b4ceb640 1 -- 172.21.15.5:0/3320027188 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f98b006bf50 msgr2=0x7f98b006c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:42.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.640+0000 7f98b4ceb640 1 --2- 172.21.15.5:0/3320027188 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f98b006bf50 0x7f98b006c360 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f98a8009dc0 tx=0x7f98a802bf20 comp rx=0 tx=0).stop 2024-03-24T10:57:42.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.640+0000 7f98b4ceb640 1 -- 172.21.15.5:0/3320027188 shutdown_connections 2024-03-24T10:57:42.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.640+0000 7f98b4ceb640 1 --2- 172.21.15.5:0/3320027188 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f98b006bf50 0x7f98b006c360 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:42.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.640+0000 7f98b4ceb640 1 -- 172.21.15.5:0/3320027188 >> 172.21.15.5:0/3320027188 conn(0x7f98b0072ef0 msgr2=0x7f98b0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:42.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.640+0000 7f98b4ceb640 1 -- 172.21.15.5:0/3320027188 shutdown_connections 2024-03-24T10:57:42.641 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.640+0000 7f98b4ceb640 1 -- 172.21.15.5:0/3320027188 wait complete. 2024-03-24T10:57:42.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.642+0000 7f98b4ceb640 1 Processor -- start 2024-03-24T10:57:42.642 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.642+0000 7f98b4ceb640 1 -- start start 2024-03-24T10:57:42.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.644+0000 7f98b4ceb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98b006bf50 0x7f98b00ce450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:42.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.644+0000 7f98b4ceb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98b01543a0 0x7f98b015c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:42.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.644+0000 7f98b4ceb640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f98b015ac00 0x7f98b00b4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:42.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.644+0000 7f98b4ceb640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f98b015b870 con 0x7f98b01543a0 2024-03-24T10:57:42.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.644+0000 7f98b4ceb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f98b00cf0c0 con 0x7f98b006bf50 2024-03-24T10:57:42.644 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.644+0000 7f98b4ceb640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f98b01f03f0 con 0x7f98b015ac00 2024-03-24T10:57:42.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.644+0000 7f98a7fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98b01543a0 0x7f98b015c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:42.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.644+0000 7f98ae59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98b006bf50 0x7f98b00ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:42.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.644+0000 7f98aed9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f98b015ac00 0x7f98b00b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:42.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.645+0000 7f98a7fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98b01543a0 0x7f98b015c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:58566/0 (socket says 172.21.15.5:58566) 2024-03-24T10:57:42.645 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.645+0000 7f98a7fff640 1 -- 172.21.15.5:0/1184075719 learned_addr learned my addr 172.21.15.5:0/1184075719 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:42.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.648+0000 7f98a7fff640 1 -- 172.21.15.5:0/1184075719 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f98b015ac00 msgr2=0x7f98b00b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:42.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.648+0000 7f98a7fff640 1 --2- 172.21.15.5:0/1184075719 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f98b015ac00 0x7f98b00b4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:42.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.648+0000 7f98a7fff640 1 -- 172.21.15.5:0/1184075719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98b006bf50 msgr2=0x7f98b00ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:42.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.648+0000 7f98a7fff640 1 --2- 172.21.15.5:0/1184075719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98b006bf50 0x7f98b00ce450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:42.648 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.648+0000 7f98a7fff640 1 -- 172.21.15.5:0/1184075719 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98a8009a70 con 0x7f98b01543a0 2024-03-24T10:57:42.650 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.650+0000 7f98a7fff640 1 --2- 172.21.15.5:0/1184075719 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98b01543a0 0x7f98b015c6e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f989800c410 tx=0x7f989800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:42.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.655+0000 7f98a6ffd640 1 -- 172.21.15.5:0/1184075719 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9898005ce0 con 0x7f98b01543a0 2024-03-24T10:57:42.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.655+0000 7f98a6ffd640 1 -- 172.21.15.5:0/1184075719 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9898005e40 con 0x7f98b01543a0 2024-03-24T10:57:42.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.655+0000 7f98b4ceb640 1 -- 172.21.15.5:0/1184075719 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98b00cd970 con 0x7f98b01543a0 2024-03-24T10:57:42.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.655+0000 7f98a6ffd640 1 -- 172.21.15.5:0/1184075719 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98980072f0 con 0x7f98b01543a0 2024-03-24T10:57:42.655 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.655+0000 7f98b4ceb640 1 -- 172.21.15.5:0/1184075719 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98b00cdf30 con 0x7f98b01543a0 2024-03-24T10:57:42.656 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.656+0000 7f98b4ceb640 1 -- 172.21.15.5:0/1184075719 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98b00ce150 con 0x7f98b01543a0 2024-03-24T10:57:42.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.680+0000 7f98a6ffd640 1 -- 172.21.15.5:0/1184075719 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9898007450 con 0x7f98b01543a0 2024-03-24T10:57:42.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.680+0000 7f98a6ffd640 1 --2- 172.21.15.5:0/1184075719 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f989006b860 0x7f989006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:42.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.680+0000 7f98ae59c640 1 --2- 172.21.15.5:0/1184075719 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f989006b860 0x7f989006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:42.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.681+0000 7f98ae59c640 1 --2- 172.21.15.5:0/1184075719 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f989006b860 0x7f989006dd20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f98a00011e0 tx=0x7f98a0009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:42.684 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.684+0000 7f98a6ffd640 1 -- 172.21.15.5:0/1184075719 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f9898058510 con 0x7f98b01543a0 2024-03-24T10:57:42.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:42.966+0000 7f98a6ffd640 1 -- 172.21.15.5:0/1184075719 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f98980547b0 con 0x7f98b01543a0 2024-03-24T10:57:43.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:43.186+0000 7f98b4ceb640 1 -- 172.21.15.5:0/1184075719 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98b00681d0 con 0x7f98b01543a0 2024-03-24T10:57:43.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:43.199+0000 7f98a6ffd640 1 -- 172.21.15.5:0/1184075719 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f9898017090 con 0x7f98b01543a0 2024-03-24T10:57:43.199 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:43.199 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:57:43.200 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:57:43.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:43.201+0000 7f98b4ceb640 1 -- 172.21.15.5:0/1184075719 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f989006b860 msgr2=0x7f989006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:43.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:43.201+0000 7f98b4ceb640 1 --2- 172.21.15.5:0/1184075719 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f989006b860 0x7f989006dd20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f98a00011e0 tx=0x7f98a0009040 comp rx=0 tx=0).stop 2024-03-24T10:57:43.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:43.201+0000 7f98b4ceb640 1 -- 172.21.15.5:0/1184075719 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98b01543a0 msgr2=0x7f98b015c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:43.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:43.201+0000 7f98b4ceb640 1 --2- 172.21.15.5:0/1184075719 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98b01543a0 0x7f98b015c6e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f989800c410 tx=0x7f989800c8d0 comp rx=0 tx=0).stop 2024-03-24T10:57:43.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:43.201+0000 7f98b4ceb640 1 -- 172.21.15.5:0/1184075719 shutdown_connections 2024-03-24T10:57:43.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:43.201+0000 7f98b4ceb640 1 --2- 172.21.15.5:0/1184075719 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f989006b860 0x7f989006dd20 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:43.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:43.201+0000 7f98b4ceb640 1 --2- 172.21.15.5:0/1184075719 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f98b01543a0 0x7f98b015c6e0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:43.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:43.201+0000 7f98b4ceb640 1 --2- 172.21.15.5:0/1184075719 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f98b015ac00 0x7f98b00b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:43.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:43.201+0000 7f98b4ceb640 1 --2- 172.21.15.5:0/1184075719 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f98b006bf50 0x7f98b00ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:43.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:43.201+0000 7f98b4ceb640 1 -- 172.21.15.5:0/1184075719 >> 172.21.15.5:0/1184075719 conn(0x7f98b0072ef0 msgr2=0x7f98b0153640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:43.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:43.201+0000 7f98b4ceb640 1 -- 172.21.15.5:0/1184075719 shutdown_connections 2024-03-24T10:57:43.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:43.201+0000 7f98b4ceb640 1 -- 172.21.15.5:0/1184075719 wait complete. 2024-03-24T10:57:43.817 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:57:43.817 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:44.818 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:44.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.921+0000 7f142ed13640 1 Processor -- start 2024-03-24T10:57:44.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.921+0000 7f142ed13640 1 -- start start 2024-03-24T10:57:44.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.921+0000 7f142ed13640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f14281543a0 0x7f142815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:44.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.921+0000 7f142ed13640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f142806d4f0 0x7f14280cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:44.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.921+0000 7f142ed13640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f14280ccaa0 0x7f142806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:44.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.921+0000 7f142ed13640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f14280c6750 con 0x7f14281543a0 2024-03-24T10:57:44.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.921+0000 7f142ed13640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f14280cc380 con 0x7f14280ccaa0 2024-03-24T10:57:44.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.921+0000 7f142ed13640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f14280cb2f0 con 0x7f142806d4f0 2024-03-24T10:57:44.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.922+0000 7f142caaf640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f14281543a0 0x7f142815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:44.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.922+0000 7f1427fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f142806d4f0 0x7f14280cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:44.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.922+0000 7f142d2b0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f14280ccaa0 0x7f142806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:44.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.923+0000 7f142caaf640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f14281543a0 0x7f142815ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:58574/0 (socket says 172.21.15.5:58574) 2024-03-24T10:57:44.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.923+0000 7f142caaf640 1 -- 172.21.15.5:0/3365934256 learned_addr learned my addr 172.21.15.5:0/3365934256 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:44.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.925+0000 7f142caaf640 1 -- 172.21.15.5:0/3365934256 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f142806d4f0 msgr2=0x7f14280cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:44.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.925+0000 7f142caaf640 1 --2- 172.21.15.5:0/3365934256 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f142806d4f0 0x7f14280cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:44.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.925+0000 7f142caaf640 1 -- 172.21.15.5:0/3365934256 >> v2:172.21.15.57:3300/0 conn(0x7f14280ccaa0 msgr2=0x7f142806bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:44.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.925+0000 7f142caaf640 1 --2- 172.21.15.5:0/3365934256 >> v2:172.21.15.57:3300/0 conn(0x7f14280ccaa0 0x7f142806bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:44.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.925+0000 7f142caaf640 1 -- 172.21.15.5:0/3365934256 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14280ca7e0 con 0x7f14281543a0 2024-03-24T10:57:44.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.927+0000 7f142caaf640 1 --2- 172.21.15.5:0/3365934256 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f14281543a0 0x7f142815ac00 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f14280628b0 tx=0x7f141c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a6468eba2c4997 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:44.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.931+0000 7f14277fe640 1 -- 172.21.15.5:0/3365934256 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141c030860 con 0x7f14281543a0 2024-03-24T10:57:44.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.931+0000 7f14277fe640 1 -- 172.21.15.5:0/3365934256 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f141c0309c0 con 0x7f14281543a0 2024-03-24T10:57:44.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.931+0000 7f14277fe640 1 -- 172.21.15.5:0/3365934256 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141c030d50 con 0x7f14281543a0 2024-03-24T10:57:44.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.932+0000 7f142ed13640 1 -- 172.21.15.5:0/3365934256 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f14281543a0 msgr2=0x7f142815ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:44.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.932+0000 7f142ed13640 1 --2- 172.21.15.5:0/3365934256 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f14281543a0 0x7f142815ac00 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f14280628b0 tx=0x7f141c02b9a0 comp rx=0 tx=0).stop 2024-03-24T10:57:44.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.932+0000 7f142ed13640 1 -- 172.21.15.5:0/3365934256 shutdown_connections 2024-03-24T10:57:44.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.932+0000 7f142ed13640 1 --2- 172.21.15.5:0/3365934256 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f142806d4f0 0x7f14280cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:44.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.932+0000 7f142ed13640 1 --2- 172.21.15.5:0/3365934256 >> v2:172.21.15.57:3300/0 conn(0x7f14280ccaa0 0x7f142806bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:44.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.932+0000 7f142ed13640 1 --2- 172.21.15.5:0/3365934256 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f14281543a0 0x7f142815ac00 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:44.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.932+0000 7f142ed13640 1 -- 172.21.15.5:0/3365934256 >> 172.21.15.5:0/3365934256 conn(0x7f1428072ef0 msgr2=0x7f1428061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:44.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.932+0000 7f142ed13640 1 -- 172.21.15.5:0/3365934256 shutdown_connections 2024-03-24T10:57:44.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.932+0000 7f142ed13640 1 -- 172.21.15.5:0/3365934256 wait complete. 2024-03-24T10:57:44.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.934+0000 7f142ed13640 1 Processor -- start 2024-03-24T10:57:44.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.934+0000 7f142ed13640 1 -- start start 2024-03-24T10:57:44.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.935+0000 7f142ed13640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f142806d4f0 0x7f1428159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:44.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.935+0000 7f142ed13640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f14280ccaa0 0x7f1428144350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:44.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.935+0000 7f142ed13640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14281543a0 0x7f1428144890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:44.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.935+0000 7f142ed13640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f14280b4fe0 con 0x7f14280ccaa0 2024-03-24T10:57:44.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.935+0000 7f142ed13640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f14280b4a40 con 0x7f14281543a0 2024-03-24T10:57:44.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.935+0000 7f142ed13640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f14280b4d00 con 0x7f142806d4f0 2024-03-24T10:57:44.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.936+0000 7f1427fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f14280ccaa0 0x7f1428144350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:44.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.936+0000 7f142d2b0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14281543a0 0x7f1428144890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:44.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.936+0000 7f1427fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f14280ccaa0 0x7f1428144350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:58576/0 (socket says 172.21.15.5:58576) 2024-03-24T10:57:44.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.936+0000 7f1427fff640 1 -- 172.21.15.5:0/3560741830 learned_addr learned my addr 172.21.15.5:0/3560741830 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:44.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.936+0000 7f142caaf640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f142806d4f0 0x7f1428159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:44.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.940+0000 7f1427fff640 1 -- 172.21.15.5:0/3560741830 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f142806d4f0 msgr2=0x7f1428159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:44.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.940+0000 7f1427fff640 1 --2- 172.21.15.5:0/3560741830 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f142806d4f0 0x7f1428159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:44.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.940+0000 7f1427fff640 1 -- 172.21.15.5:0/3560741830 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14281543a0 msgr2=0x7f1428144890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:44.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.940+0000 7f1427fff640 1 --2- 172.21.15.5:0/3560741830 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14281543a0 0x7f1428144890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:44.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.940+0000 7f1427fff640 1 -- 172.21.15.5:0/3560741830 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141c0094d0 con 0x7f14280ccaa0 2024-03-24T10:57:44.942 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.942+0000 7f1427fff640 1 --2- 172.21.15.5:0/3560741830 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f14280ccaa0 0x7f1428144350 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f141400e9b0 tx=0x7f141400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:44.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.947+0000 7f14257fa640 1 -- 172.21.15.5:0/3560741830 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1414009670 con 0x7f14280ccaa0 2024-03-24T10:57:44.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.947+0000 7f14257fa640 1 -- 172.21.15.5:0/3560741830 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f141400c9a0 con 0x7f14280ccaa0 2024-03-24T10:57:44.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.947+0000 7f14257fa640 1 -- 172.21.15.5:0/3560741830 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f141400cd50 con 0x7f14280ccaa0 2024-03-24T10:57:44.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.948+0000 7f142ed13640 1 -- 172.21.15.5:0/3560741830 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1428154c00 con 0x7f14280ccaa0 2024-03-24T10:57:44.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.948+0000 7f142ed13640 1 -- 172.21.15.5:0/3560741830 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14281560d0 con 0x7f14280ccaa0 2024-03-24T10:57:44.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:44.948+0000 7f142ed13640 1 -- 172.21.15.5:0/3560741830 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1428154a60 con 0x7f14280ccaa0 2024-03-24T10:57:45.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:45.240+0000 7f14257fa640 1 -- 172.21.15.5:0/3560741830 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1414013470 con 0x7f14280ccaa0 2024-03-24T10:57:45.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:45.241+0000 7f14257fa640 1 --2- 172.21.15.5:0/3560741830 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f140c06b890 0x7f140c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:45.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:45.241+0000 7f142caaf640 1 --2- 172.21.15.5:0/3560741830 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f140c06b890 0x7f140c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:45.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:45.241+0000 7f142caaf640 1 --2- 172.21.15.5:0/3560741830 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f140c06b890 0x7f140c06dd50 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f141c036000 tx=0x7f141c008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:45.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:45.255+0000 7f14257fa640 1 -- 172.21.15.5:0/3560741830 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f14140591a0 con 0x7f14280ccaa0 2024-03-24T10:57:45.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:45.255+0000 7f14257fa640 1 -- 172.21.15.5:0/3560741830 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1414055440 con 0x7f14280ccaa0 2024-03-24T10:57:45.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:45.476+0000 7f142ed13640 1 -- 172.21.15.5:0/3560741830 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1428154a60 con 0x7f14280ccaa0 2024-03-24T10:57:45.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:45.490+0000 7f14257fa640 1 -- 172.21.15.5:0/3560741830 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f1428154a60 con 0x7f14280ccaa0 2024-03-24T10:57:45.490 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:45.490 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:57:45.490 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:57:45.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:45.492+0000 7f142ed13640 1 -- 172.21.15.5:0/3560741830 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f140c06b890 msgr2=0x7f140c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:45.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:45.492+0000 7f142ed13640 1 --2- 172.21.15.5:0/3560741830 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f140c06b890 0x7f140c06dd50 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f141c036000 tx=0x7f141c008030 comp rx=0 tx=0).stop 2024-03-24T10:57:45.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:45.492+0000 7f142ed13640 1 -- 172.21.15.5:0/3560741830 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f14280ccaa0 msgr2=0x7f1428144350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:45.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:45.492+0000 7f142ed13640 1 --2- 172.21.15.5:0/3560741830 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f14280ccaa0 0x7f1428144350 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f141400e9b0 tx=0x7f141400ee70 comp rx=0 tx=0).stop 2024-03-24T10:57:45.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:45.492+0000 7f142ed13640 1 -- 172.21.15.5:0/3560741830 shutdown_connections 2024-03-24T10:57:45.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:45.492+0000 7f142ed13640 1 --2- 172.21.15.5:0/3560741830 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f140c06b890 0x7f140c06dd50 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:45.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:45.492+0000 7f142ed13640 1 --2- 172.21.15.5:0/3560741830 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f14280ccaa0 0x7f1428144350 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:45.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:45.492+0000 7f142ed13640 1 --2- 172.21.15.5:0/3560741830 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f14281543a0 0x7f1428144890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:45.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:45.492+0000 7f142ed13640 1 --2- 172.21.15.5:0/3560741830 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f142806d4f0 0x7f1428159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:45.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:45.492+0000 7f142ed13640 1 -- 172.21.15.5:0/3560741830 >> 172.21.15.5:0/3560741830 conn(0x7f1428072ef0 msgr2=0x7f142806bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:45.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:45.492+0000 7f142ed13640 1 -- 172.21.15.5:0/3560741830 shutdown_connections 2024-03-24T10:57:45.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:45.492+0000 7f142ed13640 1 -- 172.21.15.5:0/3560741830 wait complete. 2024-03-24T10:57:46.112 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:57:46.112 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:47.113 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:47.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.216+0000 7fc6875d0640 1 Processor -- start 2024-03-24T10:57:47.217 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.216+0000 7fc6875d0640 1 -- start start 2024-03-24T10:57:47.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.217+0000 7fc6875d0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc6801543a0 0x7fc68015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:47.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.217+0000 7fc6875d0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc68006d4f0 0x7fc6800cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:47.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.217+0000 7fc6875d0640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6800cb2f0 con 0x7fc6800ccaa0 2024-03-24T10:57:47.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.217+0000 7fc6875d0640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fc6800c6750 con 0x7fc6801543a0 2024-03-24T10:57:47.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.217+0000 7fc6875d0640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fc6800cc380 con 0x7fc68006d4f0 2024-03-24T10:57:47.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.217+0000 7fc68536c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc6801543a0 0x7fc68015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:47.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.218+0000 7fc684b6b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc68006d4f0 0x7fc6800cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:47.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.218+0000 7fc68536c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fc6801543a0 0x7fc68015ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:58586/0 (socket says 172.21.15.5:58586) 2024-03-24T10:57:47.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.218+0000 7fc68536c640 1 -- 172.21.15.5:0/1918815718 learned_addr learned my addr 172.21.15.5:0/1918815718 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:47.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.220+0000 7fc68536c640 1 -- 172.21.15.5:0/1918815718 >> v2:172.21.15.57:3300/0 conn(0x7fc68006d4f0 msgr2=0x7fc6800cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:47.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.220+0000 7fc68536c640 1 --2- 172.21.15.5:0/1918815718 >> v2:172.21.15.57:3300/0 conn(0x7fc68006d4f0 0x7fc6800cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:47.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.220+0000 7fc68536c640 1 -- 172.21.15.5:0/1918815718 >> v1:172.21.15.57:6789/0 conn(0x7fc6800ccaa0 legacy=0x7fc68006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:47.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.220+0000 7fc68536c640 1 -- 172.21.15.5:0/1918815718 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6800ca7e0 con 0x7fc6801543a0 2024-03-24T10:57:47.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.223+0000 7fc68536c640 1 --2- 172.21.15.5:0/1918815718 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6801543a0 0x7fc68015ac00 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fc6800c6990 tx=0x7fc67c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a588bd99528ed036 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:47.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.226+0000 7fc677fff640 1 -- 172.21.15.5:0/1918815718 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c030860 con 0x7fc6801543a0 2024-03-24T10:57:47.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.226+0000 7fc677fff640 1 -- 172.21.15.5:0/1918815718 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc67c0309c0 con 0x7fc6801543a0 2024-03-24T10:57:47.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.226+0000 7fc677fff640 1 -- 172.21.15.5:0/1918815718 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c030d50 con 0x7fc6801543a0 2024-03-24T10:57:47.227 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.227+0000 7fc6875d0640 1 -- 172.21.15.5:0/1918815718 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6801543a0 msgr2=0x7fc68015ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:47.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.227+0000 7fc6875d0640 1 --2- 172.21.15.5:0/1918815718 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6801543a0 0x7fc68015ac00 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fc6800c6990 tx=0x7fc67c02b9a0 comp rx=0 tx=0).stop 2024-03-24T10:57:47.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.227+0000 7fc6875d0640 1 -- 172.21.15.5:0/1918815718 shutdown_connections 2024-03-24T10:57:47.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.227+0000 7fc6875d0640 1 --2- 172.21.15.5:0/1918815718 >> v2:172.21.15.57:3300/0 conn(0x7fc68006d4f0 0x7fc6800cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:47.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.227+0000 7fc6875d0640 1 --2- 172.21.15.5:0/1918815718 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6801543a0 0x7fc68015ac00 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:47.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.227+0000 7fc6875d0640 1 -- 172.21.15.5:0/1918815718 >> 172.21.15.5:0/1918815718 conn(0x7fc680072ef0 msgr2=0x7fc680061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:47.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.227+0000 7fc6875d0640 1 -- 172.21.15.5:0/1918815718 shutdown_connections 2024-03-24T10:57:47.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.227+0000 7fc6875d0640 1 -- 172.21.15.5:0/1918815718 wait complete. 2024-03-24T10:57:47.228 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.228+0000 7fc6875d0640 1 Processor -- start 2024-03-24T10:57:47.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.228+0000 7fc6875d0640 1 -- start start 2024-03-24T10:57:47.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.229+0000 7fc6875d0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc68006d4f0 0x7fc6800cf4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:47.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.229+0000 7fc6875d0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6800ccaa0 0x7fc6801e3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:47.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.229+0000 7fc6875d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6801543a0 0x7fc680151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:47.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.229+0000 7fc6875d0640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fc6801450e0 con 0x7fc6800ccaa0 2024-03-24T10:57:47.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.229+0000 7fc6875d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc6800b2f60 con 0x7fc6801543a0 2024-03-24T10:57:47.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.229+0000 7fc6875d0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc6800b3240 con 0x7fc68006d4f0 2024-03-24T10:57:47.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.230+0000 7fc684b6b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6800ccaa0 0x7fc6801e3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:47.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.230+0000 7fc685b6d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6801543a0 0x7fc680151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:47.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.230+0000 7fc684b6b640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6800ccaa0 0x7fc6801e3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:58596/0 (socket says 172.21.15.5:58596) 2024-03-24T10:57:47.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.230+0000 7fc684b6b640 1 -- 172.21.15.5:0/1118034692 learned_addr learned my addr 172.21.15.5:0/1118034692 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:47.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.230+0000 7fc68536c640 1 --2- 172.21.15.5:0/1118034692 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc68006d4f0 0x7fc6800cf4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:47.233 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.233+0000 7fc684b6b640 1 -- 172.21.15.5:0/1118034692 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc68006d4f0 msgr2=0x7fc6800cf4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:47.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.233+0000 7fc684b6b640 1 --2- 172.21.15.5:0/1118034692 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc68006d4f0 0x7fc6800cf4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:47.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.233+0000 7fc684b6b640 1 -- 172.21.15.5:0/1118034692 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6801543a0 msgr2=0x7fc680151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:47.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.233+0000 7fc684b6b640 1 --2- 172.21.15.5:0/1118034692 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6801543a0 0x7fc680151580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:47.234 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.233+0000 7fc684b6b640 1 -- 172.21.15.5:0/1118034692 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc67c0094d0 con 0x7fc6800ccaa0 2024-03-24T10:57:47.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.236+0000 7fc684b6b640 1 --2- 172.21.15.5:0/1118034692 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6800ccaa0 0x7fc6801e3ea0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc67000e9b0 tx=0x7fc67000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:47.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.240+0000 7fc675ffb640 1 -- 172.21.15.5:0/1118034692 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc670009670 con 0x7fc6800ccaa0 2024-03-24T10:57:47.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.240+0000 7fc675ffb640 1 -- 172.21.15.5:0/1118034692 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc67000c9a0 con 0x7fc6800ccaa0 2024-03-24T10:57:47.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.240+0000 7fc6875d0640 1 -- 172.21.15.5:0/1118034692 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6801e43e0 con 0x7fc6800ccaa0 2024-03-24T10:57:47.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.241+0000 7fc675ffb640 1 -- 172.21.15.5:0/1118034692 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67000cd50 con 0x7fc6800ccaa0 2024-03-24T10:57:47.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.241+0000 7fc6875d0640 1 -- 172.21.15.5:0/1118034692 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6800cebe0 con 0x7fc6800ccaa0 2024-03-24T10:57:47.242 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.242+0000 7fc6875d0640 1 -- 172.21.15.5:0/1118034692 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc680152f60 con 0x7fc6800ccaa0 2024-03-24T10:57:47.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.528+0000 7fc675ffb640 1 -- 172.21.15.5:0/1118034692 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc670013470 con 0x7fc6800ccaa0 2024-03-24T10:57:47.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.529+0000 7fc675ffb640 1 --2- 172.21.15.5:0/1118034692 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc66406b860 0x7fc66406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:47.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.529+0000 7fc68536c640 1 --2- 172.21.15.5:0/1118034692 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc66406b860 0x7fc66406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:47.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.529+0000 7fc68536c640 1 --2- 172.21.15.5:0/1118034692 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc66406b860 0x7fc66406dd20 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc67c036000 tx=0x7fc67c008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:47.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.543+0000 7fc675ffb640 1 -- 172.21.15.5:0/1118034692 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fc670059360 con 0x7fc6800ccaa0 2024-03-24T10:57:47.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.543+0000 7fc675ffb640 1 -- 172.21.15.5:0/1118034692 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc670055600 con 0x7fc6800ccaa0 2024-03-24T10:57:47.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.789+0000 7fc6875d0640 1 -- 172.21.15.5:0/1118034692 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc68014f5a0 con 0x7fc6800ccaa0 2024-03-24T10:57:47.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.802+0000 7fc675ffb640 1 -- 172.21.15.5:0/1118034692 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7fc670021090 con 0x7fc6800ccaa0 2024-03-24T10:57:47.803 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:47.803 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:57:47.806 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:57:47.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.805+0000 7fc6875d0640 1 -- 172.21.15.5:0/1118034692 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc66406b860 msgr2=0x7fc66406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:47.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.805+0000 7fc6875d0640 1 --2- 172.21.15.5:0/1118034692 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc66406b860 0x7fc66406dd20 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc67c036000 tx=0x7fc67c008030 comp rx=0 tx=0).stop 2024-03-24T10:57:47.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.805+0000 7fc6875d0640 1 -- 172.21.15.5:0/1118034692 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6800ccaa0 msgr2=0x7fc6801e3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:47.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.805+0000 7fc6875d0640 1 --2- 172.21.15.5:0/1118034692 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6800ccaa0 0x7fc6801e3ea0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc67000e9b0 tx=0x7fc67000ee70 comp rx=0 tx=0).stop 2024-03-24T10:57:47.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.805+0000 7fc6875d0640 1 -- 172.21.15.5:0/1118034692 shutdown_connections 2024-03-24T10:57:47.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.805+0000 7fc6875d0640 1 --2- 172.21.15.5:0/1118034692 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc66406b860 0x7fc66406dd20 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:47.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.805+0000 7fc6875d0640 1 --2- 172.21.15.5:0/1118034692 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc6800ccaa0 0x7fc6801e3ea0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:47.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.805+0000 7fc6875d0640 1 --2- 172.21.15.5:0/1118034692 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc6801543a0 0x7fc680151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:47.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.805+0000 7fc6875d0640 1 --2- 172.21.15.5:0/1118034692 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc68006d4f0 0x7fc6800cf4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:47.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.805+0000 7fc6875d0640 1 -- 172.21.15.5:0/1118034692 >> 172.21.15.5:0/1118034692 conn(0x7fc680072ef0 msgr2=0x7fc6801e65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:47.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.805+0000 7fc6875d0640 1 -- 172.21.15.5:0/1118034692 shutdown_connections 2024-03-24T10:57:47.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:47.805+0000 7fc6875d0640 1 -- 172.21.15.5:0/1118034692 wait complete. 2024-03-24T10:57:48.423 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:57:48.423 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:49.425 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:49.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.526+0000 7fc4c566f640 1 Processor -- start 2024-03-24T10:57:49.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.526+0000 7fc4c566f640 1 -- start start 2024-03-24T10:57:49.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.527+0000 7fc4c566f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4c01543a0 0x7fc4c015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:49.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.527+0000 7fc4c566f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc4c006bf50 0x7fc4c006c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:49.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.527+0000 7fc4c566f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4c00cb2f0 con 0x7fc4c006d4f0 2024-03-24T10:57:49.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.527+0000 7fc4c566f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fc4c00c6750 con 0x7fc4c006bf50 2024-03-24T10:57:49.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.527+0000 7fc4c566f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc4c00cc380 con 0x7fc4c01543a0 2024-03-24T10:57:49.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.528+0000 7fc4bf7fe640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc4c006bf50 0x7fc4c006c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:49.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.528+0000 7fc4beffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4c01543a0 0x7fc4c015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:49.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.528+0000 7fc4be7fc640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fc4c006d4f0 0x7fc4c00cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:48404/0 (socket says 172.21.15.5:48404) 2024-03-24T10:57:49.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.528+0000 7fc4be7fc640 1 -- 172.21.15.5:0/329185268 learned_addr learned my addr 172.21.15.5:0/329185268 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:57:49.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.532+0000 7fc4bf7fe640 1 -- 172.21.15.5:0/329185268 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4c01543a0 msgr2=0x7fc4c015ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:49.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.532+0000 7fc4bf7fe640 1 --2- 172.21.15.5:0/329185268 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4c01543a0 0x7fc4c015ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:49.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.532+0000 7fc4bf7fe640 1 -- 172.21.15.5:0/329185268 >> v1:172.21.15.57:6789/0 conn(0x7fc4c006d4f0 legacy=0x7fc4c00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:49.532 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.532+0000 7fc4bf7fe640 1 -- 172.21.15.5:0/329185268 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4c00ca7e0 con 0x7fc4c006bf50 2024-03-24T10:57:49.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.535+0000 7fc4bf7fe640 1 --2- 172.21.15.5:0/329185268 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4c006bf50 0x7fc4c006c380 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8009d90 tx=0x7fc4b802bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3171e579935a37a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:49.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.540+0000 7fc4bdffb640 1 -- 172.21.15.5:0/329185268 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b802cce0 con 0x7fc4c006bf50 2024-03-24T10:57:49.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.540+0000 7fc4bdffb640 1 -- 172.21.15.5:0/329185268 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4b802ce40 con 0x7fc4c006bf50 2024-03-24T10:57:49.540 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.540+0000 7fc4bdffb640 1 -- 172.21.15.5:0/329185268 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b8034660 con 0x7fc4c006bf50 2024-03-24T10:57:49.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.541+0000 7fc4c566f640 1 -- 172.21.15.5:0/329185268 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4c006bf50 msgr2=0x7fc4c006c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:49.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.541+0000 7fc4c566f640 1 --2- 172.21.15.5:0/329185268 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4c006bf50 0x7fc4c006c380 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8009d90 tx=0x7fc4b802bf30 comp rx=0 tx=0).stop 2024-03-24T10:57:49.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.541+0000 7fc4c566f640 1 -- 172.21.15.5:0/329185268 shutdown_connections 2024-03-24T10:57:49.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.541+0000 7fc4c566f640 1 --2- 172.21.15.5:0/329185268 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4c006bf50 0x7fc4c006c380 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:49.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.541+0000 7fc4c566f640 1 --2- 172.21.15.5:0/329185268 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4c01543a0 0x7fc4c015ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:49.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.541+0000 7fc4c566f640 1 -- 172.21.15.5:0/329185268 >> 172.21.15.5:0/329185268 conn(0x7fc4c0072ef0 msgr2=0x7fc4c0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:49.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.541+0000 7fc4c566f640 1 -- 172.21.15.5:0/329185268 shutdown_connections 2024-03-24T10:57:49.542 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.541+0000 7fc4c566f640 1 -- 172.21.15.5:0/329185268 wait complete. 2024-03-24T10:57:49.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.543+0000 7fc4c566f640 1 Processor -- start 2024-03-24T10:57:49.543 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.543+0000 7fc4c566f640 1 -- start start 2024-03-24T10:57:49.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.545+0000 7fc4c566f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4c006bf50 0x7fc4c00b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:49.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.545+0000 7fc4c566f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc4c006d4f0 0x7fc4c015a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:49.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.545+0000 7fc4c566f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4c01543a0 0x7fc4c015a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:49.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.545+0000 7fc4c566f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c015b680 con 0x7fc4c006d4f0 2024-03-24T10:57:49.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.545+0000 7fc4c566f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c0159bc0 con 0x7fc4c01543a0 2024-03-24T10:57:49.545 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.545+0000 7fc4c566f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc4c015c930 con 0x7fc4c006bf50 2024-03-24T10:57:49.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.545+0000 7fc4bf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4c01543a0 0x7fc4c015a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:49.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.545+0000 7fc4beffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4c006bf50 0x7fc4c00b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:49.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.546+0000 7fc4beffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4c006bf50 0x7fc4c00b44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:40684/0 (socket says 172.21.15.5:40684) 2024-03-24T10:57:49.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.546+0000 7fc4beffd640 1 -- 172.21.15.5:0/4150968084 learned_addr learned my addr 172.21.15.5:0/4150968084 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:49.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.549+0000 7fc4bf7fe640 1 -- 172.21.15.5:0/4150968084 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4c006bf50 msgr2=0x7fc4c00b44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:49.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.549+0000 7fc4bf7fe640 1 --2- 172.21.15.5:0/4150968084 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4c006bf50 0x7fc4c00b44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:49.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.549+0000 7fc4bf7fe640 1 -- 172.21.15.5:0/4150968084 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc4c006d4f0 msgr2=0x7fc4c015a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:49.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.549+0000 7fc4bf7fe640 1 --2- 172.21.15.5:0/4150968084 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc4c006d4f0 0x7fc4c015a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:49.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.549+0000 7fc4bf7fe640 1 -- 172.21.15.5:0/4150968084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b8009a70 con 0x7fc4c01543a0 2024-03-24T10:57:49.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.552+0000 7fc4bf7fe640 1 --2- 172.21.15.5:0/4150968084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4c01543a0 0x7fc4c015a7c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8033000 tx=0x7fc4b8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:49.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.557+0000 7fc4a77fe640 1 -- 172.21.15.5:0/4150968084 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b803d600 con 0x7fc4c01543a0 2024-03-24T10:57:49.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.557+0000 7fc4a77fe640 1 -- 172.21.15.5:0/4150968084 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4b803d780 con 0x7fc4c01543a0 2024-03-24T10:57:49.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.557+0000 7fc4c566f640 1 -- 172.21.15.5:0/4150968084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4c00b2580 con 0x7fc4c01543a0 2024-03-24T10:57:49.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.557+0000 7fc4a77fe640 1 -- 172.21.15.5:0/4150968084 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b803db30 con 0x7fc4c01543a0 2024-03-24T10:57:49.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.557+0000 7fc4c566f640 1 -- 172.21.15.5:0/4150968084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4c01597e0 con 0x7fc4c01543a0 2024-03-24T10:57:49.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.558+0000 7fc4c566f640 1 -- 172.21.15.5:0/4150968084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4c0159340 con 0x7fc4c01543a0 2024-03-24T10:57:49.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.829+0000 7fc4a77fe640 1 -- 172.21.15.5:0/4150968084 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc4b803b040 con 0x7fc4c01543a0 2024-03-24T10:57:49.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.830+0000 7fc4a77fe640 1 --2- 172.21.15.5:0/4150968084 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc4ac06b810 0x7fc4ac06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:49.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.830+0000 7fc4beffd640 1 --2- 172.21.15.5:0/4150968084 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc4ac06b810 0x7fc4ac06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:49.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.830+0000 7fc4beffd640 1 --2- 172.21.15.5:0/4150968084 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc4ac06b810 0x7fc4ac06dcd0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc4c00c4e00 tx=0x7fc4b0009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:49.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.832+0000 7fc4a77fe640 1 -- 172.21.15.5:0/4150968084 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fc4b807b480 con 0x7fc4c01543a0 2024-03-24T10:57:49.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:49.843+0000 7fc4a77fe640 1 -- 172.21.15.5:0/4150968084 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc4b80d8e10 con 0x7fc4c01543a0 2024-03-24T10:57:50.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:50.071+0000 7fc4c566f640 1 -- 172.21.15.5:0/4150968084 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc4c00681d0 con 0x7fc4c01543a0 2024-03-24T10:57:50.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:50.093+0000 7fc4a77fe640 1 -- 172.21.15.5:0/4150968084 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7fc4b8039020 con 0x7fc4c01543a0 2024-03-24T10:57:50.093 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:50.093 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:57:50.094 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:57:50.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:50.095+0000 7fc4c566f640 1 -- 172.21.15.5:0/4150968084 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc4ac06b810 msgr2=0x7fc4ac06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:50.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:50.095+0000 7fc4c566f640 1 --2- 172.21.15.5:0/4150968084 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc4ac06b810 0x7fc4ac06dcd0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc4c00c4e00 tx=0x7fc4b0009190 comp rx=0 tx=0).stop 2024-03-24T10:57:50.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:50.095+0000 7fc4c566f640 1 -- 172.21.15.5:0/4150968084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4c01543a0 msgr2=0x7fc4c015a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:50.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:50.095+0000 7fc4c566f640 1 --2- 172.21.15.5:0/4150968084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4c01543a0 0x7fc4c015a7c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8033000 tx=0x7fc4b8004a40 comp rx=0 tx=0).stop 2024-03-24T10:57:50.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:50.095+0000 7fc4c566f640 1 -- 172.21.15.5:0/4150968084 shutdown_connections 2024-03-24T10:57:50.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:50.095+0000 7fc4c566f640 1 --2- 172.21.15.5:0/4150968084 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc4ac06b810 0x7fc4ac06dcd0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:50.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:50.095+0000 7fc4c566f640 1 --2- 172.21.15.5:0/4150968084 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc4c006d4f0 0x7fc4c015a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:50.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:50.095+0000 7fc4c566f640 1 --2- 172.21.15.5:0/4150968084 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc4c01543a0 0x7fc4c015a7c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:50.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:50.095+0000 7fc4c566f640 1 --2- 172.21.15.5:0/4150968084 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc4c006bf50 0x7fc4c00b44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:50.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:50.095+0000 7fc4c566f640 1 -- 172.21.15.5:0/4150968084 >> 172.21.15.5:0/4150968084 conn(0x7fc4c0072ef0 msgr2=0x7fc4c01472a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:50.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:50.096+0000 7fc4c566f640 1 -- 172.21.15.5:0/4150968084 shutdown_connections 2024-03-24T10:57:50.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:50.096+0000 7fc4c566f640 1 -- 172.21.15.5:0/4150968084 wait complete. 2024-03-24T10:57:50.711 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:57:50.711 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:51.712 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:51.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.818+0000 7f5dbecc9640 1 Processor -- start 2024-03-24T10:57:51.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.818+0000 7f5dbecc9640 1 -- start start 2024-03-24T10:57:51.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.819+0000 7f5dbecc9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5db80cae90 0x7f5db80cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:51.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.819+0000 7f5dbecc9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5db80cb270 0x7f5db80cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:51.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.819+0000 7f5dbecc9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5db80ccba0 con 0x7f5db80cd960 2024-03-24T10:57:51.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.819+0000 7f5dbecc9640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f5db80cce40 con 0x7f5db80cae90 2024-03-24T10:57:51.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.819+0000 7f5dbecc9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f5db815a590 con 0x7f5db80cb270 2024-03-24T10:57:51.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.819+0000 7f5db7fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5db80cb270 0x7f5db80cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:51.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.820+0000 7f5dbd266640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f5db80cd960 0x7f5db80cb6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:48414/0 (socket says 172.21.15.5:48414) 2024-03-24T10:57:51.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.820+0000 7f5dbd266640 1 -- 172.21.15.5:0/1391396042 learned_addr learned my addr 172.21.15.5:0/1391396042 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:57:51.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.820+0000 7f5dbca65640 1 --2- 172.21.15.5:0/1391396042 >> v2:172.21.15.57:3300/0 conn(0x7f5db80cae90 0x7f5db80cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:51.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.823+0000 7f5db7fff640 1 -- 172.21.15.5:0/1391396042 >> v2:172.21.15.57:3300/0 conn(0x7f5db80cae90 msgr2=0x7f5db80cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:51.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.823+0000 7f5db7fff640 1 --2- 172.21.15.5:0/1391396042 >> v2:172.21.15.57:3300/0 conn(0x7f5db80cae90 0x7f5db80cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:51.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.823+0000 7f5db7fff640 1 -- 172.21.15.5:0/1391396042 >> v1:172.21.15.57:6789/0 conn(0x7f5db80cd960 legacy=0x7f5db80cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:51.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.823+0000 7f5db7fff640 1 -- 172.21.15.5:0/1391396042 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db815a870 con 0x7f5db80cb270 2024-03-24T10:57:51.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.827+0000 7f5db7fff640 1 --2- 172.21.15.5:0/1391396042 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5db80cb270 0x7f5db80cc2a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5da40097a0 tx=0x7f5da402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6136ae8c95220613 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:51.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.834+0000 7f5db77fe640 1 -- 172.21.15.5:0/1391396042 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da4030610 con 0x7f5db80cb270 2024-03-24T10:57:51.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.834+0000 7f5db77fe640 1 -- 172.21.15.5:0/1391396042 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5da4030770 con 0x7f5db80cb270 2024-03-24T10:57:51.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.834+0000 7f5db77fe640 1 -- 172.21.15.5:0/1391396042 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da4030b00 con 0x7f5db80cb270 2024-03-24T10:57:51.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.835+0000 7f5dbecc9640 1 -- 172.21.15.5:0/1391396042 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5db80cb270 msgr2=0x7f5db80cc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:51.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.835+0000 7f5dbecc9640 1 --2- 172.21.15.5:0/1391396042 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5db80cb270 0x7f5db80cc2a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5da40097a0 tx=0x7f5da402b960 comp rx=0 tx=0).stop 2024-03-24T10:57:51.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.835+0000 7f5dbecc9640 1 -- 172.21.15.5:0/1391396042 shutdown_connections 2024-03-24T10:57:51.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.835+0000 7f5dbecc9640 1 --2- 172.21.15.5:0/1391396042 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5db80cb270 0x7f5db80cc2a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:51.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.835+0000 7f5dbecc9640 1 --2- 172.21.15.5:0/1391396042 >> v2:172.21.15.57:3300/0 conn(0x7f5db80cae90 0x7f5db80cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:51.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.835+0000 7f5dbecc9640 1 -- 172.21.15.5:0/1391396042 >> 172.21.15.5:0/1391396042 conn(0x7f5db80cfb80 msgr2=0x7f5db80cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:51.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.835+0000 7f5dbecc9640 1 -- 172.21.15.5:0/1391396042 shutdown_connections 2024-03-24T10:57:51.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.835+0000 7f5dbecc9640 1 -- 172.21.15.5:0/1391396042 wait complete. 2024-03-24T10:57:51.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.836+0000 7f5dbecc9640 1 Processor -- start 2024-03-24T10:57:51.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.836+0000 7f5dbecc9640 1 -- start start 2024-03-24T10:57:51.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.837+0000 7f5dbecc9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5db80cae90 0x7f5db8154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:51.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.837+0000 7f5dbecc9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5db80cb270 0x7f5db80c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:51.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.837+0000 7f5dbecc9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5db80cd960 0x7f5db815e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:51.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.837+0000 7f5dbecc9640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f5db8155450 con 0x7f5db80cd960 2024-03-24T10:57:51.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.837+0000 7f5dbecc9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5db80b4ea0 con 0x7f5db80cb270 2024-03-24T10:57:51.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.837+0000 7f5dbecc9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f5db80b5140 con 0x7f5db80cae90 2024-03-24T10:57:51.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.838+0000 7f5dbca65640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5db80cae90 0x7f5db8154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:51.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.838+0000 7f5db7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5db80cb270 0x7f5db80c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:51.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.838+0000 7f5db7fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5db80cb270 0x7f5db80c70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:41576/0 (socket says 172.21.15.5:41576) 2024-03-24T10:57:51.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.838+0000 7f5db7fff640 1 -- 172.21.15.5:0/2849854114 learned_addr learned my addr 172.21.15.5:0/2849854114 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:51.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.841+0000 7f5db7fff640 1 -- 172.21.15.5:0/2849854114 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5db80cae90 msgr2=0x7f5db8154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:51.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.841+0000 7f5db7fff640 1 --2- 172.21.15.5:0/2849854114 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5db80cae90 0x7f5db8154df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:51.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.841+0000 7f5db7fff640 1 -- 172.21.15.5:0/2849854114 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5db80cd960 msgr2=0x7f5db815e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:51.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.841+0000 7f5db7fff640 1 --2- 172.21.15.5:0/2849854114 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5db80cd960 0x7f5db815e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:51.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.841+0000 7f5db7fff640 1 -- 172.21.15.5:0/2849854114 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5da4009400 con 0x7f5db80cb270 2024-03-24T10:57:51.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.844+0000 7f5db7fff640 1 --2- 172.21.15.5:0/2849854114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5db80cb270 0x7f5db80c70f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5da402bf70 tx=0x7f5da4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:51.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.849+0000 7f5db57fa640 1 -- 172.21.15.5:0/2849854114 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da403e040 con 0x7f5db80cb270 2024-03-24T10:57:51.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.849+0000 7f5db57fa640 1 -- 172.21.15.5:0/2849854114 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5da4039cb0 con 0x7f5db80cb270 2024-03-24T10:57:51.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.849+0000 7f5dbecc9640 1 -- 172.21.15.5:0/2849854114 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db80b5420 con 0x7f5db80cb270 2024-03-24T10:57:51.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.849+0000 7f5db57fa640 1 -- 172.21.15.5:0/2849854114 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da4044630 con 0x7f5db80cb270 2024-03-24T10:57:51.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.849+0000 7f5dbecc9640 1 -- 172.21.15.5:0/2849854114 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db815d800 con 0x7f5db80cb270 2024-03-24T10:57:51.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.851+0000 7f5dbecc9640 1 -- 172.21.15.5:0/2849854114 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5db80681d0 con 0x7f5db80cb270 2024-03-24T10:57:51.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.875+0000 7f5db57fa640 1 -- 172.21.15.5:0/2849854114 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5da404d3b0 con 0x7f5db80cb270 2024-03-24T10:57:51.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.876+0000 7f5db57fa640 1 --2- 172.21.15.5:0/2849854114 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5d9c06b860 0x7f5d9c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:51.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.876+0000 7f5dbca65640 1 --2- 172.21.15.5:0/2849854114 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5d9c06b860 0x7f5d9c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:51.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.876+0000 7f5dbca65640 1 --2- 172.21.15.5:0/2849854114 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5d9c06b860 0x7f5d9c06dd20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5dac009b40 tx=0x7f5dac009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:51.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:51.880+0000 7f5db57fa640 1 -- 172.21.15.5:0/2849854114 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f5da4039e20 con 0x7f5db80cb270 2024-03-24T10:57:52.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:52.166+0000 7f5db57fa640 1 -- 172.21.15.5:0/2849854114 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5da40775b0 con 0x7f5db80cb270 2024-03-24T10:57:52.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:52.449+0000 7f5dbecc9640 1 -- 172.21.15.5:0/2849854114 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5db806cf30 con 0x7f5db80cb270 2024-03-24T10:57:52.473 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:52.473+0000 7f5db57fa640 1 -- 172.21.15.5:0/2849854114 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f5da4049090 con 0x7f5db80cb270 2024-03-24T10:57:52.473 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:52.474 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:57:52.474 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:57:52.475 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:52.475+0000 7f5dbecc9640 1 -- 172.21.15.5:0/2849854114 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5d9c06b860 msgr2=0x7f5d9c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:52.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:52.475+0000 7f5dbecc9640 1 --2- 172.21.15.5:0/2849854114 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5d9c06b860 0x7f5d9c06dd20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5dac009b40 tx=0x7f5dac009190 comp rx=0 tx=0).stop 2024-03-24T10:57:52.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:52.475+0000 7f5dbecc9640 1 -- 172.21.15.5:0/2849854114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5db80cb270 msgr2=0x7f5db80c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:52.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:52.475+0000 7f5dbecc9640 1 --2- 172.21.15.5:0/2849854114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5db80cb270 0x7f5db80c70f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5da402bf70 tx=0x7f5da4004730 comp rx=0 tx=0).stop 2024-03-24T10:57:52.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:52.475+0000 7f5dbecc9640 1 -- 172.21.15.5:0/2849854114 shutdown_connections 2024-03-24T10:57:52.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:52.475+0000 7f5dbecc9640 1 --2- 172.21.15.5:0/2849854114 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f5d9c06b860 0x7f5d9c06dd20 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:52.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:52.475+0000 7f5dbecc9640 1 --2- 172.21.15.5:0/2849854114 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f5db80cd960 0x7f5db815e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:52.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:52.475+0000 7f5dbecc9640 1 --2- 172.21.15.5:0/2849854114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5db80cb270 0x7f5db80c70f0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:52.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:52.475+0000 7f5dbecc9640 1 --2- 172.21.15.5:0/2849854114 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f5db80cae90 0x7f5db8154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:52.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:52.475+0000 7f5dbecc9640 1 -- 172.21.15.5:0/2849854114 >> 172.21.15.5:0/2849854114 conn(0x7f5db80cfb80 msgr2=0x7f5db80c79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:52.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:52.476+0000 7f5dbecc9640 1 -- 172.21.15.5:0/2849854114 shutdown_connections 2024-03-24T10:57:52.476 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:52.476+0000 7f5dbecc9640 1 -- 172.21.15.5:0/2849854114 wait complete. 2024-03-24T10:57:53.092 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:57:53.092 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:54.093 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:54.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.197+0000 7ff1b8b5c640 1 Processor -- start 2024-03-24T10:57:54.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.197+0000 7ff1b8b5c640 1 -- start start 2024-03-24T10:57:54.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.198+0000 7ff1b8b5c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff1b41543a0 0x7ff1b415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:54.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.198+0000 7ff1b8b5c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff1b406d4f0 0x7ff1b40cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:54.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.198+0000 7ff1b8b5c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff1b40ccaa0 0x7ff1b406bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:54.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.198+0000 7ff1b8b5c640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7ff1b40c6750 con 0x7ff1b40ccaa0 2024-03-24T10:57:54.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.198+0000 7ff1b8b5c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7ff1b40cc380 con 0x7ff1b406d4f0 2024-03-24T10:57:54.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.198+0000 7ff1b8b5c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff1b40cb2f0 con 0x7ff1b41543a0 2024-03-24T10:57:54.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.198+0000 7ff1b2d9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff1b40ccaa0 0x7ff1b406bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:54.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.199+0000 7ff1b259c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff1b41543a0 0x7ff1b415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:54.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.199+0000 7ff1b1d9b640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff1b406d4f0 0x7ff1b40cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:54.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.199+0000 7ff1b2d9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7ff1b40ccaa0 0x7ff1b406bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39636/0 (socket says 172.21.15.5:39636) 2024-03-24T10:57:54.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.199+0000 7ff1b2d9d640 1 -- 172.21.15.5:0/2775204599 learned_addr learned my addr 172.21.15.5:0/2775204599 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:54.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.201+0000 7ff1b2d9d640 1 -- 172.21.15.5:0/2775204599 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff1b41543a0 msgr2=0x7ff1b415ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:54.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.201+0000 7ff1b2d9d640 1 --2- 172.21.15.5:0/2775204599 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff1b41543a0 0x7ff1b415ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:54.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.201+0000 7ff1b2d9d640 1 -- 172.21.15.5:0/2775204599 >> v2:172.21.15.57:3300/0 conn(0x7ff1b406d4f0 msgr2=0x7ff1b40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:54.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.201+0000 7ff1b2d9d640 1 --2- 172.21.15.5:0/2775204599 >> v2:172.21.15.57:3300/0 conn(0x7ff1b406d4f0 0x7ff1b40cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:54.201 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.201+0000 7ff1b2d9d640 1 -- 172.21.15.5:0/2775204599 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b40ca7e0 con 0x7ff1b40ccaa0 2024-03-24T10:57:54.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.203+0000 7ff1b2d9d640 1 --2- 172.21.15.5:0/2775204599 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff1b40ccaa0 0x7ff1b406bf50 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4062a50 tx=0x7ff1ac02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8627092cc12dfe1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:54.205 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.205+0000 7ff1b159a640 1 -- 172.21.15.5:0/2775204599 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac02cb00 con 0x7ff1b40ccaa0 2024-03-24T10:57:54.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.207+0000 7ff1b159a640 1 -- 172.21.15.5:0/2775204599 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1ac02cc60 con 0x7ff1b40ccaa0 2024-03-24T10:57:54.207 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.207+0000 7ff1b159a640 1 -- 172.21.15.5:0/2775204599 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac0343f0 con 0x7ff1b40ccaa0 2024-03-24T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.208+0000 7ff1b8b5c640 1 -- 172.21.15.5:0/2775204599 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff1b40ccaa0 msgr2=0x7ff1b406bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.208+0000 7ff1b8b5c640 1 --2- 172.21.15.5:0/2775204599 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff1b40ccaa0 0x7ff1b406bf50 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff1b4062a50 tx=0x7ff1ac02bf60 comp rx=0 tx=0).stop 2024-03-24T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.208+0000 7ff1b8b5c640 1 -- 172.21.15.5:0/2775204599 shutdown_connections 2024-03-24T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.208+0000 7ff1b8b5c640 1 --2- 172.21.15.5:0/2775204599 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff1b40ccaa0 0x7ff1b406bf50 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.208+0000 7ff1b8b5c640 1 --2- 172.21.15.5:0/2775204599 >> v2:172.21.15.57:3300/0 conn(0x7ff1b406d4f0 0x7ff1b40cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.208+0000 7ff1b8b5c640 1 --2- 172.21.15.5:0/2775204599 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff1b41543a0 0x7ff1b415ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.208+0000 7ff1b8b5c640 1 -- 172.21.15.5:0/2775204599 >> 172.21.15.5:0/2775204599 conn(0x7ff1b4072ef0 msgr2=0x7ff1b4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.208+0000 7ff1b8b5c640 1 -- 172.21.15.5:0/2775204599 shutdown_connections 2024-03-24T10:57:54.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.208+0000 7ff1b8b5c640 1 -- 172.21.15.5:0/2775204599 wait complete. 2024-03-24T10:57:54.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.210+0000 7ff1b8b5c640 1 Processor -- start 2024-03-24T10:57:54.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.210+0000 7ff1b8b5c640 1 -- start start 2024-03-24T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.212+0000 7ff1b8b5c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff1b406d4f0 0x7ff1b41f09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.212+0000 7ff1b8b5c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1b40ccaa0 0x7ff1b41581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.212+0000 7ff1b8b5c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff1b41543a0 0x7ff1b40b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.212+0000 7ff1b8b5c640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b41f43d0 con 0x7ff1b406d4f0 2024-03-24T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.212+0000 7ff1b8b5c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b41f0520 con 0x7ff1b40ccaa0 2024-03-24T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.212+0000 7ff1b8b5c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff1b40cf5e0 con 0x7ff1b41543a0 2024-03-24T10:57:54.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.212+0000 7ff1b259c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff1b406d4f0 0x7ff1b41f09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:54.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.212+0000 7ff1b1d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1b40ccaa0 0x7ff1b41581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:54.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.212+0000 7ff1b2d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff1b41543a0 0x7ff1b40b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:54.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.213+0000 7ff1b259c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff1b406d4f0 0x7ff1b41f09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39646/0 (socket says 172.21.15.5:39646) 2024-03-24T10:57:54.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.213+0000 7ff1b259c640 1 -- 172.21.15.5:0/1922632946 learned_addr learned my addr 172.21.15.5:0/1922632946 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:54.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.216+0000 7ff1b259c640 1 -- 172.21.15.5:0/1922632946 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff1b41543a0 msgr2=0x7ff1b40b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:54.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.216+0000 7ff1b259c640 1 --2- 172.21.15.5:0/1922632946 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff1b41543a0 0x7ff1b40b3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:54.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.216+0000 7ff1b259c640 1 -- 172.21.15.5:0/1922632946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1b40ccaa0 msgr2=0x7ff1b41581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:54.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.216+0000 7ff1b259c640 1 --2- 172.21.15.5:0/1922632946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1b40ccaa0 0x7ff1b41581d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:54.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.216+0000 7ff1b259c640 1 -- 172.21.15.5:0/1922632946 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ac009b30 con 0x7ff1b406d4f0 2024-03-24T10:57:54.218 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.218+0000 7ff1b259c640 1 --2- 172.21.15.5:0/1922632946 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff1b406d4f0 0x7ff1b41f09b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400e870 tx=0x7ff1a400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:54.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.222+0000 7ff1a2ffd640 1 -- 172.21.15.5:0/1922632946 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a40094f0 con 0x7ff1b406d4f0 2024-03-24T10:57:54.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.222+0000 7ff1a2ffd640 1 -- 172.21.15.5:0/1922632946 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1a400c940 con 0x7ff1b406d4f0 2024-03-24T10:57:54.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.222+0000 7ff1a2ffd640 1 -- 172.21.15.5:0/1922632946 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a400ccf0 con 0x7ff1b406d4f0 2024-03-24T10:57:54.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.222+0000 7ff1b8b5c640 1 -- 172.21.15.5:0/1922632946 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1b40b38e0 con 0x7ff1b406d4f0 2024-03-24T10:57:54.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.222+0000 7ff1b8b5c640 1 -- 172.21.15.5:0/1922632946 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1b40b3b80 con 0x7ff1b406d4f0 2024-03-24T10:57:54.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.223+0000 7ff1b8b5c640 1 -- 172.21.15.5:0/1922632946 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1b4152630 con 0x7ff1b406d4f0 2024-03-24T10:57:54.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.490+0000 7ff1a2ffd640 1 -- 172.21.15.5:0/1922632946 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff1a4013470 con 0x7ff1b406d4f0 2024-03-24T10:57:54.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.490+0000 7ff1a2ffd640 1 --2- 172.21.15.5:0/1922632946 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff19406b890 0x7ff19406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:54.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.490+0000 7ff1b1d9b640 1 --2- 172.21.15.5:0/1922632946 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff19406b890 0x7ff19406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:54.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.491+0000 7ff1b1d9b640 1 --2- 172.21.15.5:0/1922632946 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff19406b890 0x7ff19406dd50 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff1b406d210 tx=0x7ff1a8009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:54.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.506+0000 7ff1a2ffd640 1 -- 172.21.15.5:0/1922632946 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7ff1a4058d50 con 0x7ff1b406d4f0 2024-03-24T10:57:54.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.507+0000 7ff1a2ffd640 1 -- 172.21.15.5:0/1922632946 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff1a4054ff0 con 0x7ff1b406d4f0 2024-03-24T10:57:54.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.730+0000 7ff1b8b5c640 1 -- 172.21.15.5:0/1922632946 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1b40cb110 con 0x7ff1b406d4f0 2024-03-24T10:57:54.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.755+0000 7ff1a2ffd640 1 -- 172.21.15.5:0/1922632946 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7ff1a4017090 con 0x7ff1b406d4f0 2024-03-24T10:57:54.758 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:54.759 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:57:54.760 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:57:54.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.757+0000 7ff1b8b5c640 1 -- 172.21.15.5:0/1922632946 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff19406b890 msgr2=0x7ff19406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:54.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.757+0000 7ff1b8b5c640 1 --2- 172.21.15.5:0/1922632946 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff19406b890 0x7ff19406dd50 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff1b406d210 tx=0x7ff1a8009190 comp rx=0 tx=0).stop 2024-03-24T10:57:54.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.757+0000 7ff1b8b5c640 1 -- 172.21.15.5:0/1922632946 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff1b406d4f0 msgr2=0x7ff1b41f09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:54.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.757+0000 7ff1b8b5c640 1 --2- 172.21.15.5:0/1922632946 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff1b406d4f0 0x7ff1b41f09b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff1a400e870 tx=0x7ff1a400ed30 comp rx=0 tx=0).stop 2024-03-24T10:57:54.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.757+0000 7ff1b8b5c640 1 -- 172.21.15.5:0/1922632946 shutdown_connections 2024-03-24T10:57:54.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.757+0000 7ff1b8b5c640 1 --2- 172.21.15.5:0/1922632946 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff19406b890 0x7ff19406dd50 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:54.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.757+0000 7ff1b8b5c640 1 --2- 172.21.15.5:0/1922632946 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff1b41543a0 0x7ff1b40b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:54.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.757+0000 7ff1b8b5c640 1 --2- 172.21.15.5:0/1922632946 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff1b40ccaa0 0x7ff1b41581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:54.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.757+0000 7ff1b8b5c640 1 --2- 172.21.15.5:0/1922632946 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff1b406d4f0 0x7ff1b41f09b0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:54.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.757+0000 7ff1b8b5c640 1 -- 172.21.15.5:0/1922632946 >> 172.21.15.5:0/1922632946 conn(0x7ff1b4072ef0 msgr2=0x7ff1b406c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:54.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.758+0000 7ff1b8b5c640 1 -- 172.21.15.5:0/1922632946 shutdown_connections 2024-03-24T10:57:54.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:54.758+0000 7ff1b8b5c640 1 -- 172.21.15.5:0/1922632946 wait complete. 2024-03-24T10:57:55.375 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:57:55.376 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:56.377 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:56.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.480+0000 7ff031847640 1 Processor -- start 2024-03-24T10:57:56.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.480+0000 7ff031847640 1 -- start start 2024-03-24T10:57:56.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.480+0000 7ff031847640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7ff02c1543a0 0x7ff02c15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:56.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.481+0000 7ff031847640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff02c06d4f0 0x7ff02c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:56.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.481+0000 7ff031847640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff02c0cb2f0 con 0x7ff02c0ccaa0 2024-03-24T10:57:56.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.481+0000 7ff031847640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7ff02c0c6750 con 0x7ff02c1543a0 2024-03-24T10:57:56.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.481+0000 7ff031847640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff02c0cc380 con 0x7ff02c06d4f0 2024-03-24T10:57:56.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.481+0000 7ff02b7fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7ff02c0ccaa0 0x7ff02c06bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:40484/0 (socket says 172.21.15.5:40484) 2024-03-24T10:57:56.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.481+0000 7ff02b7fe640 1 -- 172.21.15.5:0/3854831365 learned_addr learned my addr 172.21.15.5:0/3854831365 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:57:56.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.481+0000 7ff02affd640 1 --2- 172.21.15.5:0/3854831365 >> v2:172.21.15.57:3300/0 conn(0x7ff02c1543a0 0x7ff02c15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:56.482 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.481+0000 7ff02a7fc640 1 --2- 172.21.15.5:0/3854831365 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff02c06d4f0 0x7ff02c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:56.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.484+0000 7ff02affd640 1 -- 172.21.15.5:0/3854831365 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff02c06d4f0 msgr2=0x7ff02c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:56.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.484+0000 7ff02affd640 1 --2- 172.21.15.5:0/3854831365 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff02c06d4f0 0x7ff02c0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:56.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.484+0000 7ff02affd640 1 -- 172.21.15.5:0/3854831365 >> v1:172.21.15.5:6789/0 conn(0x7ff02c0ccaa0 legacy=0x7ff02c06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:56.484 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.484+0000 7ff02affd640 1 -- 172.21.15.5:0/3854831365 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff02c0ca7e0 con 0x7ff02c1543a0 2024-03-24T10:57:56.486 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.486+0000 7ff02affd640 1 --2- 172.21.15.5:0/3854831365 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02c1543a0 0x7ff02c15ac00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff01c0097a0 tx=0x7ff01c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53238850b708364b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:56.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.490+0000 7ff029ffb640 1 -- 172.21.15.5:0/3854831365 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01c030610 con 0x7ff02c1543a0 2024-03-24T10:57:56.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.490+0000 7ff029ffb640 1 -- 172.21.15.5:0/3854831365 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff01c030770 con 0x7ff02c1543a0 2024-03-24T10:57:56.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.490+0000 7ff029ffb640 1 -- 172.21.15.5:0/3854831365 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01c030b00 con 0x7ff02c1543a0 2024-03-24T10:57:56.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.491+0000 7ff031847640 1 -- 172.21.15.5:0/3854831365 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02c1543a0 msgr2=0x7ff02c15ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:56.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.491+0000 7ff031847640 1 --2- 172.21.15.5:0/3854831365 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02c1543a0 0x7ff02c15ac00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff01c0097a0 tx=0x7ff01c02b960 comp rx=0 tx=0).stop 2024-03-24T10:57:56.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.491+0000 7ff031847640 1 -- 172.21.15.5:0/3854831365 shutdown_connections 2024-03-24T10:57:56.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.491+0000 7ff031847640 1 --2- 172.21.15.5:0/3854831365 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff02c06d4f0 0x7ff02c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:56.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.491+0000 7ff031847640 1 --2- 172.21.15.5:0/3854831365 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02c1543a0 0x7ff02c15ac00 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:56.491 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.491+0000 7ff031847640 1 -- 172.21.15.5:0/3854831365 >> 172.21.15.5:0/3854831365 conn(0x7ff02c072ef0 msgr2=0x7ff02c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:56.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.491+0000 7ff031847640 1 -- 172.21.15.5:0/3854831365 shutdown_connections 2024-03-24T10:57:56.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.491+0000 7ff031847640 1 -- 172.21.15.5:0/3854831365 wait complete. 2024-03-24T10:57:56.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.492+0000 7ff031847640 1 Processor -- start 2024-03-24T10:57:56.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.492+0000 7ff031847640 1 -- start start 2024-03-24T10:57:56.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.493+0000 7ff031847640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff02c06d4f0 0x7ff02c159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:56.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.493+0000 7ff031847640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02c0ccaa0 0x7ff02c1efab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:56.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.493+0000 7ff031847640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff02c1543a0 0x7ff02c1efff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:56.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.493+0000 7ff031847640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7ff02c0c8210 con 0x7ff02c06d4f0 2024-03-24T10:57:56.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.493+0000 7ff031847640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7ff02c0c7310 con 0x7ff02c0ccaa0 2024-03-24T10:57:56.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.493+0000 7ff031847640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7ff02c0b2fd0 con 0x7ff02c1543a0 2024-03-24T10:57:56.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.494+0000 7ff02a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02c0ccaa0 0x7ff02c1efab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:56.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.494+0000 7ff02b7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff02c1543a0 0x7ff02c1efff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:56.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.494+0000 7ff02a7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02c0ccaa0 0x7ff02c1efab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36824/0 (socket says 172.21.15.5:36824) 2024-03-24T10:57:56.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.494+0000 7ff02a7fc640 1 -- 172.21.15.5:0/2773536905 learned_addr learned my addr 172.21.15.5:0/2773536905 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:56.495 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.494+0000 7ff02affd640 1 --2- 172.21.15.5:0/2773536905 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff02c06d4f0 0x7ff02c159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:56.497 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.498+0000 7ff02a7fc640 1 -- 172.21.15.5:0/2773536905 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff02c1543a0 msgr2=0x7ff02c1efff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:56.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.498+0000 7ff02a7fc640 1 --2- 172.21.15.5:0/2773536905 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff02c1543a0 0x7ff02c1efff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:56.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.498+0000 7ff02a7fc640 1 -- 172.21.15.5:0/2773536905 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff02c06d4f0 msgr2=0x7ff02c159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:56.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.498+0000 7ff02a7fc640 1 --2- 172.21.15.5:0/2773536905 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff02c06d4f0 0x7ff02c159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:56.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.498+0000 7ff02a7fc640 1 -- 172.21.15.5:0/2773536905 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff01c009400 con 0x7ff02c0ccaa0 2024-03-24T10:57:56.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.500+0000 7ff02a7fc640 1 --2- 172.21.15.5:0/2773536905 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02c0ccaa0 0x7ff02c1efab0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff02000e9b0 tx=0x7ff02000ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:56.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.504+0000 7ff013fff640 1 -- 172.21.15.5:0/2773536905 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff020009670 con 0x7ff02c0ccaa0 2024-03-24T10:57:56.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.504+0000 7ff013fff640 1 -- 172.21.15.5:0/2773536905 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff02000c9a0 con 0x7ff02c0ccaa0 2024-03-24T10:57:56.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.504+0000 7ff013fff640 1 -- 172.21.15.5:0/2773536905 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02000cd50 con 0x7ff02c0ccaa0 2024-03-24T10:57:56.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.504+0000 7ff031847640 1 -- 172.21.15.5:0/2773536905 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff02c0b3b10 con 0x7ff02c0ccaa0 2024-03-24T10:57:56.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.504+0000 7ff031847640 1 -- 172.21.15.5:0/2773536905 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff02c0b31e0 con 0x7ff02c0ccaa0 2024-03-24T10:57:56.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.505+0000 7ff031847640 1 -- 172.21.15.5:0/2773536905 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff02c0b3ca0 con 0x7ff02c0ccaa0 2024-03-24T10:57:56.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.773+0000 7ff013fff640 1 -- 172.21.15.5:0/2773536905 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff020013470 con 0x7ff02c0ccaa0 2024-03-24T10:57:56.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.773+0000 7ff013fff640 1 --2- 172.21.15.5:0/2773536905 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff01806b860 0x7ff01806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:56.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.773+0000 7ff02affd640 1 --2- 172.21.15.5:0/2773536905 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff01806b860 0x7ff01806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:56.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.774+0000 7ff02affd640 1 --2- 172.21.15.5:0/2773536905 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff01806b860 0x7ff01806dd20 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff01c036000 tx=0x7ff01c03b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:56.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.776+0000 7ff013fff640 1 -- 172.21.15.5:0/2773536905 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7ff020059370 con 0x7ff02c0ccaa0 2024-03-24T10:57:56.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:56.787+0000 7ff013fff640 1 -- 172.21.15.5:0/2773536905 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff020055610 con 0x7ff02c0ccaa0 2024-03-24T10:57:57.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:57.032+0000 7ff031847640 1 -- 172.21.15.5:0/2773536905 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff02c0681d0 con 0x7ff02c0ccaa0 2024-03-24T10:57:57.056 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:57.056+0000 7ff013fff640 1 -- 172.21.15.5:0/2773536905 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7ff020017090 con 0x7ff02c0ccaa0 2024-03-24T10:57:57.056 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:57.056 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:57:57.057 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:57:57.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:57.059+0000 7ff031847640 1 -- 172.21.15.5:0/2773536905 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff01806b860 msgr2=0x7ff01806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:57.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:57.059+0000 7ff031847640 1 --2- 172.21.15.5:0/2773536905 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff01806b860 0x7ff01806dd20 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff01c036000 tx=0x7ff01c03b040 comp rx=0 tx=0).stop 2024-03-24T10:57:57.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:57.059+0000 7ff031847640 1 -- 172.21.15.5:0/2773536905 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02c0ccaa0 msgr2=0x7ff02c1efab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:57.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:57.059+0000 7ff031847640 1 --2- 172.21.15.5:0/2773536905 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02c0ccaa0 0x7ff02c1efab0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff02000e9b0 tx=0x7ff02000ee70 comp rx=0 tx=0).stop 2024-03-24T10:57:57.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:57.059+0000 7ff031847640 1 -- 172.21.15.5:0/2773536905 shutdown_connections 2024-03-24T10:57:57.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:57.059+0000 7ff031847640 1 --2- 172.21.15.5:0/2773536905 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7ff01806b860 0x7ff01806dd20 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:57.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:57.059+0000 7ff031847640 1 --2- 172.21.15.5:0/2773536905 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7ff02c1543a0 0x7ff02c1efff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:57.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:57.059+0000 7ff031847640 1 --2- 172.21.15.5:0/2773536905 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff02c0ccaa0 0x7ff02c1efab0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:57.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:57.059+0000 7ff031847640 1 --2- 172.21.15.5:0/2773536905 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7ff02c06d4f0 0x7ff02c159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:57.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:57.059+0000 7ff031847640 1 -- 172.21.15.5:0/2773536905 >> 172.21.15.5:0/2773536905 conn(0x7ff02c072ef0 msgr2=0x7ff02c061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:57.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:57.059+0000 7ff031847640 1 -- 172.21.15.5:0/2773536905 shutdown_connections 2024-03-24T10:57:57.060 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:57.059+0000 7ff031847640 1 -- 172.21.15.5:0/2773536905 wait complete. 2024-03-24T10:57:57.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:57:57.674 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:57:58.675 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:57:58.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.778+0000 7f9a7b525640 1 Processor -- start 2024-03-24T10:57:58.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.778+0000 7f9a7b525640 1 -- start start 2024-03-24T10:57:58.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.779+0000 7f9a7b525640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9a74154220 0x7f9a740c4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:58.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.779+0000 7f9a7b525640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9a740c84e0 0x7f9a740cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:58.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.779+0000 7f9a7b525640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a740c9050 con 0x7f9a740c8900 2024-03-24T10:57:58.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.779+0000 7f9a7b525640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f9a7415a5f0 con 0x7f9a74154220 2024-03-24T10:57:58.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.779+0000 7f9a7b525640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f9a7415a8d0 con 0x7f9a740c84e0 2024-03-24T10:57:58.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.779+0000 7f9a792c1640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9a74154220 0x7f9a740c4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:58.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.780+0000 7f9a792c1640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f9a74154220 0x7f9a740c4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:39664/0 (socket says 172.21.15.5:39664) 2024-03-24T10:57:58.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.780+0000 7f9a792c1640 1 -- 172.21.15.5:0/3256417238 learned_addr learned my addr 172.21.15.5:0/3256417238 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:58.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.780+0000 7f9a78ac0640 1 --2- 172.21.15.5:0/3256417238 >> v2:172.21.15.57:3300/0 conn(0x7f9a740c84e0 0x7f9a740cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.782+0000 7f9a792c1640 1 -- 172.21.15.5:0/3256417238 >> v2:172.21.15.57:3300/0 conn(0x7f9a740c84e0 msgr2=0x7f9a740cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:57:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.782+0000 7f9a792c1640 1 --2- 172.21.15.5:0/3256417238 >> v2:172.21.15.57:3300/0 conn(0x7f9a740c84e0 0x7f9a740cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.782+0000 7f9a792c1640 1 -- 172.21.15.5:0/3256417238 >> v1:172.21.15.57:6789/0 conn(0x7f9a740c8900 legacy=0x7f9a7406c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:58.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.782+0000 7f9a792c1640 1 -- 172.21.15.5:0/3256417238 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a7415abb0 con 0x7f9a74154220 2024-03-24T10:57:58.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.784+0000 7f9a792c1640 1 --2- 172.21.15.5:0/3256417238 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a74154220 0x7f9a740c4300 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9a740ce590 tx=0x7f9a7002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dda466e7324a05e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:58.786 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.787+0000 7f9a6bfff640 1 -- 172.21.15.5:0/3256417238 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7002f850 con 0x7f9a74154220 2024-03-24T10:57:58.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.789+0000 7f9a6bfff640 1 -- 172.21.15.5:0/3256417238 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a7002f9b0 con 0x7f9a74154220 2024-03-24T10:57:58.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.789+0000 7f9a6bfff640 1 -- 172.21.15.5:0/3256417238 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7002fd40 con 0x7f9a74154220 2024-03-24T10:57:58.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.789+0000 7f9a7b525640 1 -- 172.21.15.5:0/3256417238 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a74154220 msgr2=0x7f9a740c4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:58.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.789+0000 7f9a7b525640 1 --2- 172.21.15.5:0/3256417238 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a74154220 0x7f9a740c4300 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9a740ce590 tx=0x7f9a7002ba30 comp rx=0 tx=0).stop 2024-03-24T10:57:58.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.790+0000 7f9a7b525640 1 -- 172.21.15.5:0/3256417238 shutdown_connections 2024-03-24T10:57:58.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.790+0000 7f9a7b525640 1 --2- 172.21.15.5:0/3256417238 >> v2:172.21.15.57:3300/0 conn(0x7f9a740c84e0 0x7f9a740cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:58.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.790+0000 7f9a7b525640 1 --2- 172.21.15.5:0/3256417238 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a74154220 0x7f9a740c4300 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:58.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.790+0000 7f9a7b525640 1 -- 172.21.15.5:0/3256417238 >> 172.21.15.5:0/3256417238 conn(0x7f9a7406f830 msgr2=0x7f9a7406fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:58.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.790+0000 7f9a7b525640 1 -- 172.21.15.5:0/3256417238 shutdown_connections 2024-03-24T10:57:58.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.790+0000 7f9a7b525640 1 -- 172.21.15.5:0/3256417238 wait complete. 2024-03-24T10:57:58.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.791+0000 7f9a7b525640 1 Processor -- start 2024-03-24T10:57:58.792 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.792+0000 7f9a7b525640 1 -- start start 2024-03-24T10:57:58.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.793+0000 7f9a7b525640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9a740c84e0 0x7f9a740ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:58.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.793+0000 7f9a7b525640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a740c8900 0x7f9a7415b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:58.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.793+0000 7f9a7b525640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a74154220 0x7f9a7415bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:58.793 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.793+0000 7f9a7b525640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f9a741efeb0 con 0x7f9a740c8900 2024-03-24T10:57:58.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.793+0000 7f9a7b525640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f9a741f0130 con 0x7f9a74154220 2024-03-24T10:57:58.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.793+0000 7f9a7b525640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f9a741f0410 con 0x7f9a740c84e0 2024-03-24T10:57:58.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.793+0000 7f9a78ac0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a740c8900 0x7f9a7415b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:58.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.793+0000 7f9a79ac2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a74154220 0x7f9a7415bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:58.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.793+0000 7f9a79ac2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a74154220 0x7f9a7415bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36838/0 (socket says 172.21.15.5:36838) 2024-03-24T10:57:58.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.794+0000 7f9a79ac2640 1 -- 172.21.15.5:0/2166363597 learned_addr learned my addr 172.21.15.5:0/2166363597 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:57:58.794 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.794+0000 7f9a792c1640 1 --2- 172.21.15.5:0/2166363597 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9a740c84e0 0x7f9a740ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.797+0000 7f9a79ac2640 1 -- 172.21.15.5:0/2166363597 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9a740c84e0 msgr2=0x7f9a740ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.797+0000 7f9a79ac2640 1 --2- 172.21.15.5:0/2166363597 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9a740c84e0 0x7f9a740ccba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.797+0000 7f9a79ac2640 1 -- 172.21.15.5:0/2166363597 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a740c8900 msgr2=0x7f9a7415b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.797+0000 7f9a79ac2640 1 --2- 172.21.15.5:0/2166363597 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a740c8900 0x7f9a7415b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:58.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.797+0000 7f9a79ac2640 1 -- 172.21.15.5:0/2166363597 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a700094d0 con 0x7f9a74154220 2024-03-24T10:57:58.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.800+0000 7f9a79ac2640 1 --2- 172.21.15.5:0/2166363597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a74154220 0x7f9a7415bb70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00c730 tx=0x7f9a6c00cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:58.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.807+0000 7f9a69ffb640 1 -- 172.21.15.5:0/2166363597 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c003b70 con 0x7f9a74154220 2024-03-24T10:57:58.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.807+0000 7f9a69ffb640 1 -- 172.21.15.5:0/2166363597 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a6c003cd0 con 0x7f9a74154220 2024-03-24T10:57:58.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.807+0000 7f9a7b525640 1 -- 172.21.15.5:0/2166363597 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a741f06f0 con 0x7f9a74154220 2024-03-24T10:57:58.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.807+0000 7f9a69ffb640 1 -- 172.21.15.5:0/2166363597 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c00aa90 con 0x7f9a74154220 2024-03-24T10:57:58.807 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.807+0000 7f9a7b525640 1 -- 172.21.15.5:0/2166363597 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a740c8150 con 0x7f9a74154220 2024-03-24T10:57:58.808 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:58.808+0000 7f9a7b525640 1 -- 172.21.15.5:0/2166363597 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a740681d0 con 0x7f9a74154220 2024-03-24T10:57:59.086 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:59.086+0000 7f9a69ffb640 1 -- 172.21.15.5:0/2166363597 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9a6c00abf0 con 0x7f9a74154220 2024-03-24T10:57:59.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:59.086+0000 7f9a69ffb640 1 --2- 172.21.15.5:0/2166363597 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9a5806b810 0x7f9a5806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:57:59.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:59.087+0000 7f9a792c1640 1 --2- 172.21.15.5:0/2166363597 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9a5806b810 0x7f9a5806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:57:59.087 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:59.087+0000 7f9a792c1640 1 --2- 172.21.15.5:0/2166363597 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9a5806b810 0x7f9a5806dcd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9a70036000 tx=0x7f9a70008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:57:59.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:59.094+0000 7f9a69ffb640 1 -- 172.21.15.5:0/2166363597 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f9a6c00ce40 con 0x7f9a74154220 2024-03-24T10:57:59.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:59.110+0000 7f9a69ffb640 1 -- 172.21.15.5:0/2166363597 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9a6c055490 con 0x7f9a74154220 2024-03-24T10:57:59.332 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:59.332+0000 7f9a7b525640 1 -- 172.21.15.5:0/2166363597 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a741521f0 con 0x7f9a74154220 2024-03-24T10:57:59.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:59.356+0000 7f9a69ffb640 1 -- 172.21.15.5:0/2166363597 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f9a6c058b50 con 0x7f9a74154220 2024-03-24T10:57:59.356 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:57:59.356 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:57:59.357 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:57:59.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:59.359+0000 7f9a7b525640 1 -- 172.21.15.5:0/2166363597 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9a5806b810 msgr2=0x7f9a5806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:59.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:59.359+0000 7f9a7b525640 1 --2- 172.21.15.5:0/2166363597 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9a5806b810 0x7f9a5806dcd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9a70036000 tx=0x7f9a70008030 comp rx=0 tx=0).stop 2024-03-24T10:57:59.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:59.359+0000 7f9a7b525640 1 -- 172.21.15.5:0/2166363597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a74154220 msgr2=0x7f9a7415bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:57:59.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:59.359+0000 7f9a7b525640 1 --2- 172.21.15.5:0/2166363597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a74154220 0x7f9a7415bb70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00c730 tx=0x7f9a6c00cbf0 comp rx=0 tx=0).stop 2024-03-24T10:57:59.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:59.359+0000 7f9a7b525640 1 -- 172.21.15.5:0/2166363597 shutdown_connections 2024-03-24T10:57:59.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:59.359+0000 7f9a7b525640 1 --2- 172.21.15.5:0/2166363597 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f9a5806b810 0x7f9a5806dcd0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:59.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:59.359+0000 7f9a7b525640 1 --2- 172.21.15.5:0/2166363597 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f9a740c8900 0x7f9a7415b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:59.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:59.359+0000 7f9a7b525640 1 --2- 172.21.15.5:0/2166363597 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a74154220 0x7f9a7415bb70 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:59.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:59.359+0000 7f9a7b525640 1 --2- 172.21.15.5:0/2166363597 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f9a740c84e0 0x7f9a740ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:57:59.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:59.359+0000 7f9a7b525640 1 -- 172.21.15.5:0/2166363597 >> 172.21.15.5:0/2166363597 conn(0x7f9a7406f830 msgr2=0x7f9a7406fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:57:59.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:59.359+0000 7f9a7b525640 1 -- 172.21.15.5:0/2166363597 shutdown_connections 2024-03-24T10:57:59.360 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:57:59.359+0000 7f9a7b525640 1 -- 172.21.15.5:0/2166363597 wait complete. 2024-03-24T10:57:59.975 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:57:59.975 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:00.976 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:01.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.079+0000 7f6993f93640 1 Processor -- start 2024-03-24T10:58:01.079 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.079+0000 7f6993f93640 1 -- start start 2024-03-24T10:58:01.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.080+0000 7f6993f93640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f698c15aac0 0x7f698c15aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:01.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.080+0000 7f6993f93640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f698c06c230 0x7f698c06c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:01.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.080+0000 7f6993f93640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f698c06cd50 con 0x7f698c0cebc0 2024-03-24T10:58:01.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.080+0000 7f6993f93640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f698c0ce790 con 0x7f698c06c230 2024-03-24T10:58:01.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.080+0000 7f6993f93640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f698c15b3e0 con 0x7f698c15aac0 2024-03-24T10:58:01.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.080+0000 7f699152e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f698c06c230 0x7f698c06c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:01.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.080+0000 7f6991d2f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f698c15aac0 0x7f698c15aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:01.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.081+0000 7f6991d2f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f698c15aac0 0x7f698c15aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36842/0 (socket says 172.21.15.5:36842) 2024-03-24T10:58:01.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.081+0000 7f6991d2f640 1 -- 172.21.15.5:0/2810201932 learned_addr learned my addr 172.21.15.5:0/2810201932 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:01.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.083+0000 7f6991d2f640 1 -- 172.21.15.5:0/2810201932 >> v2:172.21.15.5:3300/0 conn(0x7f698c06c230 msgr2=0x7f698c06c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:01.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.083+0000 7f6991d2f640 1 --2- 172.21.15.5:0/2810201932 >> v2:172.21.15.5:3300/0 conn(0x7f698c06c230 0x7f698c06c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:01.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.083+0000 7f6991d2f640 1 -- 172.21.15.5:0/2810201932 >> v1:172.21.15.5:6789/0 conn(0x7f698c0cebc0 legacy=0x7f698c0ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:01.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.083+0000 7f6991d2f640 1 -- 172.21.15.5:0/2810201932 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698c0cdb10 con 0x7f698c15aac0 2024-03-24T10:58:01.085 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.085+0000 7f6991d2f640 1 --2- 172.21.15.5:0/2810201932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f698c15aac0 0x7f698c15aea0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f698c0b2780 tx=0x7f698802ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8894471b5b41b671 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:01.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.090+0000 7f6990d2d640 1 -- 172.21.15.5:0/2810201932 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698802f850 con 0x7f698c15aac0 2024-03-24T10:58:01.089 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.090+0000 7f6990d2d640 1 -- 172.21.15.5:0/2810201932 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f698802f9b0 con 0x7f698c15aac0 2024-03-24T10:58:01.090 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.090+0000 7f6990d2d640 1 -- 172.21.15.5:0/2810201932 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698802fd40 con 0x7f698c15aac0 2024-03-24T10:58:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.091+0000 7f6993f93640 1 -- 172.21.15.5:0/2810201932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f698c15aac0 msgr2=0x7f698c15aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.091+0000 7f6993f93640 1 --2- 172.21.15.5:0/2810201932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f698c15aac0 0x7f698c15aea0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f698c0b2780 tx=0x7f698802ba30 comp rx=0 tx=0).stop 2024-03-24T10:58:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.091+0000 7f6993f93640 1 -- 172.21.15.5:0/2810201932 shutdown_connections 2024-03-24T10:58:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.091+0000 7f6993f93640 1 --2- 172.21.15.5:0/2810201932 >> v2:172.21.15.5:3300/0 conn(0x7f698c06c230 0x7f698c06c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.091+0000 7f6993f93640 1 --2- 172.21.15.5:0/2810201932 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f698c15aac0 0x7f698c15aea0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.091+0000 7f6993f93640 1 -- 172.21.15.5:0/2810201932 >> 172.21.15.5:0/2810201932 conn(0x7f698c0cfb90 msgr2=0x7f698c0cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.091+0000 7f6993f93640 1 -- 172.21.15.5:0/2810201932 shutdown_connections 2024-03-24T10:58:01.091 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.091+0000 7f6993f93640 1 -- 172.21.15.5:0/2810201932 wait complete. 2024-03-24T10:58:01.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.092+0000 7f6993f93640 1 Processor -- start 2024-03-24T10:58:01.092 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.092+0000 7f6993f93640 1 -- start start 2024-03-24T10:58:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.093+0000 7f6993f93640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f698c06c230 0x7f698c0b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.093+0000 7f6993f93640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f698c0cebc0 0x7f698c0af760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.093+0000 7f6993f93640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f698c15aac0 0x7f698c146eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.093+0000 7f6993f93640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f698c0b0280 con 0x7f698c06c230 2024-03-24T10:58:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.093+0000 7f6993f93640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f698c155740 con 0x7f698c0cebc0 2024-03-24T10:58:01.093 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.093+0000 7f6993f93640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f698c157670 con 0x7f698c15aac0 2024-03-24T10:58:01.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.093+0000 7f6991d2f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f698c06c230 0x7f698c0b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:01.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.093+0000 7f699152e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f698c0cebc0 0x7f698c0af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:01.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.093+0000 7f699152e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f698c0cebc0 0x7f698c0af760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:36854/0 (socket says 172.21.15.5:36854) 2024-03-24T10:58:01.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.093+0000 7f699152e640 1 -- 172.21.15.5:0/2137065976 learned_addr learned my addr 172.21.15.5:0/2137065976 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:01.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.094+0000 7f6992530640 1 --2- 172.21.15.5:0/2137065976 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f698c15aac0 0x7f698c146eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:01.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.097+0000 7f699152e640 1 -- 172.21.15.5:0/2137065976 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f698c15aac0 msgr2=0x7f698c146eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:01.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.097+0000 7f699152e640 1 --2- 172.21.15.5:0/2137065976 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f698c15aac0 0x7f698c146eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:01.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.097+0000 7f699152e640 1 -- 172.21.15.5:0/2137065976 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f698c06c230 msgr2=0x7f698c0b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:01.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.097+0000 7f699152e640 1 --2- 172.21.15.5:0/2137065976 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f698c06c230 0x7f698c0b0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:01.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.097+0000 7f699152e640 1 -- 172.21.15.5:0/2137065976 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69880094d0 con 0x7f698c0cebc0 2024-03-24T10:58:01.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.099+0000 7f699152e640 1 --2- 172.21.15.5:0/2137065976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f698c0cebc0 0x7f698c0af760 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f697c00e9b0 tx=0x7f697c00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:01.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.104+0000 7f69827fc640 1 -- 172.21.15.5:0/2137065976 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c009670 con 0x7f698c0cebc0 2024-03-24T10:58:01.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.104+0000 7f69827fc640 1 -- 172.21.15.5:0/2137065976 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f697c00c9a0 con 0x7f698c0cebc0 2024-03-24T10:58:01.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.104+0000 7f69827fc640 1 -- 172.21.15.5:0/2137065976 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f697c00cd50 con 0x7f698c0cebc0 2024-03-24T10:58:01.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.104+0000 7f6993f93640 1 -- 172.21.15.5:0/2137065976 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f698c157910 con 0x7f698c0cebc0 2024-03-24T10:58:01.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.104+0000 7f6993f93640 1 -- 172.21.15.5:0/2137065976 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f698c157bf0 con 0x7f698c0cebc0 2024-03-24T10:58:01.105 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.105+0000 7f6993f93640 1 -- 172.21.15.5:0/2137065976 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f698c0681d0 con 0x7f698c0cebc0 2024-03-24T10:58:01.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.130+0000 7f69827fc640 1 -- 172.21.15.5:0/2137065976 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f697c013470 con 0x7f698c0cebc0 2024-03-24T10:58:01.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.130+0000 7f69827fc640 1 --2- 172.21.15.5:0/2137065976 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f697006b860 0x7f697006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:01.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.131+0000 7f6991d2f640 1 --2- 172.21.15.5:0/2137065976 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f697006b860 0x7f697006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:01.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.131+0000 7f6991d2f640 1 --2- 172.21.15.5:0/2137065976 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f697006b860 0x7f697006dd20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f698c0b2780 tx=0x7f6988008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:01.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.133+0000 7f69827fc640 1 -- 172.21.15.5:0/2137065976 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f697c059370 con 0x7f698c0cebc0 2024-03-24T10:58:01.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.417+0000 7f69827fc640 1 -- 172.21.15.5:0/2137065976 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f697c055610 con 0x7f698c0cebc0 2024-03-24T10:58:01.638 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.638+0000 7f6993f93640 1 -- 172.21.15.5:0/2137065976 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f698c0caa40 con 0x7f698c0cebc0 2024-03-24T10:58:01.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.778+0000 7f69827fc640 1 -- 172.21.15.5:0/2137065976 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f697c017020 con 0x7f698c0cebc0 2024-03-24T10:58:01.779 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:01.779 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:01.779 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:01.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.780+0000 7f6993f93640 1 -- 172.21.15.5:0/2137065976 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f697006b860 msgr2=0x7f697006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.780+0000 7f6993f93640 1 --2- 172.21.15.5:0/2137065976 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f697006b860 0x7f697006dd20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f698c0b2780 tx=0x7f6988008030 comp rx=0 tx=0).stop 2024-03-24T10:58:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.780+0000 7f6993f93640 1 -- 172.21.15.5:0/2137065976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f698c0cebc0 msgr2=0x7f698c0af760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.780+0000 7f6993f93640 1 --2- 172.21.15.5:0/2137065976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f698c0cebc0 0x7f698c0af760 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f697c00e9b0 tx=0x7f697c00ee70 comp rx=0 tx=0).stop 2024-03-24T10:58:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.780+0000 7f6993f93640 1 -- 172.21.15.5:0/2137065976 shutdown_connections 2024-03-24T10:58:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.780+0000 7f6993f93640 1 --2- 172.21.15.5:0/2137065976 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f697006b860 0x7f697006dd20 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.780+0000 7f6993f93640 1 --2- 172.21.15.5:0/2137065976 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f698c15aac0 0x7f698c146eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.780+0000 7f6993f93640 1 --2- 172.21.15.5:0/2137065976 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f698c0cebc0 0x7f698c0af760 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.780+0000 7f6993f93640 1 --2- 172.21.15.5:0/2137065976 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f698c06c230 0x7f698c0b0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.780+0000 7f6993f93640 1 -- 172.21.15.5:0/2137065976 >> 172.21.15.5:0/2137065976 conn(0x7f698c0cfb90 msgr2=0x7f698c0c6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.780+0000 7f6993f93640 1 -- 172.21.15.5:0/2137065976 shutdown_connections 2024-03-24T10:58:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:01.780+0000 7f6993f93640 1 -- 172.21.15.5:0/2137065976 wait complete. 2024-03-24T10:58:02.395 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:02.395 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:02.960 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:58:02.963 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:58:03.396 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:03.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.499+0000 7fdb34ad4640 1 Processor -- start 2024-03-24T10:58:03.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.499+0000 7fdb34ad4640 1 -- start start 2024-03-24T10:58:03.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.500+0000 7fdb34ad4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb3015aea0 0x7fdb300cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:03.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.500+0000 7fdb34ad4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fdb300cf100 0x7fdb300ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:03.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.500+0000 7fdb34ad4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb3006cd50 con 0x7fdb3015aac0 2024-03-24T10:58:03.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.500+0000 7fdb34ad4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fdb300ce790 con 0x7fdb300cf100 2024-03-24T10:58:03.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.500+0000 7fdb34ad4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fdb300cdb10 con 0x7fdb3015aea0 2024-03-24T10:58:03.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.501+0000 7fdb2e59c640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fdb3015aac0 0x7fdb3006c230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:39254/0 (socket says 172.21.15.5:39254) 2024-03-24T10:58:03.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.501+0000 7fdb2e59c640 1 -- 172.21.15.5:0/2191398218 learned_addr learned my addr 172.21.15.5:0/2191398218 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:58:03.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.501+0000 7fdb27fff640 1 --2- 172.21.15.5:0/2191398218 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb3015aea0 0x7fdb300cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:03.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.502+0000 7fdb2ed9d640 1 --2- 172.21.15.5:0/2191398218 >> v2:172.21.15.57:3300/0 conn(0x7fdb300cf100 0x7fdb300ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:03.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.504+0000 7fdb27fff640 1 -- 172.21.15.5:0/2191398218 >> v2:172.21.15.57:3300/0 conn(0x7fdb300cf100 msgr2=0x7fdb300ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:03.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.504+0000 7fdb27fff640 1 --2- 172.21.15.5:0/2191398218 >> v2:172.21.15.57:3300/0 conn(0x7fdb300cf100 0x7fdb300ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:03.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.504+0000 7fdb27fff640 1 -- 172.21.15.5:0/2191398218 >> v1:172.21.15.57:6789/0 conn(0x7fdb3015aac0 legacy=0x7fdb3006c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:03.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.504+0000 7fdb27fff640 1 -- 172.21.15.5:0/2191398218 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb300cddf0 con 0x7fdb3015aea0 2024-03-24T10:58:03.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.506+0000 7fdb27fff640 1 --2- 172.21.15.5:0/2191398218 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb3015aea0 0x7fdb300cebc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdb180097a0 tx=0x7fdb1802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=935f2b42cca82d14 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:03.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.509+0000 7fdb2dd9b640 1 -- 172.21.15.5:0/2191398218 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb18030610 con 0x7fdb3015aea0 2024-03-24T10:58:03.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.512+0000 7fdb2dd9b640 1 -- 172.21.15.5:0/2191398218 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb18030770 con 0x7fdb3015aea0 2024-03-24T10:58:03.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.512+0000 7fdb2dd9b640 1 -- 172.21.15.5:0/2191398218 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb18030b00 con 0x7fdb3015aea0 2024-03-24T10:58:03.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.512+0000 7fdb34ad4640 1 -- 172.21.15.5:0/2191398218 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb3015aea0 msgr2=0x7fdb300cebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:03.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.512+0000 7fdb34ad4640 1 --2- 172.21.15.5:0/2191398218 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb3015aea0 0x7fdb300cebc0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdb180097a0 tx=0x7fdb1802b960 comp rx=0 tx=0).stop 2024-03-24T10:58:03.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.512+0000 7fdb34ad4640 1 -- 172.21.15.5:0/2191398218 shutdown_connections 2024-03-24T10:58:03.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.512+0000 7fdb34ad4640 1 --2- 172.21.15.5:0/2191398218 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb3015aea0 0x7fdb300cebc0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:03.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.512+0000 7fdb34ad4640 1 --2- 172.21.15.5:0/2191398218 >> v2:172.21.15.57:3300/0 conn(0x7fdb300cf100 0x7fdb300ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:03.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.512+0000 7fdb34ad4640 1 -- 172.21.15.5:0/2191398218 >> 172.21.15.5:0/2191398218 conn(0x7fdb300cfb90 msgr2=0x7fdb300cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:03.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.512+0000 7fdb34ad4640 1 -- 172.21.15.5:0/2191398218 shutdown_connections 2024-03-24T10:58:03.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.512+0000 7fdb34ad4640 1 -- 172.21.15.5:0/2191398218 wait complete. 2024-03-24T10:58:03.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.514+0000 7fdb34ad4640 1 Processor -- start 2024-03-24T10:58:03.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.514+0000 7fdb34ad4640 1 -- start start 2024-03-24T10:58:03.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.514+0000 7fdb34ad4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb300cf100 0x7fdb3014ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:03.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.514+0000 7fdb34ad4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb3015aac0 0x7fdb3014f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:03.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.514+0000 7fdb34ad4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb3015aea0 0x7fdb300b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:03.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.514+0000 7fdb34ad4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fdb301f0140 con 0x7fdb300cf100 2024-03-24T10:58:03.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.514+0000 7fdb34ad4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fdb301f0330 con 0x7fdb3015aac0 2024-03-24T10:58:03.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.514+0000 7fdb34ad4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fdb301f0610 con 0x7fdb3015aea0 2024-03-24T10:58:03.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.515+0000 7fdb2e59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb300cf100 0x7fdb3014ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:03.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.515+0000 7fdb2ed9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb3015aea0 0x7fdb300b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:03.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.515+0000 7fdb27fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb3015aac0 0x7fdb3014f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:03.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.515+0000 7fdb2ed9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb3015aea0 0x7fdb300b1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:48558/0 (socket says 172.21.15.5:48558) 2024-03-24T10:58:03.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.515+0000 7fdb2ed9d640 1 -- 172.21.15.5:0/650034068 learned_addr learned my addr 172.21.15.5:0/650034068 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:03.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.519+0000 7fdb2ed9d640 1 -- 172.21.15.5:0/650034068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb3015aac0 msgr2=0x7fdb3014f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:03.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.519+0000 7fdb2ed9d640 1 --2- 172.21.15.5:0/650034068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb3015aac0 0x7fdb3014f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:03.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.519+0000 7fdb2ed9d640 1 -- 172.21.15.5:0/650034068 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb300cf100 msgr2=0x7fdb3014ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:03.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.519+0000 7fdb2ed9d640 1 --2- 172.21.15.5:0/650034068 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb300cf100 0x7fdb3014ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:03.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.519+0000 7fdb2ed9d640 1 -- 172.21.15.5:0/650034068 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb18009400 con 0x7fdb3015aea0 2024-03-24T10:58:03.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.519+0000 7fdb2e59c640 1 --2- 172.21.15.5:0/650034068 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb300cf100 0x7fdb3014ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:58:03.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.522+0000 7fdb2ed9d640 1 --2- 172.21.15.5:0/650034068 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb3015aea0 0x7fdb300b1af0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdb28007e40 tx=0x7fdb2800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:03.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.526+0000 7fdb26ffd640 1 -- 172.21.15.5:0/650034068 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb28014620 con 0x7fdb3015aea0 2024-03-24T10:58:03.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.526+0000 7fdb26ffd640 1 -- 172.21.15.5:0/650034068 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb2800ee90 con 0x7fdb3015aea0 2024-03-24T10:58:03.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.526+0000 7fdb34ad4640 1 -- 172.21.15.5:0/650034068 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb301f08f0 con 0x7fdb3015aea0 2024-03-24T10:58:03.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.526+0000 7fdb26ffd640 1 -- 172.21.15.5:0/650034068 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb280149e0 con 0x7fdb3015aea0 2024-03-24T10:58:03.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.526+0000 7fdb34ad4640 1 -- 172.21.15.5:0/650034068 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb301f0eb0 con 0x7fdb3015aea0 2024-03-24T10:58:03.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.527+0000 7fdb34ad4640 1 -- 172.21.15.5:0/650034068 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb300732f0 con 0x7fdb3015aea0 2024-03-24T10:58:03.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.552+0000 7fdb26ffd640 1 -- 172.21.15.5:0/650034068 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdb28014b40 con 0x7fdb3015aea0 2024-03-24T10:58:03.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.552+0000 7fdb26ffd640 1 --2- 172.21.15.5:0/650034068 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdb1006b860 0x7fdb1006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:03.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.553+0000 7fdb2e59c640 1 --2- 172.21.15.5:0/650034068 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdb1006b860 0x7fdb1006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:03.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.553+0000 7fdb2e59c640 1 --2- 172.21.15.5:0/650034068 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdb1006b860 0x7fdb1006dd20 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fdb20007fd0 tx=0x7fdb20007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:03.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.555+0000 7fdb26ffd640 1 -- 172.21.15.5:0/650034068 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fdb28059dc0 con 0x7fdb3015aea0 2024-03-24T10:58:03.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:03.864+0000 7fdb26ffd640 1 -- 172.21.15.5:0/650034068 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdb28056060 con 0x7fdb3015aea0 2024-03-24T10:58:04.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:04.098+0000 7fdb34ad4640 1 -- 172.21.15.5:0/650034068 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb301f10a0 con 0x7fdb3015aea0 2024-03-24T10:58:04.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:04.120+0000 7fdb26ffd640 1 -- 172.21.15.5:0/650034068 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7fdb28059720 con 0x7fdb3015aea0 2024-03-24T10:58:04.121 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:04.121 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:04.121 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:04.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:04.123+0000 7fdb34ad4640 1 -- 172.21.15.5:0/650034068 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdb1006b860 msgr2=0x7fdb1006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:04.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:04.123+0000 7fdb34ad4640 1 --2- 172.21.15.5:0/650034068 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdb1006b860 0x7fdb1006dd20 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fdb20007fd0 tx=0x7fdb20007e80 comp rx=0 tx=0).stop 2024-03-24T10:58:04.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:04.123+0000 7fdb34ad4640 1 -- 172.21.15.5:0/650034068 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb3015aea0 msgr2=0x7fdb300b1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:04.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:04.123+0000 7fdb34ad4640 1 --2- 172.21.15.5:0/650034068 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb3015aea0 0x7fdb300b1af0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdb28007e40 tx=0x7fdb2800e4b0 comp rx=0 tx=0).stop 2024-03-24T10:58:04.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:04.123+0000 7fdb34ad4640 1 -- 172.21.15.5:0/650034068 shutdown_connections 2024-03-24T10:58:04.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:04.123+0000 7fdb34ad4640 1 --2- 172.21.15.5:0/650034068 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fdb1006b860 0x7fdb1006dd20 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:04.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:04.123+0000 7fdb34ad4640 1 --2- 172.21.15.5:0/650034068 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fdb3015aea0 0x7fdb300b1af0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:04.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:04.123+0000 7fdb34ad4640 1 --2- 172.21.15.5:0/650034068 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fdb3015aac0 0x7fdb3014f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:04.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:04.123+0000 7fdb34ad4640 1 --2- 172.21.15.5:0/650034068 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fdb300cf100 0x7fdb3014ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:04.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:04.123+0000 7fdb34ad4640 1 -- 172.21.15.5:0/650034068 >> 172.21.15.5:0/650034068 conn(0x7fdb300cfb90 msgr2=0x7fdb300cc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:04.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:04.123+0000 7fdb34ad4640 1 -- 172.21.15.5:0/650034068 shutdown_connections 2024-03-24T10:58:04.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:04.123+0000 7fdb34ad4640 1 -- 172.21.15.5:0/650034068 wait complete. 2024-03-24T10:58:04.738 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:04.738 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:05.738 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:05.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.841+0000 7f914f4af640 1 Processor -- start 2024-03-24T10:58:05.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.841+0000 7f914f4af640 1 -- start start 2024-03-24T10:58:05.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.842+0000 7f914f4af640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f914815aac0 0x7f914815aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:05.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.842+0000 7f914f4af640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f914806c670 0x7f91480ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:05.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.842+0000 7f914f4af640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f914806cd50 con 0x7f914806c230 2024-03-24T10:58:05.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.842+0000 7f914f4af640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f91480ce790 con 0x7f914806c670 2024-03-24T10:58:05.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.842+0000 7f914f4af640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f914815b3e0 con 0x7f914815aac0 2024-03-24T10:58:05.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.843+0000 7f914da4c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f914806c670 0x7f91480ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:05.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.843+0000 7f914da4c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f914806c670 0x7f91480ca5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52298/0 (socket says 172.21.15.5:52298) 2024-03-24T10:58:05.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.843+0000 7f914da4c640 1 -- 172.21.15.5:0/909317238 learned_addr learned my addr 172.21.15.5:0/909317238 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:05.844 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.844+0000 7f914d24b640 1 --2- 172.21.15.5:0/909317238 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f914815aac0 0x7f914815aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:05.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.848+0000 7f914d24b640 1 -- 172.21.15.5:0/909317238 >> v2:172.21.15.5:3300/0 conn(0x7f914806c670 msgr2=0x7f91480ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:05.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.848+0000 7f914d24b640 1 --2- 172.21.15.5:0/909317238 >> v2:172.21.15.5:3300/0 conn(0x7f914806c670 0x7f91480ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:05.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.848+0000 7f914d24b640 1 -- 172.21.15.5:0/909317238 >> v1:172.21.15.5:6789/0 conn(0x7f914806c230 legacy=0x7f91480cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:05.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.848+0000 7f914d24b640 1 -- 172.21.15.5:0/909317238 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f914806cb00 con 0x7f914815aac0 2024-03-24T10:58:05.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.850+0000 7f914d24b640 1 --2- 172.21.15.5:0/909317238 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f914815aac0 0x7f914815aea0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f91440097a0 tx=0x7f914402b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eabd627ac437536 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:05.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.855+0000 7f913ffff640 1 -- 172.21.15.5:0/909317238 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9144030610 con 0x7f914815aac0 2024-03-24T10:58:05.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.855+0000 7f913ffff640 1 -- 172.21.15.5:0/909317238 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9144030770 con 0x7f914815aac0 2024-03-24T10:58:05.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.855+0000 7f913ffff640 1 -- 172.21.15.5:0/909317238 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9144030b00 con 0x7f914815aac0 2024-03-24T10:58:05.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.856+0000 7f914f4af640 1 -- 172.21.15.5:0/909317238 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f914815aac0 msgr2=0x7f914815aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:05.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.856+0000 7f914f4af640 1 --2- 172.21.15.5:0/909317238 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f914815aac0 0x7f914815aea0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f91440097a0 tx=0x7f914402b960 comp rx=0 tx=0).stop 2024-03-24T10:58:05.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.856+0000 7f914f4af640 1 -- 172.21.15.5:0/909317238 shutdown_connections 2024-03-24T10:58:05.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.856+0000 7f914f4af640 1 --2- 172.21.15.5:0/909317238 >> v2:172.21.15.5:3300/0 conn(0x7f914806c670 0x7f91480ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:05.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.856+0000 7f914f4af640 1 --2- 172.21.15.5:0/909317238 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f914815aac0 0x7f914815aea0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:05.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.856+0000 7f914f4af640 1 -- 172.21.15.5:0/909317238 >> 172.21.15.5:0/909317238 conn(0x7f91480cfb90 msgr2=0x7f91480cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:05.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.856+0000 7f914f4af640 1 -- 172.21.15.5:0/909317238 shutdown_connections 2024-03-24T10:58:05.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.857+0000 7f914f4af640 1 -- 172.21.15.5:0/909317238 wait complete. 2024-03-24T10:58:05.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.858+0000 7f914f4af640 1 Processor -- start 2024-03-24T10:58:05.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.858+0000 7f914f4af640 1 -- start start 2024-03-24T10:58:05.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.860+0000 7f914f4af640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f914806c230 0x7f9148152070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:05.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.860+0000 7f914f4af640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f914806c670 0x7f91481f35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:05.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.860+0000 7f914f4af640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f914815aac0 0x7f91481f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:05.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.860+0000 7f914f4af640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f91480b1a70 con 0x7f914806c670 2024-03-24T10:58:05.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.860+0000 7f914f4af640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f91480b0df0 con 0x7f914806c230 2024-03-24T10:58:05.860 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.860+0000 7f914f4af640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f91480b10d0 con 0x7f914815aac0 2024-03-24T10:58:05.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.860+0000 7f914da4c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f914815aac0 0x7f91481f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:05.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.860+0000 7f914da4c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f914815aac0 0x7f91481f3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:48576/0 (socket says 172.21.15.5:48576) 2024-03-24T10:58:05.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.860+0000 7f914d24b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f914806c230 0x7f9148152070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:05.861 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.860+0000 7f914da4c640 1 -- 172.21.15.5:0/3905115483 learned_addr learned my addr 172.21.15.5:0/3905115483 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:05.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.864+0000 7f914da4c640 1 -- 172.21.15.5:0/3905115483 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f914806c230 msgr2=0x7f9148152070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:05.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.864+0000 7f914da4c640 1 --2- 172.21.15.5:0/3905115483 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f914806c230 0x7f9148152070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:05.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.864+0000 7f914da4c640 1 -- 172.21.15.5:0/3905115483 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f914806c670 msgr2=0x7f91481f35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:05.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.864+0000 7f914da4c640 1 --2- 172.21.15.5:0/3905115483 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f914806c670 0x7f91481f35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:05.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.864+0000 7f914da4c640 1 -- 172.21.15.5:0/3905115483 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9144009400 con 0x7f914815aac0 2024-03-24T10:58:05.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.866+0000 7f914da4c640 1 --2- 172.21.15.5:0/3905115483 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f914815aac0 0x7f91481f3b30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f914000efc0 tx=0x7f914000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:05.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.870+0000 7f913dffb640 1 -- 172.21.15.5:0/3905115483 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914000cd20 con 0x7f914815aac0 2024-03-24T10:58:05.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.870+0000 7f913dffb640 1 -- 172.21.15.5:0/3905115483 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f914000f040 con 0x7f914815aac0 2024-03-24T10:58:05.870 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.870+0000 7f914f4af640 1 -- 172.21.15.5:0/3905115483 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91480b13b0 con 0x7f914815aac0 2024-03-24T10:58:05.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.870+0000 7f913dffb640 1 -- 172.21.15.5:0/3905115483 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914000cea0 con 0x7f914815aac0 2024-03-24T10:58:05.871 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.870+0000 7f914f4af640 1 -- 172.21.15.5:0/3905115483 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9148154de0 con 0x7f914815aac0 2024-03-24T10:58:05.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.872+0000 7f914f4af640 1 -- 172.21.15.5:0/3905115483 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91481f33f0 con 0x7f914815aac0 2024-03-24T10:58:05.903 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.903+0000 7f913dffb640 1 -- 172.21.15.5:0/3905115483 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9140003bd0 con 0x7f914815aac0 2024-03-24T10:58:05.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.903+0000 7f913dffb640 1 --2- 172.21.15.5:0/3905115483 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f912c06b810 0x7f912c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:05.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.903+0000 7f914d24b640 1 --2- 172.21.15.5:0/3905115483 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f912c06b810 0x7f912c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:05.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.904+0000 7f914d24b640 1 --2- 172.21.15.5:0/3905115483 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f912c06b810 0x7f912c06dcd0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9144036000 tx=0x7f914403b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:05.909 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:05.909+0000 7f913dffb640 1 -- 172.21.15.5:0/3905115483 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f914005a4d0 con 0x7f914815aac0 2024-03-24T10:58:06.179 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:06.179+0000 7f913dffb640 1 -- 172.21.15.5:0/3905115483 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9140056070 con 0x7f914815aac0 2024-03-24T10:58:06.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:06.481+0000 7f914f4af640 1 -- 172.21.15.5:0/3905115483 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91480b1210 con 0x7f914815aac0 2024-03-24T10:58:06.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:06.496+0000 7f913dffb640 1 -- 172.21.15.5:0/3905115483 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f91480b1210 con 0x7f914815aac0 2024-03-24T10:58:06.497 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:06.497 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:06.497 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:06.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:06.499+0000 7f914f4af640 1 -- 172.21.15.5:0/3905115483 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f912c06b810 msgr2=0x7f912c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:06.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:06.499+0000 7f914f4af640 1 --2- 172.21.15.5:0/3905115483 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f912c06b810 0x7f912c06dcd0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9144036000 tx=0x7f914403b040 comp rx=0 tx=0).stop 2024-03-24T10:58:06.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:06.499+0000 7f914f4af640 1 -- 172.21.15.5:0/3905115483 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f914815aac0 msgr2=0x7f91481f3b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:06.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:06.499+0000 7f914f4af640 1 --2- 172.21.15.5:0/3905115483 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f914815aac0 0x7f91481f3b30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f914000efc0 tx=0x7f914000c490 comp rx=0 tx=0).stop 2024-03-24T10:58:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:06.499+0000 7f914f4af640 1 -- 172.21.15.5:0/3905115483 shutdown_connections 2024-03-24T10:58:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:06.499+0000 7f914f4af640 1 --2- 172.21.15.5:0/3905115483 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f912c06b810 0x7f912c06dcd0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:06.499+0000 7f914f4af640 1 --2- 172.21.15.5:0/3905115483 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f914806c670 0x7f91481f35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:06.499+0000 7f914f4af640 1 --2- 172.21.15.5:0/3905115483 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f914815aac0 0x7f91481f3b30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:06.499+0000 7f914f4af640 1 --2- 172.21.15.5:0/3905115483 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f914806c230 0x7f9148152070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:06.499+0000 7f914f4af640 1 -- 172.21.15.5:0/3905115483 >> 172.21.15.5:0/3905115483 conn(0x7f91480cfb90 msgr2=0x7f91481e3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:06.499+0000 7f914f4af640 1 -- 172.21.15.5:0/3905115483 shutdown_connections 2024-03-24T10:58:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:06.499+0000 7f914f4af640 1 -- 172.21.15.5:0/3905115483 wait complete. 2024-03-24T10:58:07.115 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:07.115 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:08.117 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:08.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.220+0000 7fac5a18a640 1 Processor -- start 2024-03-24T10:58:08.220 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.220+0000 7fac5a18a640 1 -- start start 2024-03-24T10:58:08.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.220+0000 7fac5a18a640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fac5406c0a0 0x7fac5415e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:08.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.220+0000 7fac5a18a640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fac5406ba80 0x7fac540c69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:08.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.221+0000 7fac5a18a640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac5415b1f0 con 0x7fac54154470 2024-03-24T10:58:08.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.221+0000 7fac5a18a640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fac5415e810 con 0x7fac5406ba80 2024-03-24T10:58:08.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.221+0000 7fac5a18a640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fac5415a5e0 con 0x7fac5406c0a0 2024-03-24T10:58:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.221+0000 7fac52ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fac5406c0a0 0x7fac5415e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.222+0000 7fac53fff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fac5406ba80 0x7fac540c69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.222+0000 7fac52ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fac5406c0a0 0x7fac5415e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:47630/0 (socket says 172.21.15.5:47630) 2024-03-24T10:58:08.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.222+0000 7fac52ffd640 1 -- 172.21.15.5:0/500994610 learned_addr learned my addr 172.21.15.5:0/500994610 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:08.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.225+0000 7fac53fff640 1 -- 172.21.15.5:0/500994610 >> v2:172.21.15.57:3300/0 conn(0x7fac5406c0a0 msgr2=0x7fac5415e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:08.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.225+0000 7fac53fff640 1 --2- 172.21.15.5:0/500994610 >> v2:172.21.15.57:3300/0 conn(0x7fac5406c0a0 0x7fac5415e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:08.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.225+0000 7fac53fff640 1 -- 172.21.15.5:0/500994610 >> v1:172.21.15.5:6789/0 conn(0x7fac54154470 legacy=0x7fac5406f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:08.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.225+0000 7fac53fff640 1 -- 172.21.15.5:0/500994610 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5415a8c0 con 0x7fac5406ba80 2024-03-24T10:58:08.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.229+0000 7fac53fff640 1 --2- 172.21.15.5:0/500994610 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac5406ba80 0x7fac540c69d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fac540c6450 tx=0x7fac4c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=991a0397115db73c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:08.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.235+0000 7fac527fc640 1 -- 172.21.15.5:0/500994610 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c02cd00 con 0x7fac5406ba80 2024-03-24T10:58:08.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.236+0000 7fac527fc640 1 -- 172.21.15.5:0/500994610 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac4c02ce60 con 0x7fac5406ba80 2024-03-24T10:58:08.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.236+0000 7fac527fc640 1 -- 172.21.15.5:0/500994610 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c034660 con 0x7fac5406ba80 2024-03-24T10:58:08.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.236+0000 7fac5a18a640 1 -- 172.21.15.5:0/500994610 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac5406ba80 msgr2=0x7fac540c69d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:08.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.236+0000 7fac5a18a640 1 --2- 172.21.15.5:0/500994610 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac5406ba80 0x7fac540c69d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fac540c6450 tx=0x7fac4c02bf50 comp rx=0 tx=0).stop 2024-03-24T10:58:08.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.236+0000 7fac5a18a640 1 -- 172.21.15.5:0/500994610 shutdown_connections 2024-03-24T10:58:08.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.236+0000 7fac5a18a640 1 --2- 172.21.15.5:0/500994610 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac5406ba80 0x7fac540c69d0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:08.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.236+0000 7fac5a18a640 1 --2- 172.21.15.5:0/500994610 >> v2:172.21.15.57:3300/0 conn(0x7fac5406c0a0 0x7fac5415e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:08.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.236+0000 7fac5a18a640 1 -- 172.21.15.5:0/500994610 >> 172.21.15.5:0/500994610 conn(0x7fac5406d790 msgr2=0x7fac5406dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:08.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.236+0000 7fac5a18a640 1 -- 172.21.15.5:0/500994610 shutdown_connections 2024-03-24T10:58:08.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.236+0000 7fac5a18a640 1 -- 172.21.15.5:0/500994610 wait complete. 2024-03-24T10:58:08.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.238+0000 7fac5a18a640 1 Processor -- start 2024-03-24T10:58:08.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.238+0000 7fac5a18a640 1 -- start start 2024-03-24T10:58:08.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.239+0000 7fac5a18a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac5406ba80 0x7fac54155ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:08.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.239+0000 7fac5a18a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac5406c0a0 0x7fac54152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:08.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.239+0000 7fac5a18a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac54154470 0x7fac54152fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:08.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.239+0000 7fac5a18a640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fac540cb2c0 con 0x7fac54154470 2024-03-24T10:58:08.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.239+0000 7fac5a18a640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fac54155800 con 0x7fac5406c0a0 2024-03-24T10:58:08.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.239+0000 7fac5a18a640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fac540c9f40 con 0x7fac5406ba80 2024-03-24T10:58:08.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.239+0000 7fac53fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac54154470 0x7fac54152fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:08.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.239+0000 7fac537fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac5406ba80 0x7fac54155ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:08.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.239+0000 7fac52ffd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac5406c0a0 0x7fac54152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:08.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.240+0000 7fac53fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac54154470 0x7fac54152fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52314/0 (socket says 172.21.15.5:52314) 2024-03-24T10:58:08.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.240+0000 7fac537fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac5406ba80 0x7fac54155ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:48580/0 (socket says 172.21.15.5:48580) 2024-03-24T10:58:08.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.240+0000 7fac53fff640 1 -- 172.21.15.5:0/1995008892 learned_addr learned my addr 172.21.15.5:0/1995008892 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:08.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.243+0000 7fac53fff640 1 -- 172.21.15.5:0/1995008892 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac5406ba80 msgr2=0x7fac54155ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.243+0000 7fac53fff640 1 --2- 172.21.15.5:0/1995008892 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac5406ba80 0x7fac54155ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.243+0000 7fac53fff640 1 -- 172.21.15.5:0/1995008892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac5406c0a0 msgr2=0x7fac54152a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.243+0000 7fac53fff640 1 --2- 172.21.15.5:0/1995008892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac5406c0a0 0x7fac54152a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:08.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.243+0000 7fac53fff640 1 -- 172.21.15.5:0/1995008892 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4c009a70 con 0x7fac54154470 2024-03-24T10:58:08.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.246+0000 7fac53fff640 1 --2- 172.21.15.5:0/1995008892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac54154470 0x7fac54152fd0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fac4c02c560 tx=0x7fac4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.248+0000 7fac3bfff640 1 -- 172.21.15.5:0/1995008892 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c03d600 con 0x7fac54154470 2024-03-24T10:58:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.248+0000 7fac5a18a640 1 -- 172.21.15.5:0/1995008892 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac540ca220 con 0x7fac54154470 2024-03-24T10:58:08.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.248+0000 7fac5a18a640 1 -- 172.21.15.5:0/1995008892 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac540ca7e0 con 0x7fac54154470 2024-03-24T10:58:08.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.249+0000 7fac5a18a640 1 -- 172.21.15.5:0/1995008892 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac540681d0 con 0x7fac54154470 2024-03-24T10:58:08.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.250+0000 7fac3bfff640 1 -- 172.21.15.5:0/1995008892 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac4c009450 con 0x7fac54154470 2024-03-24T10:58:08.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.250+0000 7fac3bfff640 1 -- 172.21.15.5:0/1995008892 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c03d960 con 0x7fac54154470 2024-03-24T10:58:08.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.519+0000 7fac3bfff640 1 -- 172.21.15.5:0/1995008892 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fac4c03b040 con 0x7fac54154470 2024-03-24T10:58:08.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.520+0000 7fac3bfff640 1 --2- 172.21.15.5:0/1995008892 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac4006b840 0x7fac4006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:08.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.520+0000 7fac537fe640 1 --2- 172.21.15.5:0/1995008892 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac4006b840 0x7fac4006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:08.520 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.520+0000 7fac537fe640 1 --2- 172.21.15.5:0/1995008892 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac4006b840 0x7fac4006dd00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fac440016e0 tx=0x7fac44009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:08.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.534+0000 7fac3bfff640 1 -- 172.21.15.5:0/1995008892 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fac4c07b4e0 con 0x7fac54154470 2024-03-24T10:58:08.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.535+0000 7fac3bfff640 1 -- 172.21.15.5:0/1995008892 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fac4c077780 con 0x7fac54154470 2024-03-24T10:58:08.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.806+0000 7fac5a18a640 1 -- 172.21.15.5:0/1995008892 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac541535b0 con 0x7fac54154470 2024-03-24T10:58:08.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.827+0000 7fac3bfff640 1 -- 172.21.15.5:0/1995008892 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7fac4c04e090 con 0x7fac54154470 2024-03-24T10:58:08.827 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:08.827 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:08.828 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:08.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.829+0000 7fac5a18a640 1 -- 172.21.15.5:0/1995008892 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac4006b840 msgr2=0x7fac4006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:08.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.829+0000 7fac5a18a640 1 --2- 172.21.15.5:0/1995008892 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac4006b840 0x7fac4006dd00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fac440016e0 tx=0x7fac44009040 comp rx=0 tx=0).stop 2024-03-24T10:58:08.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.829+0000 7fac5a18a640 1 -- 172.21.15.5:0/1995008892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac54154470 msgr2=0x7fac54152fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:08.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.829+0000 7fac5a18a640 1 --2- 172.21.15.5:0/1995008892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac54154470 0x7fac54152fd0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fac4c02c560 tx=0x7fac4c004a40 comp rx=0 tx=0).stop 2024-03-24T10:58:08.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.829+0000 7fac5a18a640 1 -- 172.21.15.5:0/1995008892 shutdown_connections 2024-03-24T10:58:08.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.829+0000 7fac5a18a640 1 --2- 172.21.15.5:0/1995008892 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac4006b840 0x7fac4006dd00 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:08.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.829+0000 7fac5a18a640 1 --2- 172.21.15.5:0/1995008892 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac54154470 0x7fac54152fd0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:08.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.829+0000 7fac5a18a640 1 --2- 172.21.15.5:0/1995008892 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac5406c0a0 0x7fac54152a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:08.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.829+0000 7fac5a18a640 1 --2- 172.21.15.5:0/1995008892 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac5406ba80 0x7fac54155ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:08.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.829+0000 7fac5a18a640 1 -- 172.21.15.5:0/1995008892 >> 172.21.15.5:0/1995008892 conn(0x7fac5406d790 msgr2=0x7fac540c8520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:08.830 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.829+0000 7fac5a18a640 1 -- 172.21.15.5:0/1995008892 shutdown_connections 2024-03-24T10:58:08.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:08.829+0000 7fac5a18a640 1 -- 172.21.15.5:0/1995008892 wait complete. 2024-03-24T10:58:09.445 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:09.445 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:10.446 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:10.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.549+0000 7f028027c640 1 Processor -- start 2024-03-24T10:58:10.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.549+0000 7f028027c640 1 -- start start 2024-03-24T10:58:10.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.550+0000 7f028027c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02780cc8f0 0x7f027815b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:10.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.550+0000 7f028027c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f02780cf110 0x7f0278153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:10.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.550+0000 7f028027c640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02780cd890 con 0x7f027806c580 2024-03-24T10:58:10.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.550+0000 7f028027c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f027806cae0 con 0x7f02780cf110 2024-03-24T10:58:10.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.550+0000 7f028027c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f02780cdb50 con 0x7f02780cc8f0 2024-03-24T10:58:10.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.551+0000 7f027d817640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f027806c580 0x7f02781538b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:56594/0 (socket says 172.21.15.5:56594) 2024-03-24T10:58:10.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.551+0000 7f027d817640 1 -- 172.21.15.5:0/1189623662 learned_addr learned my addr 172.21.15.5:0/1189623662 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:58:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.551+0000 7f027e819640 1 --2- 172.21.15.5:0/1189623662 >> v2:172.21.15.57:3300/0 conn(0x7f02780cf110 0x7f0278153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:10.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.551+0000 7f027e018640 1 --2- 172.21.15.5:0/1189623662 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02780cc8f0 0x7f027815b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:10.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.554+0000 7f027e819640 1 -- 172.21.15.5:0/1189623662 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02780cc8f0 msgr2=0x7f027815b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:10.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.554+0000 7f027e819640 1 --2- 172.21.15.5:0/1189623662 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02780cc8f0 0x7f027815b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:10.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.554+0000 7f027e819640 1 -- 172.21.15.5:0/1189623662 >> v1:172.21.15.5:6789/0 conn(0x7f027806c580 legacy=0x7f02781538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:10.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.554+0000 7f027e819640 1 -- 172.21.15.5:0/1189623662 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02780cde30 con 0x7f02780cf110 2024-03-24T10:58:10.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.556+0000 7f027e819640 1 --2- 172.21.15.5:0/1189623662 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02780cf110 0x7f0278153fc0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0270009d90 tx=0x7f027002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9ca4f289aff568f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:10.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.560+0000 7f027d016640 1 -- 172.21.15.5:0/1189623662 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f027002cc50 con 0x7f02780cf110 2024-03-24T10:58:10.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.560+0000 7f027d016640 1 -- 172.21.15.5:0/1189623662 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f027002cdb0 con 0x7f02780cf110 2024-03-24T10:58:10.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.560+0000 7f027d016640 1 -- 172.21.15.5:0/1189623662 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0270034660 con 0x7f02780cf110 2024-03-24T10:58:10.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.561+0000 7f028027c640 1 -- 172.21.15.5:0/1189623662 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02780cf110 msgr2=0x7f0278153fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:10.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.561+0000 7f028027c640 1 --2- 172.21.15.5:0/1189623662 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02780cf110 0x7f0278153fc0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0270009d90 tx=0x7f027002bea0 comp rx=0 tx=0).stop 2024-03-24T10:58:10.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.561+0000 7f028027c640 1 -- 172.21.15.5:0/1189623662 shutdown_connections 2024-03-24T10:58:10.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.561+0000 7f028027c640 1 --2- 172.21.15.5:0/1189623662 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02780cf110 0x7f0278153fc0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:10.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.561+0000 7f028027c640 1 --2- 172.21.15.5:0/1189623662 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02780cc8f0 0x7f027815b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:10.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.561+0000 7f028027c640 1 -- 172.21.15.5:0/1189623662 >> 172.21.15.5:0/1189623662 conn(0x7f02780b1820 msgr2=0x7f02780b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:10.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.561+0000 7f028027c640 1 -- 172.21.15.5:0/1189623662 shutdown_connections 2024-03-24T10:58:10.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.561+0000 7f028027c640 1 -- 172.21.15.5:0/1189623662 wait complete. 2024-03-24T10:58:10.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.562+0000 7f028027c640 1 Processor -- start 2024-03-24T10:58:10.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.562+0000 7f028027c640 1 -- start start 2024-03-24T10:58:10.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.563+0000 7f028027c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f027806c580 0x7f02780c3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:10.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.563+0000 7f028027c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02780cc8f0 0x7f02780c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:10.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.563+0000 7f028027c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02780cf110 0x7f0278150d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:10.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.563+0000 7f028027c640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f02780c9a80 con 0x7f02780cc8f0 2024-03-24T10:58:10.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.563+0000 7f028027c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f02781f3920 con 0x7f027806c580 2024-03-24T10:58:10.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.563+0000 7f028027c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f02780c6f50 con 0x7f02780cf110 2024-03-24T10:58:10.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.564+0000 7f027d817640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02780cc8f0 0x7f02780c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:10.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.564+0000 7f027e819640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02780cf110 0x7f0278150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:10.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.564+0000 7f027e018640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f027806c580 0x7f02780c3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:10.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.564+0000 7f027d817640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02780cc8f0 0x7f02780c8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:52318/0 (socket says 172.21.15.5:52318) 2024-03-24T10:58:10.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.564+0000 7f027d817640 1 -- 172.21.15.5:0/3421963915 learned_addr learned my addr 172.21.15.5:0/3421963915 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:10.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.564+0000 7f027e018640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f027806c580 0x7f02780c3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:47656/0 (socket says 172.21.15.5:47656) 2024-03-24T10:58:10.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.568+0000 7f027d817640 1 -- 172.21.15.5:0/3421963915 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02780cf110 msgr2=0x7f0278150d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:10.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.568+0000 7f027d817640 1 --2- 172.21.15.5:0/3421963915 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02780cf110 0x7f0278150d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:10.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.568+0000 7f027d817640 1 -- 172.21.15.5:0/3421963915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f027806c580 msgr2=0x7f02780c3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:10.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.568+0000 7f027d817640 1 --2- 172.21.15.5:0/3421963915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f027806c580 0x7f02780c3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:10.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.568+0000 7f027d817640 1 -- 172.21.15.5:0/3421963915 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0270009a70 con 0x7f02780cc8f0 2024-03-24T10:58:10.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.570+0000 7f027d817640 1 --2- 172.21.15.5:0/3421963915 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02780cc8f0 0x7f02780c8fa0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f026800c3c0 tx=0x7f026800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:10.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.575+0000 7f026effd640 1 -- 172.21.15.5:0/3421963915 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0268005ce0 con 0x7f02780cc8f0 2024-03-24T10:58:10.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.575+0000 7f026effd640 1 -- 172.21.15.5:0/3421963915 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0268005e40 con 0x7f02780cc8f0 2024-03-24T10:58:10.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.575+0000 7f026effd640 1 -- 172.21.15.5:0/3421963915 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02680072f0 con 0x7f02780cc8f0 2024-03-24T10:58:10.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.575+0000 7f028027c640 1 -- 172.21.15.5:0/3421963915 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02780c7230 con 0x7f02780cc8f0 2024-03-24T10:58:10.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.575+0000 7f028027c640 1 -- 172.21.15.5:0/3421963915 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02780c77f0 con 0x7f02780cc8f0 2024-03-24T10:58:10.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.576+0000 7f028027c640 1 -- 172.21.15.5:0/3421963915 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02781f3a90 con 0x7f02780cc8f0 2024-03-24T10:58:10.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.601+0000 7f026effd640 1 -- 172.21.15.5:0/3421963915 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0268007450 con 0x7f02780cc8f0 2024-03-24T10:58:10.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.601+0000 7f026effd640 1 --2- 172.21.15.5:0/3421963915 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f025c06b840 0x7f025c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:10.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.601+0000 7f027e018640 1 --2- 172.21.15.5:0/3421963915 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f025c06b840 0x7f025c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:10.602 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.602+0000 7f027e018640 1 --2- 172.21.15.5:0/3421963915 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f025c06b840 0x7f025c06dd00 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0274009b40 tx=0x7f0274009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:10.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.604+0000 7f026effd640 1 -- 172.21.15.5:0/3421963915 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f02680581a0 con 0x7f02780cc8f0 2024-03-24T10:58:10.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:10.916+0000 7f026effd640 1 -- 172.21.15.5:0/3421963915 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0268054440 con 0x7f02780cc8f0 2024-03-24T10:58:11.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:11.178+0000 7f028027c640 1 -- 172.21.15.5:0/3421963915 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02781f3a90 con 0x7f02780cc8f0 2024-03-24T10:58:11.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:11.193+0000 7f026effd640 1 -- 172.21.15.5:0/3421963915 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f02781f3a90 con 0x7f02780cc8f0 2024-03-24T10:58:11.193 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:11.193 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:11.194 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:11.195+0000 7f028027c640 1 -- 172.21.15.5:0/3421963915 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f025c06b840 msgr2=0x7f025c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:11.195+0000 7f028027c640 1 --2- 172.21.15.5:0/3421963915 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f025c06b840 0x7f025c06dd00 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0274009b40 tx=0x7f0274009190 comp rx=0 tx=0).stop 2024-03-24T10:58:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:11.195+0000 7f028027c640 1 -- 172.21.15.5:0/3421963915 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02780cc8f0 msgr2=0x7f02780c8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:11.195+0000 7f028027c640 1 --2- 172.21.15.5:0/3421963915 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02780cc8f0 0x7f02780c8fa0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f026800c3c0 tx=0x7f026800c880 comp rx=0 tx=0).stop 2024-03-24T10:58:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:11.195+0000 7f028027c640 1 -- 172.21.15.5:0/3421963915 shutdown_connections 2024-03-24T10:58:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:11.195+0000 7f028027c640 1 --2- 172.21.15.5:0/3421963915 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f025c06b840 0x7f025c06dd00 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:11.195+0000 7f028027c640 1 --2- 172.21.15.5:0/3421963915 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f02780cc8f0 0x7f02780c8fa0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:11.195+0000 7f028027c640 1 --2- 172.21.15.5:0/3421963915 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02780cf110 0x7f0278150d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:11.195+0000 7f028027c640 1 --2- 172.21.15.5:0/3421963915 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f027806c580 0x7f02780c3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:11.195+0000 7f028027c640 1 -- 172.21.15.5:0/3421963915 >> 172.21.15.5:0/3421963915 conn(0x7f02780b1820 msgr2=0x7f027815c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:11.196+0000 7f028027c640 1 -- 172.21.15.5:0/3421963915 shutdown_connections 2024-03-24T10:58:11.196 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:11.196+0000 7f028027c640 1 -- 172.21.15.5:0/3421963915 wait complete. 2024-03-24T10:58:11.811 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:11.811 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:12.812 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.915+0000 7f19556d4640 1 Processor -- start 2024-03-24T10:58:12.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.915+0000 7f19556d4640 1 -- start start 2024-03-24T10:58:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.916+0000 7f19556d4640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f19501543a0 0x7f195015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.916+0000 7f19556d4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19500cb2f0 con 0x7f195006d4f0 2024-03-24T10:58:12.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.916+0000 7f19556d4640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19500ca7e0 con 0x7f195006bf50 2024-03-24T10:58:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.916+0000 7f19556d4640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f19500c6750 con 0x7f19501543a0 2024-03-24T10:58:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.917+0000 7f194effd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f19501543a0 0x7f195015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.917+0000 7f194f7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f195006bf50 0x7f195006c330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:45750/0 (socket says 172.21.15.5:45750) 2024-03-24T10:58:12.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.917+0000 7f194f7fe640 1 -- 172.21.15.5:0/1131339219 learned_addr learned my addr 172.21.15.5:0/1131339219 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:58:12.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.921+0000 7f194dffb640 1 -- 172.21.15.5:0/1131339219 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2336883173 0 0) 0x7f19500ca7e0 con 0x7f195006bf50 2024-03-24T10:58:12.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.921+0000 7f194dffb640 1 -- 172.21.15.5:0/1131339219 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1938003520 con 0x7f195006bf50 2024-03-24T10:58:12.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.923+0000 7f194effd640 1 -- 172.21.15.5:0/1131339219 >> v1:172.21.15.57:6789/0 conn(0x7f195006bf50 legacy=0x7f195006c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:12.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.923+0000 7f194effd640 1 -- 172.21.15.5:0/1131339219 >> v1:172.21.15.5:6789/0 conn(0x7f195006d4f0 legacy=0x7f19500cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:12.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.923+0000 7f194effd640 1 -- 172.21.15.5:0/1131339219 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19500cb520 con 0x7f19501543a0 2024-03-24T10:58:12.925 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.926+0000 7f194effd640 1 --2- 172.21.15.5:0/1131339219 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f19501543a0 0x7f195015ac00 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f19400097a0 tx=0x7f194002b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa2f8424a268f4c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:12.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.931+0000 7f194dffb640 1 -- 172.21.15.5:0/1131339219 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194002f810 con 0x7f19501543a0 2024-03-24T10:58:12.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.931+0000 7f194dffb640 1 -- 172.21.15.5:0/1131339219 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f194002f970 con 0x7f19501543a0 2024-03-24T10:58:12.931 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.931+0000 7f194dffb640 1 -- 172.21.15.5:0/1131339219 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194002fd00 con 0x7f19501543a0 2024-03-24T10:58:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.931+0000 7f19556d4640 1 -- 172.21.15.5:0/1131339219 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f19501543a0 msgr2=0x7f195015ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.931+0000 7f19556d4640 1 --2- 172.21.15.5:0/1131339219 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f19501543a0 0x7f195015ac00 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f19400097a0 tx=0x7f194002b9f0 comp rx=0 tx=0).stop 2024-03-24T10:58:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.932+0000 7f19556d4640 1 -- 172.21.15.5:0/1131339219 shutdown_connections 2024-03-24T10:58:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.932+0000 7f19556d4640 1 --2- 172.21.15.5:0/1131339219 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f19501543a0 0x7f195015ac00 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.932+0000 7f19556d4640 1 -- 172.21.15.5:0/1131339219 >> 172.21.15.5:0/1131339219 conn(0x7f1950072ef0 msgr2=0x7f1950061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:12.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.932+0000 7f19556d4640 1 -- 172.21.15.5:0/1131339219 shutdown_connections 2024-03-24T10:58:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.932+0000 7f19556d4640 1 -- 172.21.15.5:0/1131339219 wait complete. 2024-03-24T10:58:12.933 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.934+0000 7f19556d4640 1 Processor -- start 2024-03-24T10:58:12.934 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.934+0000 7f19556d4640 1 -- start start 2024-03-24T10:58:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.935+0000 7f19556d4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f195006bf50 0x7f1950158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.935+0000 7f19556d4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f195006d4f0 0x7f19500b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.935+0000 7f19556d4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f19501543a0 0x7f19501f0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.935+0000 7f19556d4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f19501f0550 con 0x7f195006d4f0 2024-03-24T10:58:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.935+0000 7f19556d4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f19500cf5a0 con 0x7f195006bf50 2024-03-24T10:58:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.935+0000 7f19556d4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f19501efeb0 con 0x7f19501543a0 2024-03-24T10:58:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.935+0000 7f194e7fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f195006d4f0 0x7f19500b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.936+0000 7f194effd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f195006bf50 0x7f1950158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.936+0000 7f194f7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f19501543a0 0x7f19501f0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.936+0000 7f194e7fc640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f195006d4f0 0x7f19500b42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:56820/0 (socket says 172.21.15.5:56820) 2024-03-24T10:58:12.936 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.936+0000 7f194e7fc640 1 -- 172.21.15.5:0/2772750460 learned_addr learned my addr 172.21.15.5:0/2772750460 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:12.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.939+0000 7f194effd640 1 -- 172.21.15.5:0/2772750460 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f19501543a0 msgr2=0x7f19501f0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:12.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.939+0000 7f194effd640 1 --2- 172.21.15.5:0/2772750460 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f19501543a0 0x7f19501f0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:12.939 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.939+0000 7f194effd640 1 -- 172.21.15.5:0/2772750460 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f195006d4f0 msgr2=0x7f19500b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:12.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.939+0000 7f194effd640 1 --2- 172.21.15.5:0/2772750460 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f195006d4f0 0x7f19500b42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:12.940 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.939+0000 7f194effd640 1 -- 172.21.15.5:0/2772750460 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1940009400 con 0x7f195006bf50 2024-03-24T10:58:12.941 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.942+0000 7f194effd640 1 --2- 172.21.15.5:0/2772750460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f195006bf50 0x7f1950158cb0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1940033000 tx=0x7f1940004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:12.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.944+0000 7f19377fe640 1 -- 172.21.15.5:0/2772750460 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194003c040 con 0x7f195006bf50 2024-03-24T10:58:12.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.944+0000 7f19556d4640 1 -- 172.21.15.5:0/2772750460 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19500b4930 con 0x7f195006bf50 2024-03-24T10:58:12.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.944+0000 7f19556d4640 1 -- 172.21.15.5:0/2772750460 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19500b4ef0 con 0x7f195006bf50 2024-03-24T10:58:12.945 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.946+0000 7f19556d4640 1 -- 172.21.15.5:0/2772750460 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19501f0690 con 0x7f195006bf50 2024-03-24T10:58:12.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.947+0000 7f19377fe640 1 -- 172.21.15.5:0/2772750460 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19400378d0 con 0x7f195006bf50 2024-03-24T10:58:12.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:12.947+0000 7f19377fe640 1 -- 172.21.15.5:0/2772750460 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1940037c40 con 0x7f195006bf50 2024-03-24T10:58:13.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:13.250+0000 7f19377fe640 1 -- 172.21.15.5:0/2772750460 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f19400423f0 con 0x7f195006bf50 2024-03-24T10:58:13.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:13.250+0000 7f19377fe640 1 --2- 172.21.15.5:0/2772750460 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f193c06b810 0x7f193c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:13.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:13.250+0000 7f194e7fc640 1 --2- 172.21.15.5:0/2772750460 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f193c06b810 0x7f193c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:13.250 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:13.251+0000 7f194e7fc640 1 --2- 172.21.15.5:0/2772750460 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f193c06b810 0x7f193c06dcd0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f19500b2fd0 tx=0x7f1944009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:13.256 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:13.256+0000 7f19377fe640 1 -- 172.21.15.5:0/2772750460 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f194007ac20 con 0x7f195006bf50 2024-03-24T10:58:13.271 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:13.271+0000 7f19377fe640 1 -- 172.21.15.5:0/2772750460 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1940076ec0 con 0x7f195006bf50 2024-03-24T10:58:13.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:13.531+0000 7f19556d4640 1 -- 172.21.15.5:0/2772750460 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19500c61f0 con 0x7f195006bf50 2024-03-24T10:58:13.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:13.554+0000 7f19377fe640 1 -- 172.21.15.5:0/2772750460 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f1940047090 con 0x7f195006bf50 2024-03-24T10:58:13.554 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:13.554 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:13.555 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:13.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:13.556+0000 7f19556d4640 1 -- 172.21.15.5:0/2772750460 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f193c06b810 msgr2=0x7f193c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:13.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:13.556+0000 7f19556d4640 1 --2- 172.21.15.5:0/2772750460 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f193c06b810 0x7f193c06dcd0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f19500b2fd0 tx=0x7f1944009040 comp rx=0 tx=0).stop 2024-03-24T10:58:13.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:13.556+0000 7f19556d4640 1 -- 172.21.15.5:0/2772750460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f195006bf50 msgr2=0x7f1950158cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:13.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:13.556+0000 7f19556d4640 1 --2- 172.21.15.5:0/2772750460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f195006bf50 0x7f1950158cb0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1940033000 tx=0x7f1940004730 comp rx=0 tx=0).stop 2024-03-24T10:58:13.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:13.556+0000 7f19556d4640 1 -- 172.21.15.5:0/2772750460 shutdown_connections 2024-03-24T10:58:13.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:13.556+0000 7f19556d4640 1 --2- 172.21.15.5:0/2772750460 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f193c06b810 0x7f193c06dcd0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:13.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:13.556+0000 7f19556d4640 1 --2- 172.21.15.5:0/2772750460 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f195006d4f0 0x7f19500b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:13.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:13.556+0000 7f19556d4640 1 --2- 172.21.15.5:0/2772750460 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f19501543a0 0x7f19501f0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:13.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:13.556+0000 7f19556d4640 1 --2- 172.21.15.5:0/2772750460 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f195006bf50 0x7f1950158cb0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:13.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:13.556+0000 7f19556d4640 1 -- 172.21.15.5:0/2772750460 >> 172.21.15.5:0/2772750460 conn(0x7f1950072ef0 msgr2=0x7f19500c5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:13.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:13.556+0000 7f19556d4640 1 -- 172.21.15.5:0/2772750460 shutdown_connections 2024-03-24T10:58:13.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:13.556+0000 7f19556d4640 1 -- 172.21.15.5:0/2772750460 wait complete. 2024-03-24T10:58:14.172 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:14.172 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:15.173 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:15.276 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.276+0000 7fc96aa8d640 1 Processor -- start 2024-03-24T10:58:15.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.276+0000 7fc96aa8d640 1 -- start start 2024-03-24T10:58:15.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.277+0000 7fc96aa8d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc96406fc60 0x7fc9640c4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:15.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.277+0000 7fc96aa8d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc964157f80 0x7fc9640c5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:15.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.277+0000 7fc96aa8d640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc96415b530 con 0x7fc964152140 2024-03-24T10:58:15.277 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.277+0000 7fc96aa8d640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fc96415b810 con 0x7fc964157f80 2024-03-24T10:58:15.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.277+0000 7fc96aa8d640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc964159430 con 0x7fc96406fc60 2024-03-24T10:58:15.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.278+0000 7fc963fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc96406fc60 0x7fc9640c4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:15.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.278+0000 7fc96902a640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc964157f80 0x7fc9640c5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:15.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.278+0000 7fc968829640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fc964152140 0x7fc96415bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:40950/0 (socket says 172.21.15.5:40950) 2024-03-24T10:58:15.278 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.278+0000 7fc968829640 1 -- 172.21.15.5:0/2330266635 learned_addr learned my addr 172.21.15.5:0/2330266635 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:58:15.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.281+0000 7fc9637fe640 1 -- 172.21.15.5:0/2330266635 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4144787885 0 0) 0x7fc96415b530 con 0x7fc964152140 2024-03-24T10:58:15.281 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.281+0000 7fc9637fe640 1 -- 172.21.15.5:0/2330266635 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc954003520 con 0x7fc964152140 2024-03-24T10:58:15.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.282+0000 7fc963fff640 1 -- 172.21.15.5:0/2330266635 >> v2:172.21.15.57:3300/0 conn(0x7fc964157f80 msgr2=0x7fc9640c5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:15.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.282+0000 7fc963fff640 1 --2- 172.21.15.5:0/2330266635 >> v2:172.21.15.57:3300/0 conn(0x7fc964157f80 0x7fc9640c5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:15.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.282+0000 7fc963fff640 1 -- 172.21.15.5:0/2330266635 >> v1:172.21.15.5:6789/0 conn(0x7fc964152140 legacy=0x7fc96415bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:15.282 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.282+0000 7fc963fff640 1 -- 172.21.15.5:0/2330266635 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9641584c0 con 0x7fc96406fc60 2024-03-24T10:58:15.284 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.284+0000 7fc963fff640 1 --2- 172.21.15.5:0/2330266635 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc96406fc60 0x7fc9640c4e40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc9500097a0 tx=0x7fc95002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24f8eeb5121c488b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:15.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.288+0000 7fc9637fe640 1 -- 172.21.15.5:0/2330266635 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc950030610 con 0x7fc96406fc60 2024-03-24T10:58:15.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.288+0000 7fc9637fe640 1 -- 172.21.15.5:0/2330266635 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc950030770 con 0x7fc96406fc60 2024-03-24T10:58:15.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.288+0000 7fc9637fe640 1 -- 172.21.15.5:0/2330266635 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc950030b00 con 0x7fc96406fc60 2024-03-24T10:58:15.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.289+0000 7fc96aa8d640 1 -- 172.21.15.5:0/2330266635 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc96406fc60 msgr2=0x7fc9640c4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:15.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.289+0000 7fc96aa8d640 1 --2- 172.21.15.5:0/2330266635 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc96406fc60 0x7fc9640c4e40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc9500097a0 tx=0x7fc95002b960 comp rx=0 tx=0).stop 2024-03-24T10:58:15.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.289+0000 7fc96aa8d640 1 -- 172.21.15.5:0/2330266635 shutdown_connections 2024-03-24T10:58:15.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.289+0000 7fc96aa8d640 1 --2- 172.21.15.5:0/2330266635 >> v2:172.21.15.57:3300/0 conn(0x7fc964157f80 0x7fc9640c5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:15.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.289+0000 7fc96aa8d640 1 --2- 172.21.15.5:0/2330266635 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc96406fc60 0x7fc9640c4e40 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:15.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.289+0000 7fc96aa8d640 1 -- 172.21.15.5:0/2330266635 >> 172.21.15.5:0/2330266635 conn(0x7fc96406b130 msgr2=0x7fc96406b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:15.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.289+0000 7fc96aa8d640 1 -- 172.21.15.5:0/2330266635 shutdown_connections 2024-03-24T10:58:15.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.289+0000 7fc96aa8d640 1 -- 172.21.15.5:0/2330266635 wait complete. 2024-03-24T10:58:15.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.290+0000 7fc96aa8d640 1 Processor -- start 2024-03-24T10:58:15.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.290+0000 7fc96aa8d640 1 -- start start 2024-03-24T10:58:15.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.291+0000 7fc96aa8d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc96406fc60 0x7fc96414bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:15.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.291+0000 7fc96aa8d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc964152140 0x7fc96414c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:15.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.291+0000 7fc96aa8d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc964157f80 0x7fc964072360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:15.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.291+0000 7fc96aa8d640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fc9641ed2b0 con 0x7fc964157f80 2024-03-24T10:58:15.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.291+0000 7fc96aa8d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc9641ed4c0 con 0x7fc96406fc60 2024-03-24T10:58:15.291 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.291+0000 7fc96aa8d640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc9641ed760 con 0x7fc964152140 2024-03-24T10:58:15.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.292+0000 7fc963fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc964152140 0x7fc96414c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:15.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.292+0000 7fc96902a640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc964157f80 0x7fc964072360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:15.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.292+0000 7fc968829640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc96406fc60 0x7fc96414bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:15.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.292+0000 7fc963fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc964152140 0x7fc96414c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:35160/0 (socket says 172.21.15.5:35160) 2024-03-24T10:58:15.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.292+0000 7fc968829640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc96406fc60 0x7fc96414bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:39034/0 (socket says 172.21.15.5:39034) 2024-03-24T10:58:15.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.292+0000 7fc963fff640 1 -- 172.21.15.5:0/2111959278 learned_addr learned my addr 172.21.15.5:0/2111959278 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:15.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.296+0000 7fc96902a640 1 -- 172.21.15.5:0/2111959278 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc964152140 msgr2=0x7fc96414c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:15.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.296+0000 7fc96902a640 1 --2- 172.21.15.5:0/2111959278 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc964152140 0x7fc96414c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:15.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.296+0000 7fc96902a640 1 -- 172.21.15.5:0/2111959278 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc96406fc60 msgr2=0x7fc96414bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:15.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.296+0000 7fc96902a640 1 --2- 172.21.15.5:0/2111959278 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc96406fc60 0x7fc96414bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:15.296 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.296+0000 7fc96902a640 1 -- 172.21.15.5:0/2111959278 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc950009400 con 0x7fc964157f80 2024-03-24T10:58:15.298 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.298+0000 7fc96902a640 1 --2- 172.21.15.5:0/2111959278 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc964157f80 0x7fc964072360 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc95c00ef90 tx=0x7fc95c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:15.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.302+0000 7fc9617fa640 1 -- 172.21.15.5:0/2111959278 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95c009c00 con 0x7fc964157f80 2024-03-24T10:58:15.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.302+0000 7fc9617fa640 1 -- 172.21.15.5:0/2111959278 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc95c00f040 con 0x7fc964157f80 2024-03-24T10:58:15.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.302+0000 7fc9617fa640 1 -- 172.21.15.5:0/2111959278 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc95c0146c0 con 0x7fc964157f80 2024-03-24T10:58:15.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.302+0000 7fc96aa8d640 1 -- 172.21.15.5:0/2111959278 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9641eda40 con 0x7fc964157f80 2024-03-24T10:58:15.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.302+0000 7fc96aa8d640 1 -- 172.21.15.5:0/2111959278 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9641ee000 con 0x7fc964157f80 2024-03-24T10:58:15.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.303+0000 7fc96aa8d640 1 -- 172.21.15.5:0/2111959278 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc964062a00 con 0x7fc964157f80 2024-03-24T10:58:15.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.323+0000 7fc9617fa640 1 -- 172.21.15.5:0/2111959278 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc95c014820 con 0x7fc964157f80 2024-03-24T10:58:15.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.323+0000 7fc9617fa640 1 --2- 172.21.15.5:0/2111959278 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc94806b840 0x7fc94806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:15.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.323+0000 7fc968829640 1 --2- 172.21.15.5:0/2111959278 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc94806b840 0x7fc94806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:15.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.324+0000 7fc968829640 1 --2- 172.21.15.5:0/2111959278 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc94806b840 0x7fc94806dd00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc958005880 tx=0x7fc958009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:15.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.599+0000 7fc9617fa640 1 -- 172.21.15.5:0/2111959278 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fc95c059cf0 con 0x7fc964157f80 2024-03-24T10:58:15.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.600+0000 7fc9617fa640 1 -- 172.21.15.5:0/2111959278 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc95c055f90 con 0x7fc964157f80 2024-03-24T10:58:15.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.822+0000 7fc96aa8d640 1 -- 172.21.15.5:0/2111959278 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9640681d0 con 0x7fc964157f80 2024-03-24T10:58:15.882 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.882+0000 7fc9617fa640 1 -- 172.21.15.5:0/2111959278 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7fc95c018090 con 0x7fc964157f80 2024-03-24T10:58:15.882 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:15.882 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:15.883 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:15.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.884+0000 7fc96aa8d640 1 -- 172.21.15.5:0/2111959278 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc94806b840 msgr2=0x7fc94806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:15.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.884+0000 7fc96aa8d640 1 --2- 172.21.15.5:0/2111959278 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc94806b840 0x7fc94806dd00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fc958005880 tx=0x7fc958009040 comp rx=0 tx=0).stop 2024-03-24T10:58:15.884 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.884+0000 7fc96aa8d640 1 -- 172.21.15.5:0/2111959278 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc964157f80 msgr2=0x7fc964072360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:15.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.884+0000 7fc96aa8d640 1 --2- 172.21.15.5:0/2111959278 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc964157f80 0x7fc964072360 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc95c00ef90 tx=0x7fc95c00c490 comp rx=0 tx=0).stop 2024-03-24T10:58:15.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.884+0000 7fc96aa8d640 1 -- 172.21.15.5:0/2111959278 shutdown_connections 2024-03-24T10:58:15.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.884+0000 7fc96aa8d640 1 --2- 172.21.15.5:0/2111959278 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc94806b840 0x7fc94806dd00 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:15.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.884+0000 7fc96aa8d640 1 --2- 172.21.15.5:0/2111959278 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc964157f80 0x7fc964072360 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:15.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.884+0000 7fc96aa8d640 1 --2- 172.21.15.5:0/2111959278 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc964152140 0x7fc96414c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:15.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.884+0000 7fc96aa8d640 1 --2- 172.21.15.5:0/2111959278 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc96406fc60 0x7fc96414bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:15.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.884+0000 7fc96aa8d640 1 -- 172.21.15.5:0/2111959278 >> 172.21.15.5:0/2111959278 conn(0x7fc96406b130 msgr2=0x7fc9640c6730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:15.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.884+0000 7fc96aa8d640 1 -- 172.21.15.5:0/2111959278 shutdown_connections 2024-03-24T10:58:15.885 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:15.884+0000 7fc96aa8d640 1 -- 172.21.15.5:0/2111959278 wait complete. 2024-03-24T10:58:16.499 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:16.499 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:17.500 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:17.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.604+0000 7f528fb20640 1 Processor -- start 2024-03-24T10:58:17.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.604+0000 7f528fb20640 1 -- start start 2024-03-24T10:58:17.604 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.605+0000 7f528fb20640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f528815aac0 0x7f528815aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:17.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.605+0000 7f528fb20640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f528806cd50 con 0x7f528806c670 2024-03-24T10:58:17.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.605+0000 7f528fb20640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f528806cae0 con 0x7f528806c230 2024-03-24T10:58:17.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.605+0000 7f528fb20640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f52880ce790 con 0x7f528815aac0 2024-03-24T10:58:17.605 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.605+0000 7f528e0bd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f528806c670 0x7f52880ca5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:40956/0 (socket says 172.21.15.5:40956) 2024-03-24T10:58:17.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.605+0000 7f528e0bd640 1 -- 172.21.15.5:0/3467601307 learned_addr learned my addr 172.21.15.5:0/3467601307 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:58:17.606 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.605+0000 7f528d8bc640 1 --2- 172.21.15.5:0/3467601307 >> v2:172.21.15.57:3300/0 conn(0x7f528815aac0 0x7f528815aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:17.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.608+0000 7f528c8ba640 1 -- 172.21.15.5:0/3467601307 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4046313132 0 0) 0x7f528806cd50 con 0x7f528806c670 2024-03-24T10:58:17.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.608+0000 7f528c8ba640 1 -- 172.21.15.5:0/3467601307 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5274003520 con 0x7f528806c670 2024-03-24T10:58:17.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.610+0000 7f528d8bc640 1 -- 172.21.15.5:0/3467601307 >> v1:172.21.15.57:6789/0 conn(0x7f528806c230 legacy=0x7f52880cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:17.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.610+0000 7f528d8bc640 1 -- 172.21.15.5:0/3467601307 >> v1:172.21.15.5:6789/0 conn(0x7f528806c670 legacy=0x7f52880ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:17.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.610+0000 7f528d8bc640 1 -- 172.21.15.5:0/3467601307 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52880cdb10 con 0x7f528815aac0 2024-03-24T10:58:17.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.614+0000 7f528d8bc640 1 --2- 172.21.15.5:0/3467601307 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f528815aac0 0x7f528815aea0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f52840097a0 tx=0x7f528402b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5da1eca96cea8c23 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:17.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.619+0000 7f528c8ba640 1 -- 172.21.15.5:0/3467601307 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528402f810 con 0x7f528815aac0 2024-03-24T10:58:17.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.619+0000 7f528c8ba640 1 -- 172.21.15.5:0/3467601307 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f528402f970 con 0x7f528815aac0 2024-03-24T10:58:17.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.619+0000 7f528c8ba640 1 -- 172.21.15.5:0/3467601307 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528402fd00 con 0x7f528815aac0 2024-03-24T10:58:17.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.620+0000 7f528fb20640 1 -- 172.21.15.5:0/3467601307 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f528815aac0 msgr2=0x7f528815aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:17.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.620+0000 7f528fb20640 1 --2- 172.21.15.5:0/3467601307 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f528815aac0 0x7f528815aea0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f52840097a0 tx=0x7f528402b9f0 comp rx=0 tx=0).stop 2024-03-24T10:58:17.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.620+0000 7f528fb20640 1 -- 172.21.15.5:0/3467601307 shutdown_connections 2024-03-24T10:58:17.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.620+0000 7f528fb20640 1 --2- 172.21.15.5:0/3467601307 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f528815aac0 0x7f528815aea0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:17.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.620+0000 7f528fb20640 1 -- 172.21.15.5:0/3467601307 >> 172.21.15.5:0/3467601307 conn(0x7f52880cfb90 msgr2=0x7f52880cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:17.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.620+0000 7f528fb20640 1 -- 172.21.15.5:0/3467601307 shutdown_connections 2024-03-24T10:58:17.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.620+0000 7f528fb20640 1 -- 172.21.15.5:0/3467601307 wait complete. 2024-03-24T10:58:17.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.621+0000 7f528fb20640 1 Processor -- start 2024-03-24T10:58:17.621 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.621+0000 7f528fb20640 1 -- start start 2024-03-24T10:58:17.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.622+0000 7f528fb20640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f528806c230 0x7f52880c7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:17.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.622+0000 7f528fb20640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f528806c670 0x7f52881f32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:17.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.622+0000 7f528fb20640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f528815aac0 0x7f528814f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:17.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.622+0000 7f528fb20640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f5288157e20 con 0x7f528815aac0 2024-03-24T10:58:17.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.622+0000 7f528fb20640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f5288157990 con 0x7f528806c230 2024-03-24T10:58:17.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.622+0000 7f528fb20640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f52881570a0 con 0x7f528806c670 2024-03-24T10:58:17.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.623+0000 7f528e0bd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f528815aac0 0x7f528814f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:17.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.623+0000 7f528d8bc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f528806c230 0x7f52880c7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:17.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.623+0000 7f528e0bd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f528815aac0 0x7f528814f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:56846/0 (socket says 172.21.15.5:56846) 2024-03-24T10:58:17.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.623+0000 7f528e0bd640 1 -- 172.21.15.5:0/2761709922 learned_addr learned my addr 172.21.15.5:0/2761709922 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:17.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.623+0000 7f528d0bb640 1 --2- 172.21.15.5:0/2761709922 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f528806c670 0x7f52881f32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:17.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.626+0000 7f528e0bd640 1 -- 172.21.15.5:0/2761709922 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f528806c670 msgr2=0x7f52881f32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:17.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.626+0000 7f528e0bd640 1 --2- 172.21.15.5:0/2761709922 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f528806c670 0x7f52881f32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:17.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.626+0000 7f528e0bd640 1 -- 172.21.15.5:0/2761709922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f528806c230 msgr2=0x7f52880c7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:17.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.626+0000 7f528e0bd640 1 --2- 172.21.15.5:0/2761709922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f528806c230 0x7f52880c7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:17.627 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.626+0000 7f528e0bd640 1 -- 172.21.15.5:0/2761709922 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5284009400 con 0x7f528815aac0 2024-03-24T10:58:17.628 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.628+0000 7f528e0bd640 1 --2- 172.21.15.5:0/2761709922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f528815aac0 0x7f528814f480 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f528000cb40 tx=0x7f528000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:17.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.632+0000 7f527e7fc640 1 -- 172.21.15.5:0/2761709922 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f528000aca0 con 0x7f528815aac0 2024-03-24T10:58:17.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.632+0000 7f527e7fc640 1 -- 172.21.15.5:0/2761709922 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f528000ae00 con 0x7f528815aac0 2024-03-24T10:58:17.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.632+0000 7f528fb20640 1 -- 172.21.15.5:0/2761709922 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5288153690 con 0x7f528815aac0 2024-03-24T10:58:17.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.632+0000 7f527e7fc640 1 -- 172.21.15.5:0/2761709922 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5280012620 con 0x7f528815aac0 2024-03-24T10:58:17.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.632+0000 7f528fb20640 1 -- 172.21.15.5:0/2761709922 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5288157620 con 0x7f528815aac0 2024-03-24T10:58:17.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.633+0000 7f528fb20640 1 -- 172.21.15.5:0/2761709922 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5288157b00 con 0x7f528815aac0 2024-03-24T10:58:17.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.657+0000 7f527e7fc640 1 -- 172.21.15.5:0/2761709922 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5280007070 con 0x7f528815aac0 2024-03-24T10:58:17.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.658+0000 7f527e7fc640 1 --2- 172.21.15.5:0/2761709922 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f526c06b860 0x7f526c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:17.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.658+0000 7f528d8bc640 1 --2- 172.21.15.5:0/2761709922 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f526c06b860 0x7f526c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:17.659 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.658+0000 7f528d8bc640 1 --2- 172.21.15.5:0/2761709922 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f526c06b860 0x7f526c06dd20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5284036000 tx=0x7f5284008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:17.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.928+0000 7f527e7fc640 1 -- 172.21.15.5:0/2761709922 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f5280058de0 con 0x7f528815aac0 2024-03-24T10:58:17.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:17.929+0000 7f527e7fc640 1 -- 172.21.15.5:0/2761709922 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5280055080 con 0x7f528815aac0 2024-03-24T10:58:18.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:18.183+0000 7f528fb20640 1 -- 172.21.15.5:0/2761709922 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52880681d0 con 0x7f528815aac0 2024-03-24T10:58:18.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:18.225+0000 7f527e7fc640 1 -- 172.21.15.5:0/2761709922 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f5280017030 con 0x7f528815aac0 2024-03-24T10:58:18.227 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:18.227 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:18.227 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:18.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:18.228+0000 7f528fb20640 1 -- 172.21.15.5:0/2761709922 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f526c06b860 msgr2=0x7f526c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:18.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:18.228+0000 7f528fb20640 1 --2- 172.21.15.5:0/2761709922 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f526c06b860 0x7f526c06dd20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f5284036000 tx=0x7f5284008030 comp rx=0 tx=0).stop 2024-03-24T10:58:18.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:18.228+0000 7f528fb20640 1 -- 172.21.15.5:0/2761709922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f528815aac0 msgr2=0x7f528814f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:18.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:18.228+0000 7f528fb20640 1 --2- 172.21.15.5:0/2761709922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f528815aac0 0x7f528814f480 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f528000cb40 tx=0x7f528000a060 comp rx=0 tx=0).stop 2024-03-24T10:58:18.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:18.228+0000 7f528fb20640 1 -- 172.21.15.5:0/2761709922 shutdown_connections 2024-03-24T10:58:18.229 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:18.228+0000 7f528fb20640 1 --2- 172.21.15.5:0/2761709922 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f526c06b860 0x7f526c06dd20 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:18.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:18.228+0000 7f528fb20640 1 --2- 172.21.15.5:0/2761709922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f528815aac0 0x7f528814f480 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:18.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:18.228+0000 7f528fb20640 1 --2- 172.21.15.5:0/2761709922 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f528806c670 0x7f52881f32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:18.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:18.228+0000 7f528fb20640 1 --2- 172.21.15.5:0/2761709922 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f528806c230 0x7f52880c7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:18.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:18.228+0000 7f528fb20640 1 -- 172.21.15.5:0/2761709922 >> 172.21.15.5:0/2761709922 conn(0x7f52880cfb90 msgr2=0x7f52880c7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:18.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:18.228+0000 7f528fb20640 1 -- 172.21.15.5:0/2761709922 shutdown_connections 2024-03-24T10:58:18.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:18.228+0000 7f528fb20640 1 -- 172.21.15.5:0/2761709922 wait complete. 2024-03-24T10:58:18.844 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:18.844 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:19.845 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:19.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.946+0000 7fe6b2c9d640 1 Processor -- start 2024-03-24T10:58:19.948 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.946+0000 7fe6b2c9d640 1 -- start start 2024-03-24T10:58:19.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.948+0000 7fe6b2c9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe6ac1543a0 0x7fe6ac15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:19.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.948+0000 7fe6b2c9d640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6ac0cb2f0 con 0x7fe6ac06d4f0 2024-03-24T10:58:19.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.948+0000 7fe6b2c9d640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6ac0ca7e0 con 0x7fe6ac06bf50 2024-03-24T10:58:19.949 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.948+0000 7fe6b2c9d640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fe6ac0c6750 con 0x7fe6ac1543a0 2024-03-24T10:58:19.950 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.949+0000 7fe6b0a39640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe6ac1543a0 0x7fe6ac15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:19.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.950+0000 7fe6b0a39640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fe6ac1543a0 0x7fe6ac15ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:56860/0 (socket says 172.21.15.5:56860) 2024-03-24T10:58:19.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.950+0000 7fe6b0a39640 1 -- 172.21.15.5:0/2297487594 learned_addr learned my addr 172.21.15.5:0/2297487594 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:19.953 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.953+0000 7fe6b0a39640 1 -- 172.21.15.5:0/2297487594 >> v1:172.21.15.57:6789/0 conn(0x7fe6ac06bf50 legacy=0x7fe6ac06c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:19.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.953+0000 7fe6b0a39640 1 -- 172.21.15.5:0/2297487594 >> v1:172.21.15.5:6789/0 conn(0x7fe6ac06d4f0 legacy=0x7fe6ac0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:19.954 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.953+0000 7fe6b0a39640 1 -- 172.21.15.5:0/2297487594 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6ac0cb520 con 0x7fe6ac1543a0 2024-03-24T10:58:19.957 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.956+0000 7fe6b0a39640 1 --2- 172.21.15.5:0/2297487594 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6ac1543a0 0x7fe6ac15ac00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0009870 tx=0x7fe6a002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbad0d4d4dd85532 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:19.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.964+0000 7fe6ab7fe640 1 -- 172.21.15.5:0/2297487594 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a002f840 con 0x7fe6ac1543a0 2024-03-24T10:58:19.965 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.964+0000 7fe6ab7fe640 1 -- 172.21.15.5:0/2297487594 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6a002f9a0 con 0x7fe6ac1543a0 2024-03-24T10:58:19.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.964+0000 7fe6ab7fe640 1 -- 172.21.15.5:0/2297487594 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6a002fd30 con 0x7fe6ac1543a0 2024-03-24T10:58:19.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.965+0000 7fe6b2c9d640 1 -- 172.21.15.5:0/2297487594 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6ac1543a0 msgr2=0x7fe6ac15ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:19.966 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.965+0000 7fe6b2c9d640 1 --2- 172.21.15.5:0/2297487594 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6ac1543a0 0x7fe6ac15ac00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0009870 tx=0x7fe6a002ba20 comp rx=0 tx=0).stop 2024-03-24T10:58:19.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.965+0000 7fe6b2c9d640 1 -- 172.21.15.5:0/2297487594 shutdown_connections 2024-03-24T10:58:19.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.965+0000 7fe6b2c9d640 1 --2- 172.21.15.5:0/2297487594 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6ac1543a0 0x7fe6ac15ac00 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:19.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.965+0000 7fe6b2c9d640 1 -- 172.21.15.5:0/2297487594 >> 172.21.15.5:0/2297487594 conn(0x7fe6ac072ef0 msgr2=0x7fe6ac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:19.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.965+0000 7fe6b2c9d640 1 -- 172.21.15.5:0/2297487594 shutdown_connections 2024-03-24T10:58:19.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.965+0000 7fe6b2c9d640 1 -- 172.21.15.5:0/2297487594 wait complete. 2024-03-24T10:58:19.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.966+0000 7fe6b2c9d640 1 Processor -- start 2024-03-24T10:58:19.967 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.966+0000 7fe6b2c9d640 1 -- start start 2024-03-24T10:58:19.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.967+0000 7fe6b2c9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe6ac06bf50 0x7fe6ac152380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:19.968 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.967+0000 7fe6b2c9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6ac06d4f0 0x7fe6ac1f3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:19.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.967+0000 7fe6b2c9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6ac1543a0 0x7fe6ac1f3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:19.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.967+0000 7fe6b2c9d640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac1f3180 con 0x7fe6ac06d4f0 2024-03-24T10:58:19.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.967+0000 7fe6b2c9d640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ac0b5080 con 0x7fe6ac1543a0 2024-03-24T10:58:19.969 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.967+0000 7fe6b2c9d640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe6ac0b4520 con 0x7fe6ac06bf50 2024-03-24T10:58:19.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.968+0000 7fe6b123a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6ac1543a0 0x7fe6ac1f3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:19.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.968+0000 7fe6abfff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6ac06d4f0 0x7fe6ac1f3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:19.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.968+0000 7fe6b0a39640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe6ac06bf50 0x7fe6ac152380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:19.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.968+0000 7fe6abfff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6ac06d4f0 0x7fe6ac1f3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:56876/0 (socket says 172.21.15.5:56876) 2024-03-24T10:58:19.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.968+0000 7fe6abfff640 1 -- 172.21.15.5:0/841390850 learned_addr learned my addr 172.21.15.5:0/841390850 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:19.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.972+0000 7fe6abfff640 1 -- 172.21.15.5:0/841390850 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe6ac06bf50 msgr2=0x7fe6ac152380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:19.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.972+0000 7fe6abfff640 1 --2- 172.21.15.5:0/841390850 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe6ac06bf50 0x7fe6ac152380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:19.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.972+0000 7fe6abfff640 1 -- 172.21.15.5:0/841390850 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6ac1543a0 msgr2=0x7fe6ac1f3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:19.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.972+0000 7fe6abfff640 1 --2- 172.21.15.5:0/841390850 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6ac1543a0 0x7fe6ac1f3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:19.973 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.972+0000 7fe6abfff640 1 -- 172.21.15.5:0/841390850 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6a00094d0 con 0x7fe6ac06d4f0 2024-03-24T10:58:19.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.975+0000 7fe6abfff640 1 --2- 172.21.15.5:0/841390850 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6ac06d4f0 0x7fe6ac1f3610 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fe69800c410 tx=0x7fe69800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:19.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.977+0000 7fe6a97fa640 1 -- 172.21.15.5:0/841390850 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6980035f0 con 0x7fe6ac06d4f0 2024-03-24T10:58:19.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.977+0000 7fe6b2c9d640 1 -- 172.21.15.5:0/841390850 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6ac0b4ac0 con 0x7fe6ac06d4f0 2024-03-24T10:58:19.978 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.977+0000 7fe6b2c9d640 1 -- 172.21.15.5:0/841390850 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6ac1528c0 con 0x7fe6ac06d4f0 2024-03-24T10:58:19.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.978+0000 7fe6a97fa640 1 -- 172.21.15.5:0/841390850 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe698001120 con 0x7fe6ac06d4f0 2024-03-24T10:58:19.979 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.978+0000 7fe6a97fa640 1 -- 172.21.15.5:0/841390850 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe698003950 con 0x7fe6ac06d4f0 2024-03-24T10:58:19.980 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:19.979+0000 7fe6b2c9d640 1 -- 172.21.15.5:0/841390850 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe68c005290 con 0x7fe6ac06d4f0 2024-03-24T10:58:20.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:20.263+0000 7fe6a97fa640 1 -- 172.21.15.5:0/841390850 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe698003ab0 con 0x7fe6ac06d4f0 2024-03-24T10:58:20.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:20.264+0000 7fe6a97fa640 1 --2- 172.21.15.5:0/841390850 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe69006b840 0x7fe69006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:20.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:20.264+0000 7fe6b0a39640 1 --2- 172.21.15.5:0/841390850 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe69006b840 0x7fe69006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:20.266 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:20.265+0000 7fe6b0a39640 1 --2- 172.21.15.5:0/841390850 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe69006b840 0x7fe69006dd00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0036000 tx=0x7fe6a0008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:20.279 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:20.278+0000 7fe6a97fa640 1 -- 172.21.15.5:0/841390850 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fe6980581b0 con 0x7fe6ac06d4f0 2024-03-24T10:58:20.280 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:20.279+0000 7fe6a97fa640 1 -- 172.21.15.5:0/841390850 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe698054450 con 0x7fe6ac06d4f0 2024-03-24T10:58:20.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:20.525+0000 7fe6b2c9d640 1 -- 172.21.15.5:0/841390850 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe68c0058f0 con 0x7fe6ac06d4f0 2024-03-24T10:58:20.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:20.562+0000 7fe6a97fa640 1 -- 172.21.15.5:0/841390850 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7fe68c0058f0 con 0x7fe6ac06d4f0 2024-03-24T10:58:20.564 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:20.564 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:20.564 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:20.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:20.565+0000 7fe6b2c9d640 1 -- 172.21.15.5:0/841390850 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe69006b840 msgr2=0x7fe69006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:20.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:20.565+0000 7fe6b2c9d640 1 --2- 172.21.15.5:0/841390850 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe69006b840 0x7fe69006dd00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0036000 tx=0x7fe6a0008030 comp rx=0 tx=0).stop 2024-03-24T10:58:20.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:20.565+0000 7fe6b2c9d640 1 -- 172.21.15.5:0/841390850 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6ac06d4f0 msgr2=0x7fe6ac1f3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:20.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:20.565+0000 7fe6b2c9d640 1 --2- 172.21.15.5:0/841390850 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6ac06d4f0 0x7fe6ac1f3610 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fe69800c410 tx=0x7fe69800c8d0 comp rx=0 tx=0).stop 2024-03-24T10:58:20.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:20.565+0000 7fe6b2c9d640 1 -- 172.21.15.5:0/841390850 shutdown_connections 2024-03-24T10:58:20.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:20.565+0000 7fe6b2c9d640 1 --2- 172.21.15.5:0/841390850 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe69006b840 0x7fe69006dd00 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:20.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:20.565+0000 7fe6b2c9d640 1 --2- 172.21.15.5:0/841390850 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe6ac06d4f0 0x7fe6ac1f3610 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:20.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:20.565+0000 7fe6b2c9d640 1 --2- 172.21.15.5:0/841390850 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe6ac1543a0 0x7fe6ac1f3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:20.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:20.565+0000 7fe6b2c9d640 1 --2- 172.21.15.5:0/841390850 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe6ac06bf50 0x7fe6ac152380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:20.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:20.565+0000 7fe6b2c9d640 1 -- 172.21.15.5:0/841390850 >> 172.21.15.5:0/841390850 conn(0x7fe6ac072ef0 msgr2=0x7fe6ac061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:20.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:20.565+0000 7fe6b2c9d640 1 -- 172.21.15.5:0/841390850 shutdown_connections 2024-03-24T10:58:20.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:20.565+0000 7fe6b2c9d640 1 -- 172.21.15.5:0/841390850 wait complete. 2024-03-24T10:58:21.181 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:21.181 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:22.182 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:22.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.285+0000 7f0705b93640 1 Processor -- start 2024-03-24T10:58:22.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.285+0000 7f0705b93640 1 -- start start 2024-03-24T10:58:22.287 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.286+0000 7f0705b93640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f07001543a0 0x7f070015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:22.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.286+0000 7f0705b93640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f070006d4f0 0x7f07000cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:22.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.286+0000 7f0705b93640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f07000ccaa0 0x7f070006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:22.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.286+0000 7f0705b93640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f07000c6750 con 0x7f070006d4f0 2024-03-24T10:58:22.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.286+0000 7f0705b93640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f07000cc380 con 0x7f07000ccaa0 2024-03-24T10:58:22.288 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.286+0000 7f0705b93640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f07000cb2f0 con 0x7f07001543a0 2024-03-24T10:58:22.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.287+0000 7f06fffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f07000ccaa0 0x7f070006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:22.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.287+0000 7f06ff7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f07001543a0 0x7f070015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:22.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.287+0000 7f06feffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f070006d4f0 0x7f07000cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:22.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.288+0000 7f06fffff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f07000ccaa0 0x7f070006bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:39062/0 (socket says 172.21.15.5:39062) 2024-03-24T10:58:22.289 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.288+0000 7f06fffff640 1 -- 172.21.15.5:0/4241995184 learned_addr learned my addr 172.21.15.5:0/4241995184 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:22.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.291+0000 7f06fffff640 1 -- 172.21.15.5:0/4241995184 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f07001543a0 msgr2=0x7f070015ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:22.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.291+0000 7f06fffff640 1 --2- 172.21.15.5:0/4241995184 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f07001543a0 0x7f070015ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:22.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.291+0000 7f06fffff640 1 -- 172.21.15.5:0/4241995184 >> v2:172.21.15.5:3300/0 conn(0x7f070006d4f0 msgr2=0x7f07000cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:22.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.291+0000 7f06fffff640 1 --2- 172.21.15.5:0/4241995184 >> v2:172.21.15.5:3300/0 conn(0x7f070006d4f0 0x7f07000cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:22.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.291+0000 7f06fffff640 1 -- 172.21.15.5:0/4241995184 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07000ca7e0 con 0x7f07000ccaa0 2024-03-24T10:58:22.294 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.293+0000 7f06fffff640 1 --2- 172.21.15.5:0/4241995184 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07000ccaa0 0x7f070006bf50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f070006de60 tx=0x7f06f802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=593ae4e16e26efcf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:22.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.298+0000 7f06fe7fc640 1 -- 172.21.15.5:0/4241995184 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f8030860 con 0x7f07000ccaa0 2024-03-24T10:58:22.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.298+0000 7f06fe7fc640 1 -- 172.21.15.5:0/4241995184 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06f80309c0 con 0x7f07000ccaa0 2024-03-24T10:58:22.299 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.298+0000 7f06fe7fc640 1 -- 172.21.15.5:0/4241995184 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f8030d50 con 0x7f07000ccaa0 2024-03-24T10:58:22.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.299+0000 7f0705b93640 1 -- 172.21.15.5:0/4241995184 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07000ccaa0 msgr2=0x7f070006bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:22.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.299+0000 7f0705b93640 1 --2- 172.21.15.5:0/4241995184 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07000ccaa0 0x7f070006bf50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f070006de60 tx=0x7f06f802b9a0 comp rx=0 tx=0).stop 2024-03-24T10:58:22.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.299+0000 7f0705b93640 1 -- 172.21.15.5:0/4241995184 shutdown_connections 2024-03-24T10:58:22.300 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.299+0000 7f0705b93640 1 --2- 172.21.15.5:0/4241995184 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07000ccaa0 0x7f070006bf50 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:22.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.299+0000 7f0705b93640 1 --2- 172.21.15.5:0/4241995184 >> v2:172.21.15.5:3300/0 conn(0x7f070006d4f0 0x7f07000cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:22.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.299+0000 7f0705b93640 1 --2- 172.21.15.5:0/4241995184 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f07001543a0 0x7f070015ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:22.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.299+0000 7f0705b93640 1 -- 172.21.15.5:0/4241995184 >> 172.21.15.5:0/4241995184 conn(0x7f0700072ef0 msgr2=0x7f0700061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:22.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.299+0000 7f0705b93640 1 -- 172.21.15.5:0/4241995184 shutdown_connections 2024-03-24T10:58:22.301 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.299+0000 7f0705b93640 1 -- 172.21.15.5:0/4241995184 wait complete. 2024-03-24T10:58:22.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.300+0000 7f0705b93640 1 Processor -- start 2024-03-24T10:58:22.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.300+0000 7f0705b93640 1 -- start start 2024-03-24T10:58:22.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.301+0000 7f0705b93640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f070006d4f0 0x7f07000cee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:22.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.301+0000 7f0705b93640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07000ccaa0 0x7f07001f06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:22.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.301+0000 7f0705b93640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07001543a0 0x7f0700143e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:22.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.301+0000 7f0705b93640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f07000b40e0 con 0x7f07000ccaa0 2024-03-24T10:58:22.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.301+0000 7f0705b93640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f07000c8fa0 con 0x7f07001543a0 2024-03-24T10:58:22.302 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.301+0000 7f0705b93640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f07001f0220 con 0x7f070006d4f0 2024-03-24T10:58:22.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.301+0000 7f06feffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07000ccaa0 0x7f07001f06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:22.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.302+0000 7f06ff7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f070006d4f0 0x7f07000cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:22.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.302+0000 7f06fffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07001543a0 0x7f0700143e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:22.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.302+0000 7f06fffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07001543a0 0x7f0700143e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:39076/0 (socket says 172.21.15.5:39076) 2024-03-24T10:58:22.303 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.302+0000 7f06fffff640 1 -- 172.21.15.5:0/1937458914 learned_addr learned my addr 172.21.15.5:0/1937458914 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:22.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.305+0000 7f06fffff640 1 -- 172.21.15.5:0/1937458914 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f070006d4f0 msgr2=0x7f07000cee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:22.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.305+0000 7f06fffff640 1 --2- 172.21.15.5:0/1937458914 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f070006d4f0 0x7f07000cee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:22.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.305+0000 7f06fffff640 1 -- 172.21.15.5:0/1937458914 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07000ccaa0 msgr2=0x7f07001f06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:22.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.305+0000 7f06fffff640 1 --2- 172.21.15.5:0/1937458914 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07000ccaa0 0x7f07001f06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:22.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.305+0000 7f06fffff640 1 -- 172.21.15.5:0/1937458914 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06f80094d0 con 0x7f07001543a0 2024-03-24T10:58:22.308 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.308+0000 7f06fffff640 1 --2- 172.21.15.5:0/1937458914 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07001543a0 0x7f0700143e20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f06f802b9e0 tx=0x7f06f8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:22.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.313+0000 7f06e7fff640 1 -- 172.21.15.5:0/1937458914 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f8039b90 con 0x7f07001543a0 2024-03-24T10:58:22.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.313+0000 7f06e7fff640 1 -- 172.21.15.5:0/1937458914 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06f8039cf0 con 0x7f07001543a0 2024-03-24T10:58:22.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.313+0000 7f06e7fff640 1 -- 172.21.15.5:0/1937458914 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06f8043640 con 0x7f07001543a0 2024-03-24T10:58:22.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.313+0000 7f0705b93640 1 -- 172.21.15.5:0/1937458914 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07001eff10 con 0x7f07001543a0 2024-03-24T10:58:22.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.313+0000 7f0705b93640 1 -- 172.21.15.5:0/1937458914 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07000c9490 con 0x7f07001543a0 2024-03-24T10:58:22.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.314+0000 7f0705b93640 1 -- 172.21.15.5:0/1937458914 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07001f03d0 con 0x7f07001543a0 2024-03-24T10:58:22.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.583+0000 7f06e7fff640 1 -- 172.21.15.5:0/1937458914 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f06f80437a0 con 0x7f07001543a0 2024-03-24T10:58:22.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.584+0000 7f06e7fff640 1 --2- 172.21.15.5:0/1937458914 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f06ec06b860 0x7f06ec06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:22.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.584+0000 7f06ff7fe640 1 --2- 172.21.15.5:0/1937458914 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f06ec06b860 0x7f06ec06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:22.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.584+0000 7f06ff7fe640 1 --2- 172.21.15.5:0/1937458914 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f06ec06b860 0x7f06ec06dd20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f06f0011340 tx=0x7f06f000b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:22.589 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.588+0000 7f06e7fff640 1 -- 172.21.15.5:0/1937458914 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f06f803e070 con 0x7f07001543a0 2024-03-24T10:58:22.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.600+0000 7f06e7fff640 1 -- 172.21.15.5:0/1937458914 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f06f80783c0 con 0x7f07001543a0 2024-03-24T10:58:22.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.838+0000 7f0705b93640 1 -- 172.21.15.5:0/1937458914 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07001f0390 con 0x7f07001543a0 2024-03-24T10:58:22.876 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.874+0000 7f06e7fff640 1 -- 172.21.15.5:0/1937458914 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f06f8034070 con 0x7f07001543a0 2024-03-24T10:58:22.876 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:22.876 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:22.876 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:22.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.877+0000 7f0705b93640 1 -- 172.21.15.5:0/1937458914 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f06ec06b860 msgr2=0x7f06ec06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:22.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.877+0000 7f0705b93640 1 --2- 172.21.15.5:0/1937458914 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f06ec06b860 0x7f06ec06dd20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f06f0011340 tx=0x7f06f000b040 comp rx=0 tx=0).stop 2024-03-24T10:58:22.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.877+0000 7f0705b93640 1 -- 172.21.15.5:0/1937458914 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07001543a0 msgr2=0x7f0700143e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:22.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.877+0000 7f0705b93640 1 --2- 172.21.15.5:0/1937458914 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07001543a0 0x7f0700143e20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f06f802b9e0 tx=0x7f06f8004730 comp rx=0 tx=0).stop 2024-03-24T10:58:22.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.877+0000 7f0705b93640 1 -- 172.21.15.5:0/1937458914 shutdown_connections 2024-03-24T10:58:22.878 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.877+0000 7f0705b93640 1 --2- 172.21.15.5:0/1937458914 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f06ec06b860 0x7f06ec06dd20 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:22.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.877+0000 7f0705b93640 1 --2- 172.21.15.5:0/1937458914 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f07000ccaa0 0x7f07001f06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:22.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.877+0000 7f0705b93640 1 --2- 172.21.15.5:0/1937458914 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f07001543a0 0x7f0700143e20 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:22.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.877+0000 7f0705b93640 1 --2- 172.21.15.5:0/1937458914 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f070006d4f0 0x7f07000cee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:22.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.877+0000 7f0705b93640 1 -- 172.21.15.5:0/1937458914 >> 172.21.15.5:0/1937458914 conn(0x7f0700072ef0 msgr2=0x7f0700061b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:22.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.877+0000 7f0705b93640 1 -- 172.21.15.5:0/1937458914 shutdown_connections 2024-03-24T10:58:22.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:22.877+0000 7f0705b93640 1 -- 172.21.15.5:0/1937458914 wait complete. 2024-03-24T10:58:23.493 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:23.493 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:24.495 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:24.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.597+0000 7f4e282c9640 1 Processor -- start 2024-03-24T10:58:24.597 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.597+0000 7f4e282c9640 1 -- start start 2024-03-24T10:58:24.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.598+0000 7f4e282c9640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4e2015aea0 0x7f4e200cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:24.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.598+0000 7f4e282c9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e200cf100 0x7f4e200ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:24.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.598+0000 7f4e282c9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e2006cd50 con 0x7f4e2015aac0 2024-03-24T10:58:24.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.598+0000 7f4e282c9640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f4e200ce790 con 0x7f4e2015aea0 2024-03-24T10:58:24.598 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.598+0000 7f4e282c9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4e200cdb10 con 0x7f4e200cf100 2024-03-24T10:58:24.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.598+0000 7f4e25864640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f4e2015aea0 0x7f4e200cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:24.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.598+0000 7f4e26866640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e200cf100 0x7f4e200ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:24.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.599+0000 7f4e26065640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f4e2015aac0 0x7f4e2006c230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:53330/0 (socket says 172.21.15.5:53330) 2024-03-24T10:58:24.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.599+0000 7f4e26065640 1 -- 172.21.15.5:0/1808081371 learned_addr learned my addr 172.21.15.5:0/1808081371 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:58:24.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.601+0000 7f4e26866640 1 -- 172.21.15.5:0/1808081371 >> v2:172.21.15.5:3300/0 conn(0x7f4e2015aea0 msgr2=0x7f4e200cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:24.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.601+0000 7f4e26866640 1 --2- 172.21.15.5:0/1808081371 >> v2:172.21.15.5:3300/0 conn(0x7f4e2015aea0 0x7f4e200cebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:24.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.601+0000 7f4e26866640 1 -- 172.21.15.5:0/1808081371 >> v1:172.21.15.57:6789/0 conn(0x7f4e2015aac0 legacy=0x7f4e2006c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:24.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.601+0000 7f4e26866640 1 -- 172.21.15.5:0/1808081371 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e200cddf0 con 0x7f4e200cf100 2024-03-24T10:58:24.603 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.604+0000 7f4e26866640 1 --2- 172.21.15.5:0/1808081371 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e200cf100 0x7f4e200ca5a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4e18009d90 tx=0x7f4e1802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e03a8d8cc769ca3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:24.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.608+0000 7f4e25063640 1 -- 172.21.15.5:0/1808081371 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1802cce0 con 0x7f4e200cf100 2024-03-24T10:58:24.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.608+0000 7f4e25063640 1 -- 172.21.15.5:0/1808081371 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e1802ce40 con 0x7f4e200cf100 2024-03-24T10:58:24.608 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.608+0000 7f4e25063640 1 -- 172.21.15.5:0/1808081371 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e18034660 con 0x7f4e200cf100 2024-03-24T10:58:24.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.609+0000 7f4e282c9640 1 -- 172.21.15.5:0/1808081371 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e200cf100 msgr2=0x7f4e200ca5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:24.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.609+0000 7f4e282c9640 1 --2- 172.21.15.5:0/1808081371 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e200cf100 0x7f4e200ca5a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4e18009d90 tx=0x7f4e1802bf30 comp rx=0 tx=0).stop 2024-03-24T10:58:24.609 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.609+0000 7f4e282c9640 1 -- 172.21.15.5:0/1808081371 shutdown_connections 2024-03-24T10:58:24.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.609+0000 7f4e282c9640 1 --2- 172.21.15.5:0/1808081371 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e200cf100 0x7f4e200ca5a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:24.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.609+0000 7f4e282c9640 1 --2- 172.21.15.5:0/1808081371 >> v2:172.21.15.5:3300/0 conn(0x7f4e2015aea0 0x7f4e200cebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:24.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.609+0000 7f4e282c9640 1 -- 172.21.15.5:0/1808081371 >> 172.21.15.5:0/1808081371 conn(0x7f4e200cfb90 msgr2=0x7f4e200cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:24.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.609+0000 7f4e282c9640 1 -- 172.21.15.5:0/1808081371 shutdown_connections 2024-03-24T10:58:24.610 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.609+0000 7f4e282c9640 1 -- 172.21.15.5:0/1808081371 wait complete. 2024-03-24T10:58:24.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.611+0000 7f4e282c9640 1 Processor -- start 2024-03-24T10:58:24.611 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.611+0000 7f4e282c9640 1 -- start start 2024-03-24T10:58:24.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.612+0000 7f4e282c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e200cf100 0x7f4e201f3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:24.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.612+0000 7f4e282c9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e2015aac0 0x7f4e201f0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:24.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.612+0000 7f4e282c9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e2015aea0 0x7f4e2015d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:24.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.612+0000 7f4e282c9640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f4e201f06c0 con 0x7f4e2015aea0 2024-03-24T10:58:24.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.612+0000 7f4e282c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4e201f00d0 con 0x7f4e200cf100 2024-03-24T10:58:24.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.612+0000 7f4e282c9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4e201f4590 con 0x7f4e2015aac0 2024-03-24T10:58:24.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.613+0000 7f4e26866640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e2015aea0 0x7f4e2015d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:24.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.613+0000 7f4e26065640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e200cf100 0x7f4e201f3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:24.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.613+0000 7f4e25864640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e2015aac0 0x7f4e201f0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:24.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.613+0000 7f4e26866640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e2015aea0 0x7f4e2015d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:37154/0 (socket says 172.21.15.5:37154) 2024-03-24T10:58:24.614 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.613+0000 7f4e26866640 1 -- 172.21.15.5:0/2863109349 learned_addr learned my addr 172.21.15.5:0/2863109349 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:24.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.617+0000 7f4e25864640 1 -- 172.21.15.5:0/2863109349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e200cf100 msgr2=0x7f4e201f3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:24.617 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.617+0000 7f4e25864640 1 --2- 172.21.15.5:0/2863109349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e200cf100 0x7f4e201f3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:24.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.617+0000 7f4e25864640 1 -- 172.21.15.5:0/2863109349 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e2015aea0 msgr2=0x7f4e2015d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:24.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.617+0000 7f4e25864640 1 --2- 172.21.15.5:0/2863109349 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e2015aea0 0x7f4e2015d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:24.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.617+0000 7f4e25864640 1 -- 172.21.15.5:0/2863109349 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e18009a70 con 0x7f4e2015aac0 2024-03-24T10:58:24.620 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.620+0000 7f4e25864640 1 --2- 172.21.15.5:0/2863109349 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e2015aac0 0x7f4e201f0ae0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4e1000e870 tx=0x7f4e1000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.625+0000 7f4e16ffd640 1 -- 172.21.15.5:0/2863109349 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1000ee60 con 0x7f4e2015aac0 2024-03-24T10:58:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.625+0000 7f4e16ffd640 1 -- 172.21.15.5:0/2863109349 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e1000c940 con 0x7f4e2015aac0 2024-03-24T10:58:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.625+0000 7f4e16ffd640 1 -- 172.21.15.5:0/2863109349 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1000ccd0 con 0x7f4e2015aac0 2024-03-24T10:58:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.625+0000 7f4e282c9640 1 -- 172.21.15.5:0/2863109349 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e201f1020 con 0x7f4e2015aac0 2024-03-24T10:58:24.625 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.625+0000 7f4e282c9640 1 -- 172.21.15.5:0/2863109349 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e201f3fd0 con 0x7f4e2015aac0 2024-03-24T10:58:24.626 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.626+0000 7f4e282c9640 1 -- 172.21.15.5:0/2863109349 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e201f0800 con 0x7f4e2015aac0 2024-03-24T10:58:24.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.650+0000 7f4e16ffd640 1 -- 172.21.15.5:0/2863109349 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4e10013470 con 0x7f4e2015aac0 2024-03-24T10:58:24.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.651+0000 7f4e16ffd640 1 --2- 172.21.15.5:0/2863109349 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e0406b860 0x7f4e0406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:24.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.651+0000 7f4e26065640 1 --2- 172.21.15.5:0/2863109349 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e0406b860 0x7f4e0406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:24.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.651+0000 7f4e26065640 1 --2- 172.21.15.5:0/2863109349 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e0406b860 0x7f4e0406dd20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c007fd0 tx=0x7f4e1c007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:24.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.924+0000 7f4e16ffd640 1 -- 172.21.15.5:0/2863109349 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f4e1000ce30 con 0x7f4e2015aac0 2024-03-24T10:58:24.944 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:24.944+0000 7f4e16ffd640 1 -- 172.21.15.5:0/2863109349 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4e10055240 con 0x7f4e2015aac0 2024-03-24T10:58:25.166 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:25.166+0000 7f4e282c9640 1 -- 172.21.15.5:0/2863109349 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e201f0800 con 0x7f4e2015aac0 2024-03-24T10:58:25.181 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:25.181+0000 7f4e16ffd640 1 -- 172.21.15.5:0/2863109349 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f4e201f0800 con 0x7f4e2015aac0 2024-03-24T10:58:25.182 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:25.182 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:25.182 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:25.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:25.184+0000 7f4e282c9640 1 -- 172.21.15.5:0/2863109349 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e0406b860 msgr2=0x7f4e0406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:25.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:25.184+0000 7f4e282c9640 1 --2- 172.21.15.5:0/2863109349 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e0406b860 0x7f4e0406dd20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c007fd0 tx=0x7f4e1c007e80 comp rx=0 tx=0).stop 2024-03-24T10:58:25.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:25.184+0000 7f4e282c9640 1 -- 172.21.15.5:0/2863109349 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e2015aac0 msgr2=0x7f4e201f0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:25.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:25.184+0000 7f4e282c9640 1 --2- 172.21.15.5:0/2863109349 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e2015aac0 0x7f4e201f0ae0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4e1000e870 tx=0x7f4e1000ed30 comp rx=0 tx=0).stop 2024-03-24T10:58:25.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:25.184+0000 7f4e282c9640 1 -- 172.21.15.5:0/2863109349 shutdown_connections 2024-03-24T10:58:25.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:25.184+0000 7f4e282c9640 1 --2- 172.21.15.5:0/2863109349 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e0406b860 0x7f4e0406dd20 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:25.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:25.184+0000 7f4e282c9640 1 --2- 172.21.15.5:0/2863109349 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e2015aea0 0x7f4e2015d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:25.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:25.184+0000 7f4e282c9640 1 --2- 172.21.15.5:0/2863109349 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e2015aac0 0x7f4e201f0ae0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:25.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:25.184+0000 7f4e282c9640 1 --2- 172.21.15.5:0/2863109349 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e200cf100 0x7f4e201f3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:25.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:25.184+0000 7f4e282c9640 1 -- 172.21.15.5:0/2863109349 >> 172.21.15.5:0/2863109349 conn(0x7f4e200cfb90 msgr2=0x7f4e200c85b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:25.185 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:25.184+0000 7f4e282c9640 1 -- 172.21.15.5:0/2863109349 shutdown_connections 2024-03-24T10:58:25.186 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:25.184+0000 7f4e282c9640 1 -- 172.21.15.5:0/2863109349 wait complete. 2024-03-24T10:58:25.803 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:25.803 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:26.804 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:26.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.906+0000 7f6aeb777640 1 Processor -- start 2024-03-24T10:58:26.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.906+0000 7f6aeb777640 1 -- start start 2024-03-24T10:58:26.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.907+0000 7f6aeb777640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6ae415ac00 0x7f6ae40cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:26.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.907+0000 7f6aeb777640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ae40cb2f0 con 0x7f6ae41543a0 2024-03-24T10:58:26.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.907+0000 7f6aeb777640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ae415b070 con 0x7f6ae40ccaa0 2024-03-24T10:58:26.907 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.907+0000 7f6aeb777640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f6ae40c6750 con 0x7f6ae415ac00 2024-03-24T10:58:26.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.908+0000 7f6ae8d12640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f6ae415ac00 0x7f6ae40cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:26.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.908+0000 7f6ae9d14640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f6ae40ccaa0 0x7f6ae406bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:53334/0 (socket says 172.21.15.5:53334) 2024-03-24T10:58:26.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.908+0000 7f6ae9d14640 1 -- 172.21.15.5:0/1830867429 learned_addr learned my addr 172.21.15.5:0/1830867429 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:58:26.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.911+0000 7f6ae8d12640 1 -- 172.21.15.5:0/1830867429 >> v1:172.21.15.57:6789/0 conn(0x7f6ae40ccaa0 legacy=0x7f6ae406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:26.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.911+0000 7f6ae8d12640 1 -- 172.21.15.5:0/1830867429 >> v1:172.21.15.5:6789/0 conn(0x7f6ae41543a0 legacy=0x7f6ae406d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:26.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.911+0000 7f6ae8d12640 1 -- 172.21.15.5:0/1830867429 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae40ca7e0 con 0x7f6ae415ac00 2024-03-24T10:58:26.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.915+0000 7f6ae8d12640 1 --2- 172.21.15.5:0/1830867429 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ae415ac00 0x7f6ae40cc560 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6ad40097a0 tx=0x7f6ad402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb6238d56d85f8a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:26.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.920+0000 7f6adbfff640 1 -- 172.21.15.5:0/1830867429 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4031830 con 0x7f6ae415ac00 2024-03-24T10:58:26.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.920+0000 7f6adbfff640 1 -- 172.21.15.5:0/1830867429 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ad4031990 con 0x7f6ae415ac00 2024-03-24T10:58:26.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.920+0000 7f6adbfff640 1 -- 172.21.15.5:0/1830867429 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad4031d20 con 0x7f6ae415ac00 2024-03-24T10:58:26.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.921+0000 7f6aeb777640 1 -- 172.21.15.5:0/1830867429 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ae415ac00 msgr2=0x7f6ae40cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:26.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.921+0000 7f6aeb777640 1 --2- 172.21.15.5:0/1830867429 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ae415ac00 0x7f6ae40cc560 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6ad40097a0 tx=0x7f6ad402b8d0 comp rx=0 tx=0).stop 2024-03-24T10:58:26.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.921+0000 7f6aeb777640 1 -- 172.21.15.5:0/1830867429 shutdown_connections 2024-03-24T10:58:26.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.921+0000 7f6aeb777640 1 --2- 172.21.15.5:0/1830867429 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ae415ac00 0x7f6ae40cc560 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:26.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.921+0000 7f6aeb777640 1 -- 172.21.15.5:0/1830867429 >> 172.21.15.5:0/1830867429 conn(0x7f6ae4072ef0 msgr2=0x7f6ae4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:26.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.921+0000 7f6aeb777640 1 -- 172.21.15.5:0/1830867429 shutdown_connections 2024-03-24T10:58:26.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.921+0000 7f6aeb777640 1 -- 172.21.15.5:0/1830867429 wait complete. 2024-03-24T10:58:26.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.922+0000 7f6aeb777640 1 Processor -- start 2024-03-24T10:58:26.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.922+0000 7f6aeb777640 1 -- start start 2024-03-24T10:58:26.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.923+0000 7f6aeb777640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6ae40ccaa0 0x7f6ae41f3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:26.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.923+0000 7f6aeb777640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ae41543a0 0x7f6ae40b4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:26.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.923+0000 7f6aeb777640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ae415ac00 0x7f6ae4149910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:26.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.923+0000 7f6aeb777640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae415a4f0 con 0x7f6ae41543a0 2024-03-24T10:58:26.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.923+0000 7f6aeb777640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae4159f70 con 0x7f6ae415ac00 2024-03-24T10:58:26.923 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.923+0000 7f6aeb777640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f6ae415a210 con 0x7f6ae40ccaa0 2024-03-24T10:58:26.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.924+0000 7f6ae8d12640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ae41543a0 0x7f6ae40b4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:26.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.924+0000 7f6ae9d14640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ae415ac00 0x7f6ae4149910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:26.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.924+0000 7f6ae9513640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6ae40ccaa0 0x7f6ae41f3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:26.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.924+0000 7f6ae8d12640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ae41543a0 0x7f6ae40b4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:37178/0 (socket says 172.21.15.5:37178) 2024-03-24T10:58:26.924 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.924+0000 7f6ae8d12640 1 -- 172.21.15.5:0/2688343057 learned_addr learned my addr 172.21.15.5:0/2688343057 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:26.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.929+0000 7f6ae8d12640 1 -- 172.21.15.5:0/2688343057 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6ae40ccaa0 msgr2=0x7f6ae41f3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:26.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.929+0000 7f6ae8d12640 1 --2- 172.21.15.5:0/2688343057 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6ae40ccaa0 0x7f6ae41f3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:26.929 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.929+0000 7f6ae8d12640 1 -- 172.21.15.5:0/2688343057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ae415ac00 msgr2=0x7f6ae4149910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:26.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.929+0000 7f6ae8d12640 1 --2- 172.21.15.5:0/2688343057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ae415ac00 0x7f6ae4149910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:26.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.929+0000 7f6ae8d12640 1 -- 172.21.15.5:0/2688343057 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ad4009400 con 0x7f6ae41543a0 2024-03-24T10:58:26.930 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.929+0000 7f6ae9d14640 1 --2- 172.21.15.5:0/2688343057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ae415ac00 0x7f6ae4149910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:58:26.932 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.932+0000 7f6ae8d12640 1 --2- 172.21.15.5:0/2688343057 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ae41543a0 0x7f6ae40b4b10 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4000c00 tx=0x7f6ad40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:26.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.937+0000 7f6ad9ffb640 1 -- 172.21.15.5:0/2688343057 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad403aa60 con 0x7f6ae41543a0 2024-03-24T10:58:26.937 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.937+0000 7f6ad9ffb640 1 -- 172.21.15.5:0/2688343057 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ad403abc0 con 0x7f6ae41543a0 2024-03-24T10:58:26.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.937+0000 7f6ad9ffb640 1 -- 172.21.15.5:0/2688343057 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad40443f0 con 0x7f6ae41543a0 2024-03-24T10:58:26.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.937+0000 7f6aeb777640 1 -- 172.21.15.5:0/2688343057 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ae414b260 con 0x7f6ae41543a0 2024-03-24T10:58:26.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.937+0000 7f6aeb777640 1 -- 172.21.15.5:0/2688343057 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ae4157f60 con 0x7f6ae41543a0 2024-03-24T10:58:26.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.938+0000 7f6aeb777640 1 -- 172.21.15.5:0/2688343057 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ae4158120 con 0x7f6ae41543a0 2024-03-24T10:58:26.970 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.969+0000 7f6ad9ffb640 1 -- 172.21.15.5:0/2688343057 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6ad404d450 con 0x7f6ae41543a0 2024-03-24T10:58:26.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.969+0000 7f6ad9ffb640 1 --2- 172.21.15.5:0/2688343057 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6ac806b810 0x7f6ac806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:26.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.969+0000 7f6ae9513640 1 --2- 172.21.15.5:0/2688343057 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6ac806b810 0x7f6ac806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:26.971 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.970+0000 7f6ae9513640 1 --2- 172.21.15.5:0/2688343057 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6ac806b810 0x7f6ac806dcd0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0004f60 tx=0x7f6ae0007e30 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:26.972 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:26.972+0000 7f6ad9ffb640 1 -- 172.21.15.5:0/2688343057 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f6ad407aec0 con 0x7f6ae41543a0 2024-03-24T10:58:27.230 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:27.230+0000 7f6ad9ffb640 1 -- 172.21.15.5:0/2688343057 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6ad4077160 con 0x7f6ae41543a0 2024-03-24T10:58:27.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:27.452+0000 7f6aeb777640 1 -- 172.21.15.5:0/2688343057 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ae414b0c0 con 0x7f6ae41543a0 2024-03-24T10:58:27.465 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:27.465+0000 7f6ad9ffb640 1 -- 172.21.15.5:0/2688343057 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f6ae414b0c0 con 0x7f6ae41543a0 2024-03-24T10:58:27.465 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:27.466 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:27.466 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:27.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:27.467+0000 7f6aeb777640 1 -- 172.21.15.5:0/2688343057 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6ac806b810 msgr2=0x7f6ac806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:27.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:27.467+0000 7f6aeb777640 1 --2- 172.21.15.5:0/2688343057 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6ac806b810 0x7f6ac806dcd0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6ae0004f60 tx=0x7f6ae0007e30 comp rx=0 tx=0).stop 2024-03-24T10:58:27.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:27.467+0000 7f6aeb777640 1 -- 172.21.15.5:0/2688343057 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ae41543a0 msgr2=0x7f6ae40b4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:27.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:27.467+0000 7f6aeb777640 1 --2- 172.21.15.5:0/2688343057 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ae41543a0 0x7f6ae40b4b10 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4000c00 tx=0x7f6ad40046c0 comp rx=0 tx=0).stop 2024-03-24T10:58:27.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:27.468+0000 7f6aeb777640 1 -- 172.21.15.5:0/2688343057 shutdown_connections 2024-03-24T10:58:27.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:27.468+0000 7f6aeb777640 1 --2- 172.21.15.5:0/2688343057 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f6ac806b810 0x7f6ac806dcd0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:27.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:27.468+0000 7f6aeb777640 1 --2- 172.21.15.5:0/2688343057 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f6ae41543a0 0x7f6ae40b4b10 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:27.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:27.468+0000 7f6aeb777640 1 --2- 172.21.15.5:0/2688343057 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6ae415ac00 0x7f6ae4149910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:27.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:27.468+0000 7f6aeb777640 1 --2- 172.21.15.5:0/2688343057 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f6ae40ccaa0 0x7f6ae41f3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:27.468 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:27.468+0000 7f6aeb777640 1 -- 172.21.15.5:0/2688343057 >> 172.21.15.5:0/2688343057 conn(0x7f6ae4072ef0 msgr2=0x7f6ae4152d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:27.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:27.468+0000 7f6aeb777640 1 -- 172.21.15.5:0/2688343057 shutdown_connections 2024-03-24T10:58:27.469 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:27.468+0000 7f6aeb777640 1 -- 172.21.15.5:0/2688343057 wait complete. 2024-03-24T10:58:28.083 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:28.083 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:29.084 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:29.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.187+0000 7f026ba7c640 1 Processor -- start 2024-03-24T10:58:29.187 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.187+0000 7f026ba7c640 1 -- start start 2024-03-24T10:58:29.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.188+0000 7f026ba7c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f026415ac00 0x7f02640cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:29.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.188+0000 7f026ba7c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02640ccaa0 0x7f026406bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:29.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.188+0000 7f026ba7c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02640cb2f0 con 0x7f02641543a0 2024-03-24T10:58:29.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.188+0000 7f026ba7c640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f02640c6750 con 0x7f026415ac00 2024-03-24T10:58:29.188 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.188+0000 7f026ba7c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f02640cceb0 con 0x7f02640ccaa0 2024-03-24T10:58:29.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.189+0000 7f0269017640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f026415ac00 0x7f02640cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:29.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.189+0000 7f026a019640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02640ccaa0 0x7f026406bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:29.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.189+0000 7f0269818640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f02641543a0 0x7f026406d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:53348/0 (socket says 172.21.15.5:53348) 2024-03-24T10:58:29.189 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.189+0000 7f0269818640 1 -- 172.21.15.5:0/3185978743 learned_addr learned my addr 172.21.15.5:0/3185978743 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:58:29.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.191+0000 7f0269017640 1 -- 172.21.15.5:0/3185978743 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02640ccaa0 msgr2=0x7f026406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:29.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.191+0000 7f0269017640 1 --2- 172.21.15.5:0/3185978743 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02640ccaa0 0x7f026406bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:29.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.191+0000 7f0269017640 1 -- 172.21.15.5:0/3185978743 >> v1:172.21.15.57:6789/0 conn(0x7f02641543a0 legacy=0x7f026406d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:29.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.191+0000 7f0269017640 1 -- 172.21.15.5:0/3185978743 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f026415b070 con 0x7f026415ac00 2024-03-24T10:58:29.193 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.194+0000 7f0269017640 1 --2- 172.21.15.5:0/3185978743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f026415ac00 0x7f02640cc560 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f02540097a0 tx=0x7f025402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=851031dcc0349345 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:29.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.198+0000 7f0268816640 1 -- 172.21.15.5:0/3185978743 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0254031830 con 0x7f026415ac00 2024-03-24T10:58:29.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.198+0000 7f0268816640 1 -- 172.21.15.5:0/3185978743 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0254031990 con 0x7f026415ac00 2024-03-24T10:58:29.198 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.198+0000 7f0268816640 1 -- 172.21.15.5:0/3185978743 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0254031d20 con 0x7f026415ac00 2024-03-24T10:58:29.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.199+0000 7f026ba7c640 1 -- 172.21.15.5:0/3185978743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f026415ac00 msgr2=0x7f02640cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:29.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.199+0000 7f026ba7c640 1 --2- 172.21.15.5:0/3185978743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f026415ac00 0x7f02640cc560 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f02540097a0 tx=0x7f025402b8d0 comp rx=0 tx=0).stop 2024-03-24T10:58:29.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.199+0000 7f026ba7c640 1 -- 172.21.15.5:0/3185978743 shutdown_connections 2024-03-24T10:58:29.199 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.199+0000 7f026ba7c640 1 --2- 172.21.15.5:0/3185978743 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02640ccaa0 0x7f026406bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:29.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.199+0000 7f026ba7c640 1 --2- 172.21.15.5:0/3185978743 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f026415ac00 0x7f02640cc560 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:29.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.199+0000 7f026ba7c640 1 -- 172.21.15.5:0/3185978743 >> 172.21.15.5:0/3185978743 conn(0x7f0264072ef0 msgr2=0x7f0264061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:29.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.199+0000 7f026ba7c640 1 -- 172.21.15.5:0/3185978743 shutdown_connections 2024-03-24T10:58:29.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.199+0000 7f026ba7c640 1 -- 172.21.15.5:0/3185978743 wait complete. 2024-03-24T10:58:29.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.200+0000 7f026ba7c640 1 Processor -- start 2024-03-24T10:58:29.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.201+0000 7f026ba7c640 1 -- start start 2024-03-24T10:58:29.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.201+0000 7f026ba7c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02640ccaa0 0x7f02640b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:29.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.201+0000 7f026ba7c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02641543a0 0x7f02641f32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:29.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.201+0000 7f026ba7c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f026415ac00 0x7f0264146650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:29.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.201+0000 7f026ba7c640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f0264158770 con 0x7f026415ac00 2024-03-24T10:58:29.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.201+0000 7f026ba7c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f02640c77c0 con 0x7f02640ccaa0 2024-03-24T10:58:29.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.201+0000 7f026ba7c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0264157f60 con 0x7f02641543a0 2024-03-24T10:58:29.202 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.202+0000 7f026a019640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f026415ac00 0x7f0264146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:29.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.202+0000 7f0269017640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02641543a0 0x7f02641f32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:29.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.202+0000 7f026a019640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f026415ac00 0x7f0264146650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:37192/0 (socket says 172.21.15.5:37192) 2024-03-24T10:58:29.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.202+0000 7f026a019640 1 -- 172.21.15.5:0/2790404644 learned_addr learned my addr 172.21.15.5:0/2790404644 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:29.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.202+0000 7f0269818640 1 --2- 172.21.15.5:0/2790404644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02640ccaa0 0x7f02640b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:29.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.206+0000 7f026a019640 1 -- 172.21.15.5:0/2790404644 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02641543a0 msgr2=0x7f02641f32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:29.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.206+0000 7f026a019640 1 --2- 172.21.15.5:0/2790404644 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02641543a0 0x7f02641f32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:29.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.206+0000 7f026a019640 1 -- 172.21.15.5:0/2790404644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02640ccaa0 msgr2=0x7f02640b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:29.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.206+0000 7f026a019640 1 --2- 172.21.15.5:0/2790404644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02640ccaa0 0x7f02640b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:29.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.206+0000 7f026a019640 1 -- 172.21.15.5:0/2790404644 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0254009400 con 0x7f026415ac00 2024-03-24T10:58:29.208 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.208+0000 7f026a019640 1 --2- 172.21.15.5:0/2790404644 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f026415ac00 0x7f0264146650 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f025c00efc0 tx=0x7f025c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:29.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.213+0000 7f025a7fc640 1 -- 172.21.15.5:0/2790404644 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025c00ced0 con 0x7f026415ac00 2024-03-24T10:58:29.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.213+0000 7f025a7fc640 1 -- 172.21.15.5:0/2790404644 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f025c00f040 con 0x7f026415ac00 2024-03-24T10:58:29.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.213+0000 7f026ba7c640 1 -- 172.21.15.5:0/2790404644 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02641f3c80 con 0x7f026415ac00 2024-03-24T10:58:29.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.213+0000 7f025a7fc640 1 -- 172.21.15.5:0/2790404644 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025c0146c0 con 0x7f026415ac00 2024-03-24T10:58:29.213 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.213+0000 7f026ba7c640 1 -- 172.21.15.5:0/2790404644 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0264158490 con 0x7f026415ac00 2024-03-24T10:58:29.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.214+0000 7f026ba7c640 1 -- 172.21.15.5:0/2790404644 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02640681d0 con 0x7f026415ac00 2024-03-24T10:58:29.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.239+0000 7f025a7fc640 1 -- 172.21.15.5:0/2790404644 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f025c014820 con 0x7f026415ac00 2024-03-24T10:58:29.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.239+0000 7f025a7fc640 1 --2- 172.21.15.5:0/2790404644 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f024806b860 0x7f024806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:29.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.239+0000 7f0269818640 1 --2- 172.21.15.5:0/2790404644 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f024806b860 0x7f024806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:29.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.240+0000 7f0269818640 1 --2- 172.21.15.5:0/2790404644 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f024806b860 0x7f024806dd20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0260007fd0 tx=0x7f0260007e80 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:29.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.241+0000 7f025a7fc640 1 -- 172.21.15.5:0/2790404644 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f025c059c30 con 0x7f026415ac00 2024-03-24T10:58:29.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.515+0000 7f025a7fc640 1 -- 172.21.15.5:0/2790404644 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f025c055ed0 con 0x7f026415ac00 2024-03-24T10:58:29.742 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.742+0000 7f026ba7c640 1 -- 172.21.15.5:0/2790404644 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0264150310 con 0x7f026415ac00 2024-03-24T10:58:29.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.777+0000 7f025a7fc640 1 -- 172.21.15.5:0/2790404644 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f025c018020 con 0x7f026415ac00 2024-03-24T10:58:29.778 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:29.778 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:29.778 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:29.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.779+0000 7f026ba7c640 1 -- 172.21.15.5:0/2790404644 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f024806b860 msgr2=0x7f024806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:29.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.779+0000 7f026ba7c640 1 --2- 172.21.15.5:0/2790404644 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f024806b860 0x7f024806dd20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0260007fd0 tx=0x7f0260007e80 comp rx=0 tx=0).stop 2024-03-24T10:58:29.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.779+0000 7f026ba7c640 1 -- 172.21.15.5:0/2790404644 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f026415ac00 msgr2=0x7f0264146650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:29.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.779+0000 7f026ba7c640 1 --2- 172.21.15.5:0/2790404644 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f026415ac00 0x7f0264146650 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f025c00efc0 tx=0x7f025c00c490 comp rx=0 tx=0).stop 2024-03-24T10:58:29.779 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.779+0000 7f026ba7c640 1 -- 172.21.15.5:0/2790404644 shutdown_connections 2024-03-24T10:58:29.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.779+0000 7f026ba7c640 1 --2- 172.21.15.5:0/2790404644 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f024806b860 0x7f024806dd20 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:29.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.779+0000 7f026ba7c640 1 --2- 172.21.15.5:0/2790404644 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f026415ac00 0x7f0264146650 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:29.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.779+0000 7f026ba7c640 1 --2- 172.21.15.5:0/2790404644 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f02641543a0 0x7f02641f32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:29.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.779+0000 7f026ba7c640 1 --2- 172.21.15.5:0/2790404644 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f02640ccaa0 0x7f02640b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:29.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.779+0000 7f026ba7c640 1 -- 172.21.15.5:0/2790404644 >> 172.21.15.5:0/2790404644 conn(0x7f0264072ef0 msgr2=0x7f0264151200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:29.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.779+0000 7f026ba7c640 1 -- 172.21.15.5:0/2790404644 shutdown_connections 2024-03-24T10:58:29.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:29.779+0000 7f026ba7c640 1 -- 172.21.15.5:0/2790404644 wait complete. 2024-03-24T10:58:30.395 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:30.395 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:31.398 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:31.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.500+0000 7f53ba40e640 1 Processor -- start 2024-03-24T10:58:31.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.500+0000 7f53ba40e640 1 -- start start 2024-03-24T10:58:31.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.500+0000 7f53ba40e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f53b415a310 0x7f53b406c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:31.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.500+0000 7f53ba40e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f53b406c780 0x7f53b406f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:31.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.501+0000 7f53ba40e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b40cb0c0 con 0x7f53b40cace0 2024-03-24T10:58:31.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.501+0000 7f53ba40e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f53b406f110 con 0x7f53b406c780 2024-03-24T10:58:31.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.501+0000 7f53ba40e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f53b415b320 con 0x7f53b415a310 2024-03-24T10:58:31.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.501+0000 7f53b89ab640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f53b406c780 0x7f53b406f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:31.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.501+0000 7f53b37fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f53b415a310 0x7f53b406c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:31.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.501+0000 7f53b3fff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f53b40cace0 0x7f53b41432d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:48024/0 (socket says 172.21.15.5:48024) 2024-03-24T10:58:31.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.501+0000 7f53b3fff640 1 -- 172.21.15.5:0/2136036282 learned_addr learned my addr 172.21.15.5:0/2136036282 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:58:31.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.504+0000 7f53b89ab640 1 -- 172.21.15.5:0/2136036282 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f53b415a310 msgr2=0x7f53b406c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:31.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.504+0000 7f53b89ab640 1 --2- 172.21.15.5:0/2136036282 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f53b415a310 0x7f53b406c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:31.504 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.504+0000 7f53b89ab640 1 -- 172.21.15.5:0/2136036282 >> v1:172.21.15.5:6789/0 conn(0x7f53b40cace0 legacy=0x7f53b41432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:31.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.504+0000 7f53b89ab640 1 -- 172.21.15.5:0/2136036282 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b406cf70 con 0x7f53b406c780 2024-03-24T10:58:31.508 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.509+0000 7f53b89ab640 1 --2- 172.21.15.5:0/2136036282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53b406c780 0x7f53b406f530 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009d90 tx=0x7f53ac02bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=834e60360fe5c3d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:31.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.513+0000 7f53b2ffd640 1 -- 172.21.15.5:0/2136036282 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ac02cce0 con 0x7f53b406c780 2024-03-24T10:58:31.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.513+0000 7f53b2ffd640 1 -- 172.21.15.5:0/2136036282 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53ac02ce40 con 0x7f53b406c780 2024-03-24T10:58:31.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.513+0000 7f53b2ffd640 1 -- 172.21.15.5:0/2136036282 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ac034660 con 0x7f53b406c780 2024-03-24T10:58:31.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.514+0000 7f53ba40e640 1 -- 172.21.15.5:0/2136036282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53b406c780 msgr2=0x7f53b406f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:31.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.514+0000 7f53ba40e640 1 --2- 172.21.15.5:0/2136036282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53b406c780 0x7f53b406f530 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009d90 tx=0x7f53ac02bf30 comp rx=0 tx=0).stop 2024-03-24T10:58:31.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.514+0000 7f53ba40e640 1 -- 172.21.15.5:0/2136036282 shutdown_connections 2024-03-24T10:58:31.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.514+0000 7f53ba40e640 1 --2- 172.21.15.5:0/2136036282 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53b406c780 0x7f53b406f530 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:31.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.514+0000 7f53ba40e640 1 --2- 172.21.15.5:0/2136036282 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f53b415a310 0x7f53b406c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:31.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.514+0000 7f53ba40e640 1 -- 172.21.15.5:0/2136036282 >> 172.21.15.5:0/2136036282 conn(0x7f53b4071240 msgr2=0x7f53b4071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:31.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.514+0000 7f53ba40e640 1 -- 172.21.15.5:0/2136036282 shutdown_connections 2024-03-24T10:58:31.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.514+0000 7f53ba40e640 1 -- 172.21.15.5:0/2136036282 wait complete. 2024-03-24T10:58:31.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.515+0000 7f53ba40e640 1 Processor -- start 2024-03-24T10:58:31.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.516+0000 7f53ba40e640 1 -- start start 2024-03-24T10:58:31.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.516+0000 7f53ba40e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53b406c780 0x7f53b40af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:31.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.516+0000 7f53ba40e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53b40cace0 0x7f53b413c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:31.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.516+0000 7f53ba40e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f53b4138f60 0x7f53b4140000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:31.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.516+0000 7f53ba40e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f53b40aeed0 con 0x7f53b406c780 2024-03-24T10:58:31.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.516+0000 7f53ba40e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f53b40aea40 con 0x7f53b40cace0 2024-03-24T10:58:31.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.516+0000 7f53ba40e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f53b41421a0 con 0x7f53b4138f60 2024-03-24T10:58:31.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.517+0000 7f53b3fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53b406c780 0x7f53b40af9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:31.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.517+0000 7f53b37fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53b40cace0 0x7f53b413c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:31.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.517+0000 7f53b89ab640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f53b4138f60 0x7f53b4140000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:31.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.517+0000 7f53b3fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53b406c780 0x7f53b40af9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:37200/0 (socket says 172.21.15.5:37200) 2024-03-24T10:58:31.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.517+0000 7f53b3fff640 1 -- 172.21.15.5:0/3751248104 learned_addr learned my addr 172.21.15.5:0/3751248104 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:31.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.521+0000 7f53b3fff640 1 -- 172.21.15.5:0/3751248104 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f53b4138f60 msgr2=0x7f53b4140000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:31.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.521+0000 7f53b3fff640 1 --2- 172.21.15.5:0/3751248104 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f53b4138f60 0x7f53b4140000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:31.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.521+0000 7f53b3fff640 1 -- 172.21.15.5:0/3751248104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53b40cace0 msgr2=0x7f53b413c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:31.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.521+0000 7f53b3fff640 1 --2- 172.21.15.5:0/3751248104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53b40cace0 0x7f53b413c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:31.521 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.521+0000 7f53b3fff640 1 -- 172.21.15.5:0/3751248104 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ac009a70 con 0x7f53b406c780 2024-03-24T10:58:31.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.523+0000 7f53b3fff640 1 --2- 172.21.15.5:0/3751248104 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53b406c780 0x7f53b40af9a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f53a400c3c0 tx=0x7f53a400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:31.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.528+0000 7f53b0ff9640 1 -- 172.21.15.5:0/3751248104 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a4005ce0 con 0x7f53b406c780 2024-03-24T10:58:31.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.528+0000 7f53b0ff9640 1 -- 172.21.15.5:0/3751248104 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53a4005e40 con 0x7f53b406c780 2024-03-24T10:58:31.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.528+0000 7f53ba40e640 1 -- 172.21.15.5:0/3751248104 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53b4141ec0 con 0x7f53b406c780 2024-03-24T10:58:31.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.528+0000 7f53b0ff9640 1 -- 172.21.15.5:0/3751248104 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a40072d0 con 0x7f53b406c780 2024-03-24T10:58:31.528 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.528+0000 7f53ba40e640 1 -- 172.21.15.5:0/3751248104 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53b40ae6d0 con 0x7f53b406c780 2024-03-24T10:58:31.529 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.529+0000 7f53ba40e640 1 -- 172.21.15.5:0/3751248104 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53b415b000 con 0x7f53b406c780 2024-03-24T10:58:31.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.549+0000 7f53b0ff9640 1 -- 172.21.15.5:0/3751248104 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f53a40074f0 con 0x7f53b406c780 2024-03-24T10:58:31.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.549+0000 7f53b0ff9640 1 --2- 172.21.15.5:0/3751248104 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f53a006b810 0x7f53a006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:31.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.549+0000 7f53b37fe640 1 --2- 172.21.15.5:0/3751248104 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f53a006b810 0x7f53a006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:31.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.550+0000 7f53b37fe640 1 --2- 172.21.15.5:0/3751248104 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f53a006b810 0x7f53a006dcd0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f53b406fe70 tx=0x7f53a8009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:31.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.552+0000 7f53b0ff9640 1 -- 172.21.15.5:0/3751248104 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f53a4058340 con 0x7f53b406c780 2024-03-24T10:58:31.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:31.855+0000 7f53b0ff9640 1 -- 172.21.15.5:0/3751248104 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f53a40545e0 con 0x7f53b406c780 2024-03-24T10:58:32.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:32.084+0000 7f53ba40e640 1 -- 172.21.15.5:0/3751248104 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53b40cb350 con 0x7f53b406c780 2024-03-24T10:58:32.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:32.111+0000 7f53b0ff9640 1 -- 172.21.15.5:0/3751248104 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f53a4017070 con 0x7f53b406c780 2024-03-24T10:58:32.111 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:32.111 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:32.112 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:32.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:32.113+0000 7f53ba40e640 1 -- 172.21.15.5:0/3751248104 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f53a006b810 msgr2=0x7f53a006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:32.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:32.113+0000 7f53ba40e640 1 --2- 172.21.15.5:0/3751248104 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f53a006b810 0x7f53a006dcd0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f53b406fe70 tx=0x7f53a8009190 comp rx=0 tx=0).stop 2024-03-24T10:58:32.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:32.113+0000 7f53ba40e640 1 -- 172.21.15.5:0/3751248104 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53b406c780 msgr2=0x7f53b40af9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:32.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:32.113+0000 7f53ba40e640 1 --2- 172.21.15.5:0/3751248104 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53b406c780 0x7f53b40af9a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f53a400c3c0 tx=0x7f53a400c880 comp rx=0 tx=0).stop 2024-03-24T10:58:32.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:32.113+0000 7f53ba40e640 1 -- 172.21.15.5:0/3751248104 shutdown_connections 2024-03-24T10:58:32.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:32.113+0000 7f53ba40e640 1 --2- 172.21.15.5:0/3751248104 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f53a006b810 0x7f53a006dcd0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:32.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:32.113+0000 7f53ba40e640 1 --2- 172.21.15.5:0/3751248104 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f53b4138f60 0x7f53b4140000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:32.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:32.113+0000 7f53ba40e640 1 --2- 172.21.15.5:0/3751248104 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f53b40cace0 0x7f53b413c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:32.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:32.113+0000 7f53ba40e640 1 --2- 172.21.15.5:0/3751248104 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f53b406c780 0x7f53b40af9a0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:32.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:32.113+0000 7f53ba40e640 1 -- 172.21.15.5:0/3751248104 >> 172.21.15.5:0/3751248104 conn(0x7f53b4071240 msgr2=0x7f53b4159560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:32.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:32.113+0000 7f53ba40e640 1 -- 172.21.15.5:0/3751248104 shutdown_connections 2024-03-24T10:58:32.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:32.114+0000 7f53ba40e640 1 -- 172.21.15.5:0/3751248104 wait complete. 2024-03-24T10:58:32.730 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:32.730 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:33.018 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:58:33.020 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:58:33.731 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:33.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.834+0000 7f4e2bca4640 1 Processor -- start 2024-03-24T10:58:33.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.835+0000 7f4e2bca4640 1 -- start start 2024-03-24T10:58:33.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.836+0000 7f4e2bca4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4e240cb270 0x7f4e240cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:33.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.836+0000 7f4e2bca4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e240cd960 0x7f4e240cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:33.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.836+0000 7f4e2bca4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e240ccba0 con 0x7f4e240cae90 2024-03-24T10:58:33.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.836+0000 7f4e2bca4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f4e240cc9b0 con 0x7f4e240cb270 2024-03-24T10:58:33.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.836+0000 7f4e2bca4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4e240cce40 con 0x7f4e240cd960 2024-03-24T10:58:33.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.837+0000 7f4e29a40640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f4e240cae90 0x7f4e240cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:47246/0 (socket says 172.21.15.5:47246) 2024-03-24T10:58:33.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.837+0000 7f4e29a40640 1 -- 172.21.15.5:0/2479476840 learned_addr learned my addr 172.21.15.5:0/2479476840 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:58:33.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.837+0000 7f4e2a241640 1 --2- 172.21.15.5:0/2479476840 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e240cd960 0x7f4e240cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:33.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.837+0000 7f4e2923f640 1 --2- 172.21.15.5:0/2479476840 >> v2:172.21.15.57:3300/0 conn(0x7f4e240cb270 0x7f4e240cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:33.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.840+0000 7f4e28a3e640 1 -- 172.21.15.5:0/2479476840 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2326958272 0 0) 0x7f4e240ccba0 con 0x7f4e240cae90 2024-03-24T10:58:33.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.840+0000 7f4e28a3e640 1 -- 172.21.15.5:0/2479476840 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e10003520 con 0x7f4e240cae90 2024-03-24T10:58:33.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.840+0000 7f4e2923f640 1 -- 172.21.15.5:0/2479476840 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e240cd960 msgr2=0x7f4e240cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:33.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.840+0000 7f4e2923f640 1 --2- 172.21.15.5:0/2479476840 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e240cd960 0x7f4e240cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:33.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.840+0000 7f4e2923f640 1 -- 172.21.15.5:0/2479476840 >> v1:172.21.15.5:6789/0 conn(0x7f4e240cae90 legacy=0x7f4e240cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:33.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.840+0000 7f4e2923f640 1 -- 172.21.15.5:0/2479476840 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e2415a590 con 0x7f4e240cb270 2024-03-24T10:58:33.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.842+0000 7f4e2923f640 1 --2- 172.21.15.5:0/2479476840 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e240cb270 0x7f4e240cf530 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4e140097a0 tx=0x7f4e1402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe53916bc796243b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:33.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.847+0000 7f4e28a3e640 1 -- 172.21.15.5:0/2479476840 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e14030610 con 0x7f4e240cb270 2024-03-24T10:58:33.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.847+0000 7f4e28a3e640 1 -- 172.21.15.5:0/2479476840 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e14030770 con 0x7f4e240cb270 2024-03-24T10:58:33.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.847+0000 7f4e28a3e640 1 -- 172.21.15.5:0/2479476840 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e14030b00 con 0x7f4e240cb270 2024-03-24T10:58:33.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.848+0000 7f4e2bca4640 1 -- 172.21.15.5:0/2479476840 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e240cb270 msgr2=0x7f4e240cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:33.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.848+0000 7f4e2bca4640 1 --2- 172.21.15.5:0/2479476840 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e240cb270 0x7f4e240cf530 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4e140097a0 tx=0x7f4e1402b960 comp rx=0 tx=0).stop 2024-03-24T10:58:33.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.848+0000 7f4e2bca4640 1 -- 172.21.15.5:0/2479476840 shutdown_connections 2024-03-24T10:58:33.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.848+0000 7f4e2bca4640 1 --2- 172.21.15.5:0/2479476840 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e240cd960 0x7f4e240cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:33.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.848+0000 7f4e2bca4640 1 --2- 172.21.15.5:0/2479476840 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e240cb270 0x7f4e240cf530 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:33.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.848+0000 7f4e2bca4640 1 -- 172.21.15.5:0/2479476840 >> 172.21.15.5:0/2479476840 conn(0x7f4e240cfb80 msgr2=0x7f4e240cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:33.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.848+0000 7f4e2bca4640 1 -- 172.21.15.5:0/2479476840 shutdown_connections 2024-03-24T10:58:33.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.848+0000 7f4e2bca4640 1 -- 172.21.15.5:0/2479476840 wait complete. 2024-03-24T10:58:33.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.850+0000 7f4e2bca4640 1 Processor -- start 2024-03-24T10:58:33.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.850+0000 7f4e2bca4640 1 -- start start 2024-03-24T10:58:33.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.851+0000 7f4e2bca4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e240cae90 0x7f4e24151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:33.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.851+0000 7f4e2bca4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e240cb270 0x7f4e240ab1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:33.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.851+0000 7f4e2bca4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e240cd960 0x7f4e240ad880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:33.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.851+0000 7f4e2bca4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f4e24151780 con 0x7f4e240cd960 2024-03-24T10:58:33.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.851+0000 7f4e2bca4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f4e24154770 con 0x7f4e240cb270 2024-03-24T10:58:33.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.851+0000 7f4e2bca4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f4e240ae690 con 0x7f4e240cae90 2024-03-24T10:58:33.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.852+0000 7f4e2923f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e240cb270 0x7f4e240ab1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:33.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.852+0000 7f4e2a241640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e240cd960 0x7f4e240ad880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:33.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.852+0000 7f4e29a40640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e240cae90 0x7f4e24151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:33.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.852+0000 7f4e2923f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e240cb270 0x7f4e240ab1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:41372/0 (socket says 172.21.15.5:41372) 2024-03-24T10:58:33.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.852+0000 7f4e2923f640 1 -- 172.21.15.5:0/2259393770 learned_addr learned my addr 172.21.15.5:0/2259393770 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:33.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.856+0000 7f4e2923f640 1 -- 172.21.15.5:0/2259393770 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e240cae90 msgr2=0x7f4e24151240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:33.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.856+0000 7f4e2923f640 1 --2- 172.21.15.5:0/2259393770 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e240cae90 0x7f4e24151240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:33.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.856+0000 7f4e2923f640 1 -- 172.21.15.5:0/2259393770 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e240cd960 msgr2=0x7f4e240ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:33.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.856+0000 7f4e2923f640 1 --2- 172.21.15.5:0/2259393770 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e240cd960 0x7f4e240ad880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:33.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.856+0000 7f4e2923f640 1 -- 172.21.15.5:0/2259393770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e14009400 con 0x7f4e240cb270 2024-03-24T10:58:33.858 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.859+0000 7f4e2923f640 1 --2- 172.21.15.5:0/2259393770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e240cb270 0x7f4e240ab1f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4e1402bf70 tx=0x7f4e14004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:33.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.863+0000 7f4e1a7fc640 1 -- 172.21.15.5:0/2259393770 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1403e040 con 0x7f4e240cb270 2024-03-24T10:58:33.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.863+0000 7f4e1a7fc640 1 -- 172.21.15.5:0/2259393770 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e14039a40 con 0x7f4e240cb270 2024-03-24T10:58:33.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.864+0000 7f4e1a7fc640 1 -- 172.21.15.5:0/2259393770 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e14039df0 con 0x7f4e240cb270 2024-03-24T10:58:33.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.864+0000 7f4e2bca4640 1 -- 172.21.15.5:0/2259393770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e240ae970 con 0x7f4e240cb270 2024-03-24T10:58:33.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.864+0000 7f4e2bca4640 1 -- 172.21.15.5:0/2259393770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e241efda0 con 0x7f4e240cb270 2024-03-24T10:58:33.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:33.864+0000 7f4e2bca4640 1 -- 172.21.15.5:0/2259393770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e240681d0 con 0x7f4e240cb270 2024-03-24T10:58:34.190 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:34.190+0000 7f4e1a7fc640 1 -- 172.21.15.5:0/2259393770 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4e1404d3b0 con 0x7f4e240cb270 2024-03-24T10:58:34.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:34.190+0000 7f4e1a7fc640 1 --2- 172.21.15.5:0/2259393770 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e0806b860 0x7f4e0806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:34.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:34.191+0000 7f4e29a40640 1 --2- 172.21.15.5:0/2259393770 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e0806b860 0x7f4e0806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:34.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:34.191+0000 7f4e29a40640 1 --2- 172.21.15.5:0/2259393770 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e0806b860 0x7f4e0806dd20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f4e20001200 tx=0x7f4e20000bc0 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:34.197 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:34.197+0000 7f4e1a7fc640 1 -- 172.21.15.5:0/2259393770 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f4e1407b260 con 0x7f4e240cb270 2024-03-24T10:58:34.209 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:34.209+0000 7f4e1a7fc640 1 -- 172.21.15.5:0/2259393770 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4e14077500 con 0x7f4e240cb270 2024-03-24T10:58:34.496 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:34.496+0000 7f4e2bca4640 1 -- 172.21.15.5:0/2259393770 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e240c6470 con 0x7f4e240cb270 2024-03-24T10:58:34.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:34.512+0000 7f4e1a7fc640 1 -- 172.21.15.5:0/2259393770 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f4e14049090 con 0x7f4e240cb270 2024-03-24T10:58:34.512 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:34.512 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:34.512 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:34.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:34.514+0000 7f4e2bca4640 1 -- 172.21.15.5:0/2259393770 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e0806b860 msgr2=0x7f4e0806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:34.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:34.514+0000 7f4e2bca4640 1 --2- 172.21.15.5:0/2259393770 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e0806b860 0x7f4e0806dd20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f4e20001200 tx=0x7f4e20000bc0 comp rx=0 tx=0).stop 2024-03-24T10:58:34.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:34.514+0000 7f4e2bca4640 1 -- 172.21.15.5:0/2259393770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e240cb270 msgr2=0x7f4e240ab1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:34.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:34.514+0000 7f4e2bca4640 1 --2- 172.21.15.5:0/2259393770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e240cb270 0x7f4e240ab1f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4e1402bf70 tx=0x7f4e14004730 comp rx=0 tx=0).stop 2024-03-24T10:58:34.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:34.514+0000 7f4e2bca4640 1 -- 172.21.15.5:0/2259393770 shutdown_connections 2024-03-24T10:58:34.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:34.514+0000 7f4e2bca4640 1 --2- 172.21.15.5:0/2259393770 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f4e0806b860 0x7f4e0806dd20 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:34.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:34.514+0000 7f4e2bca4640 1 --2- 172.21.15.5:0/2259393770 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f4e240cd960 0x7f4e240ad880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:34.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:34.514+0000 7f4e2bca4640 1 --2- 172.21.15.5:0/2259393770 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e240cb270 0x7f4e240ab1f0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:34.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:34.514+0000 7f4e2bca4640 1 --2- 172.21.15.5:0/2259393770 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f4e240cae90 0x7f4e24151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:34.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:34.514+0000 7f4e2bca4640 1 -- 172.21.15.5:0/2259393770 >> 172.21.15.5:0/2259393770 conn(0x7f4e240cfb80 msgr2=0x7f4e240c6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:34.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:34.514+0000 7f4e2bca4640 1 -- 172.21.15.5:0/2259393770 shutdown_connections 2024-03-24T10:58:34.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:34.514+0000 7f4e2bca4640 1 -- 172.21.15.5:0/2259393770 wait complete. 2024-03-24T10:58:35.131 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:35.131 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:36.132 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:36.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.234+0000 7f441f0ed640 1 Processor -- start 2024-03-24T10:58:36.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.234+0000 7f441f0ed640 1 -- start start 2024-03-24T10:58:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.235+0000 7f441f0ed640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f441815aac0 0x7f441815aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.235+0000 7f441f0ed640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f441806c230 0x7f441806c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.235+0000 7f441f0ed640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f44180cebc0 0x7f44180cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.235+0000 7f441f0ed640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f44180ce790 con 0x7f441806c230 2024-03-24T10:58:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.235+0000 7f441f0ed640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f441815b3e0 con 0x7f441815aac0 2024-03-24T10:58:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.235+0000 7f441f0ed640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f441806cd50 con 0x7f44180cebc0 2024-03-24T10:58:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.236+0000 7f4417fff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f441806c230 0x7f441806c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.236+0000 7f441ce89640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f441815aac0 0x7f441815aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.236+0000 7f441d68a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f44180cebc0 0x7f44180cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:36.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.236+0000 7f441ce89640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f441815aac0 0x7f441815aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:41382/0 (socket says 172.21.15.5:41382) 2024-03-24T10:58:36.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.236+0000 7f441ce89640 1 -- 172.21.15.5:0/1367392934 learned_addr learned my addr 172.21.15.5:0/1367392934 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:36.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.239+0000 7f441ce89640 1 -- 172.21.15.5:0/1367392934 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f44180cebc0 msgr2=0x7f44180cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:36.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.239+0000 7f441ce89640 1 --2- 172.21.15.5:0/1367392934 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f44180cebc0 0x7f44180cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:36.239 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.239+0000 7f441ce89640 1 -- 172.21.15.5:0/1367392934 >> v2:172.21.15.5:3300/0 conn(0x7f441806c230 msgr2=0x7f441806c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:36.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.239+0000 7f441ce89640 1 --2- 172.21.15.5:0/1367392934 >> v2:172.21.15.5:3300/0 conn(0x7f441806c230 0x7f441806c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:36.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.239+0000 7f441ce89640 1 -- 172.21.15.5:0/1367392934 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44180cdb10 con 0x7f441815aac0 2024-03-24T10:58:36.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.241+0000 7f441ce89640 1 --2- 172.21.15.5:0/1367392934 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f441815aac0 0x7f441815aea0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f44180b2780 tx=0x7f440c02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af68050a6ea895a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:36.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.246+0000 7f44177fe640 1 -- 172.21.15.5:0/1367392934 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440c030860 con 0x7f441815aac0 2024-03-24T10:58:36.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.246+0000 7f44177fe640 1 -- 172.21.15.5:0/1367392934 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f440c0309c0 con 0x7f441815aac0 2024-03-24T10:58:36.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.246+0000 7f44177fe640 1 -- 172.21.15.5:0/1367392934 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440c030d50 con 0x7f441815aac0 2024-03-24T10:58:36.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.247+0000 7f441f0ed640 1 -- 172.21.15.5:0/1367392934 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f441815aac0 msgr2=0x7f441815aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.247+0000 7f441f0ed640 1 --2- 172.21.15.5:0/1367392934 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f441815aac0 0x7f441815aea0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f44180b2780 tx=0x7f440c02b9a0 comp rx=0 tx=0).stop 2024-03-24T10:58:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.247+0000 7f441f0ed640 1 -- 172.21.15.5:0/1367392934 shutdown_connections 2024-03-24T10:58:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.247+0000 7f441f0ed640 1 --2- 172.21.15.5:0/1367392934 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f44180cebc0 0x7f44180cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.247+0000 7f441f0ed640 1 --2- 172.21.15.5:0/1367392934 >> v2:172.21.15.5:3300/0 conn(0x7f441806c230 0x7f441806c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.247+0000 7f441f0ed640 1 --2- 172.21.15.5:0/1367392934 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f441815aac0 0x7f441815aea0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.247+0000 7f441f0ed640 1 -- 172.21.15.5:0/1367392934 >> 172.21.15.5:0/1367392934 conn(0x7f44180cfb90 msgr2=0x7f44180cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.247+0000 7f441f0ed640 1 -- 172.21.15.5:0/1367392934 shutdown_connections 2024-03-24T10:58:36.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.248+0000 7f441f0ed640 1 -- 172.21.15.5:0/1367392934 wait complete. 2024-03-24T10:58:36.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.249+0000 7f441f0ed640 1 Processor -- start 2024-03-24T10:58:36.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.249+0000 7f441f0ed640 1 -- start start 2024-03-24T10:58:36.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.251+0000 7f441f0ed640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f441806c230 0x7f44180ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:36.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.251+0000 7f441f0ed640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f44180cebc0 0x7f44180b0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:36.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.251+0000 7f441f0ed640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f441815aac0 0x7f4418154d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:36.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.251+0000 7f441f0ed640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f44181efe60 con 0x7f441806c230 2024-03-24T10:58:36.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.251+0000 7f441f0ed640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f44180afdb0 con 0x7f441815aac0 2024-03-24T10:58:36.253 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.251+0000 7f441f0ed640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f44180af7f0 con 0x7f44180cebc0 2024-03-24T10:58:36.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.251+0000 7f441ce89640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f441806c230 0x7f44180ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:36.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.251+0000 7f441d68a640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f441815aac0 0x7f4418154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:36.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.252+0000 7f4417fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f44180cebc0 0x7f44180b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:36.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.252+0000 7f441ce89640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f441806c230 0x7f44180ca9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38948/0 (socket says 172.21.15.5:38948) 2024-03-24T10:58:36.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.252+0000 7f441ce89640 1 -- 172.21.15.5:0/1421277501 learned_addr learned my addr 172.21.15.5:0/1421277501 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:36.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.257+0000 7f441ce89640 1 -- 172.21.15.5:0/1421277501 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f44180cebc0 msgr2=0x7f44180b0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:36.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.257+0000 7f441ce89640 1 --2- 172.21.15.5:0/1421277501 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f44180cebc0 0x7f44180b0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:36.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.257+0000 7f441ce89640 1 -- 172.21.15.5:0/1421277501 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f441815aac0 msgr2=0x7f4418154d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:36.259 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.257+0000 7f441ce89640 1 --2- 172.21.15.5:0/1421277501 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f441815aac0 0x7f4418154d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:36.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.257+0000 7f441ce89640 1 -- 172.21.15.5:0/1421277501 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f440c0094d0 con 0x7f441806c230 2024-03-24T10:58:36.260 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.257+0000 7f441d68a640 1 --2- 172.21.15.5:0/1421277501 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f441815aac0 0x7f4418154d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:58:36.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.264+0000 7f441ce89640 1 --2- 172.21.15.5:0/1421277501 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f441806c230 0x7f44180ca9d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f440c000c00 tx=0x7f440c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:36.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.267+0000 7f44157fa640 1 -- 172.21.15.5:0/1421277501 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440c039a90 con 0x7f441806c230 2024-03-24T10:58:36.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.267+0000 7f441f0ed640 1 -- 172.21.15.5:0/1421277501 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44180afad0 con 0x7f441806c230 2024-03-24T10:58:36.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.267+0000 7f441f0ed640 1 -- 172.21.15.5:0/1421277501 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44181f0350 con 0x7f441806c230 2024-03-24T10:58:36.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.268+0000 7f441f0ed640 1 -- 172.21.15.5:0/1421277501 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44180681d0 con 0x7f441806c230 2024-03-24T10:58:36.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.559+0000 7f44157fa640 1 -- 172.21.15.5:0/1421277501 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f440c039bf0 con 0x7f441806c230 2024-03-24T10:58:36.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.559+0000 7f44157fa640 1 -- 172.21.15.5:0/1421277501 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440c04d440 con 0x7f441806c230 2024-03-24T10:58:36.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.559+0000 7f44157fa640 1 -- 172.21.15.5:0/1421277501 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f440c043410 con 0x7f441806c230 2024-03-24T10:58:36.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.560+0000 7f44157fa640 1 --2- 172.21.15.5:0/1421277501 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f43fc06b810 0x7f43fc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:36.560 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.560+0000 7f4417fff640 1 --2- 172.21.15.5:0/1421277501 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f43fc06b810 0x7f43fc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:36.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.560+0000 7f4417fff640 1 --2- 172.21.15.5:0/1421277501 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f43fc06b810 0x7f43fc06dcd0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4418157280 tx=0x7f4404009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:36.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.580+0000 7f44157fa640 1 -- 172.21.15.5:0/1421277501 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f440c07b460 con 0x7f441806c230 2024-03-24T10:58:36.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.580+0000 7f44157fa640 1 -- 172.21.15.5:0/1421277501 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f440c077700 con 0x7f441806c230 2024-03-24T10:58:36.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.810+0000 7f441f0ed640 1 -- 172.21.15.5:0/1421277501 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44180cf770 con 0x7f441806c230 2024-03-24T10:58:36.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.828+0000 7f44157fa640 1 -- 172.21.15.5:0/1421277501 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f440c048090 con 0x7f441806c230 2024-03-24T10:58:36.829 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:36.829 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:36.829 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:36.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.830+0000 7f441f0ed640 1 -- 172.21.15.5:0/1421277501 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f43fc06b810 msgr2=0x7f43fc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:36.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.830+0000 7f441f0ed640 1 --2- 172.21.15.5:0/1421277501 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f43fc06b810 0x7f43fc06dcd0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4418157280 tx=0x7f4404009190 comp rx=0 tx=0).stop 2024-03-24T10:58:36.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.830+0000 7f441f0ed640 1 -- 172.21.15.5:0/1421277501 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f441806c230 msgr2=0x7f44180ca9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:36.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.830+0000 7f441f0ed640 1 --2- 172.21.15.5:0/1421277501 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f441806c230 0x7f44180ca9d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f440c000c00 tx=0x7f440c004730 comp rx=0 tx=0).stop 2024-03-24T10:58:36.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.831+0000 7f441f0ed640 1 -- 172.21.15.5:0/1421277501 shutdown_connections 2024-03-24T10:58:36.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.831+0000 7f441f0ed640 1 --2- 172.21.15.5:0/1421277501 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f43fc06b810 0x7f43fc06dcd0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:36.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.831+0000 7f441f0ed640 1 --2- 172.21.15.5:0/1421277501 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f441815aac0 0x7f4418154d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:36.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.831+0000 7f441f0ed640 1 --2- 172.21.15.5:0/1421277501 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f44180cebc0 0x7f44180b0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:36.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.831+0000 7f441f0ed640 1 --2- 172.21.15.5:0/1421277501 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f441806c230 0x7f44180ca9d0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:36.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.831+0000 7f441f0ed640 1 -- 172.21.15.5:0/1421277501 >> 172.21.15.5:0/1421277501 conn(0x7f44180cfb90 msgr2=0x7f44180c80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:36.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.831+0000 7f441f0ed640 1 -- 172.21.15.5:0/1421277501 shutdown_connections 2024-03-24T10:58:36.832 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:36.831+0000 7f441f0ed640 1 -- 172.21.15.5:0/1421277501 wait complete. 2024-03-24T10:58:37.446 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:37.446 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:38.447 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:38.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.549+0000 7fda4a2af640 1 Processor -- start 2024-03-24T10:58:38.549 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.549+0000 7fda4a2af640 1 -- start start 2024-03-24T10:58:38.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.550+0000 7fda4a2af640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda440cb270 0x7fda440cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:38.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.550+0000 7fda4a2af640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fda440cd960 0x7fda440cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:38.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.550+0000 7fda4a2af640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda440ccba0 con 0x7fda440cae90 2024-03-24T10:58:38.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.550+0000 7fda4a2af640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fda440cc9b0 con 0x7fda440cd960 2024-03-24T10:58:38.550 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.550+0000 7fda4a2af640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fda440cce40 con 0x7fda440cb270 2024-03-24T10:58:38.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.551+0000 7fda43fff640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fda440cae90 0x7fda440cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:47262/0 (socket says 172.21.15.5:47262) 2024-03-24T10:58:38.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.551+0000 7fda43fff640 1 -- 172.21.15.5:0/4176202819 learned_addr learned my addr 172.21.15.5:0/4176202819 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:58:38.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.551+0000 7fda4884c640 1 --2- 172.21.15.5:0/4176202819 >> v2:172.21.15.57:3300/0 conn(0x7fda440cd960 0x7fda440cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:38.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.551+0000 7fda437fe640 1 --2- 172.21.15.5:0/4176202819 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda440cb270 0x7fda440cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:38.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.553+0000 7fda42ffd640 1 -- 172.21.15.5:0/4176202819 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 245567103 0 0) 0x7fda440ccba0 con 0x7fda440cae90 2024-03-24T10:58:38.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.553+0000 7fda42ffd640 1 -- 172.21.15.5:0/4176202819 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda2c003520 con 0x7fda440cae90 2024-03-24T10:58:38.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.555+0000 7fda437fe640 1 -- 172.21.15.5:0/4176202819 >> v2:172.21.15.57:3300/0 conn(0x7fda440cd960 msgr2=0x7fda440cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:38.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.555+0000 7fda437fe640 1 --2- 172.21.15.5:0/4176202819 >> v2:172.21.15.57:3300/0 conn(0x7fda440cd960 0x7fda440cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:38.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.555+0000 7fda437fe640 1 -- 172.21.15.5:0/4176202819 >> v1:172.21.15.5:6789/0 conn(0x7fda440cae90 legacy=0x7fda440cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:38.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.555+0000 7fda437fe640 1 -- 172.21.15.5:0/4176202819 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda4415a590 con 0x7fda440cb270 2024-03-24T10:58:38.558 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.558+0000 7fda437fe640 1 --2- 172.21.15.5:0/4176202819 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda440cb270 0x7fda440cf530 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fda380097a0 tx=0x7fda3802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c4db75e334e2491 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:38.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.562+0000 7fda42ffd640 1 -- 172.21.15.5:0/4176202819 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda38030610 con 0x7fda440cb270 2024-03-24T10:58:38.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.562+0000 7fda42ffd640 1 -- 172.21.15.5:0/4176202819 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda38030770 con 0x7fda440cb270 2024-03-24T10:58:38.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.562+0000 7fda42ffd640 1 -- 172.21.15.5:0/4176202819 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda38030b00 con 0x7fda440cb270 2024-03-24T10:58:38.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.563+0000 7fda4a2af640 1 -- 172.21.15.5:0/4176202819 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda440cb270 msgr2=0x7fda440cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:38.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.563+0000 7fda4a2af640 1 --2- 172.21.15.5:0/4176202819 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda440cb270 0x7fda440cf530 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fda380097a0 tx=0x7fda3802b960 comp rx=0 tx=0).stop 2024-03-24T10:58:38.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.563+0000 7fda4a2af640 1 -- 172.21.15.5:0/4176202819 shutdown_connections 2024-03-24T10:58:38.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.563+0000 7fda4a2af640 1 --2- 172.21.15.5:0/4176202819 >> v2:172.21.15.57:3300/0 conn(0x7fda440cd960 0x7fda440cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:38.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.563+0000 7fda4a2af640 1 --2- 172.21.15.5:0/4176202819 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda440cb270 0x7fda440cf530 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:38.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.563+0000 7fda4a2af640 1 -- 172.21.15.5:0/4176202819 >> 172.21.15.5:0/4176202819 conn(0x7fda440cfb80 msgr2=0x7fda440cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:38.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.564+0000 7fda4a2af640 1 -- 172.21.15.5:0/4176202819 shutdown_connections 2024-03-24T10:58:38.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.564+0000 7fda4a2af640 1 -- 172.21.15.5:0/4176202819 wait complete. 2024-03-24T10:58:38.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.565+0000 7fda4a2af640 1 Processor -- start 2024-03-24T10:58:38.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.566+0000 7fda4a2af640 1 -- start start 2024-03-24T10:58:38.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.567+0000 7fda4a2af640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda440cae90 0x7fda440ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:38.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.567+0000 7fda4a2af640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda440cb270 0x7fda440abad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:38.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.567+0000 7fda4a2af640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda440cd960 0x7fda440b10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:38.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.567+0000 7fda4a2af640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fda441f3510 con 0x7fda440cd960 2024-03-24T10:58:38.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.567+0000 7fda4a2af640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fda441f3a00 con 0x7fda440cae90 2024-03-24T10:58:38.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.567+0000 7fda4a2af640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fda441f3720 con 0x7fda440cb270 2024-03-24T10:58:38.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.567+0000 7fda4884c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda440cd960 0x7fda440b10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:38.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.567+0000 7fda437fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda440cb270 0x7fda440abad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:38.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.568+0000 7fda43fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda440cae90 0x7fda440ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:38.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.568+0000 7fda4884c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda440cd960 0x7fda440b10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38958/0 (socket says 172.21.15.5:38958) 2024-03-24T10:58:38.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.568+0000 7fda4884c640 1 -- 172.21.15.5:0/1197132916 learned_addr learned my addr 172.21.15.5:0/1197132916 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:38.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.571+0000 7fda4884c640 1 -- 172.21.15.5:0/1197132916 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda440cb270 msgr2=0x7fda440abad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:38.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.571+0000 7fda4884c640 1 --2- 172.21.15.5:0/1197132916 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda440cb270 0x7fda440abad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:38.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.571+0000 7fda4884c640 1 -- 172.21.15.5:0/1197132916 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda440cae90 msgr2=0x7fda440ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:38.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.571+0000 7fda4884c640 1 --2- 172.21.15.5:0/1197132916 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda440cae90 0x7fda440ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:38.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.571+0000 7fda4884c640 1 -- 172.21.15.5:0/1197132916 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda38009400 con 0x7fda440cd960 2024-03-24T10:58:38.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.573+0000 7fda4884c640 1 --2- 172.21.15.5:0/1197132916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda440cd960 0x7fda440b10a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fda3c00efc0 tx=0x7fda3c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:38.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.575+0000 7fda40ff9640 1 -- 172.21.15.5:0/1197132916 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c009ba0 con 0x7fda440cd960 2024-03-24T10:58:38.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.575+0000 7fda4a2af640 1 -- 172.21.15.5:0/1197132916 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda441ef620 con 0x7fda440cd960 2024-03-24T10:58:38.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.576+0000 7fda4a2af640 1 -- 172.21.15.5:0/1197132916 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda441efbe0 con 0x7fda440cd960 2024-03-24T10:58:38.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.576+0000 7fda4a2af640 1 -- 172.21.15.5:0/1197132916 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda440681d0 con 0x7fda440cd960 2024-03-24T10:58:38.599 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.599+0000 7fda40ff9640 1 -- 172.21.15.5:0/1197132916 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda3c00f040 con 0x7fda440cd960 2024-03-24T10:58:38.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.599+0000 7fda40ff9640 1 -- 172.21.15.5:0/1197132916 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c014680 con 0x7fda440cd960 2024-03-24T10:58:38.600 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.600+0000 7fda40ff9640 1 -- 172.21.15.5:0/1197132916 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fda3c014820 con 0x7fda440cd960 2024-03-24T10:58:38.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.601+0000 7fda40ff9640 1 --2- 172.21.15.5:0/1197132916 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda3006b810 0x7fda3006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:38.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.601+0000 7fda43fff640 1 --2- 172.21.15.5:0/1197132916 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda3006b810 0x7fda3006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:38.601 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.601+0000 7fda43fff640 1 --2- 172.21.15.5:0/1197132916 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda3006b810 0x7fda3006dcd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fda34005880 tx=0x7fda34009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:38.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.864+0000 7fda40ff9640 1 -- 172.21.15.5:0/1197132916 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fda3c059f20 con 0x7fda440cd960 2024-03-24T10:58:38.864 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:38.864+0000 7fda40ff9640 1 -- 172.21.15.5:0/1197132916 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fda3c0561c0 con 0x7fda440cd960 2024-03-24T10:58:39.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:39.082+0000 7fda4a2af640 1 -- 172.21.15.5:0/1197132916 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda44159e60 con 0x7fda440cd960 2024-03-24T10:58:39.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:39.095+0000 7fda40ff9640 1 -- 172.21.15.5:0/1197132916 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7fda3c059880 con 0x7fda440cd960 2024-03-24T10:58:39.095 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:39.095 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:39.095 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:39.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:39.098+0000 7fda4a2af640 1 -- 172.21.15.5:0/1197132916 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda3006b810 msgr2=0x7fda3006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:39.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:39.098+0000 7fda4a2af640 1 --2- 172.21.15.5:0/1197132916 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda3006b810 0x7fda3006dcd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fda34005880 tx=0x7fda34009040 comp rx=0 tx=0).stop 2024-03-24T10:58:39.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:39.098+0000 7fda4a2af640 1 -- 172.21.15.5:0/1197132916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda440cd960 msgr2=0x7fda440b10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:39.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:39.098+0000 7fda4a2af640 1 --2- 172.21.15.5:0/1197132916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda440cd960 0x7fda440b10a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fda3c00efc0 tx=0x7fda3c00c490 comp rx=0 tx=0).stop 2024-03-24T10:58:39.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:39.098+0000 7fda4a2af640 1 -- 172.21.15.5:0/1197132916 shutdown_connections 2024-03-24T10:58:39.098 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:39.098+0000 7fda4a2af640 1 --2- 172.21.15.5:0/1197132916 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fda3006b810 0x7fda3006dcd0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:39.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:39.098+0000 7fda4a2af640 1 --2- 172.21.15.5:0/1197132916 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fda440cd960 0x7fda440b10a0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:39.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:39.098+0000 7fda4a2af640 1 --2- 172.21.15.5:0/1197132916 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fda440cb270 0x7fda440abad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:39.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:39.098+0000 7fda4a2af640 1 --2- 172.21.15.5:0/1197132916 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fda440cae90 0x7fda440ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:39.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:39.098+0000 7fda4a2af640 1 -- 172.21.15.5:0/1197132916 >> 172.21.15.5:0/1197132916 conn(0x7fda440cfb80 msgr2=0x7fda440c7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:39.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:39.098+0000 7fda4a2af640 1 -- 172.21.15.5:0/1197132916 shutdown_connections 2024-03-24T10:58:39.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:39.098+0000 7fda4a2af640 1 -- 172.21.15.5:0/1197132916 wait complete. 2024-03-24T10:58:39.713 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:39.713 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:40.714 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:40.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.817+0000 7f370f6ed640 1 Processor -- start 2024-03-24T10:58:40.817 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.817+0000 7f370f6ed640 1 -- start start 2024-03-24T10:58:40.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.818+0000 7f370f6ed640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f370806bf50 0x7f370806c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:40.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.818+0000 7f370f6ed640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37080cb2f0 con 0x7f370815ac00 2024-03-24T10:58:40.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.818+0000 7f370f6ed640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f370815b070 con 0x7f37081543a0 2024-03-24T10:58:40.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.818+0000 7f370f6ed640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f37080c6750 con 0x7f370806bf50 2024-03-24T10:58:40.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.818+0000 7f370cc88640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f370815ac00 0x7f37080cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:47274/0 (socket says 172.21.15.5:47274) 2024-03-24T10:58:40.818 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.818+0000 7f370cc88640 1 -- 172.21.15.5:0/3823080936 learned_addr learned my addr 172.21.15.5:0/3823080936 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:58:40.819 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.819+0000 7f370dc8a640 1 --2- 172.21.15.5:0/3823080936 >> v2:172.21.15.57:3300/0 conn(0x7f370806bf50 0x7f370806c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:40.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.821+0000 7f36fffff640 1 -- 172.21.15.5:0/3823080936 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1243948572 0 0) 0x7f37080cb2f0 con 0x7f370815ac00 2024-03-24T10:58:40.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.821+0000 7f36fffff640 1 -- 172.21.15.5:0/3823080936 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36f4003520 con 0x7f370815ac00 2024-03-24T10:58:40.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.823+0000 7f370dc8a640 1 -- 172.21.15.5:0/3823080936 >> v1:172.21.15.57:6789/0 conn(0x7f37081543a0 legacy=0x7f370806d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:40.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.823+0000 7f370dc8a640 1 -- 172.21.15.5:0/3823080936 >> v1:172.21.15.5:6789/0 conn(0x7f370815ac00 legacy=0x7f37080cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:40.823 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.823+0000 7f370dc8a640 1 -- 172.21.15.5:0/3823080936 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37080ca7e0 con 0x7f370806bf50 2024-03-24T10:58:40.827 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.828+0000 7f370dc8a640 1 --2- 172.21.15.5:0/3823080936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f370806bf50 0x7f370806c330 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3700009d90 tx=0x7f370002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa60a25bb3067ce7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:40.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.833+0000 7f36fffff640 1 -- 172.21.15.5:0/3823080936 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370002cd00 con 0x7f370806bf50 2024-03-24T10:58:40.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.833+0000 7f36fffff640 1 -- 172.21.15.5:0/3823080936 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f370002ce60 con 0x7f370806bf50 2024-03-24T10:58:40.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.833+0000 7f36fffff640 1 -- 172.21.15.5:0/3823080936 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3700034660 con 0x7f370806bf50 2024-03-24T10:58:40.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.833+0000 7f370f6ed640 1 -- 172.21.15.5:0/3823080936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f370806bf50 msgr2=0x7f370806c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:40.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.833+0000 7f370f6ed640 1 --2- 172.21.15.5:0/3823080936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f370806bf50 0x7f370806c330 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f3700009d90 tx=0x7f370002bf50 comp rx=0 tx=0).stop 2024-03-24T10:58:40.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.833+0000 7f370f6ed640 1 -- 172.21.15.5:0/3823080936 shutdown_connections 2024-03-24T10:58:40.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.833+0000 7f370f6ed640 1 --2- 172.21.15.5:0/3823080936 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f370806bf50 0x7f370806c330 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:40.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.833+0000 7f370f6ed640 1 -- 172.21.15.5:0/3823080936 >> 172.21.15.5:0/3823080936 conn(0x7f3708072ef0 msgr2=0x7f3708061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:40.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.834+0000 7f370f6ed640 1 -- 172.21.15.5:0/3823080936 shutdown_connections 2024-03-24T10:58:40.834 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.834+0000 7f370f6ed640 1 -- 172.21.15.5:0/3823080936 wait complete. 2024-03-24T10:58:40.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.835+0000 7f370f6ed640 1 Processor -- start 2024-03-24T10:58:40.835 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.835+0000 7f370f6ed640 1 -- start start 2024-03-24T10:58:40.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.835+0000 7f370f6ed640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f370806bf50 0x7f37080ceb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:40.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.835+0000 7f370f6ed640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37081543a0 0x7f3708143b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:40.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.835+0000 7f370f6ed640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f370815ac00 0x7f3708144040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:40.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.835+0000 7f370f6ed640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f3708156290 con 0x7f370806bf50 2024-03-24T10:58:40.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.835+0000 7f370f6ed640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f37080b4c60 con 0x7f37081543a0 2024-03-24T10:58:40.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.835+0000 7f370f6ed640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f37080b3a60 con 0x7f370815ac00 2024-03-24T10:58:40.836 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.836+0000 7f370d489640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f370806bf50 0x7f37080ceb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:40.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.836+0000 7f370dc8a640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f370815ac00 0x7f3708144040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:40.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.836+0000 7f370cc88640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37081543a0 0x7f3708143b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:40.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.836+0000 7f370d489640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f370806bf50 0x7f37080ceb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:38964/0 (socket says 172.21.15.5:38964) 2024-03-24T10:58:40.837 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.836+0000 7f370d489640 1 -- 172.21.15.5:0/1863835391 learned_addr learned my addr 172.21.15.5:0/1863835391 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:40.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.840+0000 7f370d489640 1 -- 172.21.15.5:0/1863835391 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f370815ac00 msgr2=0x7f3708144040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:40.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.840+0000 7f370d489640 1 --2- 172.21.15.5:0/1863835391 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f370815ac00 0x7f3708144040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:40.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.840+0000 7f370d489640 1 -- 172.21.15.5:0/1863835391 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37081543a0 msgr2=0x7f3708143b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:40.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.840+0000 7f370d489640 1 --2- 172.21.15.5:0/1863835391 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37081543a0 0x7f3708143b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:40.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.840+0000 7f370d489640 1 -- 172.21.15.5:0/1863835391 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3700009a70 con 0x7f370806bf50 2024-03-24T10:58:40.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.842+0000 7f370d489640 1 --2- 172.21.15.5:0/1863835391 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f370806bf50 0x7f37080ceb90 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f370400c450 tx=0x7f370400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:40.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.846+0000 7f36fdffb640 1 -- 172.21.15.5:0/1863835391 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37040052d0 con 0x7f370806bf50 2024-03-24T10:58:40.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.846+0000 7f36fdffb640 1 -- 172.21.15.5:0/1863835391 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3704005430 con 0x7f370806bf50 2024-03-24T10:58:40.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.846+0000 7f370f6ed640 1 -- 172.21.15.5:0/1863835391 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37080b3cd0 con 0x7f370806bf50 2024-03-24T10:58:40.846 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.846+0000 7f36fdffb640 1 -- 172.21.15.5:0/1863835391 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3704012870 con 0x7f370806bf50 2024-03-24T10:58:40.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.846+0000 7f370f6ed640 1 -- 172.21.15.5:0/1863835391 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3708155f20 con 0x7f370806bf50 2024-03-24T10:58:40.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:40.847+0000 7f370f6ed640 1 -- 172.21.15.5:0/1863835391 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37081560e0 con 0x7f370806bf50 2024-03-24T10:58:41.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:41.114+0000 7f36fdffb640 1 -- 172.21.15.5:0/1863835391 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f37040129d0 con 0x7f370806bf50 2024-03-24T10:58:41.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:41.114+0000 7f36fdffb640 1 --2- 172.21.15.5:0/1863835391 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f36ec06b860 0x7f36ec06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:41.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:41.114+0000 7f370cc88640 1 --2- 172.21.15.5:0/1863835391 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f36ec06b860 0x7f36ec06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:41.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:41.115+0000 7f370cc88640 1 --2- 172.21.15.5:0/1863835391 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f36ec06b860 0x7f36ec06dd20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f370814e2c0 tx=0x7f36f8007d30 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:41.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:41.131+0000 7f36fdffb640 1 -- 172.21.15.5:0/1863835391 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f3704058e80 con 0x7f370806bf50 2024-03-24T10:58:41.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:41.131+0000 7f36fdffb640 1 -- 172.21.15.5:0/1863835391 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3704055120 con 0x7f370806bf50 2024-03-24T10:58:41.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:41.351+0000 7f370f6ed640 1 -- 172.21.15.5:0/1863835391 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37080681d0 con 0x7f370806bf50 2024-03-24T10:58:41.372 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:41.372+0000 7f36fdffb640 1 -- 172.21.15.5:0/1863835391 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f37040587e0 con 0x7f370806bf50 2024-03-24T10:58:41.372 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:41.372 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:41.373 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:41.374 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:41.374+0000 7f370f6ed640 1 -- 172.21.15.5:0/1863835391 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f36ec06b860 msgr2=0x7f36ec06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:41.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:41.374+0000 7f370f6ed640 1 --2- 172.21.15.5:0/1863835391 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f36ec06b860 0x7f36ec06dd20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f370814e2c0 tx=0x7f36f8007d30 comp rx=0 tx=0).stop 2024-03-24T10:58:41.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:41.374+0000 7f370f6ed640 1 -- 172.21.15.5:0/1863835391 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f370806bf50 msgr2=0x7f37080ceb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:41.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:41.374+0000 7f370f6ed640 1 --2- 172.21.15.5:0/1863835391 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f370806bf50 0x7f37080ceb90 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f370400c450 tx=0x7f370400c910 comp rx=0 tx=0).stop 2024-03-24T10:58:41.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:41.374+0000 7f370f6ed640 1 -- 172.21.15.5:0/1863835391 shutdown_connections 2024-03-24T10:58:41.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:41.374+0000 7f370f6ed640 1 --2- 172.21.15.5:0/1863835391 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f36ec06b860 0x7f36ec06dd20 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:41.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:41.374+0000 7f370f6ed640 1 --2- 172.21.15.5:0/1863835391 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f370815ac00 0x7f3708144040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:41.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:41.374+0000 7f370f6ed640 1 --2- 172.21.15.5:0/1863835391 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f37081543a0 0x7f3708143b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:41.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:41.374+0000 7f370f6ed640 1 --2- 172.21.15.5:0/1863835391 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f370806bf50 0x7f37080ceb90 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:41.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:41.374+0000 7f370f6ed640 1 -- 172.21.15.5:0/1863835391 >> 172.21.15.5:0/1863835391 conn(0x7f3708072ef0 msgr2=0x7f37080cc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:41.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:41.374+0000 7f370f6ed640 1 -- 172.21.15.5:0/1863835391 shutdown_connections 2024-03-24T10:58:41.375 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:41.374+0000 7f370f6ed640 1 -- 172.21.15.5:0/1863835391 wait complete. 2024-03-24T10:58:41.990 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:41.990 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:42.992 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:43.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.094+0000 7f8e1f674640 1 Processor -- start 2024-03-24T10:58:43.094 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.094+0000 7f8e1f674640 1 -- start start 2024-03-24T10:58:43.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.095+0000 7f8e1f674640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8e1806c580 0x7f8e181538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:43.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.095+0000 7f8e1f674640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8e180cf110 0x7f8e18153df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:43.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.095+0000 7f8e1f674640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e1815bce0 con 0x7f8e180cc8f0 2024-03-24T10:58:43.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.095+0000 7f8e1f674640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f8e180cd890 con 0x7f8e1806c580 2024-03-24T10:58:43.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.095+0000 7f8e1f674640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f8e1806cae0 con 0x7f8e180cf110 2024-03-24T10:58:43.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.096+0000 7f8e1cc0f640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8e1806c580 0x7f8e181538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:43.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.096+0000 7f8e1dc11640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8e180cf110 0x7f8e18153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:43.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.096+0000 7f8e1d410640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f8e180cc8f0 0x7f8e1815b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:46426/0 (socket says 172.21.15.5:46426) 2024-03-24T10:58:43.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.096+0000 7f8e1d410640 1 -- 172.21.15.5:0/2685276693 learned_addr learned my addr 172.21.15.5:0/2685276693 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:58:43.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.099+0000 7f8e1cc0f640 1 -- 172.21.15.5:0/2685276693 >> v2:172.21.15.57:3300/0 conn(0x7f8e180cf110 msgr2=0x7f8e18153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:43.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.099+0000 7f8e1cc0f640 1 --2- 172.21.15.5:0/2685276693 >> v2:172.21.15.57:3300/0 conn(0x7f8e180cf110 0x7f8e18153df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:43.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.099+0000 7f8e1cc0f640 1 -- 172.21.15.5:0/2685276693 >> v1:172.21.15.57:6789/0 conn(0x7f8e180cc8f0 legacy=0x7f8e1815b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:43.099 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.099+0000 7f8e1cc0f640 1 -- 172.21.15.5:0/2685276693 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e180cdb50 con 0x7f8e1806c580 2024-03-24T10:58:43.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.101+0000 7f8e1cc0f640 1 --2- 172.21.15.5:0/2685276693 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e1806c580 0x7f8e181538b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f8e1806e100 tx=0x7f8e0802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f241e1bbbf0e09 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:43.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.106+0000 7f8e0ffff640 1 -- 172.21.15.5:0/2685276693 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e08030610 con 0x7f8e1806c580 2024-03-24T10:58:43.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.106+0000 7f8e0ffff640 1 -- 172.21.15.5:0/2685276693 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e08030770 con 0x7f8e1806c580 2024-03-24T10:58:43.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.106+0000 7f8e0ffff640 1 -- 172.21.15.5:0/2685276693 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e08030b00 con 0x7f8e1806c580 2024-03-24T10:58:43.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.106+0000 7f8e1f674640 1 -- 172.21.15.5:0/2685276693 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e1806c580 msgr2=0x7f8e181538b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:43.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.106+0000 7f8e1f674640 1 --2- 172.21.15.5:0/2685276693 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e1806c580 0x7f8e181538b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f8e1806e100 tx=0x7f8e0802b960 comp rx=0 tx=0).stop 2024-03-24T10:58:43.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.106+0000 7f8e1f674640 1 -- 172.21.15.5:0/2685276693 shutdown_connections 2024-03-24T10:58:43.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.106+0000 7f8e1f674640 1 --2- 172.21.15.5:0/2685276693 >> v2:172.21.15.57:3300/0 conn(0x7f8e180cf110 0x7f8e18153df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:43.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.106+0000 7f8e1f674640 1 --2- 172.21.15.5:0/2685276693 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e1806c580 0x7f8e181538b0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:43.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.106+0000 7f8e1f674640 1 -- 172.21.15.5:0/2685276693 >> 172.21.15.5:0/2685276693 conn(0x7f8e180b1820 msgr2=0x7f8e180b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:43.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.107+0000 7f8e1f674640 1 -- 172.21.15.5:0/2685276693 shutdown_connections 2024-03-24T10:58:43.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.107+0000 7f8e1f674640 1 -- 172.21.15.5:0/2685276693 wait complete. 2024-03-24T10:58:43.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.108+0000 7f8e1f674640 1 Processor -- start 2024-03-24T10:58:43.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.108+0000 7f8e1f674640 1 -- start start 2024-03-24T10:58:43.109 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.109+0000 7f8e1f674640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e1806c580 0x7f8e181e30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:43.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.109+0000 7f8e1f674640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e180cc8f0 0x7f8e18152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:43.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.109+0000 7f8e1f674640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8e180cf110 0x7f8e18155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:43.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.109+0000 7f8e1f674640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f8e181efab0 con 0x7f8e1806c580 2024-03-24T10:58:43.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.109+0000 7f8e1f674640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8e18155a20 con 0x7f8e180cc8f0 2024-03-24T10:58:43.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.109+0000 7f8e1f674640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f8e181efcc0 con 0x7f8e180cf110 2024-03-24T10:58:43.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.109+0000 7f8e1d410640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e1806c580 0x7f8e181e30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:43.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.109+0000 7f8e1cc0f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e180cc8f0 0x7f8e18152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:43.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.110+0000 7f8e1d410640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e1806c580 0x7f8e181e30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57872/0 (socket says 172.21.15.5:57872) 2024-03-24T10:58:43.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.110+0000 7f8e1dc11640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8e180cf110 0x7f8e18155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:43.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.110+0000 7f8e1d410640 1 -- 172.21.15.5:0/1090580399 learned_addr learned my addr 172.21.15.5:0/1090580399 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:43.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.113+0000 7f8e1d410640 1 -- 172.21.15.5:0/1090580399 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8e180cf110 msgr2=0x7f8e18155eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:43.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.113+0000 7f8e1d410640 1 --2- 172.21.15.5:0/1090580399 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8e180cf110 0x7f8e18155eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:43.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.113+0000 7f8e1d410640 1 -- 172.21.15.5:0/1090580399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e180cc8f0 msgr2=0x7f8e18152fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:43.113 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.113+0000 7f8e1d410640 1 --2- 172.21.15.5:0/1090580399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e180cc8f0 0x7f8e18152fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:43.114 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.113+0000 7f8e1d410640 1 -- 172.21.15.5:0/1090580399 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e08009400 con 0x7f8e1806c580 2024-03-24T10:58:43.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.115+0000 7f8e1d410640 1 --2- 172.21.15.5:0/1090580399 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e1806c580 0x7f8e181e30a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400c4a0 tx=0x7f8e1400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:43.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.117+0000 7f8e0dffb640 1 -- 172.21.15.5:0/1090580399 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e14005ce0 con 0x7f8e1806c580 2024-03-24T10:58:43.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.117+0000 7f8e1f674640 1 -- 172.21.15.5:0/1090580399 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e181effa0 con 0x7f8e1806c580 2024-03-24T10:58:43.117 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.118+0000 7f8e1f674640 1 -- 172.21.15.5:0/1090580399 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e181f0560 con 0x7f8e1806c580 2024-03-24T10:58:43.118 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.118+0000 7f8e1f674640 1 -- 172.21.15.5:0/1090580399 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e180681d0 con 0x7f8e1806c580 2024-03-24T10:58:43.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.386+0000 7f8e0dffb640 1 -- 172.21.15.5:0/1090580399 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e14005e40 con 0x7f8e1806c580 2024-03-24T10:58:43.386 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.386+0000 7f8e0dffb640 1 -- 172.21.15.5:0/1090580399 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e14007270 con 0x7f8e1806c580 2024-03-24T10:58:43.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.387+0000 7f8e0dffb640 1 -- 172.21.15.5:0/1090580399 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8e14007490 con 0x7f8e1806c580 2024-03-24T10:58:43.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.387+0000 7f8e0dffb640 1 --2- 172.21.15.5:0/1090580399 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8dfc06b810 0x7f8dfc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:43.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.387+0000 7f8e1cc0f640 1 --2- 172.21.15.5:0/1090580399 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8dfc06b810 0x7f8dfc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:43.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.388+0000 7f8e1cc0f640 1 --2- 172.21.15.5:0/1090580399 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8dfc06b810 0x7f8dfc06dcd0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8e08036000 tx=0x7f8e08008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:43.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.402+0000 7f8e0dffb640 1 -- 172.21.15.5:0/1090580399 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f8e14058760 con 0x7f8e1806c580 2024-03-24T10:58:43.402 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.402+0000 7f8e0dffb640 1 -- 172.21.15.5:0/1090580399 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8e14054a00 con 0x7f8e1806c580 2024-03-24T10:58:43.651 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.651+0000 7f8e1f674640 1 -- 172.21.15.5:0/1090580399 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e180cf720 con 0x7f8e1806c580 2024-03-24T10:58:43.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.698+0000 7f8e0dffb640 1 -- 172.21.15.5:0/1090580399 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f8e14021090 con 0x7f8e1806c580 2024-03-24T10:58:43.700 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:43.700 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:43.700 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:43.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.701+0000 7f8e1f674640 1 -- 172.21.15.5:0/1090580399 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8dfc06b810 msgr2=0x7f8dfc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:43.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.701+0000 7f8e1f674640 1 --2- 172.21.15.5:0/1090580399 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8dfc06b810 0x7f8dfc06dcd0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8e08036000 tx=0x7f8e08008030 comp rx=0 tx=0).stop 2024-03-24T10:58:43.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.701+0000 7f8e1f674640 1 -- 172.21.15.5:0/1090580399 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e1806c580 msgr2=0x7f8e181e30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:43.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.701+0000 7f8e1f674640 1 --2- 172.21.15.5:0/1090580399 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e1806c580 0x7f8e181e30a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f8e1400c4a0 tx=0x7f8e1400c960 comp rx=0 tx=0).stop 2024-03-24T10:58:43.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.701+0000 7f8e1f674640 1 -- 172.21.15.5:0/1090580399 shutdown_connections 2024-03-24T10:58:43.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.701+0000 7f8e1f674640 1 --2- 172.21.15.5:0/1090580399 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8dfc06b810 0x7f8dfc06dcd0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:43.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.701+0000 7f8e1f674640 1 --2- 172.21.15.5:0/1090580399 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8e180cf110 0x7f8e18155eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:43.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.701+0000 7f8e1f674640 1 --2- 172.21.15.5:0/1090580399 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8e180cc8f0 0x7f8e18152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:43.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.701+0000 7f8e1f674640 1 --2- 172.21.15.5:0/1090580399 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8e1806c580 0x7f8e181e30a0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:43.703 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.701+0000 7f8e1f674640 1 -- 172.21.15.5:0/1090580399 >> 172.21.15.5:0/1090580399 conn(0x7f8e180b1820 msgr2=0x7f8e1815be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:43.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.701+0000 7f8e1f674640 1 -- 172.21.15.5:0/1090580399 shutdown_connections 2024-03-24T10:58:43.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:43.701+0000 7f8e1f674640 1 -- 172.21.15.5:0/1090580399 wait complete. 2024-03-24T10:58:44.318 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:44.318 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:45.319 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:45.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.423+0000 7fe70adf4640 1 Processor -- start 2024-03-24T10:58:45.424 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.423+0000 7fe70adf4640 1 -- start start 2024-03-24T10:58:45.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.424+0000 7fe70adf4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe7041543a0 0x7fe70415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:45.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.424+0000 7fe70adf4640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fe70406d4f0 0x7fe7040cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:45.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.424+0000 7fe70adf4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7040cb2f0 con 0x7fe7040ccaa0 2024-03-24T10:58:45.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.424+0000 7fe70adf4640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fe7040c6750 con 0x7fe70406d4f0 2024-03-24T10:58:45.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.424+0000 7fe70adf4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe7040cc380 con 0x7fe7041543a0 2024-03-24T10:58:45.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.425+0000 7fe708b90640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe7041543a0 0x7fe70415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:45.425 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.425+0000 7fe709391640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fe7040ccaa0 0x7fe70406bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:33332/0 (socket says 172.21.15.5:33332) 2024-03-24T10:58:45.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.425+0000 7fe709391640 1 -- 172.21.15.5:0/912190834 learned_addr learned my addr 172.21.15.5:0/912190834 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:58:45.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.425+0000 7fe703fff640 1 --2- 172.21.15.5:0/912190834 >> v2:172.21.15.57:3300/0 conn(0x7fe70406d4f0 0x7fe7040cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:45.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.428+0000 7fe703fff640 1 -- 172.21.15.5:0/912190834 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe7041543a0 msgr2=0x7fe70415ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:45.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.428+0000 7fe703fff640 1 --2- 172.21.15.5:0/912190834 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe7041543a0 0x7fe70415ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:45.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.428+0000 7fe703fff640 1 -- 172.21.15.5:0/912190834 >> v1:172.21.15.5:6789/0 conn(0x7fe7040ccaa0 legacy=0x7fe70406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:45.428 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.428+0000 7fe703fff640 1 -- 172.21.15.5:0/912190834 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7040ca7e0 con 0x7fe70406d4f0 2024-03-24T10:58:45.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.430+0000 7fe703fff640 1 --2- 172.21.15.5:0/912190834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe70406d4f0 0x7fe7040cc560 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fe6f00097a0 tx=0x7fe6f002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb513ce0b7a7135 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:45.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.435+0000 7fe7037fe640 1 -- 172.21.15.5:0/912190834 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0030610 con 0x7fe70406d4f0 2024-03-24T10:58:45.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.435+0000 7fe7037fe640 1 -- 172.21.15.5:0/912190834 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6f0030770 con 0x7fe70406d4f0 2024-03-24T10:58:45.437 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.435+0000 7fe7037fe640 1 -- 172.21.15.5:0/912190834 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0030b00 con 0x7fe70406d4f0 2024-03-24T10:58:45.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.436+0000 7fe70adf4640 1 -- 172.21.15.5:0/912190834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe70406d4f0 msgr2=0x7fe7040cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:45.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.436+0000 7fe70adf4640 1 --2- 172.21.15.5:0/912190834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe70406d4f0 0x7fe7040cc560 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fe6f00097a0 tx=0x7fe6f002b960 comp rx=0 tx=0).stop 2024-03-24T10:58:45.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.436+0000 7fe70adf4640 1 -- 172.21.15.5:0/912190834 shutdown_connections 2024-03-24T10:58:45.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.436+0000 7fe70adf4640 1 --2- 172.21.15.5:0/912190834 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe70406d4f0 0x7fe7040cc560 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:45.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.436+0000 7fe70adf4640 1 --2- 172.21.15.5:0/912190834 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe7041543a0 0x7fe70415ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:45.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.436+0000 7fe70adf4640 1 -- 172.21.15.5:0/912190834 >> 172.21.15.5:0/912190834 conn(0x7fe704072ef0 msgr2=0x7fe704061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:45.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.436+0000 7fe70adf4640 1 -- 172.21.15.5:0/912190834 shutdown_connections 2024-03-24T10:58:45.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.436+0000 7fe70adf4640 1 -- 172.21.15.5:0/912190834 wait complete. 2024-03-24T10:58:45.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.437+0000 7fe70adf4640 1 Processor -- start 2024-03-24T10:58:45.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.437+0000 7fe70adf4640 1 -- start start 2024-03-24T10:58:45.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.438+0000 7fe70adf4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe70406d4f0 0x7fe704159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:45.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.438+0000 7fe70adf4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7040ccaa0 0x7fe7041f2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:45.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.438+0000 7fe70adf4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe7041543a0 0x7fe7041495c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:45.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.438+0000 7fe70adf4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fe704158d80 con 0x7fe7040ccaa0 2024-03-24T10:58:45.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.438+0000 7fe70adf4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fe7041f3a70 con 0x7fe7041543a0 2024-03-24T10:58:45.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.438+0000 7fe70adf4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fe70415b350 con 0x7fe70406d4f0 2024-03-24T10:58:45.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.438+0000 7fe703fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7040ccaa0 0x7fe7041f2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:45.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.439+0000 7fe709391640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe7041543a0 0x7fe7041495c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:45.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.439+0000 7fe708b90640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe70406d4f0 0x7fe704159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:45.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.439+0000 7fe703fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7040ccaa0 0x7fe7041f2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57882/0 (socket says 172.21.15.5:57882) 2024-03-24T10:58:45.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.439+0000 7fe703fff640 1 -- 172.21.15.5:0/536655013 learned_addr learned my addr 172.21.15.5:0/536655013 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:45.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.442+0000 7fe703fff640 1 -- 172.21.15.5:0/536655013 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe70406d4f0 msgr2=0x7fe704159210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:45.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.442+0000 7fe703fff640 1 --2- 172.21.15.5:0/536655013 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe70406d4f0 0x7fe704159210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:45.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.442+0000 7fe703fff640 1 -- 172.21.15.5:0/536655013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe7041543a0 msgr2=0x7fe7041495c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:45.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.442+0000 7fe703fff640 1 --2- 172.21.15.5:0/536655013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe7041543a0 0x7fe7041495c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:45.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.443+0000 7fe703fff640 1 -- 172.21.15.5:0/536655013 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6f0009400 con 0x7fe7040ccaa0 2024-03-24T10:58:45.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.445+0000 7fe703fff640 1 --2- 172.21.15.5:0/536655013 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7040ccaa0 0x7fe7041f2fb0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe6f002bf70 tx=0x7fe6f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:45.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.449+0000 7fe7017fa640 1 -- 172.21.15.5:0/536655013 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f003e070 con 0x7fe7040ccaa0 2024-03-24T10:58:45.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.449+0000 7fe7017fa640 1 -- 172.21.15.5:0/536655013 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6f00397a0 con 0x7fe7040ccaa0 2024-03-24T10:58:45.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.449+0000 7fe7017fa640 1 -- 172.21.15.5:0/536655013 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0039b30 con 0x7fe7040ccaa0 2024-03-24T10:58:45.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.449+0000 7fe70adf4640 1 -- 172.21.15.5:0/536655013 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe70415b630 con 0x7fe7040ccaa0 2024-03-24T10:58:45.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.449+0000 7fe70adf4640 1 -- 172.21.15.5:0/536655013 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe704158a10 con 0x7fe7040ccaa0 2024-03-24T10:58:45.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.450+0000 7fe70adf4640 1 -- 172.21.15.5:0/536655013 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7040681d0 con 0x7fe7040ccaa0 2024-03-24T10:58:45.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.733+0000 7fe7017fa640 1 -- 172.21.15.5:0/536655013 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe6f0039c90 con 0x7fe7040ccaa0 2024-03-24T10:58:45.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.734+0000 7fe7017fa640 1 --2- 172.21.15.5:0/536655013 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe6e806b860 0x7fe6e806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:45.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.734+0000 7fe708b90640 1 --2- 172.21.15.5:0/536655013 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe6e806b860 0x7fe6e806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:45.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.734+0000 7fe708b90640 1 --2- 172.21.15.5:0/536655013 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe6e806b860 0x7fe6e806dd20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8009b40 tx=0x7fe6f8009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:45.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.749+0000 7fe7017fa640 1 -- 172.21.15.5:0/536655013 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fe6f007ad00 con 0x7fe7040ccaa0 2024-03-24T10:58:45.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.749+0000 7fe7017fa640 1 -- 172.21.15.5:0/536655013 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe6f0076fa0 con 0x7fe7040ccaa0 2024-03-24T10:58:45.975 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.975+0000 7fe70adf4640 1 -- 172.21.15.5:0/536655013 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe704153a20 con 0x7fe7040ccaa0 2024-03-24T10:58:45.996 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.996+0000 7fe7017fa640 1 -- 172.21.15.5:0/536655013 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7fe6f0048070 con 0x7fe7040ccaa0 2024-03-24T10:58:45.997 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:45.997 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:45.997 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:45.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.999+0000 7fe70adf4640 1 -- 172.21.15.5:0/536655013 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe6e806b860 msgr2=0x7fe6e806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:45.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.999+0000 7fe70adf4640 1 --2- 172.21.15.5:0/536655013 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe6e806b860 0x7fe6e806dd20 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8009b40 tx=0x7fe6f8009190 comp rx=0 tx=0).stop 2024-03-24T10:58:45.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.999+0000 7fe70adf4640 1 -- 172.21.15.5:0/536655013 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7040ccaa0 msgr2=0x7fe7041f2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:45.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.999+0000 7fe70adf4640 1 --2- 172.21.15.5:0/536655013 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7040ccaa0 0x7fe7041f2fb0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe6f002bf70 tx=0x7fe6f00046c0 comp rx=0 tx=0).stop 2024-03-24T10:58:45.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.999+0000 7fe70adf4640 1 -- 172.21.15.5:0/536655013 shutdown_connections 2024-03-24T10:58:45.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.999+0000 7fe70adf4640 1 --2- 172.21.15.5:0/536655013 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fe6e806b860 0x7fe6e806dd20 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:45.999 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.999+0000 7fe70adf4640 1 --2- 172.21.15.5:0/536655013 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fe7040ccaa0 0x7fe7041f2fb0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:46.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.999+0000 7fe70adf4640 1 --2- 172.21.15.5:0/536655013 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fe7041543a0 0x7fe7041495c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:46.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.999+0000 7fe70adf4640 1 --2- 172.21.15.5:0/536655013 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fe70406d4f0 0x7fe704159210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:46.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.999+0000 7fe70adf4640 1 -- 172.21.15.5:0/536655013 >> 172.21.15.5:0/536655013 conn(0x7fe704072ef0 msgr2=0x7fe704152c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:46.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.999+0000 7fe70adf4640 1 -- 172.21.15.5:0/536655013 shutdown_connections 2024-03-24T10:58:46.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:45.999+0000 7fe70adf4640 1 -- 172.21.15.5:0/536655013 wait complete. 2024-03-24T10:58:46.614 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:46.614 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:47.616 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:47.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.717+0000 7f0be054f640 1 Processor -- start 2024-03-24T10:58:47.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.717+0000 7f0be054f640 1 -- start start 2024-03-24T10:58:47.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.718+0000 7f0be054f640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0bd80cc8f0 0x7f0bd815b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:47.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.718+0000 7f0be054f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0bd806c580 0x7f0bd81538b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:47.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.718+0000 7f0be054f640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bd80cd890 con 0x7f0bd80cf110 2024-03-24T10:58:47.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.718+0000 7f0be054f640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f0bd806cae0 con 0x7f0bd80cc8f0 2024-03-24T10:58:47.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.718+0000 7f0be054f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0bd80cdb50 con 0x7f0bd806c580 2024-03-24T10:58:47.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.719+0000 7f0bde2eb640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0bd80cc8f0 0x7f0bd815b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:47.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.719+0000 7f0bddaea640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0bd806c580 0x7f0bd81538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:47.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.719+0000 7f0bde2eb640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f0bd80cc8f0 0x7f0bd815b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57896/0 (socket says 172.21.15.5:57896) 2024-03-24T10:58:47.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.719+0000 7f0bde2eb640 1 -- 172.21.15.5:0/3863429508 learned_addr learned my addr 172.21.15.5:0/3863429508 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:47.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.722+0000 7f0bde2eb640 1 -- 172.21.15.5:0/3863429508 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0bd806c580 msgr2=0x7f0bd81538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:47.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.722+0000 7f0bde2eb640 1 --2- 172.21.15.5:0/3863429508 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0bd806c580 0x7f0bd81538b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:47.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.722+0000 7f0bde2eb640 1 -- 172.21.15.5:0/3863429508 >> v1:172.21.15.5:6789/0 conn(0x7f0bd80cf110 legacy=0x7f0bd8153df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:47.722 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.722+0000 7f0bde2eb640 1 -- 172.21.15.5:0/3863429508 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bd80cde30 con 0x7f0bd80cc8f0 2024-03-24T10:58:47.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.724+0000 7f0bde2eb640 1 --2- 172.21.15.5:0/3863429508 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0bd80cc8f0 0x7f0bd815b5d0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4009870 tx=0x7f0bd402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb4f8ec27b2c207 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:47.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.729+0000 7f0bdd2e9640 1 -- 172.21.15.5:0/3863429508 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd4031640 con 0x7f0bd80cc8f0 2024-03-24T10:58:47.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.729+0000 7f0bdd2e9640 1 -- 172.21.15.5:0/3863429508 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bd40317a0 con 0x7f0bd80cc8f0 2024-03-24T10:58:47.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.729+0000 7f0bdd2e9640 1 -- 172.21.15.5:0/3863429508 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd4031b30 con 0x7f0bd80cc8f0 2024-03-24T10:58:47.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.730+0000 7f0be054f640 1 -- 172.21.15.5:0/3863429508 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0bd80cc8f0 msgr2=0x7f0bd815b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:47.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.730+0000 7f0be054f640 1 --2- 172.21.15.5:0/3863429508 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0bd80cc8f0 0x7f0bd815b5d0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f0bd4009870 tx=0x7f0bd402b8f0 comp rx=0 tx=0).stop 2024-03-24T10:58:47.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.730+0000 7f0be054f640 1 -- 172.21.15.5:0/3863429508 shutdown_connections 2024-03-24T10:58:47.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.730+0000 7f0be054f640 1 --2- 172.21.15.5:0/3863429508 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0bd806c580 0x7f0bd81538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:47.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.730+0000 7f0be054f640 1 --2- 172.21.15.5:0/3863429508 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0bd80cc8f0 0x7f0bd815b5d0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:47.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.730+0000 7f0be054f640 1 -- 172.21.15.5:0/3863429508 >> 172.21.15.5:0/3863429508 conn(0x7f0bd80b1820 msgr2=0x7f0bd80b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:47.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.730+0000 7f0be054f640 1 -- 172.21.15.5:0/3863429508 shutdown_connections 2024-03-24T10:58:47.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.730+0000 7f0be054f640 1 -- 172.21.15.5:0/3863429508 wait complete. 2024-03-24T10:58:47.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.731+0000 7f0be054f640 1 Processor -- start 2024-03-24T10:58:47.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.732+0000 7f0be054f640 1 -- start start 2024-03-24T10:58:47.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.732+0000 7f0be054f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0bd806c580 0x7f0bd80cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:47.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.732+0000 7f0be054f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0bd80cc8f0 0x7f0bd80c4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:47.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.732+0000 7f0be054f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0bd80cf110 0x7f0bd81510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:47.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.732+0000 7f0be054f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd80cb9f0 con 0x7f0bd80cf110 2024-03-24T10:58:47.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.732+0000 7f0be054f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd81529d0 con 0x7f0bd806c580 2024-03-24T10:58:47.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.732+0000 7f0be054f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0bd80c4d40 con 0x7f0bd80cc8f0 2024-03-24T10:58:47.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.733+0000 7f0bdeaec640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0bd80cf110 0x7f0bd81510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:47.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.733+0000 7f0bde2eb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0bd806c580 0x7f0bd80cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:47.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.733+0000 7f0bdeaec640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0bd80cf110 0x7f0bd81510e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57898/0 (socket says 172.21.15.5:57898) 2024-03-24T10:58:47.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.733+0000 7f0bdeaec640 1 -- 172.21.15.5:0/3418072831 learned_addr learned my addr 172.21.15.5:0/3418072831 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:47.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.734+0000 7f0bddaea640 1 --2- 172.21.15.5:0/3418072831 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0bd80cc8f0 0x7f0bd80c4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:47.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.737+0000 7f0bde2eb640 1 -- 172.21.15.5:0/3418072831 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0bd80cc8f0 msgr2=0x7f0bd80c4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:47.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.737+0000 7f0bde2eb640 1 --2- 172.21.15.5:0/3418072831 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0bd80cc8f0 0x7f0bd80c4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:47.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.737+0000 7f0bde2eb640 1 -- 172.21.15.5:0/3418072831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0bd80cf110 msgr2=0x7f0bd81510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:47.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.737+0000 7f0bde2eb640 1 --2- 172.21.15.5:0/3418072831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0bd80cf110 0x7f0bd81510e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:47.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.737+0000 7f0bde2eb640 1 -- 172.21.15.5:0/3418072831 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bd40094d0 con 0x7f0bd806c580 2024-03-24T10:58:47.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.740+0000 7f0bde2eb640 1 --2- 172.21.15.5:0/3418072831 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0bd806c580 0x7f0bd80cb4b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0bd808d3c0 tx=0x7f0bd40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:47.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.745+0000 7f0bceffd640 1 -- 172.21.15.5:0/3418072831 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd403ac60 con 0x7f0bd806c580 2024-03-24T10:58:47.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.745+0000 7f0bceffd640 1 -- 172.21.15.5:0/3418072831 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bd403adc0 con 0x7f0bd806c580 2024-03-24T10:58:47.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.745+0000 7f0be054f640 1 -- 172.21.15.5:0/3418072831 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bd80c6db0 con 0x7f0bd806c580 2024-03-24T10:58:47.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.745+0000 7f0bceffd640 1 -- 172.21.15.5:0/3418072831 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd4042610 con 0x7f0bd806c580 2024-03-24T10:58:47.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.745+0000 7f0be054f640 1 -- 172.21.15.5:0/3418072831 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bd80c7350 con 0x7f0bd806c580 2024-03-24T10:58:47.746 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.746+0000 7f0be054f640 1 -- 172.21.15.5:0/3418072831 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bd80732f0 con 0x7f0bd806c580 2024-03-24T10:58:47.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.776+0000 7f0bceffd640 1 -- 172.21.15.5:0/3418072831 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0bd402f050 con 0x7f0bd806c580 2024-03-24T10:58:47.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.777+0000 7f0bceffd640 1 --2- 172.21.15.5:0/3418072831 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0bbc06b860 0x7f0bbc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:47.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.777+0000 7f0bddaea640 1 --2- 172.21.15.5:0/3418072831 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0bbc06b860 0x7f0bbc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:47.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.777+0000 7f0bddaea640 1 --2- 172.21.15.5:0/3418072831 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0bbc06b860 0x7f0bbc06dd20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0bd81535e0 tx=0x7f0bc8009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:47.780 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:47.780+0000 7f0bceffd640 1 -- 172.21.15.5:0/3418072831 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f0bd407ac30 con 0x7f0bd806c580 2024-03-24T10:58:48.065 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:48.065+0000 7f0bceffd640 1 -- 172.21.15.5:0/3418072831 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0bd4076ed0 con 0x7f0bd806c580 2024-03-24T10:58:48.324 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:48.324+0000 7f0be054f640 1 -- 172.21.15.5:0/3418072831 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0bd80b4b20 con 0x7f0bd806c580 2024-03-24T10:58:48.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:48.377+0000 7f0bceffd640 1 -- 172.21.15.5:0/3418072831 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f0bd407a590 con 0x7f0bd806c580 2024-03-24T10:58:48.377 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:48.378 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:48.378 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:48.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:48.379+0000 7f0be054f640 1 -- 172.21.15.5:0/3418072831 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0bbc06b860 msgr2=0x7f0bbc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:48.382 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:48.379+0000 7f0be054f640 1 --2- 172.21.15.5:0/3418072831 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0bbc06b860 0x7f0bbc06dd20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f0bd81535e0 tx=0x7f0bc8009190 comp rx=0 tx=0).stop 2024-03-24T10:58:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:48.379+0000 7f0be054f640 1 -- 172.21.15.5:0/3418072831 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0bd806c580 msgr2=0x7f0bd80cb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:48.379+0000 7f0be054f640 1 --2- 172.21.15.5:0/3418072831 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0bd806c580 0x7f0bd80cb4b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0bd808d3c0 tx=0x7f0bd40046c0 comp rx=0 tx=0).stop 2024-03-24T10:58:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:48.379+0000 7f0be054f640 1 -- 172.21.15.5:0/3418072831 shutdown_connections 2024-03-24T10:58:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:48.379+0000 7f0be054f640 1 --2- 172.21.15.5:0/3418072831 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0bbc06b860 0x7f0bbc06dd20 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:48.379+0000 7f0be054f640 1 --2- 172.21.15.5:0/3418072831 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0bd80cf110 0x7f0bd81510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:48.379+0000 7f0be054f640 1 --2- 172.21.15.5:0/3418072831 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0bd80cc8f0 0x7f0bd80c4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:48.379+0000 7f0be054f640 1 --2- 172.21.15.5:0/3418072831 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0bd806c580 0x7f0bd80cb4b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:48.379+0000 7f0be054f640 1 -- 172.21.15.5:0/3418072831 >> 172.21.15.5:0/3418072831 conn(0x7f0bd80b1820 msgr2=0x7f0bd80b2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:48.379+0000 7f0be054f640 1 -- 172.21.15.5:0/3418072831 shutdown_connections 2024-03-24T10:58:48.383 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:48.379+0000 7f0be054f640 1 -- 172.21.15.5:0/3418072831 wait complete. 2024-03-24T10:58:48.996 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:48.996 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:49.997 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:50.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.100+0000 7f26f2674640 1 Processor -- start 2024-03-24T10:58:50.100 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.100+0000 7f26f2674640 1 -- start start 2024-03-24T10:58:50.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.100+0000 7f26f2674640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f26ec15aac0 0x7f26ec15aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:50.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.100+0000 7f26f2674640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26ec06c670 0x7f26ec0ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:50.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.100+0000 7f26f2674640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26ec06cd50 con 0x7f26ec06c230 2024-03-24T10:58:50.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.100+0000 7f26f2674640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f26ec0ce790 con 0x7f26ec15aac0 2024-03-24T10:58:50.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.100+0000 7f26f2674640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f26ec15b3e0 con 0x7f26ec06c670 2024-03-24T10:58:50.101 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.101+0000 7f26ebfff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f26ec15aac0 0x7f26ec15aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:50.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.101+0000 7f26ebfff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f26ec15aac0 0x7f26ec15aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:57914/0 (socket says 172.21.15.5:57914) 2024-03-24T10:58:50.102 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.101+0000 7f26ebfff640 1 -- 172.21.15.5:0/3628300511 learned_addr learned my addr 172.21.15.5:0/3628300511 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:50.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.104+0000 7f26ebfff640 1 -- 172.21.15.5:0/3628300511 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26ec06c670 msgr2=0x7f26ec0ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:50.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.104+0000 7f26ebfff640 1 --2- 172.21.15.5:0/3628300511 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26ec06c670 0x7f26ec0ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:50.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.104+0000 7f26ebfff640 1 -- 172.21.15.5:0/3628300511 >> v1:172.21.15.5:6789/0 conn(0x7f26ec06c230 legacy=0x7f26ec0cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:50.104 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.104+0000 7f26ebfff640 1 -- 172.21.15.5:0/3628300511 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26ec06cb00 con 0x7f26ec15aac0 2024-03-24T10:58:50.106 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.106+0000 7f26ebfff640 1 --2- 172.21.15.5:0/3628300511 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26ec15aac0 0x7f26ec15aea0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0098a0 tx=0x7f26dc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0fbe2ad8dd217fa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:50.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.140+0000 7f26eaffd640 1 -- 172.21.15.5:0/3628300511 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc030840 con 0x7f26ec15aac0 2024-03-24T10:58:50.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.140+0000 7f26eaffd640 1 -- 172.21.15.5:0/3628300511 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26dc0309a0 con 0x7f26ec15aac0 2024-03-24T10:58:50.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.140+0000 7f26eaffd640 1 -- 172.21.15.5:0/3628300511 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc030d30 con 0x7f26ec15aac0 2024-03-24T10:58:50.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.141+0000 7f26f2674640 1 -- 172.21.15.5:0/3628300511 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26ec15aac0 msgr2=0x7f26ec15aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:50.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.141+0000 7f26f2674640 1 --2- 172.21.15.5:0/3628300511 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26ec15aac0 0x7f26ec15aea0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0098a0 tx=0x7f26dc02b980 comp rx=0 tx=0).stop 2024-03-24T10:58:50.141 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.141+0000 7f26f2674640 1 -- 172.21.15.5:0/3628300511 shutdown_connections 2024-03-24T10:58:50.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.141+0000 7f26f2674640 1 --2- 172.21.15.5:0/3628300511 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26ec06c670 0x7f26ec0ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:50.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.141+0000 7f26f2674640 1 --2- 172.21.15.5:0/3628300511 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26ec15aac0 0x7f26ec15aea0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:50.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.141+0000 7f26f2674640 1 -- 172.21.15.5:0/3628300511 >> 172.21.15.5:0/3628300511 conn(0x7f26ec0cfb90 msgr2=0x7f26ec0cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:50.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.141+0000 7f26f2674640 1 -- 172.21.15.5:0/3628300511 shutdown_connections 2024-03-24T10:58:50.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.141+0000 7f26f2674640 1 -- 172.21.15.5:0/3628300511 wait complete. 2024-03-24T10:58:50.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.143+0000 7f26f2674640 1 Processor -- start 2024-03-24T10:58:50.143 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.143+0000 7f26f2674640 1 -- start start 2024-03-24T10:58:50.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.144+0000 7f26f2674640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26ec06c230 0x7f26ec0b0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:50.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.144+0000 7f26f2674640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26ec06c670 0x7f26ec1f03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:50.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.144+0000 7f26f2674640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26ec15aac0 0x7f26ec1f0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:50.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.144+0000 7f26f2674640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec1effb0 con 0x7f26ec15aac0 2024-03-24T10:58:50.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.144+0000 7f26f2674640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec15e250 con 0x7f26ec06c230 2024-03-24T10:58:50.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.144+0000 7f26f2674640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f26ec0c9ee0 con 0x7f26ec06c670 2024-03-24T10:58:50.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.145+0000 7f26eb7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26ec06c670 0x7f26ec1f03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:50.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.145+0000 7f26ebfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26ec06c230 0x7f26ec0b0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:50.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.145+0000 7f26f0c11640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26ec15aac0 0x7f26ec1f0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:50.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.145+0000 7f26eb7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26ec06c670 0x7f26ec1f03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:53970/0 (socket says 172.21.15.5:53970) 2024-03-24T10:58:50.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.145+0000 7f26eb7fe640 1 -- 172.21.15.5:0/2840906459 learned_addr learned my addr 172.21.15.5:0/2840906459 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:50.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.151+0000 7f26f0c11640 1 -- 172.21.15.5:0/2840906459 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26ec06c670 msgr2=0x7f26ec1f03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:50.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.151+0000 7f26f0c11640 1 --2- 172.21.15.5:0/2840906459 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26ec06c670 0x7f26ec1f03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:50.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.151+0000 7f26f0c11640 1 -- 172.21.15.5:0/2840906459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26ec06c230 msgr2=0x7f26ec0b0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:50.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.151+0000 7f26f0c11640 1 --2- 172.21.15.5:0/2840906459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26ec06c230 0x7f26ec0b0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:50.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.151+0000 7f26f0c11640 1 -- 172.21.15.5:0/2840906459 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26dc0094d0 con 0x7f26ec15aac0 2024-03-24T10:58:50.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.154+0000 7f26f0c11640 1 --2- 172.21.15.5:0/2840906459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26ec15aac0 0x7f26ec1f0920 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f26e400be20 tx=0x7f26e400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:50.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.158+0000 7f26e8ff9640 1 -- 172.21.15.5:0/2840906459 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e400ce70 con 0x7f26ec15aac0 2024-03-24T10:58:50.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.158+0000 7f26e8ff9640 1 -- 172.21.15.5:0/2840906459 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26e4014440 con 0x7f26ec15aac0 2024-03-24T10:58:50.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.158+0000 7f26e8ff9640 1 -- 172.21.15.5:0/2840906459 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26e4014810 con 0x7f26ec15aac0 2024-03-24T10:58:50.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.158+0000 7f26f2674640 1 -- 172.21.15.5:0/2840906459 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26ec15e4c0 con 0x7f26ec15aac0 2024-03-24T10:58:50.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.158+0000 7f26f2674640 1 -- 172.21.15.5:0/2840906459 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26ec15bd00 con 0x7f26ec15aac0 2024-03-24T10:58:50.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.159+0000 7f26f2674640 1 -- 172.21.15.5:0/2840906459 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26ec15bef0 con 0x7f26ec15aac0 2024-03-24T10:58:50.191 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.191+0000 7f26e8ff9640 1 -- 172.21.15.5:0/2840906459 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f26e401d3b0 con 0x7f26ec15aac0 2024-03-24T10:58:50.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.191+0000 7f26e8ff9640 1 --2- 172.21.15.5:0/2840906459 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f26d806b860 0x7f26d806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:50.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.192+0000 7f26ebfff640 1 --2- 172.21.15.5:0/2840906459 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f26d806b860 0x7f26d806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:50.192 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.192+0000 7f26ebfff640 1 --2- 172.21.15.5:0/2840906459 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f26d806b860 0x7f26d806dd20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f26dc036000 tx=0x7f26dc008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:50.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.451+0000 7f26e8ff9640 1 -- 172.21.15.5:0/2840906459 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f26e4058fe0 con 0x7f26ec15aac0 2024-03-24T10:58:50.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.451+0000 7f26e8ff9640 1 -- 172.21.15.5:0/2840906459 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f26e4055280 con 0x7f26ec15aac0 2024-03-24T10:58:50.693 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.693+0000 7f26f2674640 1 -- 172.21.15.5:0/2840906459 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f26ec0681d0 con 0x7f26ec15aac0 2024-03-24T10:58:50.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.715+0000 7f26e8ff9640 1 -- 172.21.15.5:0/2840906459 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f26e4019090 con 0x7f26ec15aac0 2024-03-24T10:58:50.715 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:50.715 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:50.715 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:50.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.717+0000 7f26f2674640 1 -- 172.21.15.5:0/2840906459 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f26d806b860 msgr2=0x7f26d806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:50.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.717+0000 7f26f2674640 1 --2- 172.21.15.5:0/2840906459 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f26d806b860 0x7f26d806dd20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f26dc036000 tx=0x7f26dc008030 comp rx=0 tx=0).stop 2024-03-24T10:58:50.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.717+0000 7f26f2674640 1 -- 172.21.15.5:0/2840906459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26ec15aac0 msgr2=0x7f26ec1f0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:50.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.717+0000 7f26f2674640 1 --2- 172.21.15.5:0/2840906459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26ec15aac0 0x7f26ec1f0920 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f26e400be20 tx=0x7f26e400bf10 comp rx=0 tx=0).stop 2024-03-24T10:58:50.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.717+0000 7f26f2674640 1 -- 172.21.15.5:0/2840906459 shutdown_connections 2024-03-24T10:58:50.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.717+0000 7f26f2674640 1 --2- 172.21.15.5:0/2840906459 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f26d806b860 0x7f26d806dd20 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:50.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.717+0000 7f26f2674640 1 --2- 172.21.15.5:0/2840906459 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f26ec15aac0 0x7f26ec1f0920 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:50.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.717+0000 7f26f2674640 1 --2- 172.21.15.5:0/2840906459 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f26ec06c670 0x7f26ec1f03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:50.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.717+0000 7f26f2674640 1 --2- 172.21.15.5:0/2840906459 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26ec06c230 0x7f26ec0b0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:50.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.717+0000 7f26f2674640 1 -- 172.21.15.5:0/2840906459 >> 172.21.15.5:0/2840906459 conn(0x7f26ec0cfb90 msgr2=0x7f26ec0c86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:50.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.717+0000 7f26f2674640 1 -- 172.21.15.5:0/2840906459 shutdown_connections 2024-03-24T10:58:50.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:50.717+0000 7f26f2674640 1 -- 172.21.15.5:0/2840906459 wait complete. 2024-03-24T10:58:51.333 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:51.333 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:52.333 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:52.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.436+0000 7fae69d1e640 1 Processor -- start 2024-03-24T10:58:52.436 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.436+0000 7fae69d1e640 1 -- start start 2024-03-24T10:58:52.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.438+0000 7fae69d1e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fae641543a0 0x7fae6415ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:52.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.438+0000 7fae69d1e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fae6406d4f0 0x7fae640cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:52.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.438+0000 7fae69d1e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fae640ccaa0 0x7fae6406bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:52.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.438+0000 7fae69d1e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fae640c6750 con 0x7fae641543a0 2024-03-24T10:58:52.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.438+0000 7fae69d1e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fae640cceb0 con 0x7fae6406d4f0 2024-03-24T10:58:52.438 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.438+0000 7fae69d1e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fae640cc380 con 0x7fae640ccaa0 2024-03-24T10:58:52.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.439+0000 7fae637fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fae641543a0 0x7fae6415ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:52.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.439+0000 7fae62ffd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fae6406d4f0 0x7fae640cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:52.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.439+0000 7fae63fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fae640ccaa0 0x7fae6406bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:52.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.439+0000 7fae63fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fae640ccaa0 0x7fae6406bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:53980/0 (socket says 172.21.15.5:53980) 2024-03-24T10:58:52.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.439+0000 7fae63fff640 1 -- 172.21.15.5:0/3502407922 learned_addr learned my addr 172.21.15.5:0/3502407922 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:52.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.441+0000 7fae637fe640 1 -- 172.21.15.5:0/3502407922 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fae640ccaa0 msgr2=0x7fae6406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:52.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.441+0000 7fae637fe640 1 --2- 172.21.15.5:0/3502407922 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fae640ccaa0 0x7fae6406bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:52.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.441+0000 7fae637fe640 1 -- 172.21.15.5:0/3502407922 >> v2:172.21.15.57:3300/0 conn(0x7fae6406d4f0 msgr2=0x7fae640cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:52.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.441+0000 7fae637fe640 1 --2- 172.21.15.5:0/3502407922 >> v2:172.21.15.57:3300/0 conn(0x7fae6406d4f0 0x7fae640cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:52.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.441+0000 7fae637fe640 1 -- 172.21.15.5:0/3502407922 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae640cb2f0 con 0x7fae641543a0 2024-03-24T10:58:52.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.444+0000 7fae637fe640 1 --2- 172.21.15.5:0/3502407922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae641543a0 0x7fae6415ac00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fae640c6990 tx=0x7fae5402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c5b029c04b749c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:52.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.448+0000 7fae627fc640 1 -- 172.21.15.5:0/3502407922 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae54031830 con 0x7fae641543a0 2024-03-24T10:58:52.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.448+0000 7fae627fc640 1 -- 172.21.15.5:0/3502407922 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae54031990 con 0x7fae641543a0 2024-03-24T10:58:52.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.448+0000 7fae627fc640 1 -- 172.21.15.5:0/3502407922 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae54031d20 con 0x7fae641543a0 2024-03-24T10:58:52.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.449+0000 7fae69d1e640 1 -- 172.21.15.5:0/3502407922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae641543a0 msgr2=0x7fae6415ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:52.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.449+0000 7fae69d1e640 1 --2- 172.21.15.5:0/3502407922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae641543a0 0x7fae6415ac00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fae640c6990 tx=0x7fae5402b8d0 comp rx=0 tx=0).stop 2024-03-24T10:58:52.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.449+0000 7fae69d1e640 1 -- 172.21.15.5:0/3502407922 shutdown_connections 2024-03-24T10:58:52.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.449+0000 7fae69d1e640 1 --2- 172.21.15.5:0/3502407922 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fae640ccaa0 0x7fae6406bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:52.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.449+0000 7fae69d1e640 1 --2- 172.21.15.5:0/3502407922 >> v2:172.21.15.57:3300/0 conn(0x7fae6406d4f0 0x7fae640cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:52.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.449+0000 7fae69d1e640 1 --2- 172.21.15.5:0/3502407922 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae641543a0 0x7fae6415ac00 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:52.449 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.449+0000 7fae69d1e640 1 -- 172.21.15.5:0/3502407922 >> 172.21.15.5:0/3502407922 conn(0x7fae64072ef0 msgr2=0x7fae64061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:52.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.449+0000 7fae69d1e640 1 -- 172.21.15.5:0/3502407922 shutdown_connections 2024-03-24T10:58:52.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.449+0000 7fae69d1e640 1 -- 172.21.15.5:0/3502407922 wait complete. 2024-03-24T10:58:52.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.450+0000 7fae69d1e640 1 Processor -- start 2024-03-24T10:58:52.450 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.450+0000 7fae69d1e640 1 -- start start 2024-03-24T10:58:52.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.451+0000 7fae69d1e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae6406d4f0 0x7fae64157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:52.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.451+0000 7fae69d1e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fae640ccaa0 0x7fae64157660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:52.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.451+0000 7fae69d1e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae641543a0 0x7fae640b4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:52.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.451+0000 7fae69d1e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fae640c6e90 con 0x7fae6406d4f0 2024-03-24T10:58:52.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.451+0000 7fae69d1e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fae6414ecc0 con 0x7fae641543a0 2024-03-24T10:58:52.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.451+0000 7fae69d1e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fae640b26c0 con 0x7fae640ccaa0 2024-03-24T10:58:52.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.451+0000 7fae637fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae6406d4f0 0x7fae64157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:52.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.452+0000 7fae62ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fae640ccaa0 0x7fae64157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:52.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.452+0000 7fae63fff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae641543a0 0x7fae640b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:52.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.452+0000 7fae62ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fae640ccaa0 0x7fae64157660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:53994/0 (socket says 172.21.15.5:53994) 2024-03-24T10:58:52.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.452+0000 7fae62ffd640 1 -- 172.21.15.5:0/2826956551 learned_addr learned my addr 172.21.15.5:0/2826956551 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:52.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.456+0000 7fae62ffd640 1 -- 172.21.15.5:0/2826956551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae641543a0 msgr2=0x7fae640b4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:52.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.456+0000 7fae62ffd640 1 --2- 172.21.15.5:0/2826956551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae641543a0 0x7fae640b4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:52.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.456+0000 7fae62ffd640 1 -- 172.21.15.5:0/2826956551 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae6406d4f0 msgr2=0x7fae64157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:52.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.456+0000 7fae62ffd640 1 --2- 172.21.15.5:0/2826956551 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae6406d4f0 0x7fae64157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:52.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.456+0000 7fae62ffd640 1 -- 172.21.15.5:0/2826956551 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae54009400 con 0x7fae640ccaa0 2024-03-24T10:58:52.459 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.459+0000 7fae62ffd640 1 --2- 172.21.15.5:0/2826956551 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fae640ccaa0 0x7fae64157660 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fae5800e9b0 tx=0x7fae5800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:52.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.462+0000 7fae4bfff640 1 -- 172.21.15.5:0/2826956551 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae58009670 con 0x7fae640ccaa0 2024-03-24T10:58:52.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.462+0000 7fae69d1e640 1 -- 172.21.15.5:0/2826956551 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae640c70e0 con 0x7fae640ccaa0 2024-03-24T10:58:52.462 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.462+0000 7fae69d1e640 1 -- 172.21.15.5:0/2826956551 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fae640c98e0 con 0x7fae640ccaa0 2024-03-24T10:58:52.463 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.463+0000 7fae69d1e640 1 -- 172.21.15.5:0/2826956551 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae640681d0 con 0x7fae640ccaa0 2024-03-24T10:58:52.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.464+0000 7fae4bfff640 1 -- 172.21.15.5:0/2826956551 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae5800c9a0 con 0x7fae640ccaa0 2024-03-24T10:58:52.464 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.464+0000 7fae4bfff640 1 -- 172.21.15.5:0/2826956551 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5800cd10 con 0x7fae640ccaa0 2024-03-24T10:58:52.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.746+0000 7fae4bfff640 1 -- 172.21.15.5:0/2826956551 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fae58013470 con 0x7fae640ccaa0 2024-03-24T10:58:52.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.747+0000 7fae4bfff640 1 --2- 172.21.15.5:0/2826956551 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fae5006b810 0x7fae5006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:52.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.747+0000 7fae637fe640 1 --2- 172.21.15.5:0/2826956551 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fae5006b810 0x7fae5006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:52.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.747+0000 7fae637fe640 1 --2- 172.21.15.5:0/2826956551 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fae5006b810 0x7fae5006dcd0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fae54036000 tx=0x7fae5403c040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:52.750 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.750+0000 7fae4bfff640 1 -- 172.21.15.5:0/2826956551 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fae58059350 con 0x7fae640ccaa0 2024-03-24T10:58:52.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.762+0000 7fae4bfff640 1 -- 172.21.15.5:0/2826956551 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fae580555f0 con 0x7fae640ccaa0 2024-03-24T10:58:52.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.983+0000 7fae69d1e640 1 -- 172.21.15.5:0/2826956551 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae64152b10 con 0x7fae640ccaa0 2024-03-24T10:58:52.998 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:52.998+0000 7fae4bfff640 1 -- 172.21.15.5:0/2826956551 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7fae58021090 con 0x7fae640ccaa0 2024-03-24T10:58:52.998 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:52.998 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:52.999 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:53.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:53.000+0000 7fae69d1e640 1 -- 172.21.15.5:0/2826956551 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fae5006b810 msgr2=0x7fae5006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:53.000 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:53.000+0000 7fae69d1e640 1 --2- 172.21.15.5:0/2826956551 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fae5006b810 0x7fae5006dcd0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fae54036000 tx=0x7fae5403c040 comp rx=0 tx=0).stop 2024-03-24T10:58:53.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:53.000+0000 7fae69d1e640 1 -- 172.21.15.5:0/2826956551 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fae640ccaa0 msgr2=0x7fae64157660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:53.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:53.000+0000 7fae69d1e640 1 --2- 172.21.15.5:0/2826956551 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fae640ccaa0 0x7fae64157660 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fae5800e9b0 tx=0x7fae5800ee70 comp rx=0 tx=0).stop 2024-03-24T10:58:53.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:53.000+0000 7fae69d1e640 1 -- 172.21.15.5:0/2826956551 shutdown_connections 2024-03-24T10:58:53.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:53.000+0000 7fae69d1e640 1 --2- 172.21.15.5:0/2826956551 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fae5006b810 0x7fae5006dcd0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:53.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:53.000+0000 7fae69d1e640 1 --2- 172.21.15.5:0/2826956551 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fae641543a0 0x7fae640b4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:53.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:53.000+0000 7fae69d1e640 1 --2- 172.21.15.5:0/2826956551 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fae640ccaa0 0x7fae64157660 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:53.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:53.000+0000 7fae69d1e640 1 --2- 172.21.15.5:0/2826956551 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fae6406d4f0 0x7fae64157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:53.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:53.000+0000 7fae69d1e640 1 -- 172.21.15.5:0/2826956551 >> 172.21.15.5:0/2826956551 conn(0x7fae64072ef0 msgr2=0x7fae640c5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:53.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:53.000+0000 7fae69d1e640 1 -- 172.21.15.5:0/2826956551 shutdown_connections 2024-03-24T10:58:53.001 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:53.000+0000 7fae69d1e640 1 -- 172.21.15.5:0/2826956551 wait complete. 2024-03-24T10:58:53.615 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:53.615 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:54.616 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:54.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.718+0000 7fac5abaf640 1 Processor -- start 2024-03-24T10:58:54.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.718+0000 7fac5abaf640 1 -- start start 2024-03-24T10:58:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.720+0000 7fac5abaf640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fac540cb270 0x7fac540cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.720+0000 7fac5abaf640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac540cd960 0x7fac540cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.720+0000 7fac5abaf640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac540ccba0 con 0x7fac540cae90 2024-03-24T10:58:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.720+0000 7fac5abaf640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fac540cc9b0 con 0x7fac540cb270 2024-03-24T10:58:54.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.720+0000 7fac5abaf640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fac540cce40 con 0x7fac540cd960 2024-03-24T10:58:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.720+0000 7fac5914c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac540cd960 0x7fac540cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.721+0000 7fac5914c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac540cd960 0x7fac540cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:40780/0 (socket says 172.21.15.5:40780) 2024-03-24T10:58:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.721+0000 7fac5914c640 1 -- 172.21.15.5:0/1236737753 learned_addr learned my addr 172.21.15.5:0/1236737753 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:54.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.721+0000 7fac53fff640 1 --2- 172.21.15.5:0/1236737753 >> v2:172.21.15.5:3300/0 conn(0x7fac540cb270 0x7fac540cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:54.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.723+0000 7fac5914c640 1 -- 172.21.15.5:0/1236737753 >> v2:172.21.15.5:3300/0 conn(0x7fac540cb270 msgr2=0x7fac540cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:54.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.723+0000 7fac5914c640 1 --2- 172.21.15.5:0/1236737753 >> v2:172.21.15.5:3300/0 conn(0x7fac540cb270 0x7fac540cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:54.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.723+0000 7fac5914c640 1 -- 172.21.15.5:0/1236737753 >> v1:172.21.15.5:6789/0 conn(0x7fac540cae90 legacy=0x7fac540cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:58:54.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.723+0000 7fac5914c640 1 -- 172.21.15.5:0/1236737753 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5415a590 con 0x7fac540cd960 2024-03-24T10:58:54.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.725+0000 7fac5914c640 1 --2- 172.21.15.5:0/1236737753 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac540cd960 0x7fac540cb6f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fac540cd680 tx=0x7fac4c02bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=528c212f917e4180 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:54.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.730+0000 7fac537fe640 1 -- 172.21.15.5:0/1236737753 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c02caf0 con 0x7fac540cd960 2024-03-24T10:58:54.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.730+0000 7fac537fe640 1 -- 172.21.15.5:0/1236737753 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac4c02cc50 con 0x7fac540cd960 2024-03-24T10:58:54.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.730+0000 7fac537fe640 1 -- 172.21.15.5:0/1236737753 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4c0343f0 con 0x7fac540cd960 2024-03-24T10:58:54.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.730+0000 7fac5abaf640 1 -- 172.21.15.5:0/1236737753 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac540cd960 msgr2=0x7fac540cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:54.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.730+0000 7fac5abaf640 1 --2- 172.21.15.5:0/1236737753 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac540cd960 0x7fac540cb6f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fac540cd680 tx=0x7fac4c02bf50 comp rx=0 tx=0).stop 2024-03-24T10:58:54.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.731+0000 7fac5abaf640 1 -- 172.21.15.5:0/1236737753 shutdown_connections 2024-03-24T10:58:54.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.731+0000 7fac5abaf640 1 --2- 172.21.15.5:0/1236737753 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac540cd960 0x7fac540cb6f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:54.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.731+0000 7fac5abaf640 1 --2- 172.21.15.5:0/1236737753 >> v2:172.21.15.5:3300/0 conn(0x7fac540cb270 0x7fac540cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:54.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.731+0000 7fac5abaf640 1 -- 172.21.15.5:0/1236737753 >> 172.21.15.5:0/1236737753 conn(0x7fac540cfb80 msgr2=0x7fac540cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:54.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.731+0000 7fac5abaf640 1 -- 172.21.15.5:0/1236737753 shutdown_connections 2024-03-24T10:58:54.731 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.731+0000 7fac5abaf640 1 -- 172.21.15.5:0/1236737753 wait complete. 2024-03-24T10:58:54.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.732+0000 7fac5abaf640 1 Processor -- start 2024-03-24T10:58:54.732 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.732+0000 7fac5abaf640 1 -- start start 2024-03-24T10:58:54.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.733+0000 7fac5abaf640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac540cae90 0x7fac541f3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:54.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.733+0000 7fac5abaf640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac540cb270 0x7fac540b1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:54.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.733+0000 7fac5abaf640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac540cd960 0x7fac541f0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:54.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.733+0000 7fac5abaf640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fac541edcb0 con 0x7fac540cb270 2024-03-24T10:58:54.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.733+0000 7fac5abaf640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fac541efc70 con 0x7fac540cd960 2024-03-24T10:58:54.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.733+0000 7fac5abaf640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fac540b4f40 con 0x7fac540cae90 2024-03-24T10:58:54.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.733+0000 7fac53fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac540cb270 0x7fac540b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:54.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.733+0000 7fac5894b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac540cae90 0x7fac541f3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:54.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.734+0000 7fac5894b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac540cae90 0x7fac541f3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:40784/0 (socket says 172.21.15.5:40784) 2024-03-24T10:58:54.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.734+0000 7fac5894b640 1 -- 172.21.15.5:0/127797730 learned_addr learned my addr 172.21.15.5:0/127797730 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:54.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.734+0000 7fac5914c640 1 --2- 172.21.15.5:0/127797730 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac540cd960 0x7fac541f0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:54.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.737+0000 7fac5894b640 1 -- 172.21.15.5:0/127797730 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac540cd960 msgr2=0x7fac541f0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:54.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.737+0000 7fac5894b640 1 --2- 172.21.15.5:0/127797730 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac540cd960 0x7fac541f0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:54.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.737+0000 7fac5894b640 1 -- 172.21.15.5:0/127797730 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac540cb270 msgr2=0x7fac540b1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:54.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.737+0000 7fac5894b640 1 --2- 172.21.15.5:0/127797730 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac540cb270 0x7fac540b1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:54.737 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.737+0000 7fac5894b640 1 -- 172.21.15.5:0/127797730 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4c009b30 con 0x7fac540cae90 2024-03-24T10:58:54.740 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.740+0000 7fac5894b640 1 --2- 172.21.15.5:0/127797730 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac540cae90 0x7fac541f3c20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fac4800c410 tx=0x7fac4800c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:54.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.744+0000 7fac517fa640 1 -- 172.21.15.5:0/127797730 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac480035f0 con 0x7fac540cae90 2024-03-24T10:58:54.744 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.744+0000 7fac517fa640 1 -- 172.21.15.5:0/127797730 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac48003750 con 0x7fac540cae90 2024-03-24T10:58:54.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.744+0000 7fac517fa640 1 -- 172.21.15.5:0/127797730 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac48003b40 con 0x7fac540cae90 2024-03-24T10:58:54.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.744+0000 7fac5abaf640 1 -- 172.21.15.5:0/127797730 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac540b5220 con 0x7fac540cae90 2024-03-24T10:58:54.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.744+0000 7fac5abaf640 1 -- 172.21.15.5:0/127797730 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fac540b57e0 con 0x7fac540cae90 2024-03-24T10:58:54.745 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.745+0000 7fac5abaf640 1 -- 172.21.15.5:0/127797730 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac540681d0 con 0x7fac540cae90 2024-03-24T10:58:54.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.771+0000 7fac517fa640 1 -- 172.21.15.5:0/127797730 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fac48003ca0 con 0x7fac540cae90 2024-03-24T10:58:54.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.772+0000 7fac517fa640 1 --2- 172.21.15.5:0/127797730 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac3806b890 0x7fac3806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:54.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.772+0000 7fac53fff640 1 --2- 172.21.15.5:0/127797730 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac3806b890 0x7fac3806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:54.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.772+0000 7fac53fff640 1 --2- 172.21.15.5:0/127797730 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac3806b890 0x7fac3806dd50 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fac540cf780 tx=0x7fac40009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:54.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:54.774+0000 7fac517fa640 1 -- 172.21.15.5:0/127797730 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fac480010c0 con 0x7fac540cae90 2024-03-24T10:58:55.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:55.055+0000 7fac517fa640 1 -- 172.21.15.5:0/127797730 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fac48054740 con 0x7fac540cae90 2024-03-24T10:58:55.274 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:55.274+0000 7fac5abaf640 1 -- 172.21.15.5:0/127797730 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac540b59a0 con 0x7fac540cae90 2024-03-24T10:58:55.290 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:55.290+0000 7fac517fa640 1 -- 172.21.15.5:0/127797730 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7fac48017020 con 0x7fac540cae90 2024-03-24T10:58:55.290 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:55.290 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:55.291 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:55.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:55.292+0000 7fac5abaf640 1 -- 172.21.15.5:0/127797730 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac3806b890 msgr2=0x7fac3806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:55.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:55.292+0000 7fac5abaf640 1 --2- 172.21.15.5:0/127797730 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac3806b890 0x7fac3806dd50 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fac540cf780 tx=0x7fac40009190 comp rx=0 tx=0).stop 2024-03-24T10:58:55.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:55.292+0000 7fac5abaf640 1 -- 172.21.15.5:0/127797730 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac540cae90 msgr2=0x7fac541f3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:55.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:55.292+0000 7fac5abaf640 1 --2- 172.21.15.5:0/127797730 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac540cae90 0x7fac541f3c20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fac4800c410 tx=0x7fac4800c8d0 comp rx=0 tx=0).stop 2024-03-24T10:58:55.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:55.292+0000 7fac5abaf640 1 -- 172.21.15.5:0/127797730 shutdown_connections 2024-03-24T10:58:55.292 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:55.292+0000 7fac5abaf640 1 --2- 172.21.15.5:0/127797730 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fac3806b890 0x7fac3806dd50 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:55.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:55.292+0000 7fac5abaf640 1 --2- 172.21.15.5:0/127797730 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fac540cb270 0x7fac540b1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:55.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:55.292+0000 7fac5abaf640 1 --2- 172.21.15.5:0/127797730 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fac540cd960 0x7fac541f0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:55.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:55.292+0000 7fac5abaf640 1 --2- 172.21.15.5:0/127797730 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fac540cae90 0x7fac541f3c20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:55.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:55.292+0000 7fac5abaf640 1 -- 172.21.15.5:0/127797730 >> 172.21.15.5:0/127797730 conn(0x7fac540cfb80 msgr2=0x7fac54159da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:55.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:55.292+0000 7fac5abaf640 1 -- 172.21.15.5:0/127797730 shutdown_connections 2024-03-24T10:58:55.293 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:55.292+0000 7fac5abaf640 1 -- 172.21.15.5:0/127797730 wait complete. 2024-03-24T10:58:55.908 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:55.908 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:56.909 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:57.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.012+0000 7f93fe584640 1 Processor -- start 2024-03-24T10:58:57.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.012+0000 7f93fe584640 1 -- start start 2024-03-24T10:58:57.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.013+0000 7f93fe584640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f93f81543a0 0x7f93f815ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:57.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.013+0000 7f93fe584640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93f80cb2f0 con 0x7f93f806bf50 2024-03-24T10:58:57.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.013+0000 7f93fe584640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93f80ca7e0 con 0x7f93f806d4f0 2024-03-24T10:58:57.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.013+0000 7f93fe584640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f93f80c6750 con 0x7f93f81543a0 2024-03-24T10:58:57.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.014+0000 7f93fcb21640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f93f806bf50 0x7f93f806c330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:35638/0 (socket says 172.21.15.5:35638) 2024-03-24T10:58:57.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.014+0000 7f93fcb21640 1 -- 172.21.15.5:0/612653359 learned_addr learned my addr 172.21.15.5:0/612653359 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:58:57.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.014+0000 7f93f7fff640 1 --2- 172.21.15.5:0/612653359 >> v2:172.21.15.57:3300/0 conn(0x7f93f81543a0 0x7f93f815ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.017+0000 7f93f6ffd640 1 -- 172.21.15.5:0/612653359 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 222315983 0 0) 0x7f93f80cb2f0 con 0x7f93f806bf50 2024-03-24T10:58:57.017 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.017+0000 7f93f6ffd640 1 -- 172.21.15.5:0/612653359 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93e0003520 con 0x7f93f806bf50 2024-03-24T10:58:57.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.018+0000 7f93f7fff640 1 -- 172.21.15.5:0/612653359 >> v1:172.21.15.57:6789/0 conn(0x7f93f806d4f0 legacy=0x7f93f80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:57.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.018+0000 7f93f7fff640 1 -- 172.21.15.5:0/612653359 >> v1:172.21.15.5:6789/0 conn(0x7f93f806bf50 legacy=0x7f93f806c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:57.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.018+0000 7f93f7fff640 1 -- 172.21.15.5:0/612653359 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93f80cb520 con 0x7f93f81543a0 2024-03-24T10:58:57.020 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.021+0000 7f93f7fff640 1 --2- 172.21.15.5:0/612653359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93f81543a0 0x7f93f815ac00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f93e80097a0 tx=0x7f93e802b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d2ad4aa1936816c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:57.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.026+0000 7f93f6ffd640 1 -- 172.21.15.5:0/612653359 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e802f810 con 0x7f93f81543a0 2024-03-24T10:58:57.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.028+0000 7f93f6ffd640 1 -- 172.21.15.5:0/612653359 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93e802f970 con 0x7f93f81543a0 2024-03-24T10:58:57.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.028+0000 7f93f6ffd640 1 -- 172.21.15.5:0/612653359 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e802fce0 con 0x7f93f81543a0 2024-03-24T10:58:57.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.028+0000 7f93fe584640 1 -- 172.21.15.5:0/612653359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93f81543a0 msgr2=0x7f93f815ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:57.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.028+0000 7f93fe584640 1 --2- 172.21.15.5:0/612653359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93f81543a0 0x7f93f815ac00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f93e80097a0 tx=0x7f93e802b9f0 comp rx=0 tx=0).stop 2024-03-24T10:58:57.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.028+0000 7f93fe584640 1 -- 172.21.15.5:0/612653359 shutdown_connections 2024-03-24T10:58:57.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.028+0000 7f93fe584640 1 --2- 172.21.15.5:0/612653359 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93f81543a0 0x7f93f815ac00 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:57.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.028+0000 7f93fe584640 1 -- 172.21.15.5:0/612653359 >> 172.21.15.5:0/612653359 conn(0x7f93f8072ef0 msgr2=0x7f93f8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:57.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.029+0000 7f93fe584640 1 -- 172.21.15.5:0/612653359 shutdown_connections 2024-03-24T10:58:57.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.029+0000 7f93fe584640 1 -- 172.21.15.5:0/612653359 wait complete. 2024-03-24T10:58:57.029 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.030+0000 7f93fe584640 1 Processor -- start 2024-03-24T10:58:57.030 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.030+0000 7f93fe584640 1 -- start start 2024-03-24T10:58:57.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.031+0000 7f93fe584640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f93f806bf50 0x7f93f80ced10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:57.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.031+0000 7f93fe584640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93f806d4f0 0x7f93f80cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:57.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.031+0000 7f93fe584640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93f81543a0 0x7f93f80cdf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:57.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.031+0000 7f93fe584640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f93f80c6ed0 con 0x7f93f81543a0 2024-03-24T10:58:57.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.031+0000 7f93fe584640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f93f8159060 con 0x7f93f806d4f0 2024-03-24T10:58:57.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.031+0000 7f93fe584640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f93f80b4fb0 con 0x7f93f806bf50 2024-03-24T10:58:57.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.031+0000 7f93f77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93f806d4f0 0x7f93f80cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:57.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.031+0000 7f93f7fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f93f806bf50 0x7f93f80ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:57.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.031+0000 7f93fcb21640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93f81543a0 0x7f93f80cdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:57.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.032+0000 7f93f77fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93f806d4f0 0x7f93f80cd9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:39814/0 (socket says 172.21.15.5:39814) 2024-03-24T10:58:57.032 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.032+0000 7f93f77fe640 1 -- 172.21.15.5:0/402508080 learned_addr learned my addr 172.21.15.5:0/402508080 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:57.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.035+0000 7f93f77fe640 1 -- 172.21.15.5:0/402508080 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f93f806bf50 msgr2=0x7f93f80ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:57.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.035+0000 7f93f77fe640 1 --2- 172.21.15.5:0/402508080 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f93f806bf50 0x7f93f80ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:57.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.035+0000 7f93f77fe640 1 -- 172.21.15.5:0/402508080 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93f81543a0 msgr2=0x7f93f80cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:57.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.035+0000 7f93f77fe640 1 --2- 172.21.15.5:0/402508080 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93f81543a0 0x7f93f80cdf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:57.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.035+0000 7f93f77fe640 1 -- 172.21.15.5:0/402508080 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e8009400 con 0x7f93f806d4f0 2024-03-24T10:58:57.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.038+0000 7f93f77fe640 1 --2- 172.21.15.5:0/402508080 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93f806d4f0 0x7f93f80cd9d0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f93ec00c400 tx=0x7f93ec00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:57.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.042+0000 7f93f4ff9640 1 -- 172.21.15.5:0/402508080 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ec0052d0 con 0x7f93f806d4f0 2024-03-24T10:58:57.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.042+0000 7f93f4ff9640 1 -- 172.21.15.5:0/402508080 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93ec005450 con 0x7f93f806d4f0 2024-03-24T10:58:57.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.042+0000 7f93fe584640 1 -- 172.21.15.5:0/402508080 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93f80b4a10 con 0x7f93f806d4f0 2024-03-24T10:58:57.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.042+0000 7f93f4ff9640 1 -- 172.21.15.5:0/402508080 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ec007260 con 0x7f93f806d4f0 2024-03-24T10:58:57.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.042+0000 7f93fe584640 1 -- 172.21.15.5:0/402508080 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93f8158a10 con 0x7f93f806d4f0 2024-03-24T10:58:57.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.043+0000 7f93fe584640 1 -- 172.21.15.5:0/402508080 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93f80681d0 con 0x7f93f806d4f0 2024-03-24T10:58:57.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.068+0000 7f93f4ff9640 1 -- 172.21.15.5:0/402508080 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f93ec0073c0 con 0x7f93f806d4f0 2024-03-24T10:58:57.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.069+0000 7f93f4ff9640 1 --2- 172.21.15.5:0/402508080 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f93e406b890 0x7f93e406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:57.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.069+0000 7f93f7fff640 1 --2- 172.21.15.5:0/402508080 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f93e406b890 0x7f93e406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:57.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.069+0000 7f93f7fff640 1 --2- 172.21.15.5:0/402508080 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f93e406b890 0x7f93e406dd50 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f93e8036000 tx=0x7f93e8008030 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:57.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.341+0000 7f93f4ff9640 1 -- 172.21.15.5:0/402508080 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f93ec0585e0 con 0x7f93f806d4f0 2024-03-24T10:58:57.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.352+0000 7f93f4ff9640 1 -- 172.21.15.5:0/402508080 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f93ec054880 con 0x7f93f806d4f0 2024-03-24T10:58:57.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.572+0000 7f93fe584640 1 -- 172.21.15.5:0/402508080 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93f81591a0 con 0x7f93f806d4f0 2024-03-24T10:58:57.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.588+0000 7f93f4ff9640 1 -- 172.21.15.5:0/402508080 <== mon.1 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f93ec020020 con 0x7f93f806d4f0 2024-03-24T10:58:57.589 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:58:57.589 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:58:57.589 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:58:57.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.591+0000 7f93fe584640 1 -- 172.21.15.5:0/402508080 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f93e406b890 msgr2=0x7f93e406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:57.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.591+0000 7f93fe584640 1 --2- 172.21.15.5:0/402508080 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f93e406b890 0x7f93e406dd50 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f93e8036000 tx=0x7f93e8008030 comp rx=0 tx=0).stop 2024-03-24T10:58:57.591 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.591+0000 7f93fe584640 1 -- 172.21.15.5:0/402508080 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93f806d4f0 msgr2=0x7f93f80cd9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:57.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.591+0000 7f93fe584640 1 --2- 172.21.15.5:0/402508080 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93f806d4f0 0x7f93f80cd9d0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f93ec00c400 tx=0x7f93ec00c8c0 comp rx=0 tx=0).stop 2024-03-24T10:58:57.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.591+0000 7f93fe584640 1 -- 172.21.15.5:0/402508080 shutdown_connections 2024-03-24T10:58:57.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.591+0000 7f93fe584640 1 --2- 172.21.15.5:0/402508080 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f93e406b890 0x7f93e406dd50 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:57.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.591+0000 7f93fe584640 1 --2- 172.21.15.5:0/402508080 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f93f81543a0 0x7f93f80cdf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:57.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.591+0000 7f93fe584640 1 --2- 172.21.15.5:0/402508080 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f93f806d4f0 0x7f93f80cd9d0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:57.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.591+0000 7f93fe584640 1 --2- 172.21.15.5:0/402508080 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f93f806bf50 0x7f93f80ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:57.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.591+0000 7f93fe584640 1 -- 172.21.15.5:0/402508080 >> 172.21.15.5:0/402508080 conn(0x7f93f8072ef0 msgr2=0x7f93f8152540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:57.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.591+0000 7f93fe584640 1 -- 172.21.15.5:0/402508080 shutdown_connections 2024-03-24T10:58:57.592 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:57.591+0000 7f93fe584640 1 -- 172.21.15.5:0/402508080 wait complete. 2024-03-24T10:58:58.206 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:58:58.206 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:58:59.207 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:58:59.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.309+0000 7f2fa81e9640 1 Processor -- start 2024-03-24T10:58:59.309 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.309+0000 7f2fa81e9640 1 -- start start 2024-03-24T10:58:59.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.311+0000 7f2fa81e9640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2fa00cb270 0x7f2fa00cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:59.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.311+0000 7f2fa81e9640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fa00ccba0 con 0x7f2fa00cae90 2024-03-24T10:58:59.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.311+0000 7f2fa81e9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fa00cce40 con 0x7f2fa00cd960 2024-03-24T10:58:59.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.311+0000 7f2fa81e9640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f2fa00cc9b0 con 0x7f2fa00cb270 2024-03-24T10:58:59.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.311+0000 7f2fa5f85640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f2fa00cae90 0x7f2fa00cc2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:35648/0 (socket says 172.21.15.5:35648) 2024-03-24T10:58:59.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.311+0000 7f2fa5f85640 1 -- 172.21.15.5:0/2156714053 learned_addr learned my addr 172.21.15.5:0/2156714053 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:58:59.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.311+0000 7f2fa5784640 1 --2- 172.21.15.5:0/2156714053 >> v2:172.21.15.57:3300/0 conn(0x7f2fa00cb270 0x7f2fa00cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:59.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.314+0000 7f2fa4f83640 1 -- 172.21.15.5:0/2156714053 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3002192850 0 0) 0x7f2fa00ccba0 con 0x7f2fa00cae90 2024-03-24T10:58:59.314 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.314+0000 7f2fa4f83640 1 -- 172.21.15.5:0/2156714053 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f8c003520 con 0x7f2fa00cae90 2024-03-24T10:58:59.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.316+0000 7f2fa5784640 1 -- 172.21.15.5:0/2156714053 >> v1:172.21.15.57:6789/0 conn(0x7f2fa00cd960 legacy=0x7f2fa00cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:59.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.316+0000 7f2fa5784640 1 -- 172.21.15.5:0/2156714053 >> v1:172.21.15.5:6789/0 conn(0x7f2fa00cae90 legacy=0x7f2fa00cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:59.316 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.316+0000 7f2fa5784640 1 -- 172.21.15.5:0/2156714053 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa015a590 con 0x7f2fa00cb270 2024-03-24T10:58:59.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.320+0000 7f2fa5784640 1 --2- 172.21.15.5:0/2156714053 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fa00cb270 0x7f2fa00cf530 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2f900097a0 tx=0x7f2f9002b9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=395c7922e6bab88 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:59.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.325+0000 7f2fa4f83640 1 -- 172.21.15.5:0/2156714053 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9002f810 con 0x7f2fa00cb270 2024-03-24T10:58:59.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.325+0000 7f2fa4f83640 1 -- 172.21.15.5:0/2156714053 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f9002f970 con 0x7f2fa00cb270 2024-03-24T10:58:59.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.325+0000 7f2fa4f83640 1 -- 172.21.15.5:0/2156714053 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9002fd00 con 0x7f2fa00cb270 2024-03-24T10:58:59.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.326+0000 7f2fa81e9640 1 -- 172.21.15.5:0/2156714053 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fa00cb270 msgr2=0x7f2fa00cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:59.326 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.326+0000 7f2fa81e9640 1 --2- 172.21.15.5:0/2156714053 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fa00cb270 0x7f2fa00cf530 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f2f900097a0 tx=0x7f2f9002b9f0 comp rx=0 tx=0).stop 2024-03-24T10:58:59.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.326+0000 7f2fa81e9640 1 -- 172.21.15.5:0/2156714053 shutdown_connections 2024-03-24T10:58:59.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.326+0000 7f2fa81e9640 1 --2- 172.21.15.5:0/2156714053 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fa00cb270 0x7f2fa00cf530 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:59.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.326+0000 7f2fa81e9640 1 -- 172.21.15.5:0/2156714053 >> 172.21.15.5:0/2156714053 conn(0x7f2fa00cfb80 msgr2=0x7f2fa00cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:58:59.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.326+0000 7f2fa81e9640 1 -- 172.21.15.5:0/2156714053 shutdown_connections 2024-03-24T10:58:59.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.326+0000 7f2fa81e9640 1 -- 172.21.15.5:0/2156714053 wait complete. 2024-03-24T10:58:59.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.328+0000 7f2fa81e9640 1 Processor -- start 2024-03-24T10:58:59.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.328+0000 7f2fa81e9640 1 -- start start 2024-03-24T10:58:59.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.329+0000 7f2fa81e9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fa00cae90 0x7f2fa00b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:59.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.329+0000 7f2fa81e9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2fa00cb270 0x7f2fa00ce030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:59.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.329+0000 7f2fa81e9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2fa00cd960 0x7f2fa01efde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:59.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.329+0000 7f2fa81e9640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa01ef990 con 0x7f2fa00cd960 2024-03-24T10:58:59.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.329+0000 7f2fa81e9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa00b3cf0 con 0x7f2fa00cae90 2024-03-24T10:58:59.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.329+0000 7f2fa81e9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f2fa01f3080 con 0x7f2fa00cb270 2024-03-24T10:58:59.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.330+0000 7f2fa6786640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2fa00cd960 0x7f2fa01efde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:59.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.330+0000 7f2fa5784640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2fa00cb270 0x7f2fa00ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:59.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.330+0000 7f2fa5f85640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fa00cae90 0x7f2fa00b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:59.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.330+0000 7f2fa5784640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2fa00cb270 0x7f2fa00ce030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:40798/0 (socket says 172.21.15.5:40798) 2024-03-24T10:58:59.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.330+0000 7f2fa5784640 1 -- 172.21.15.5:0/3458272694 learned_addr learned my addr 172.21.15.5:0/3458272694 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:58:59.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.334+0000 7f2fa5784640 1 -- 172.21.15.5:0/3458272694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fa00cae90 msgr2=0x7f2fa00b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:59.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.334+0000 7f2fa5784640 1 --2- 172.21.15.5:0/3458272694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fa00cae90 0x7f2fa00b37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:59.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.334+0000 7f2fa5784640 1 -- 172.21.15.5:0/3458272694 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2fa00cd960 msgr2=0x7f2fa01efde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:58:59.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.334+0000 7f2fa5784640 1 --2- 172.21.15.5:0/3458272694 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2fa00cd960 0x7f2fa01efde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:58:59.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.334+0000 7f2fa5784640 1 -- 172.21.15.5:0/3458272694 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f90009400 con 0x7f2fa00cb270 2024-03-24T10:58:59.336 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.336+0000 7f2fa5784640 1 --2- 172.21.15.5:0/3458272694 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2fa00cb270 0x7f2fa00ce030 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2f9002c000 tx=0x7f2f90004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:59.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.341+0000 7f2f967fc640 1 -- 172.21.15.5:0/3458272694 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9003c040 con 0x7f2fa00cb270 2024-03-24T10:58:59.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.342+0000 7f2f967fc640 1 -- 172.21.15.5:0/3458272694 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f900378f0 con 0x7f2fa00cb270 2024-03-24T10:58:59.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.342+0000 7f2fa81e9640 1 -- 172.21.15.5:0/3458272694 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fa01f3360 con 0x7f2fa00cb270 2024-03-24T10:58:59.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.342+0000 7f2f967fc640 1 -- 172.21.15.5:0/3458272694 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f90037ca0 con 0x7f2fa00cb270 2024-03-24T10:58:59.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.342+0000 7f2fa81e9640 1 -- 172.21.15.5:0/3458272694 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fa01f3920 con 0x7f2fa00cb270 2024-03-24T10:58:59.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.343+0000 7f2fa81e9640 1 -- 172.21.15.5:0/3458272694 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fa006d5a0 con 0x7f2fa00cb270 2024-03-24T10:58:59.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.622+0000 7f2f967fc640 1 -- 172.21.15.5:0/3458272694 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2f9004b3b0 con 0x7f2fa00cb270 2024-03-24T10:58:59.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.622+0000 7f2f967fc640 1 --2- 172.21.15.5:0/3458272694 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2f8406b810 0x7f2f8406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:58:59.622 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.622+0000 7f2fa5f85640 1 --2- 172.21.15.5:0/3458272694 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2f8406b810 0x7f2f8406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:58:59.623 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.622+0000 7f2fa5f85640 1 --2- 172.21.15.5:0/3458272694 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2f8406b810 0x7f2f8406dcd0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0011e0 tx=0x7f2f9c007d30 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:58:59.630 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.630+0000 7f2f967fc640 1 -- 172.21.15.5:0/3458272694 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f2f9007ad60 con 0x7f2fa00cb270 2024-03-24T10:58:59.649 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.649+0000 7f2f967fc640 1 -- 172.21.15.5:0/3458272694 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2f90077000 con 0x7f2fa00cb270 2024-03-24T10:58:59.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:58:59.889+0000 7f2fa81e9640 1 -- 172.21.15.5:0/3458272694 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2fa01efb00 con 0x7f2fa00cb270 2024-03-24T10:59:00.039 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:00.039+0000 7f2f967fc640 1 -- 172.21.15.5:0/3458272694 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f2fa01efb00 con 0x7f2fa00cb270 2024-03-24T10:59:00.039 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:59:00.039 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:59:00.040 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:59:00.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:00.041+0000 7f2fa81e9640 1 -- 172.21.15.5:0/3458272694 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2f8406b810 msgr2=0x7f2f8406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:00.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:00.041+0000 7f2fa81e9640 1 --2- 172.21.15.5:0/3458272694 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2f8406b810 0x7f2f8406dcd0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0011e0 tx=0x7f2f9c007d30 comp rx=0 tx=0).stop 2024-03-24T10:59:00.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:00.041+0000 7f2fa81e9640 1 -- 172.21.15.5:0/3458272694 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2fa00cb270 msgr2=0x7f2fa00ce030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:00.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:00.041+0000 7f2fa81e9640 1 --2- 172.21.15.5:0/3458272694 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2fa00cb270 0x7f2fa00ce030 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2f9002c000 tx=0x7f2f90004730 comp rx=0 tx=0).stop 2024-03-24T10:59:00.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:00.041+0000 7f2fa81e9640 1 -- 172.21.15.5:0/3458272694 shutdown_connections 2024-03-24T10:59:00.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:00.041+0000 7f2fa81e9640 1 --2- 172.21.15.5:0/3458272694 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2f8406b810 0x7f2f8406dcd0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:00.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:00.041+0000 7f2fa81e9640 1 --2- 172.21.15.5:0/3458272694 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2fa00cd960 0x7f2fa01efde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:00.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:00.041+0000 7f2fa81e9640 1 --2- 172.21.15.5:0/3458272694 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2fa00cb270 0x7f2fa00ce030 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:00.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:00.041+0000 7f2fa81e9640 1 --2- 172.21.15.5:0/3458272694 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2fa00cae90 0x7f2fa00b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:00.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:00.041+0000 7f2fa81e9640 1 -- 172.21.15.5:0/3458272694 >> 172.21.15.5:0/3458272694 conn(0x7f2fa00cfb80 msgr2=0x7f2fa00cff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:00.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:00.041+0000 7f2fa81e9640 1 -- 172.21.15.5:0/3458272694 shutdown_connections 2024-03-24T10:59:00.042 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:00.041+0000 7f2fa81e9640 1 -- 172.21.15.5:0/3458272694 wait complete. 2024-03-24T10:59:00.657 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:59:00.657 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:59:01.658 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:59:01.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.760+0000 7f66bc563640 1 Processor -- start 2024-03-24T10:59:01.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.760+0000 7f66bc563640 1 -- start start 2024-03-24T10:59:01.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.761+0000 7f66bc563640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f66b415ac00 0x7f66b40cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:01.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.761+0000 7f66bc563640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66b40cb2f0 con 0x7f66b40ccaa0 2024-03-24T10:59:01.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.761+0000 7f66bc563640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66b415b070 con 0x7f66b41543a0 2024-03-24T10:59:01.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.761+0000 7f66bc563640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f66b40c6750 con 0x7f66b415ac00 2024-03-24T10:59:01.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.762+0000 7f66b9afe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f66b415ac00 0x7f66b40cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:01.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.762+0000 7f66ba2ff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f66b41543a0 0x7f66b406d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:51870/0 (socket says 172.21.15.5:51870) 2024-03-24T10:59:01.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.762+0000 7f66ba2ff640 1 -- 172.21.15.5:0/2348220260 learned_addr learned my addr 172.21.15.5:0/2348220260 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:59:01.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.765+0000 7f66b92fd640 1 -- 172.21.15.5:0/2348220260 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1713816958 0 0) 0x7f66b415b070 con 0x7f66b41543a0 2024-03-24T10:59:01.765 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.765+0000 7f66b92fd640 1 -- 172.21.15.5:0/2348220260 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66a0003520 con 0x7f66b41543a0 2024-03-24T10:59:01.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.766+0000 7f66b9afe640 1 -- 172.21.15.5:0/2348220260 >> v1:172.21.15.57:6789/0 conn(0x7f66b41543a0 legacy=0x7f66b406d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:01.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.766+0000 7f66b9afe640 1 -- 172.21.15.5:0/2348220260 >> v1:172.21.15.5:6789/0 conn(0x7f66b40ccaa0 legacy=0x7f66b406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:01.766 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.766+0000 7f66b9afe640 1 -- 172.21.15.5:0/2348220260 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b40ca7e0 con 0x7f66b415ac00 2024-03-24T10:59:01.770 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.771+0000 7f66b9afe640 1 --2- 172.21.15.5:0/2348220260 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66b415ac00 0x7f66b40cc560 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f66a40097a0 tx=0x7f66a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a60f4939d2e796f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.781+0000 7f66b92fd640 1 -- 172.21.15.5:0/2348220260 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4031830 con 0x7f66b415ac00 2024-03-24T10:59:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.781+0000 7f66b92fd640 1 -- 172.21.15.5:0/2348220260 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66a4031990 con 0x7f66b415ac00 2024-03-24T10:59:01.781 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.781+0000 7f66b92fd640 1 -- 172.21.15.5:0/2348220260 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4031d20 con 0x7f66b415ac00 2024-03-24T10:59:01.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.782+0000 7f66bc563640 1 -- 172.21.15.5:0/2348220260 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66b415ac00 msgr2=0x7f66b40cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:01.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.782+0000 7f66bc563640 1 --2- 172.21.15.5:0/2348220260 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66b415ac00 0x7f66b40cc560 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f66a40097a0 tx=0x7f66a402b8d0 comp rx=0 tx=0).stop 2024-03-24T10:59:01.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.782+0000 7f66bc563640 1 -- 172.21.15.5:0/2348220260 shutdown_connections 2024-03-24T10:59:01.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.782+0000 7f66bc563640 1 --2- 172.21.15.5:0/2348220260 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66b415ac00 0x7f66b40cc560 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:01.782 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.782+0000 7f66bc563640 1 -- 172.21.15.5:0/2348220260 >> 172.21.15.5:0/2348220260 conn(0x7f66b4072ef0 msgr2=0x7f66b4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:01.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.782+0000 7f66bc563640 1 -- 172.21.15.5:0/2348220260 shutdown_connections 2024-03-24T10:59:01.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.782+0000 7f66bc563640 1 -- 172.21.15.5:0/2348220260 wait complete. 2024-03-24T10:59:01.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.783+0000 7f66bc563640 1 Processor -- start 2024-03-24T10:59:01.783 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.783+0000 7f66bc563640 1 -- start start 2024-03-24T10:59:01.784 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.784+0000 7f66bc563640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66b40ccaa0 0x7f66b4157e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:01.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.784+0000 7f66bc563640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66b41543a0 0x7f66b40cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:01.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.784+0000 7f66bc563640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f66b415ac00 0x7f66b40ce010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:01.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.784+0000 7f66bc563640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f66b4157980 con 0x7f66b40ccaa0 2024-03-24T10:59:01.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.784+0000 7f66bc563640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f66b40b48a0 con 0x7f66b41543a0 2024-03-24T10:59:01.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.784+0000 7f66bc563640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f66b40cf1a0 con 0x7f66b415ac00 2024-03-24T10:59:01.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.784+0000 7f66ba2ff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66b40ccaa0 0x7f66b4157e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:01.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.785+0000 7f66b9afe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66b41543a0 0x7f66b40cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:01.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.785+0000 7f66bab00640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f66b415ac00 0x7f66b40ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:01.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.785+0000 7f66ba2ff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66b40ccaa0 0x7f66b4157e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:54308/0 (socket says 172.21.15.5:54308) 2024-03-24T10:59:01.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.785+0000 7f66ba2ff640 1 -- 172.21.15.5:0/3502533060 learned_addr learned my addr 172.21.15.5:0/3502533060 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:01.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.788+0000 7f66ba2ff640 1 -- 172.21.15.5:0/3502533060 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f66b415ac00 msgr2=0x7f66b40ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:01.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.788+0000 7f66ba2ff640 1 --2- 172.21.15.5:0/3502533060 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f66b415ac00 0x7f66b40ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:01.788 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.788+0000 7f66ba2ff640 1 -- 172.21.15.5:0/3502533060 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66b41543a0 msgr2=0x7f66b40cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:01.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.788+0000 7f66ba2ff640 1 --2- 172.21.15.5:0/3502533060 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66b41543a0 0x7f66b40cdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:01.789 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.788+0000 7f66ba2ff640 1 -- 172.21.15.5:0/3502533060 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a4009400 con 0x7f66b40ccaa0 2024-03-24T10:59:01.790 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.791+0000 7f66ba2ff640 1 --2- 172.21.15.5:0/3502533060 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66b40ccaa0 0x7f66b4157e70 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f66b000c460 tx=0x7f66b000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:01.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.795+0000 7f66aaffd640 1 -- 172.21.15.5:0/3502533060 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b0005ce0 con 0x7f66b40ccaa0 2024-03-24T10:59:01.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.795+0000 7f66aaffd640 1 -- 172.21.15.5:0/3502533060 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66b0005e40 con 0x7f66b40ccaa0 2024-03-24T10:59:01.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.795+0000 7f66bc563640 1 -- 172.21.15.5:0/3502533060 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b4155070 con 0x7f66b40ccaa0 2024-03-24T10:59:01.795 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.795+0000 7f66aaffd640 1 -- 172.21.15.5:0/3502533060 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b0007290 con 0x7f66b40ccaa0 2024-03-24T10:59:01.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.795+0000 7f66bc563640 1 -- 172.21.15.5:0/3502533060 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b40b3c90 con 0x7f66b40ccaa0 2024-03-24T10:59:01.796 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.796+0000 7f66bc563640 1 -- 172.21.15.5:0/3502533060 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66b4151480 con 0x7f66b40ccaa0 2024-03-24T10:59:01.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.821+0000 7f66aaffd640 1 -- 172.21.15.5:0/3502533060 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f66b00073f0 con 0x7f66b40ccaa0 2024-03-24T10:59:01.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.821+0000 7f66aaffd640 1 --2- 172.21.15.5:0/3502533060 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f669806b860 0x7f669806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:01.821 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.821+0000 7f66b9afe640 1 --2- 172.21.15.5:0/3502533060 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f669806b860 0x7f669806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:01.822 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.822+0000 7f66b9afe640 1 --2- 172.21.15.5:0/3502533060 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f669806b860 0x7f669806dd20 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f66a4036000 tx=0x7f66a403c040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:01.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:01.825+0000 7f66aaffd640 1 -- 172.21.15.5:0/3502533060 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f66b00583b0 con 0x7f66b40ccaa0 2024-03-24T10:59:02.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:02.132+0000 7f66aaffd640 1 -- 172.21.15.5:0/3502533060 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f66b0054650 con 0x7f66b40ccaa0 2024-03-24T10:59:02.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:02.358+0000 7f66bc563640 1 -- 172.21.15.5:0/3502533060 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66b40cad60 con 0x7f66b40ccaa0 2024-03-24T10:59:02.417 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:02.417+0000 7f66aaffd640 1 -- 172.21.15.5:0/3502533060 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f66b0017070 con 0x7f66b40ccaa0 2024-03-24T10:59:02.417 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:59:02.417 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:59:02.418 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:59:02.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:02.419+0000 7f66bc563640 1 -- 172.21.15.5:0/3502533060 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f669806b860 msgr2=0x7f669806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:02.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:02.419+0000 7f66bc563640 1 --2- 172.21.15.5:0/3502533060 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f669806b860 0x7f669806dd20 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f66a4036000 tx=0x7f66a403c040 comp rx=0 tx=0).stop 2024-03-24T10:59:02.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:02.419+0000 7f66bc563640 1 -- 172.21.15.5:0/3502533060 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66b40ccaa0 msgr2=0x7f66b4157e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:02.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:02.419+0000 7f66bc563640 1 --2- 172.21.15.5:0/3502533060 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66b40ccaa0 0x7f66b4157e70 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f66b000c460 tx=0x7f66b000c920 comp rx=0 tx=0).stop 2024-03-24T10:59:02.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:02.419+0000 7f66bc563640 1 -- 172.21.15.5:0/3502533060 shutdown_connections 2024-03-24T10:59:02.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:02.419+0000 7f66bc563640 1 --2- 172.21.15.5:0/3502533060 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f669806b860 0x7f669806dd20 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:02.420 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:02.419+0000 7f66bc563640 1 --2- 172.21.15.5:0/3502533060 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f66b415ac00 0x7f66b40ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:02.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:02.419+0000 7f66bc563640 1 --2- 172.21.15.5:0/3502533060 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66b41543a0 0x7f66b40cdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:02.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:02.419+0000 7f66bc563640 1 --2- 172.21.15.5:0/3502533060 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66b40ccaa0 0x7f66b4157e70 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:02.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:02.419+0000 7f66bc563640 1 -- 172.21.15.5:0/3502533060 >> 172.21.15.5:0/3502533060 conn(0x7f66b4072ef0 msgr2=0x7f66b414ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:02.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:02.419+0000 7f66bc563640 1 -- 172.21.15.5:0/3502533060 shutdown_connections 2024-03-24T10:59:02.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:02.419+0000 7f66bc563640 1 -- 172.21.15.5:0/3502533060 wait complete. 2024-03-24T10:59:03.035 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:59:03.035 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:59:03.201 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:59:03.203 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:59:04.036 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T10:59:04.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.137+0000 7f3494d5c640 1 Processor -- start 2024-03-24T10:59:04.138 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.138+0000 7f3494d5c640 1 -- start start 2024-03-24T10:59:04.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.139+0000 7f3494d5c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f34900c9050 0x7f34900c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:04.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.139+0000 7f3494d5c640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f349015df30 con 0x7f3490154330 2024-03-24T10:59:04.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.139+0000 7f3494d5c640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f349015a960 con 0x7f349006bd60 2024-03-24T10:59:04.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.139+0000 7f3494d5c640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f3490154710 con 0x7f34900c9050 2024-03-24T10:59:04.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.140+0000 7f348e59c640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f3490154330 0x7f349015a250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:54502/0 (socket says 172.21.15.5:54502) 2024-03-24T10:59:04.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.140+0000 7f348e59c640 1 -- 172.21.15.5:0/3037436026 learned_addr learned my addr 172.21.15.5:0/3037436026 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:59:04.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.140+0000 7f348ed9d640 1 --2- 172.21.15.5:0/3037436026 >> v2:172.21.15.57:3300/0 conn(0x7f34900c9050 0x7f34900c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:04.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.142+0000 7f348d59a640 1 -- 172.21.15.5:0/3037436026 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1354969685 0 0) 0x7f349015df30 con 0x7f3490154330 2024-03-24T10:59:04.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.142+0000 7f348d59a640 1 -- 172.21.15.5:0/3037436026 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3478003520 con 0x7f3490154330 2024-03-24T10:59:04.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.145+0000 7f348ed9d640 1 -- 172.21.15.5:0/3037436026 >> v1:172.21.15.57:6789/0 conn(0x7f349006bd60 legacy=0x7f34900c8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:04.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.145+0000 7f348ed9d640 1 -- 172.21.15.5:0/3037436026 >> v1:172.21.15.5:6789/0 conn(0x7f3490154330 legacy=0x7f349015a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:04.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.145+0000 7f348ed9d640 1 -- 172.21.15.5:0/3037436026 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f349015b1a0 con 0x7f34900c9050 2024-03-24T10:59:04.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.150+0000 7f348ed9d640 1 --2- 172.21.15.5:0/3037436026 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34900c9050 0x7f34900c6fc0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3488009d90 tx=0x7f348802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ba90b797fbcdba5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:04.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.155+0000 7f348d59a640 1 -- 172.21.15.5:0/3037436026 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348802cd00 con 0x7f34900c9050 2024-03-24T10:59:04.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.155+0000 7f348d59a640 1 -- 172.21.15.5:0/3037436026 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f348802ce60 con 0x7f34900c9050 2024-03-24T10:59:04.155 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.155+0000 7f348d59a640 1 -- 172.21.15.5:0/3037436026 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3488034660 con 0x7f34900c9050 2024-03-24T10:59:04.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.156+0000 7f3494d5c640 1 -- 172.21.15.5:0/3037436026 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34900c9050 msgr2=0x7f34900c6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:04.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.156+0000 7f3494d5c640 1 --2- 172.21.15.5:0/3037436026 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34900c9050 0x7f34900c6fc0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3488009d90 tx=0x7f348802bf50 comp rx=0 tx=0).stop 2024-03-24T10:59:04.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.156+0000 7f3494d5c640 1 -- 172.21.15.5:0/3037436026 shutdown_connections 2024-03-24T10:59:04.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.156+0000 7f3494d5c640 1 --2- 172.21.15.5:0/3037436026 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f34900c9050 0x7f34900c6fc0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:04.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.156+0000 7f3494d5c640 1 -- 172.21.15.5:0/3037436026 >> 172.21.15.5:0/3037436026 conn(0x7f349006f8b0 msgr2=0x7f349006fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:04.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.156+0000 7f3494d5c640 1 -- 172.21.15.5:0/3037436026 shutdown_connections 2024-03-24T10:59:04.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.156+0000 7f3494d5c640 1 -- 172.21.15.5:0/3037436026 wait complete. 2024-03-24T10:59:04.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.157+0000 7f3494d5c640 1 Processor -- start 2024-03-24T10:59:04.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.157+0000 7f3494d5c640 1 -- start start 2024-03-24T10:59:04.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.158+0000 7f3494d5c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f349006bd60 0x7f34901508a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:04.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.158+0000 7f3494d5c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f34900c9050 0x7f34900c7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:04.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.158+0000 7f3494d5c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3490154330 0x7f34900c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:04.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.158+0000 7f3494d5c640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f34901efdb0 con 0x7f349006bd60 2024-03-24T10:59:04.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.158+0000 7f3494d5c640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f34901effc0 con 0x7f3490154330 2024-03-24T10:59:04.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.158+0000 7f3494d5c640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f34901f02a0 con 0x7f34900c9050 2024-03-24T10:59:04.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.158+0000 7f348dd9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f34900c9050 0x7f34900c7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:04.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.158+0000 7f348ed9d640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3490154330 0x7f34900c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:04.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.158+0000 7f348e59c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f349006bd60 0x7f34901508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:04.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.159+0000 7f348dd9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f34900c9050 0x7f34900c7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:48512/0 (socket says 172.21.15.5:48512) 2024-03-24T10:59:04.159 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.159+0000 7f348dd9b640 1 -- 172.21.15.5:0/3352910635 learned_addr learned my addr 172.21.15.5:0/3352910635 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:04.162 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.163+0000 7f348e59c640 1 -- 172.21.15.5:0/3352910635 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f34900c9050 msgr2=0x7f34900c7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:04.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.163+0000 7f348e59c640 1 --2- 172.21.15.5:0/3352910635 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f34900c9050 0x7f34900c7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:04.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.163+0000 7f348e59c640 1 -- 172.21.15.5:0/3352910635 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3490154330 msgr2=0x7f34900c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:04.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.163+0000 7f348e59c640 1 --2- 172.21.15.5:0/3352910635 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3490154330 0x7f34900c7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:04.163 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.163+0000 7f348e59c640 1 -- 172.21.15.5:0/3352910635 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3488009a70 con 0x7f349006bd60 2024-03-24T10:59:04.165 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.165+0000 7f348e59c640 1 --2- 172.21.15.5:0/3352910635 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f349006bd60 0x7f34901508a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f348000c3c0 tx=0x7f3480001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:04.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.169+0000 7f347effd640 1 -- 172.21.15.5:0/3352910635 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3480005ce0 con 0x7f349006bd60 2024-03-24T10:59:04.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.169+0000 7f347effd640 1 -- 172.21.15.5:0/3352910635 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3480005e40 con 0x7f349006bd60 2024-03-24T10:59:04.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.169+0000 7f347effd640 1 -- 172.21.15.5:0/3352910635 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34800072d0 con 0x7f349006bd60 2024-03-24T10:59:04.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.169+0000 7f3494d5c640 1 -- 172.21.15.5:0/3352910635 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34901f0580 con 0x7f349006bd60 2024-03-24T10:59:04.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.169+0000 7f3494d5c640 1 -- 172.21.15.5:0/3352910635 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34901f0b40 con 0x7f349006bd60 2024-03-24T10:59:04.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.170+0000 7f3494d5c640 1 -- 172.21.15.5:0/3352910635 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3490074da0 con 0x7f349006bd60 2024-03-24T10:59:04.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.441+0000 7f347effd640 1 -- 172.21.15.5:0/3352910635 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f34800074f0 con 0x7f349006bd60 2024-03-24T10:59:04.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.441+0000 7f347effd640 1 --2- 172.21.15.5:0/3352910635 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f347006b810 0x7f347006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:04.441 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.441+0000 7f348dd9b640 1 --2- 172.21.15.5:0/3352910635 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f347006b810 0x7f347006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:04.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.442+0000 7f348dd9b640 1 --2- 172.21.15.5:0/3352910635 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f347006b810 0x7f347006dcd0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f349015a930 tx=0x7f3484009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:04.480 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.480+0000 7f347effd640 1 -- 172.21.15.5:0/3352910635 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f34800583f0 con 0x7f349006bd60 2024-03-24T10:59:04.481 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.481+0000 7f347effd640 1 -- 172.21.15.5:0/3352910635 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3480054690 con 0x7f349006bd60 2024-03-24T10:59:04.719 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.719+0000 7f3494d5c640 1 -- 172.21.15.5:0/3352910635 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34900681d0 con 0x7f349006bd60 2024-03-24T10:59:04.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.759+0000 7f347effd640 1 -- 172.21.15.5:0/3352910635 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6259 (secure 0 0 0) 0x7f3480017020 con 0x7f349006bd60 2024-03-24T10:59:04.760 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:59:04.760 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T10:45:01.393211+0000","modified":"2024-03-24T10:57:20.978745+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4258,"mds_1":4248,"mds_2":4254,"mds_3":4252,"mds_4":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4248":{"gid":4248,"name":"c","rank":1,"incarnation":18,"state":"up:resolve","state_seq":21,"addr":"172.21.15.5:6837/1627030005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6836","nonce":1627030005},{"type":"v1","addr":"172.21.15.5:6837","nonce":1627030005}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4251":{"gid":4251,"name":"b","rank":4,"incarnation":16,"state":"up:active","state_seq":7,"addr":"172.21.15.57:6836/556535972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":556535972},{"type":"v1","addr":"172.21.15.57:6836","nonce":556535972}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4252":{"gid":4252,"name":"e","rank":3,"incarnation":14,"state":"up:active","state_seq":7,"addr":"172.21.15.5:6835/3042643804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6834","nonce":3042643804},{"type":"v1","addr":"172.21.15.5:6835","nonce":3042643804}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:17.929977+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4254":{"gid":4254,"name":"d","rank":2,"incarnation":12,"state":"up:active","state_seq":6,"addr":"172.21.15.57:6833/173074461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6832","nonce":173074461},{"type":"v1","addr":"172.21.15.57:6833","nonce":173074461}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:46:12.913281+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4258":{"gid":4258,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":28,"addr":"172.21.15.5:6839/1054678798","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6838","nonce":1054678798},{"type":"v1","addr":"172.21.15.5:6839","nonce":1054678798}]},"join_fscid":-1,"laggy_since":"2024-03-24T10:57:20.978744+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4258,"qdb_cluster":[4251,4252,4254,4258]},"id":1}]} 2024-03-24T10:59:04.760 INFO:teuthology.orchestra.run.smithi005.stderr:dumped fsmap epoch 24 2024-03-24T10:59:04.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.761+0000 7f3494d5c640 1 -- 172.21.15.5:0/3352910635 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f347006b810 msgr2=0x7f347006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:04.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.761+0000 7f3494d5c640 1 --2- 172.21.15.5:0/3352910635 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f347006b810 0x7f347006dcd0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f349015a930 tx=0x7f3484009040 comp rx=0 tx=0).stop 2024-03-24T10:59:04.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.761+0000 7f3494d5c640 1 -- 172.21.15.5:0/3352910635 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f349006bd60 msgr2=0x7f34901508a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:04.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.761+0000 7f3494d5c640 1 --2- 172.21.15.5:0/3352910635 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f349006bd60 0x7f34901508a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f348000c3c0 tx=0x7f3480001e70 comp rx=0 tx=0).stop 2024-03-24T10:59:04.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.761+0000 7f3494d5c640 1 -- 172.21.15.5:0/3352910635 shutdown_connections 2024-03-24T10:59:04.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.761+0000 7f3494d5c640 1 --2- 172.21.15.5:0/3352910635 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f347006b810 0x7f347006dcd0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:04.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.761+0000 7f3494d5c640 1 --2- 172.21.15.5:0/3352910635 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3490154330 0x7f34900c7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:04.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.761+0000 7f3494d5c640 1 --2- 172.21.15.5:0/3352910635 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f34900c9050 0x7f34900c7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:04.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.761+0000 7f3494d5c640 1 --2- 172.21.15.5:0/3352910635 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f349006bd60 0x7f34901508a0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:04.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.761+0000 7f3494d5c640 1 -- 172.21.15.5:0/3352910635 >> 172.21.15.5:0/3352910635 conn(0x7f349006f8b0 msgr2=0x7f3490148dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:04.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.761+0000 7f3494d5c640 1 -- 172.21.15.5:0/3352910635 shutdown_connections 2024-03-24T10:59:04.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:04.761+0000 7f3494d5c640 1 -- 172.21.15.5:0/3352910635 wait complete. 2024-03-24T10:59:05.380 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T10:45:01.393211+0000', 'modified': '2024-03-24T10:57:20.978745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 24, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4258, 'mds_1': 4248, 'mds_2': 4254, 'mds_3': 4252, 'mds_4': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4248': {'gid': 4248, 'name': 'c', 'rank': 1, 'incarnation': 18, 'state': 'up:resolve', 'state_seq': 21, 'addr': '172.21.15.5:6837/1627030005', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6836', 'nonce': 1627030005}, {'type': 'v1', 'addr': '172.21.15.5:6837', 'nonce': 1627030005}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4251': {'gid': 4251, 'name': 'b', 'rank': 4, 'incarnation': 16, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.57:6836/556535972', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6834', 'nonce': 556535972}, {'type': 'v1', 'addr': '172.21.15.57:6836', 'nonce': 556535972}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929677+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4252': {'gid': 4252, 'name': 'e', 'rank': 3, 'incarnation': 14, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.5:6835/3042643804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6834', 'nonce': 3042643804}, {'type': 'v1', 'addr': '172.21.15.5:6835', 'nonce': 3042643804}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:17.929977+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4254': {'gid': 4254, 'name': 'd', 'rank': 2, 'incarnation': 12, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.57:6833/173074461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.57:6832', 'nonce': 173074461}, {'type': 'v1', 'addr': '172.21.15.57:6833', 'nonce': 173074461}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:46:12.913281+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4258': {'gid': 4258, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.5:6839/1054678798', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.5:6838', 'nonce': 1054678798}, {'type': 'v1', 'addr': '172.21.15.5:6839', 'nonce': 1054678798}]}, 'join_fscid': -1, 'laggy_since': '2024-03-24T10:57:20.978744+0000', 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4258, 'qdb_cluster': [4251, 4252, 4254, 4258]} 2024-03-24T10:59:05.380 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4248:up:resolve 2024-03-24T10:59:06.382 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 24, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-03-24T10:45:01.393211+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 24, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4248": { "addr": "172.21.15.5:6837/1627030005", "addrs": { "addrvec": [ { "addr": "172.21.15.5:6836", "nonce": 1627030005, "type": "v2" }, { "addr": "172.21.15.5:6837", "nonce": 1627030005, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4248, "incarnation": 18, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:resolve", "state_seq": 21 }, "gid_4251": { "addr": "172.21.15.57:6836/556535972", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6834", "nonce": 556535972, "type": "v2" }, { "addr": "172.21.15.57:6836", "nonce": 556535972, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4251, "incarnation": 16, "join_fscid": -1, "laggy_since": "2024-03-24T10:46:17.929677+0000", "name": "b", "rank": 4, "state": "up:active", "state_seq": 7 }, "gid_4252": { "addr": "172.21.15.5:6835/3042643804", "addrs": { "addrvec": [ { "addr": "172.21.15.5:6834", "nonce": 3042643804, "type": "v2" }, { "addr": "172.21.15.5:6835", "nonce": 3042643804, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4252, "incarnation": 14, "join_fscid": -1, "laggy_since": "2024-03-24T10:46:17.929977+0000", "name": "e", "rank": 3, "state": "up:active", "state_seq": 7 }, "gid_4254": { "addr": "172.21.15.57:6833/173074461", "addrs": { "addrvec": [ { "addr": "172.21.15.57:6832", "nonce": 173074461, "type": "v2" }, { "addr": "172.21.15.57:6833", "nonce": 173074461, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4254, "incarnation": 12, "join_fscid": -1, "laggy_since": "2024-03-24T10:46:12.913281+0000", "name": "d", "rank": 2, "state": "up:active", "state_seq": 6 }, "gid_4258": { "addr": "172.21.15.5:6839/1054678798", "addrs": { "addrvec": [ { "addr": "172.21.15.5:6838", "nonce": 1054678798, "type": "v2" }, { "addr": "172.21.15.5:6839", "nonce": 1054678798, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4258, "incarnation": 4, "join_fscid": -1, "laggy_since": "2024-03-24T10:57:20.978744+0000", "name": "a", "rank": 0, "state": "up:active", "state_seq": 28 } }, "last_failure": 0, "last_failure_osd_epoch": 24, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-03-24T10:57:20.978745+0000", "qdb_cluster": [ 4251, 4252, 4254, 4258 ], "qdb_leader": 4258, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4258, "mds_1": 4248, "mds_2": 4254, "mds_3": 4252, "mds_4": 4251 } } } ], "standbys": [] } 2024-03-24T10:59:06.385 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-24T10:59:06.385 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:59:06.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.488+0000 7f3e48eae640 1 Processor -- start 2024-03-24T10:59:06.488 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.488+0000 7f3e48eae640 1 -- start start 2024-03-24T10:59:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.489+0000 7f3e48eae640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e44150fd0 0x7f3e440c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.489+0000 7f3e48eae640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3e441513b0 0x7f3e4415e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.489+0000 7f3e48eae640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e4406ca40 con 0x7f3e4415aea0 2024-03-24T10:59:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.489+0000 7f3e48eae640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f3e4415e100 con 0x7f3e441513b0 2024-03-24T10:59:06.489 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.489+0000 7f3e48eae640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f3e4406b240 con 0x7f3e44150fd0 2024-03-24T10:59:06.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.489+0000 7f3e3bfff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3e441513b0 0x7f3e4415e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:06.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.490+0000 7f3e4259c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e44150fd0 0x7f3e440c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:06.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.490+0000 7f3e4259c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e44150fd0 0x7f3e440c70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:48514/0 (socket says 172.21.15.5:48514) 2024-03-24T10:59:06.490 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.490+0000 7f3e4259c640 1 -- 172.21.15.5:0/3714017562 learned_addr learned my addr 172.21.15.5:0/3714017562 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:06.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.492+0000 7f3e4259c640 1 -- 172.21.15.5:0/3714017562 >> v2:172.21.15.5:3300/0 conn(0x7f3e441513b0 msgr2=0x7f3e4415e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:06.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.492+0000 7f3e4259c640 1 --2- 172.21.15.5:0/3714017562 >> v2:172.21.15.5:3300/0 conn(0x7f3e441513b0 0x7f3e4415e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:06.492 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.492+0000 7f3e4259c640 1 -- 172.21.15.5:0/3714017562 >> v1:172.21.15.5:6789/0 conn(0x7f3e4415aea0 legacy=0x7f3e4406bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:06.493 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.492+0000 7f3e4259c640 1 -- 172.21.15.5:0/3714017562 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e4406b520 con 0x7f3e44150fd0 2024-03-24T10:59:06.494 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.494+0000 7f3e4259c640 1 --2- 172.21.15.5:0/3714017562 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e44150fd0 0x7f3e440c70f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3e340098a0 tx=0x7f3e3402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acaed4fe618a8b63 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:06.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.499+0000 7f3e41d9b640 1 -- 172.21.15.5:0/3714017562 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e34031640 con 0x7f3e44150fd0 2024-03-24T10:59:06.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.499+0000 7f3e41d9b640 1 -- 172.21.15.5:0/3714017562 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e340317a0 con 0x7f3e44150fd0 2024-03-24T10:59:06.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.499+0000 7f3e41d9b640 1 -- 172.21.15.5:0/3714017562 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e34031b30 con 0x7f3e44150fd0 2024-03-24T10:59:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.500+0000 7f3e48eae640 1 -- 172.21.15.5:0/3714017562 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e44150fd0 msgr2=0x7f3e440c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.500+0000 7f3e48eae640 1 --2- 172.21.15.5:0/3714017562 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e44150fd0 0x7f3e440c70f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3e340098a0 tx=0x7f3e3402b8f0 comp rx=0 tx=0).stop 2024-03-24T10:59:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.500+0000 7f3e48eae640 1 -- 172.21.15.5:0/3714017562 shutdown_connections 2024-03-24T10:59:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.500+0000 7f3e48eae640 1 --2- 172.21.15.5:0/3714017562 >> v2:172.21.15.5:3300/0 conn(0x7f3e441513b0 0x7f3e4415e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.500+0000 7f3e48eae640 1 --2- 172.21.15.5:0/3714017562 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e44150fd0 0x7f3e440c70f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.500+0000 7f3e48eae640 1 -- 172.21.15.5:0/3714017562 >> 172.21.15.5:0/3714017562 conn(0x7f3e4406f8b0 msgr2=0x7f3e4406fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.500+0000 7f3e48eae640 1 -- 172.21.15.5:0/3714017562 shutdown_connections 2024-03-24T10:59:06.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.500+0000 7f3e48eae640 1 -- 172.21.15.5:0/3714017562 wait complete. 2024-03-24T10:59:06.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.501+0000 7f3e48eae640 1 Processor -- start 2024-03-24T10:59:06.501 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.501+0000 7f3e48eae640 1 -- start start 2024-03-24T10:59:06.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.502+0000 7f3e48eae640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e44150fd0 0x7f3e4414c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:06.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.502+0000 7f3e48eae640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e441513b0 0x7f3e4414ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:06.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.502+0000 7f3e48eae640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e4415aea0 0x7f3e440c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:06.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.502+0000 7f3e48eae640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4414b9c0 con 0x7f3e44150fd0 2024-03-24T10:59:06.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.502+0000 7f3e48eae640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3e44145290 con 0x7f3e441513b0 2024-03-24T10:59:06.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.502+0000 7f3e48eae640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f3e4414c580 con 0x7f3e4415aea0 2024-03-24T10:59:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.502+0000 7f3e4259c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e44150fd0 0x7f3e4414c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.503+0000 7f3e42d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e4415aea0 0x7f3e440c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.503+0000 7f3e3bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e441513b0 0x7f3e4414ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.503+0000 7f3e4259c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e44150fd0 0x7f3e4414c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44058/0 (socket says 172.21.15.5:44058) 2024-03-24T10:59:06.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.503+0000 7f3e4259c640 1 -- 172.21.15.5:0/3173227629 learned_addr learned my addr 172.21.15.5:0/3173227629 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:06.506 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.507+0000 7f3e42d9d640 1 -- 172.21.15.5:0/3173227629 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e441513b0 msgr2=0x7f3e4414ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:06.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.507+0000 7f3e42d9d640 1 --2- 172.21.15.5:0/3173227629 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e441513b0 0x7f3e4414ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:06.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.507+0000 7f3e42d9d640 1 -- 172.21.15.5:0/3173227629 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e44150fd0 msgr2=0x7f3e4414c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:06.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.507+0000 7f3e42d9d640 1 --2- 172.21.15.5:0/3173227629 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e44150fd0 0x7f3e4414c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:06.507 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.507+0000 7f3e42d9d640 1 -- 172.21.15.5:0/3173227629 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e340094d0 con 0x7f3e4415aea0 2024-03-24T10:59:06.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.509+0000 7f3e42d9d640 1 --2- 172.21.15.5:0/3173227629 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e4415aea0 0x7f3e440c55f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00c780 tx=0x7f3e3c00cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.513+0000 7f3e3affd640 1 -- 172.21.15.5:0/3173227629 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c003b70 con 0x7f3e4415aea0 2024-03-24T10:59:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.514+0000 7f3e3affd640 1 -- 172.21.15.5:0/3173227629 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e3c003cf0 con 0x7f3e4415aea0 2024-03-24T10:59:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.514+0000 7f3e3affd640 1 -- 172.21.15.5:0/3173227629 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c0052b0 con 0x7f3e4415aea0 2024-03-24T10:59:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.514+0000 7f3e48eae640 1 -- 172.21.15.5:0/3173227629 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e4415cb90 con 0x7f3e4415aea0 2024-03-24T10:59:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.514+0000 7f3e48eae640 1 -- 172.21.15.5:0/3173227629 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e4415c660 con 0x7f3e4415aea0 2024-03-24T10:59:06.514 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.515+0000 7f3e48eae640 1 -- 172.21.15.5:0/3173227629 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e44145440 con 0x7f3e4415aea0 2024-03-24T10:59:06.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.535+0000 7f3e3affd640 1 -- 172.21.15.5:0/3173227629 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3e3c005410 con 0x7f3e4415aea0 2024-03-24T10:59:06.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.535+0000 7f3e3affd640 1 --2- 172.21.15.5:0/3173227629 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3e2406b840 0x7f3e2406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:06.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.536+0000 7f3e4259c640 1 --2- 172.21.15.5:0/3173227629 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3e2406b840 0x7f3e2406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:06.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.536+0000 7f3e4259c640 1 --2- 172.21.15.5:0/3173227629 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3e2406b840 0x7f3e2406dd00 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3e340090b0 tx=0x7f3e3403c040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:06.785 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.785+0000 7f3e3affd640 1 -- 172.21.15.5:0/3173227629 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f3e3c0582e0 con 0x7f3e4415aea0 2024-03-24T10:59:06.798 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.797+0000 7f3e3affd640 1 -- 172.21.15.5:0/3173227629 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3e3c054580 con 0x7f3e4415aea0 2024-03-24T10:59:06.982 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.981+0000 7f3e48eae640 1 -- 172.21.15.5:0/3173227629 --> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3e4406b800 con 0x7f3e2406b840 2024-03-24T10:59:06.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.994+0000 7f3e3affd640 1 -- 172.21.15.5:0/3173227629 <== mgr.4120 v2:172.21.15.5:6832/1102439181 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251997 (secure 0 0 0) 0x7f3e34031440 con 0x7f3e2406b840 2024-03-24T10:59:06.999 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:59:07.010 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-03-24T10:59:07.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.998+0000 7f3e48eae640 1 -- 172.21.15.5:0/3173227629 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3e2406b840 msgr2=0x7f3e2406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:07.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.998+0000 7f3e48eae640 1 --2- 172.21.15.5:0/3173227629 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3e2406b840 0x7f3e2406dd00 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3e340090b0 tx=0x7f3e3403c040 comp rx=0 tx=0).stop 2024-03-24T10:59:07.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.998+0000 7f3e48eae640 1 -- 172.21.15.5:0/3173227629 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e4415aea0 msgr2=0x7f3e440c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:07.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.998+0000 7f3e48eae640 1 --2- 172.21.15.5:0/3173227629 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e4415aea0 0x7f3e440c55f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00c780 tx=0x7f3e3c00cc40 comp rx=0 tx=0).stop 2024-03-24T10:59:07.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.998+0000 7f3e48eae640 1 -- 172.21.15.5:0/3173227629 shutdown_connections 2024-03-24T10:59:07.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.998+0000 7f3e48eae640 1 --2- 172.21.15.5:0/3173227629 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3e2406b840 0x7f3e2406dd00 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.998+0000 7f3e48eae640 1 --2- 172.21.15.5:0/3173227629 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3e4415aea0 0x7f3e440c55f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.998+0000 7f3e48eae640 1 --2- 172.21.15.5:0/3173227629 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3e441513b0 0x7f3e4414ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.998+0000 7f3e48eae640 1 --2- 172.21.15.5:0/3173227629 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3e44150fd0 0x7f3e4414c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.015 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.998+0000 7f3e48eae640 1 -- 172.21.15.5:0/3173227629 >> 172.21.15.5:0/3173227629 conn(0x7f3e4406f8b0 msgr2=0x7f3e4406fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:07.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.998+0000 7f3e48eae640 1 -- 172.21.15.5:0/3173227629 shutdown_connections 2024-03-24T10:59:07.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:06.998+0000 7f3e48eae640 1 -- 172.21.15.5:0/3173227629 wait complete. 2024-03-24T10:59:07.016 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":448,"stamp":"2024-03-24T10:59:05.544851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":363,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":301,"ondisk_log_size":301,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239296,"kb_used_data":2336,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754735424,"statfs":{"total":773094113280,"available":772849074176,"internally_reserved":0,"allocated":2392064,"data_stored":1937476,"data_compressed":65430,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":242535986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002273"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.230562+0000","last_change":"2024-03-24T10:45:01.894006+0000","last_active":"2024-03-24T10:55:26.230562+0000","last_peered":"2024-03-24T10:55:26.230562+0000","last_clean":"2024-03-24T10:55:26.230562+0000","last_became_active":"2024-03-24T10:45:01.893734+0000","last_became_peered":"2024-03-24T10:45:01.893734+0000","last_unstale":"2024-03-24T10:55:26.230562+0000","last_undegraded":"2024-03-24T10:55:26.230562+0000","last_fullsized":"2024-03-24T10:55:26.230562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:05:51.579388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.323823+0000","last_change":"2024-03-24T10:45:01.931707+0000","last_active":"2024-03-24T10:55:26.323823+0000","last_peered":"2024-03-24T10:55:26.323823+0000","last_clean":"2024-03-24T10:55:26.323823+0000","last_became_active":"2024-03-24T10:45:01.931449+0000","last_became_peered":"2024-03-24T10:45:01.931449+0000","last_unstale":"2024-03-24T10:55:26.323823+0000","last_undegraded":"2024-03-24T10:55:26.323823+0000","last_fullsized":"2024-03-24T10:55:26.323823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.215675+0000","last_change":"2024-03-24T10:45:01.895929+0000","last_active":"2024-03-24T10:55:29.215675+0000","last_peered":"2024-03-24T10:55:29.215675+0000","last_clean":"2024-03-24T10:55:29.215675+0000","last_became_active":"2024-03-24T10:45:01.895654+0000","last_became_peered":"2024-03-24T10:45:01.895654+0000","last_unstale":"2024-03-24T10:55:29.215675+0000","last_undegraded":"2024-03-24T10:55:29.215675+0000","last_fullsized":"2024-03-24T10:55:29.215675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:01.243946+0000","last_change":"2024-03-24T10:45:01.952468+0000","last_active":"2024-03-24T10:57:01.243946+0000","last_peered":"2024-03-24T10:57:01.243946+0000","last_clean":"2024-03-24T10:57:01.243946+0000","last_became_active":"2024-03-24T10:45:01.952219+0000","last_became_peered":"2024-03-24T10:45:01.952219+0000","last_unstale":"2024-03-24T10:57:01.243946+0000","last_undegraded":"2024-03-24T10:57:01.243946+0000","last_fullsized":"2024-03-24T10:57:01.243946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:27:12.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.228132+0000","last_change":"2024-03-24T10:45:01.895885+0000","last_active":"2024-03-24T10:55:01.228132+0000","last_peered":"2024-03-24T10:55:01.228132+0000","last_clean":"2024-03-24T10:55:01.228132+0000","last_became_active":"2024-03-24T10:45:01.895622+0000","last_became_peered":"2024-03-24T10:45:01.895622+0000","last_unstale":"2024-03-24T10:55:01.228132+0000","last_undegraded":"2024-03-24T10:55:01.228132+0000","last_fullsized":"2024-03-24T10:55:01.228132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:16.523177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:41.320408+0000","last_change":"2024-03-24T10:45:40.169714+0000","last_active":"2024-03-24T10:55:41.320408+0000","last_peered":"2024-03-24T10:55:41.320408+0000","last_clean":"2024-03-24T10:55:41.320408+0000","last_became_active":"2024-03-24T10:45:40.169447+0000","last_became_peered":"2024-03-24T10:45:40.169447+0000","last_unstale":"2024-03-24T10:55:41.320408+0000","last_undegraded":"2024-03-24T10:55:41.320408+0000","last_fullsized":"2024-03-24T10:55:41.320408+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:27:12.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.228112+0000","last_change":"2024-03-24T10:45:01.892745+0000","last_active":"2024-03-24T10:55:01.228112+0000","last_peered":"2024-03-24T10:55:01.228112+0000","last_clean":"2024-03-24T10:55:01.228112+0000","last_became_active":"2024-03-24T10:45:01.892452+0000","last_became_peered":"2024-03-24T10:45:01.892452+0000","last_unstale":"2024-03-24T10:55:01.228112+0000","last_undegraded":"2024-03-24T10:55:01.228112+0000","last_fullsized":"2024-03-24T10:55:01.228112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.198373+0000","last_change":"2024-03-24T10:45:01.953432+0000","last_active":"2024-03-24T10:55:01.198373+0000","last_peered":"2024-03-24T10:55:01.198373+0000","last_clean":"2024-03-24T10:55:01.198373+0000","last_became_active":"2024-03-24T10:45:01.953149+0000","last_became_peered":"2024-03-24T10:45:01.953149+0000","last_unstale":"2024-03-24T10:55:01.198373+0000","last_undegraded":"2024-03-24T10:55:01.198373+0000","last_fullsized":"2024-03-24T10:55:01.198373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.226282+0000","last_change":"2024-03-24T10:45:01.956488+0000","last_active":"2024-03-24T10:54:41.226282+0000","last_peered":"2024-03-24T10:54:41.226282+0000","last_clean":"2024-03-24T10:54:41.226282+0000","last_became_active":"2024-03-24T10:45:01.956207+0000","last_became_peered":"2024-03-24T10:45:01.956207+0000","last_unstale":"2024-03-24T10:54:41.226282+0000","last_undegraded":"2024-03-24T10:54:41.226282+0000","last_fullsized":"2024-03-24T10:54:41.226282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:23:26.523809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:44.289850+0000","last_change":"2024-03-24T10:45:01.956484+0000","last_active":"2024-03-24T10:54:44.289850+0000","last_peered":"2024-03-24T10:54:44.289850+0000","last_clean":"2024-03-24T10:54:44.289850+0000","last_became_active":"2024-03-24T10:45:01.956209+0000","last_became_peered":"2024-03-24T10:45:01.956209+0000","last_unstale":"2024-03-24T10:54:44.289850+0000","last_undegraded":"2024-03-24T10:54:44.289850+0000","last_fullsized":"2024-03-24T10:54:44.289850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:44.289823+0000","last_change":"2024-03-24T10:45:01.957103+0000","last_active":"2024-03-24T10:54:44.289823+0000","last_peered":"2024-03-24T10:54:44.289823+0000","last_clean":"2024-03-24T10:54:44.289823+0000","last_became_active":"2024-03-24T10:45:01.956849+0000","last_became_peered":"2024-03-24T10:45:01.956849+0000","last_unstale":"2024-03-24T10:54:44.289823+0000","last_undegraded":"2024-03-24T10:54:44.289823+0000","last_fullsized":"2024-03-24T10:54:44.289823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:44.262492+0000","last_change":"2024-03-24T10:45:01.896701+0000","last_active":"2024-03-24T10:54:44.262492+0000","last_peered":"2024-03-24T10:54:44.262492+0000","last_clean":"2024-03-24T10:54:44.262492+0000","last_became_active":"2024-03-24T10:45:01.896400+0000","last_became_peered":"2024-03-24T10:45:01.896400+0000","last_unstale":"2024-03-24T10:54:44.262492+0000","last_undegraded":"2024-03-24T10:54:44.262492+0000","last_fullsized":"2024-03-24T10:54:44.262492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:44.380352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.196593+0000","last_change":"2024-03-24T10:45:01.894954+0000","last_active":"2024-03-24T10:54:41.196593+0000","last_peered":"2024-03-24T10:54:41.196593+0000","last_clean":"2024-03-24T10:54:41.196593+0000","last_became_active":"2024-03-24T10:45:01.894632+0000","last_became_peered":"2024-03-24T10:45:01.894632+0000","last_unstale":"2024-03-24T10:54:41.196593+0000","last_undegraded":"2024-03-24T10:54:41.196593+0000","last_fullsized":"2024-03-24T10:54:41.196593+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.221985+0000","last_change":"2024-03-24T10:45:01.929588+0000","last_active":"2024-03-24T10:56:44.221985+0000","last_peered":"2024-03-24T10:56:44.221985+0000","last_clean":"2024-03-24T10:56:44.221985+0000","last_became_active":"2024-03-24T10:45:01.928460+0000","last_became_peered":"2024-03-24T10:45:01.928460+0000","last_unstale":"2024-03-24T10:56:44.221985+0000","last_undegraded":"2024-03-24T10:56:44.221985+0000","last_fullsized":"2024-03-24T10:56:44.221985+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:47:48.896914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.329359+0000","last_change":"2024-03-24T10:45:01.928721+0000","last_active":"2024-03-24T10:56:41.329359+0000","last_peered":"2024-03-24T10:56:41.329359+0000","last_clean":"2024-03-24T10:56:41.329359+0000","last_became_active":"2024-03-24T10:45:01.928467+0000","last_became_peered":"2024-03-24T10:45:01.928467+0000","last_unstale":"2024-03-24T10:56:41.329359+0000","last_undegraded":"2024-03-24T10:56:41.329359+0000","last_fullsized":"2024-03-24T10:56:41.329359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.242233+0000","last_change":"2024-03-24T10:45:01.956030+0000","last_active":"2024-03-24T10:56:41.242233+0000","last_peered":"2024-03-24T10:56:41.242233+0000","last_clean":"2024-03-24T10:56:41.242233+0000","last_became_active":"2024-03-24T10:45:01.955768+0000","last_became_peered":"2024-03-24T10:45:01.955768+0000","last_unstale":"2024-03-24T10:56:41.242233+0000","last_undegraded":"2024-03-24T10:56:41.242233+0000","last_fullsized":"2024-03-24T10:56:41.242233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.242276+0000","last_change":"2024-03-24T10:45:01.956181+0000","last_active":"2024-03-24T10:56:41.242276+0000","last_peered":"2024-03-24T10:56:41.242276+0000","last_clean":"2024-03-24T10:56:41.242276+0000","last_became_active":"2024-03-24T10:45:01.955890+0000","last_became_peered":"2024-03-24T10:45:01.955890+0000","last_unstale":"2024-03-24T10:56:41.242276+0000","last_undegraded":"2024-03-24T10:56:41.242276+0000","last_fullsized":"2024-03-24T10:56:41.242276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:15:11.107915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.325044+0000","last_change":"2024-03-24T10:45:01.896663+0000","last_active":"2024-03-24T10:56:41.325044+0000","last_peered":"2024-03-24T10:56:41.325044+0000","last_clean":"2024-03-24T10:56:41.325044+0000","last_became_active":"2024-03-24T10:45:01.896391+0000","last_became_peered":"2024-03-24T10:45:01.896391+0000","last_unstale":"2024-03-24T10:56:41.325044+0000","last_undegraded":"2024-03-24T10:56:41.325044+0000","last_fullsized":"2024-03-24T10:56:41.325044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:30.942758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.235423+0000","last_change":"2024-03-24T10:45:01.895951+0000","last_active":"2024-03-24T10:56:16.235423+0000","last_peered":"2024-03-24T10:56:16.235423+0000","last_clean":"2024-03-24T10:56:16.235423+0000","last_became_active":"2024-03-24T10:45:01.895668+0000","last_became_peered":"2024-03-24T10:45:01.895668+0000","last_unstale":"2024-03-24T10:56:16.235423+0000","last_undegraded":"2024-03-24T10:56:16.235423+0000","last_fullsized":"2024-03-24T10:56:16.235423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:56:43.462458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:19.297955+0000","last_change":"2024-03-24T10:45:01.898126+0000","last_active":"2024-03-24T10:56:19.297955+0000","last_peered":"2024-03-24T10:56:19.297955+0000","last_clean":"2024-03-24T10:56:19.297955+0000","last_became_active":"2024-03-24T10:45:01.897878+0000","last_became_peered":"2024-03-24T10:45:01.897878+0000","last_unstale":"2024-03-24T10:56:19.297955+0000","last_undegraded":"2024-03-24T10:56:19.297955+0000","last_fullsized":"2024-03-24T10:56:19.297955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:19.220025+0000","last_change":"2024-03-24T10:45:01.896109+0000","last_active":"2024-03-24T10:56:19.220025+0000","last_peered":"2024-03-24T10:56:19.220025+0000","last_clean":"2024-03-24T10:56:19.220025+0000","last_became_active":"2024-03-24T10:45:01.895696+0000","last_became_peered":"2024-03-24T10:45:01.895696+0000","last_unstale":"2024-03-24T10:56:19.220025+0000","last_undegraded":"2024-03-24T10:56:19.220025+0000","last_fullsized":"2024-03-24T10:56:19.220025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.240130+0000","last_change":"2024-03-24T10:45:01.958119+0000","last_active":"2024-03-24T10:56:16.240130+0000","last_peered":"2024-03-24T10:56:16.240130+0000","last_clean":"2024-03-24T10:56:16.240130+0000","last_became_active":"2024-03-24T10:45:01.957832+0000","last_became_peered":"2024-03-24T10:45:01.957832+0000","last_unstale":"2024-03-24T10:56:16.240130+0000","last_undegraded":"2024-03-24T10:56:16.240130+0000","last_fullsized":"2024-03-24T10:56:16.240130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:12.476490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.204495+0000","last_change":"2024-03-24T10:45:01.842442+0000","last_active":"2024-03-24T10:56:16.204495+0000","last_peered":"2024-03-24T10:56:16.204495+0000","last_clean":"2024-03-24T10:56:16.204495+0000","last_became_active":"2024-03-24T10:45:01.842199+0000","last_became_peered":"2024-03-24T10:45:01.842199+0000","last_unstale":"2024-03-24T10:56:16.204495+0000","last_undegraded":"2024-03-24T10:56:16.204495+0000","last_fullsized":"2024-03-24T10:56:16.204495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.237949+0000","last_change":"2024-03-24T10:45:01.952378+0000","last_active":"2024-03-24T10:55:51.237949+0000","last_peered":"2024-03-24T10:55:51.237949+0000","last_clean":"2024-03-24T10:55:51.237949+0000","last_became_active":"2024-03-24T10:45:01.952123+0000","last_became_peered":"2024-03-24T10:45:01.952123+0000","last_unstale":"2024-03-24T10:55:51.237949+0000","last_undegraded":"2024-03-24T10:55:51.237949+0000","last_fullsized":"2024-03-24T10:55:51.237949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.202466+0000","last_change":"2024-03-24T10:45:01.842299+0000","last_active":"2024-03-24T10:55:51.202466+0000","last_peered":"2024-03-24T10:55:51.202466+0000","last_clean":"2024-03-24T10:55:51.202466+0000","last_became_active":"2024-03-24T10:45:01.842097+0000","last_became_peered":"2024-03-24T10:45:01.842097+0000","last_unstale":"2024-03-24T10:55:51.202466+0000","last_undegraded":"2024-03-24T10:55:51.202466+0000","last_fullsized":"2024-03-24T10:55:51.202466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:16.341360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:54.295744+0000","last_change":"2024-03-24T10:45:01.893579+0000","last_active":"2024-03-24T10:55:54.295744+0000","last_peered":"2024-03-24T10:55:54.295744+0000","last_clean":"2024-03-24T10:55:54.295744+0000","last_became_active":"2024-03-24T10:45:01.893273+0000","last_became_peered":"2024-03-24T10:45:01.893273+0000","last_unstale":"2024-03-24T10:55:54.295744+0000","last_undegraded":"2024-03-24T10:55:54.295744+0000","last_fullsized":"2024-03-24T10:55:54.295744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.325622+0000","last_change":"2024-03-24T10:45:01.930032+0000","last_active":"2024-03-24T10:55:51.325622+0000","last_peered":"2024-03-24T10:55:51.325622+0000","last_clean":"2024-03-24T10:55:51.325622+0000","last_became_active":"2024-03-24T10:45:01.929770+0000","last_became_peered":"2024-03-24T10:45:01.929770+0000","last_unstale":"2024-03-24T10:55:51.325622+0000","last_undegraded":"2024-03-24T10:55:51.325622+0000","last_fullsized":"2024-03-24T10:55:51.325622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.202455+0000","last_change":"2024-03-24T10:45:01.842460+0000","last_active":"2024-03-24T10:55:51.202455+0000","last_peered":"2024-03-24T10:55:51.202455+0000","last_clean":"2024-03-24T10:55:51.202455+0000","last_became_active":"2024-03-24T10:45:01.842165+0000","last_became_peered":"2024-03-24T10:45:01.842165+0000","last_unstale":"2024-03-24T10:55:51.202455+0000","last_undegraded":"2024-03-24T10:55:51.202455+0000","last_fullsized":"2024-03-24T10:55:51.202455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.215656+0000","last_change":"2024-03-24T10:45:01.895868+0000","last_active":"2024-03-24T10:55:29.215656+0000","last_peered":"2024-03-24T10:55:29.215656+0000","last_clean":"2024-03-24T10:55:29.215656+0000","last_became_active":"2024-03-24T10:45:01.895290+0000","last_became_peered":"2024-03-24T10:45:01.895290+0000","last_unstale":"2024-03-24T10:55:29.215656+0000","last_undegraded":"2024-03-24T10:55:29.215656+0000","last_fullsized":"2024-03-24T10:55:29.215656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.235695+0000","last_change":"2024-03-24T10:45:01.957724+0000","last_active":"2024-03-24T10:55:26.235695+0000","last_peered":"2024-03-24T10:55:26.235695+0000","last_clean":"2024-03-24T10:55:26.235695+0000","last_became_active":"2024-03-24T10:45:01.957429+0000","last_became_peered":"2024-03-24T10:45:01.957429+0000","last_unstale":"2024-03-24T10:55:26.235695+0000","last_undegraded":"2024-03-24T10:55:26.235695+0000","last_fullsized":"2024-03-24T10:55:26.235695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:22.089070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.319191+0000","last_change":"2024-03-24T10:45:01.952159+0000","last_active":"2024-03-24T10:55:26.319191+0000","last_peered":"2024-03-24T10:55:26.319191+0000","last_clean":"2024-03-24T10:55:26.319191+0000","last_became_active":"2024-03-24T10:45:01.951961+0000","last_became_peered":"2024-03-24T10:45:01.951961+0000","last_unstale":"2024-03-24T10:55:26.319191+0000","last_undegraded":"2024-03-24T10:55:26.319191+0000","last_fullsized":"2024-03-24T10:55:26.319191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:44.380352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.293413+0000","last_change":"2024-03-24T10:45:01.893452+0000","last_active":"2024-03-24T10:55:29.293413+0000","last_peered":"2024-03-24T10:55:29.293413+0000","last_clean":"2024-03-24T10:55:29.293413+0000","last_became_active":"2024-03-24T10:45:01.893139+0000","last_became_peered":"2024-03-24T10:45:01.893139+0000","last_unstale":"2024-03-24T10:55:29.293413+0000","last_undegraded":"2024-03-24T10:55:29.293413+0000","last_fullsized":"2024-03-24T10:55:29.293413+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:04.264076+0000","last_change":"2024-03-24T10:45:01.928827+0000","last_active":"2024-03-24T10:55:04.264076+0000","last_peered":"2024-03-24T10:55:04.264076+0000","last_clean":"2024-03-24T10:55:04.264076+0000","last_became_active":"2024-03-24T10:45:01.928521+0000","last_became_peered":"2024-03-24T10:45:01.928521+0000","last_unstale":"2024-03-24T10:55:04.264076+0000","last_undegraded":"2024-03-24T10:55:04.264076+0000","last_fullsized":"2024-03-24T10:55:04.264076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:59.643246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"24'7","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.198403+0000","last_change":"2024-03-24T10:45:00.637993+0000","last_active":"2024-03-24T10:55:01.198403+0000","last_peered":"2024-03-24T10:55:01.198403+0000","last_clean":"2024-03-24T10:55:01.198403+0000","last_became_active":"2024-03-24T10:45:00.637803+0000","last_became_peered":"2024-03-24T10:45:00.637803+0000","last_unstale":"2024-03-24T10:55:01.198403+0000","last_undegraded":"2024-03-24T10:55:01.198403+0000","last_fullsized":"2024-03-24T10:55:01.198403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:28:28.854541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:04.213682+0000","last_change":"2024-03-24T10:45:01.897686+0000","last_active":"2024-03-24T10:55:04.213682+0000","last_peered":"2024-03-24T10:55:04.213682+0000","last_clean":"2024-03-24T10:55:04.213682+0000","last_became_active":"2024-03-24T10:45:01.897320+0000","last_became_peered":"2024-03-24T10:45:01.897320+0000","last_unstale":"2024-03-24T10:55:04.213682+0000","last_undegraded":"2024-03-24T10:55:04.213682+0000","last_fullsized":"2024-03-24T10:55:04.213682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:04.291481+0000","last_change":"2024-03-24T10:45:00.638329+0000","last_active":"2024-03-24T10:55:04.291481+0000","last_peered":"2024-03-24T10:55:04.291481+0000","last_clean":"2024-03-24T10:55:04.291481+0000","last_became_active":"2024-03-24T10:45:00.638006+0000","last_became_peered":"2024-03-24T10:45:00.638006+0000","last_unstale":"2024-03-24T10:55:04.291481+0000","last_undegraded":"2024-03-24T10:55:04.291481+0000","last_fullsized":"2024-03-24T10:55:04.291481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.196573+0000","last_change":"2024-03-24T10:45:00.633703+0000","last_active":"2024-03-24T10:54:41.196573+0000","last_peered":"2024-03-24T10:54:41.196573+0000","last_clean":"2024-03-24T10:54:41.196573+0000","last_became_active":"2024-03-24T10:45:00.633427+0000","last_became_peered":"2024-03-24T10:45:00.633427+0000","last_unstale":"2024-03-24T10:54:41.196573+0000","last_undegraded":"2024-03-24T10:54:41.196573+0000","last_fullsized":"2024-03-24T10:54:41.196573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:30.350191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.196539+0000","last_change":"2024-03-24T10:45:01.893902+0000","last_active":"2024-03-24T10:54:41.196539+0000","last_peered":"2024-03-24T10:54:41.196539+0000","last_clean":"2024-03-24T10:54:41.196539+0000","last_became_active":"2024-03-24T10:45:01.893617+0000","last_became_peered":"2024-03-24T10:45:01.893617+0000","last_unstale":"2024-03-24T10:54:41.196539+0000","last_undegraded":"2024-03-24T10:54:41.196539+0000","last_fullsized":"2024-03-24T10:54:41.196539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"24'4","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.231681+0000","last_change":"2024-03-24T10:45:00.638778+0000","last_active":"2024-03-24T10:54:41.231681+0000","last_peered":"2024-03-24T10:54:41.231681+0000","last_clean":"2024-03-24T10:54:41.231681+0000","last_became_active":"2024-03-24T10:45:00.638089+0000","last_became_peered":"2024-03-24T10:45:00.638089+0000","last_unstale":"2024-03-24T10:54:41.231681+0000","last_undegraded":"2024-03-24T10:54:41.231681+0000","last_fullsized":"2024-03-24T10:54:41.231681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.320204+0000","last_change":"2024-03-24T10:45:39.901726+0000","last_active":"2024-03-24T10:54:41.320204+0000","last_peered":"2024-03-24T10:54:41.320204+0000","last_clean":"2024-03-24T10:54:41.320204+0000","last_became_active":"2024-03-24T10:45:39.901416+0000","last_became_peered":"2024-03-24T10:45:39.901416+0000","last_unstale":"2024-03-24T10:54:41.320204+0000","last_undegraded":"2024-03-24T10:54:41.320204+0000","last_fullsized":"2024-03-24T10:54:41.320204+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:23:26.523809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'4","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:44.289869+0000","last_change":"2024-03-24T10:45:00.639822+0000","last_active":"2024-03-24T10:54:44.289869+0000","last_peered":"2024-03-24T10:54:44.289869+0000","last_clean":"2024-03-24T10:54:44.289869+0000","last_became_active":"2024-03-24T10:45:00.639600+0000","last_became_peered":"2024-03-24T10:45:00.639600+0000","last_unstale":"2024-03-24T10:54:44.289869+0000","last_undegraded":"2024-03-24T10:54:44.289869+0000","last_fullsized":"2024-03-24T10:54:44.289869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.231724+0000","last_change":"2024-03-24T10:45:01.954015+0000","last_active":"2024-03-24T10:54:41.231724+0000","last_peered":"2024-03-24T10:54:41.231724+0000","last_clean":"2024-03-24T10:54:41.231724+0000","last_became_active":"2024-03-24T10:45:01.953767+0000","last_became_peered":"2024-03-24T10:45:01.953767+0000","last_unstale":"2024-03-24T10:54:41.231724+0000","last_undegraded":"2024-03-24T10:54:41.231724+0000","last_fullsized":"2024-03-24T10:54:41.231724+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.231756+0000","last_change":"2024-03-24T10:45:00.636553+0000","last_active":"2024-03-24T10:54:41.231756+0000","last_peered":"2024-03-24T10:54:41.231756+0000","last_clean":"2024-03-24T10:54:41.231756+0000","last_became_active":"2024-03-24T10:45:00.636282+0000","last_became_peered":"2024-03-24T10:45:00.636282+0000","last_unstale":"2024-03-24T10:54:41.231756+0000","last_undegraded":"2024-03-24T10:54:41.231756+0000","last_fullsized":"2024-03-24T10:54:41.231756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:44.212223+0000","last_change":"2024-03-24T10:45:01.957905+0000","last_active":"2024-03-24T10:54:44.212223+0000","last_peered":"2024-03-24T10:54:44.212223+0000","last_clean":"2024-03-24T10:54:44.212223+0000","last_became_active":"2024-03-24T10:45:01.957676+0000","last_became_peered":"2024-03-24T10:45:01.957676+0000","last_unstale":"2024-03-24T10:54:44.212223+0000","last_undegraded":"2024-03-24T10:54:44.212223+0000","last_fullsized":"2024-03-24T10:54:44.212223+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.320240+0000","last_change":"2024-03-24T10:45:00.635197+0000","last_active":"2024-03-24T10:54:41.320240+0000","last_peered":"2024-03-24T10:54:41.320240+0000","last_clean":"2024-03-24T10:54:41.320240+0000","last_became_active":"2024-03-24T10:45:00.634989+0000","last_became_peered":"2024-03-24T10:45:00.634989+0000","last_unstale":"2024-03-24T10:54:41.320240+0000","last_undegraded":"2024-03-24T10:54:41.320240+0000","last_fullsized":"2024-03-24T10:54:41.320240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.272421+0000","last_change":"2024-03-24T10:45:01.853785+0000","last_active":"2024-03-24T10:56:44.272421+0000","last_peered":"2024-03-24T10:56:44.272421+0000","last_clean":"2024-03-24T10:56:44.272421+0000","last_became_active":"2024-03-24T10:45:01.853504+0000","last_became_peered":"2024-03-24T10:45:01.853504+0000","last_unstale":"2024-03-24T10:56:44.272421+0000","last_undegraded":"2024-03-24T10:56:44.272421+0000","last_fullsized":"2024-03-24T10:56:44.272421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:30.942758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":36,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.300179+0000","last_change":"2024-03-24T10:45:00.632381+0000","last_active":"2024-03-24T10:56:44.300179+0000","last_peered":"2024-03-24T10:56:44.300179+0000","last_clean":"2024-03-24T10:56:44.300179+0000","last_became_active":"2024-03-24T10:45:00.632108+0000","last_became_peered":"2024-03-24T10:45:00.632108+0000","last_unstale":"2024-03-24T10:56:44.300179+0000","last_undegraded":"2024-03-24T10:56:44.300179+0000","last_fullsized":"2024-03-24T10:56:44.300179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:06.142451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.226243+0000","last_change":"2024-03-24T10:45:01.853591+0000","last_active":"2024-03-24T10:54:41.226243+0000","last_peered":"2024-03-24T10:54:41.226243+0000","last_clean":"2024-03-24T10:54:41.226243+0000","last_became_active":"2024-03-24T10:45:01.853279+0000","last_became_peered":"2024-03-24T10:45:01.853279+0000","last_unstale":"2024-03-24T10:54:41.226243+0000","last_undegraded":"2024-03-24T10:54:41.226243+0000","last_fullsized":"2024-03-24T10:54:41.226243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:07:24.149961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'9","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.222018+0000","last_change":"2024-03-24T10:45:00.634192+0000","last_active":"2024-03-24T10:56:44.222018+0000","last_peered":"2024-03-24T10:56:44.222018+0000","last_clean":"2024-03-24T10:56:44.222018+0000","last_became_active":"2024-03-24T10:45:00.633893+0000","last_became_peered":"2024-03-24T10:45:00.633893+0000","last_unstale":"2024-03-24T10:56:44.222018+0000","last_undegraded":"2024-03-24T10:56:44.222018+0000","last_fullsized":"2024-03-24T10:56:44.222018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:30.350191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.222005+0000","last_change":"2024-03-24T10:45:01.893426+0000","last_active":"2024-03-24T10:56:44.222005+0000","last_peered":"2024-03-24T10:56:44.222005+0000","last_clean":"2024-03-24T10:56:44.222005+0000","last_became_active":"2024-03-24T10:45:01.893178+0000","last_became_peered":"2024-03-24T10:45:01.893178+0000","last_unstale":"2024-03-24T10:56:44.222005+0000","last_undegraded":"2024-03-24T10:56:44.222005+0000","last_fullsized":"2024-03-24T10:56:44.222005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:13.887947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"24'6","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.242255+0000","last_change":"2024-03-24T10:45:00.638214+0000","last_active":"2024-03-24T10:56:41.242255+0000","last_peered":"2024-03-24T10:56:41.242255+0000","last_clean":"2024-03-24T10:56:41.242255+0000","last_became_active":"2024-03-24T10:45:00.637973+0000","last_became_peered":"2024-03-24T10:45:00.637973+0000","last_unstale":"2024-03-24T10:56:41.242255+0000","last_undegraded":"2024-03-24T10:56:41.242255+0000","last_fullsized":"2024-03-24T10:56:41.242255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:15.749451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.237815+0000","last_change":"2024-03-24T10:45:01.856756+0000","last_active":"2024-03-24T10:56:41.237815+0000","last_peered":"2024-03-24T10:56:41.237815+0000","last_clean":"2024-03-24T10:56:41.237815+0000","last_became_active":"2024-03-24T10:45:01.856482+0000","last_became_peered":"2024-03-24T10:45:01.856482+0000","last_unstale":"2024-03-24T10:56:41.237815+0000","last_undegraded":"2024-03-24T10:56:41.237815+0000","last_fullsized":"2024-03-24T10:56:41.237815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:12.476490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.300155+0000","last_change":"2024-03-24T10:45:00.636481+0000","last_active":"2024-03-24T10:56:44.300155+0000","last_peered":"2024-03-24T10:56:44.300155+0000","last_clean":"2024-03-24T10:56:44.300155+0000","last_became_active":"2024-03-24T10:45:00.636206+0000","last_became_peered":"2024-03-24T10:45:00.636206+0000","last_unstale":"2024-03-24T10:56:44.300155+0000","last_undegraded":"2024-03-24T10:56:44.300155+0000","last_fullsized":"2024-03-24T10:56:44.300155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.300128+0000","last_change":"2024-03-24T10:45:01.897279+0000","last_active":"2024-03-24T10:56:44.300128+0000","last_peered":"2024-03-24T10:56:44.300128+0000","last_clean":"2024-03-24T10:56:44.300128+0000","last_became_active":"2024-03-24T10:45:01.896736+0000","last_became_peered":"2024-03-24T10:45:01.896736+0000","last_unstale":"2024-03-24T10:56:44.300128+0000","last_undegraded":"2024-03-24T10:56:44.300128+0000","last_fullsized":"2024-03-24T10:56:44.300128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.325004+0000","last_change":"2024-03-24T10:45:00.564066+0000","last_active":"2024-03-24T10:56:41.325004+0000","last_peered":"2024-03-24T10:56:41.325004+0000","last_clean":"2024-03-24T10:56:41.325004+0000","last_became_active":"2024-03-24T10:45:00.563812+0000","last_became_peered":"2024-03-24T10:45:00.563812+0000","last_unstale":"2024-03-24T10:56:41.325004+0000","last_undegraded":"2024-03-24T10:56:41.325004+0000","last_fullsized":"2024-03-24T10:56:41.325004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:28:28.854541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.237784+0000","last_change":"2024-03-24T10:45:01.854429+0000","last_active":"2024-03-24T10:56:41.237784+0000","last_peered":"2024-03-24T10:56:41.237784+0000","last_clean":"2024-03-24T10:56:41.237784+0000","last_became_active":"2024-03-24T10:45:01.854179+0000","last_became_peered":"2024-03-24T10:45:01.854179+0000","last_unstale":"2024-03-24T10:56:41.237784+0000","last_undegraded":"2024-03-24T10:56:41.237784+0000","last_fullsized":"2024-03-24T10:56:41.237784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:19.270242+0000","last_change":"2024-03-24T10:45:00.634223+0000","last_active":"2024-03-24T10:56:19.270242+0000","last_peered":"2024-03-24T10:56:19.270242+0000","last_clean":"2024-03-24T10:56:19.270242+0000","last_became_active":"2024-03-24T10:45:00.633355+0000","last_became_peered":"2024-03-24T10:45:00.633355+0000","last_unstale":"2024-03-24T10:56:19.270242+0000","last_undegraded":"2024-03-24T10:56:19.270242+0000","last_fullsized":"2024-03-24T10:56:19.270242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:04.930115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.320259+0000","last_change":"2024-03-24T10:45:40.518338+0000","last_active":"2024-03-24T10:54:41.320259+0000","last_peered":"2024-03-24T10:54:41.320259+0000","last_clean":"2024-03-24T10:54:41.320259+0000","last_became_active":"2024-03-24T10:45:40.518035+0000","last_became_peered":"2024-03-24T10:45:40.518035+0000","last_unstale":"2024-03-24T10:54:41.320259+0000","last_undegraded":"2024-03-24T10:54:41.320259+0000","last_fullsized":"2024-03-24T10:54:41.320259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:29:31.508911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.323256+0000","last_change":"2024-03-24T10:45:00.626484+0000","last_active":"2024-03-24T10:56:16.323256+0000","last_peered":"2024-03-24T10:56:16.323256+0000","last_clean":"2024-03-24T10:56:16.323256+0000","last_became_active":"2024-03-24T10:45:00.626195+0000","last_became_peered":"2024-03-24T10:45:00.626195+0000","last_unstale":"2024-03-24T10:56:16.323256+0000","last_undegraded":"2024-03-24T10:56:16.323256+0000","last_fullsized":"2024-03-24T10:56:16.323256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.235444+0000","last_change":"2024-03-24T10:45:01.856995+0000","last_active":"2024-03-24T10:56:16.235444+0000","last_peered":"2024-03-24T10:56:16.235444+0000","last_clean":"2024-03-24T10:56:16.235444+0000","last_became_active":"2024-03-24T10:45:01.856531+0000","last_became_peered":"2024-03-24T10:45:01.856531+0000","last_unstale":"2024-03-24T10:56:16.235444+0000","last_undegraded":"2024-03-24T10:56:16.235444+0000","last_fullsized":"2024-03-24T10:56:16.235444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:29:31.508911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.327524+0000","last_change":"2024-03-24T10:45:00.628438+0000","last_active":"2024-03-24T10:56:16.327524+0000","last_peered":"2024-03-24T10:56:16.327524+0000","last_clean":"2024-03-24T10:56:16.327524+0000","last_became_active":"2024-03-24T10:45:00.628158+0000","last_became_peered":"2024-03-24T10:45:00.628158+0000","last_unstale":"2024-03-24T10:56:16.327524+0000","last_undegraded":"2024-03-24T10:56:16.327524+0000","last_fullsized":"2024-03-24T10:56:16.327524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:29.826202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.315968+0000","last_change":"2024-03-24T10:45:39.901453+0000","last_active":"2024-03-24T10:54:41.315968+0000","last_peered":"2024-03-24T10:54:41.315968+0000","last_clean":"2024-03-24T10:54:41.315968+0000","last_became_active":"2024-03-24T10:45:39.901119+0000","last_became_peered":"2024-03-24T10:45:39.901119+0000","last_unstale":"2024-03-24T10:54:41.315968+0000","last_undegraded":"2024-03-24T10:54:41.315968+0000","last_fullsized":"2024-03-24T10:54:41.315968+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:16.341360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.323237+0000","last_change":"2024-03-24T10:45:00.639093+0000","last_active":"2024-03-24T10:56:16.323237+0000","last_peered":"2024-03-24T10:56:16.323237+0000","last_clean":"2024-03-24T10:56:16.323237+0000","last_became_active":"2024-03-24T10:45:00.638785+0000","last_became_peered":"2024-03-24T10:45:00.638785+0000","last_unstale":"2024-03-24T10:56:16.323237+0000","last_undegraded":"2024-03-24T10:56:16.323237+0000","last_fullsized":"2024-03-24T10:56:16.323237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.235406+0000","last_change":"2024-03-24T10:45:01.953772+0000","last_active":"2024-03-24T10:56:16.235406+0000","last_peered":"2024-03-24T10:56:16.235406+0000","last_clean":"2024-03-24T10:56:16.235406+0000","last_became_active":"2024-03-24T10:45:01.953402+0000","last_became_peered":"2024-03-24T10:45:01.953402+0000","last_unstale":"2024-03-24T10:56:16.235406+0000","last_undegraded":"2024-03-24T10:56:16.235406+0000","last_fullsized":"2024-03-24T10:56:16.235406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:22.089070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":183,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:51.238749+0000","last_change":"2024-03-24T10:44:48.241074+0000","last_active":"2024-03-24T10:56:51.238749+0000","last_peered":"2024-03-24T10:56:51.238749+0000","last_clean":"2024-03-24T10:56:51.238749+0000","last_became_active":"2024-03-24T10:44:48.239986+0000","last_became_peered":"2024-03-24T10:44:48.239986+0000","last_unstale":"2024-03-24T10:56:51.238749+0000","last_undegraded":"2024-03-24T10:56:51.238749+0000","last_fullsized":"2024-03-24T10:56:51.238749+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:46.142247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:46.142247+0000","last_clean_scrub_stamp":"2024-03-24T10:44:46.142247+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:05:50.110100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.235730+0000","last_change":"2024-03-24T10:45:00.637276+0000","last_active":"2024-03-24T10:55:26.235730+0000","last_peered":"2024-03-24T10:55:26.235730+0000","last_clean":"2024-03-24T10:55:26.235730+0000","last_became_active":"2024-03-24T10:45:00.636734+0000","last_became_peered":"2024-03-24T10:45:00.636734+0000","last_unstale":"2024-03-24T10:55:26.235730+0000","last_undegraded":"2024-03-24T10:55:26.235730+0000","last_fullsized":"2024-03-24T10:55:26.235730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.200440+0000","last_change":"2024-03-24T10:45:01.958505+0000","last_active":"2024-03-24T10:55:26.200440+0000","last_peered":"2024-03-24T10:55:26.200440+0000","last_clean":"2024-03-24T10:55:26.200440+0000","last_became_active":"2024-03-24T10:45:01.958268+0000","last_became_peered":"2024-03-24T10:45:01.958268+0000","last_unstale":"2024-03-24T10:55:26.200440+0000","last_undegraded":"2024-03-24T10:55:26.200440+0000","last_fullsized":"2024-03-24T10:55:26.200440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:27:12.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"24'5","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:01.239735+0000","last_change":"2024-03-24T10:45:00.597856+0000","last_active":"2024-03-24T10:57:01.239735+0000","last_peered":"2024-03-24T10:57:01.239735+0000","last_clean":"2024-03-24T10:57:01.239735+0000","last_became_active":"2024-03-24T10:45:00.597640+0000","last_became_peered":"2024-03-24T10:45:00.597640+0000","last_unstale":"2024-03-24T10:57:01.239735+0000","last_undegraded":"2024-03-24T10:57:01.239735+0000","last_fullsized":"2024-03-24T10:57:01.239735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:29.826202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.215622+0000","last_change":"2024-03-24T10:45:01.897502+0000","last_active":"2024-03-24T10:55:29.215622+0000","last_peered":"2024-03-24T10:55:29.215622+0000","last_clean":"2024-03-24T10:55:29.215622+0000","last_became_active":"2024-03-24T10:45:01.897250+0000","last_became_peered":"2024-03-24T10:45:01.897250+0000","last_unstale":"2024-03-24T10:55:29.215622+0000","last_undegraded":"2024-03-24T10:55:29.215622+0000","last_fullsized":"2024-03-24T10:55:29.215622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:09:46.341672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"24'6","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.319220+0000","last_change":"2024-03-24T10:45:00.627166+0000","last_active":"2024-03-24T10:55:26.319220+0000","last_peered":"2024-03-24T10:55:26.319220+0000","last_clean":"2024-03-24T10:55:26.319220+0000","last_became_active":"2024-03-24T10:45:00.626881+0000","last_became_peered":"2024-03-24T10:45:00.626881+0000","last_unstale":"2024-03-24T10:55:26.319220+0000","last_undegraded":"2024-03-24T10:55:26.319220+0000","last_fullsized":"2024-03-24T10:55:26.319220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:01.330987+0000","last_change":"2024-03-24T10:45:01.930271+0000","last_active":"2024-03-24T10:57:01.330987+0000","last_peered":"2024-03-24T10:57:01.330987+0000","last_clean":"2024-03-24T10:57:01.330987+0000","last_became_active":"2024-03-24T10:45:01.929998+0000","last_became_peered":"2024-03-24T10:45:01.929998+0000","last_unstale":"2024-03-24T10:57:01.330987+0000","last_undegraded":"2024-03-24T10:57:01.330987+0000","last_fullsized":"2024-03-24T10:57:01.330987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.235722+0000","last_change":"2024-03-24T10:45:00.638309+0000","last_active":"2024-03-24T10:55:26.235722+0000","last_peered":"2024-03-24T10:55:26.235722+0000","last_clean":"2024-03-24T10:55:26.235722+0000","last_became_active":"2024-03-24T10:45:00.638041+0000","last_became_peered":"2024-03-24T10:45:00.638041+0000","last_unstale":"2024-03-24T10:55:26.235722+0000","last_undegraded":"2024-03-24T10:55:26.235722+0000","last_fullsized":"2024-03-24T10:55:26.235722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:14.049099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.266124+0000","last_change":"2024-03-24T10:45:01.897722+0000","last_active":"2024-03-24T10:55:29.266124+0000","last_peered":"2024-03-24T10:55:29.266124+0000","last_clean":"2024-03-24T10:55:29.266124+0000","last_became_active":"2024-03-24T10:45:01.897159+0000","last_became_peered":"2024-03-24T10:45:01.897159+0000","last_unstale":"2024-03-24T10:55:29.266124+0000","last_undegraded":"2024-03-24T10:55:29.266124+0000","last_fullsized":"2024-03-24T10:55:29.266124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:42.293342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.323798+0000","last_change":"2024-03-24T10:45:00.629267+0000","last_active":"2024-03-24T10:55:26.323798+0000","last_peered":"2024-03-24T10:55:26.323798+0000","last_clean":"2024-03-24T10:55:26.323798+0000","last_became_active":"2024-03-24T10:45:00.628971+0000","last_became_peered":"2024-03-24T10:45:00.628971+0000","last_unstale":"2024-03-24T10:55:26.323798+0000","last_undegraded":"2024-03-24T10:55:26.323798+0000","last_fullsized":"2024-03-24T10:55:26.323798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:41.176786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.235742+0000","last_change":"2024-03-24T10:45:01.952527+0000","last_active":"2024-03-24T10:55:26.235742+0000","last_peered":"2024-03-24T10:55:26.235742+0000","last_clean":"2024-03-24T10:55:26.235742+0000","last_became_active":"2024-03-24T10:45:01.952192+0000","last_became_peered":"2024-03-24T10:45:01.952192+0000","last_unstale":"2024-03-24T10:55:26.235742+0000","last_undegraded":"2024-03-24T10:55:26.235742+0000","last_fullsized":"2024-03-24T10:55:26.235742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.230535+0000","last_change":"2024-03-24T10:45:00.631941+0000","last_active":"2024-03-24T10:55:26.230535+0000","last_peered":"2024-03-24T10:55:26.230535+0000","last_clean":"2024-03-24T10:55:26.230535+0000","last_became_active":"2024-03-24T10:45:00.631670+0000","last_became_peered":"2024-03-24T10:45:00.631670+0000","last_unstale":"2024-03-24T10:55:26.230535+0000","last_undegraded":"2024-03-24T10:55:26.230535+0000","last_fullsized":"2024-03-24T10:55:26.230535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:09:45.225115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.323774+0000","last_change":"2024-03-24T10:45:01.930483+0000","last_active":"2024-03-24T10:55:26.323774+0000","last_peered":"2024-03-24T10:55:26.323774+0000","last_clean":"2024-03-24T10:55:26.323774+0000","last_became_active":"2024-03-24T10:45:01.930220+0000","last_became_peered":"2024-03-24T10:45:01.930220+0000","last_unstale":"2024-03-24T10:55:26.323774+0000","last_undegraded":"2024-03-24T10:55:26.323774+0000","last_fullsized":"2024-03-24T10:55:26.323774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.325636+0000","last_change":"2024-03-24T10:45:00.633650+0000","last_active":"2024-03-24T10:55:51.325636+0000","last_peered":"2024-03-24T10:55:51.325636+0000","last_clean":"2024-03-24T10:55:51.325636+0000","last_became_active":"2024-03-24T10:45:00.633329+0000","last_became_peered":"2024-03-24T10:45:00.633329+0000","last_unstale":"2024-03-24T10:55:51.325636+0000","last_undegraded":"2024-03-24T10:55:51.325636+0000","last_fullsized":"2024-03-24T10:55:51.325636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.202473+0000","last_change":"2024-03-24T10:45:01.853487+0000","last_active":"2024-03-24T10:55:51.202473+0000","last_peered":"2024-03-24T10:55:51.202473+0000","last_clean":"2024-03-24T10:55:51.202473+0000","last_became_active":"2024-03-24T10:45:01.853257+0000","last_became_peered":"2024-03-24T10:45:01.853257+0000","last_unstale":"2024-03-24T10:55:51.202473+0000","last_undegraded":"2024-03-24T10:55:51.202473+0000","last_fullsized":"2024-03-24T10:55:51.202473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:54.295762+0000","last_change":"2024-03-24T10:45:00.639282+0000","last_active":"2024-03-24T10:55:54.295762+0000","last_peered":"2024-03-24T10:55:54.295762+0000","last_clean":"2024-03-24T10:55:54.295762+0000","last_became_active":"2024-03-24T10:45:00.638999+0000","last_became_peered":"2024-03-24T10:45:00.638999+0000","last_unstale":"2024-03-24T10:55:54.295762+0000","last_undegraded":"2024-03-24T10:55:54.295762+0000","last_fullsized":"2024-03-24T10:55:54.295762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:41.176786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:54.217821+0000","last_change":"2024-03-24T10:45:01.928900+0000","last_active":"2024-03-24T10:55:54.217821+0000","last_peered":"2024-03-24T10:55:54.217821+0000","last_clean":"2024-03-24T10:55:54.217821+0000","last_became_active":"2024-03-24T10:45:01.928610+0000","last_became_peered":"2024-03-24T10:45:01.928610+0000","last_unstale":"2024-03-24T10:55:54.217821+0000","last_undegraded":"2024-03-24T10:55:54.217821+0000","last_fullsized":"2024-03-24T10:55:54.217821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.232929+0000","last_change":"2024-03-24T10:45:00.631327+0000","last_active":"2024-03-24T10:55:51.232929+0000","last_peered":"2024-03-24T10:55:51.232929+0000","last_clean":"2024-03-24T10:55:51.232929+0000","last_became_active":"2024-03-24T10:45:00.631078+0000","last_became_peered":"2024-03-24T10:45:00.631078+0000","last_unstale":"2024-03-24T10:55:51.232929+0000","last_undegraded":"2024-03-24T10:55:51.232929+0000","last_fullsized":"2024-03-24T10:55:51.232929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:43.263796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:54.268364+0000","last_change":"2024-03-24T10:45:01.897329+0000","last_active":"2024-03-24T10:55:54.268364+0000","last_peered":"2024-03-24T10:55:54.268364+0000","last_clean":"2024-03-24T10:55:54.268364+0000","last_became_active":"2024-03-24T10:45:01.897063+0000","last_became_peered":"2024-03-24T10:45:01.897063+0000","last_unstale":"2024-03-24T10:55:54.268364+0000","last_undegraded":"2024-03-24T10:55:54.268364+0000","last_fullsized":"2024-03-24T10:55:54.268364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:07.259007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.325593+0000","last_change":"2024-03-24T10:45:00.597948+0000","last_active":"2024-03-24T10:55:51.325593+0000","last_peered":"2024-03-24T10:55:51.325593+0000","last_clean":"2024-03-24T10:55:51.325593+0000","last_became_active":"2024-03-24T10:45:00.597671+0000","last_became_peered":"2024-03-24T10:45:00.597671+0000","last_unstale":"2024-03-24T10:55:51.325593+0000","last_undegraded":"2024-03-24T10:55:51.325593+0000","last_fullsized":"2024-03-24T10:55:51.325593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:06.142451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.202427+0000","last_change":"2024-03-24T10:45:01.927822+0000","last_active":"2024-03-24T10:55:51.202427+0000","last_peered":"2024-03-24T10:55:51.202427+0000","last_clean":"2024-03-24T10:55:51.202427+0000","last_became_active":"2024-03-24T10:45:01.927538+0000","last_became_peered":"2024-03-24T10:45:01.927538+0000","last_unstale":"2024-03-24T10:55:51.202427+0000","last_undegraded":"2024-03-24T10:55:51.202427+0000","last_fullsized":"2024-03-24T10:55:51.202427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'3","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.325612+0000","last_change":"2024-03-24T10:45:00.626818+0000","last_active":"2024-03-24T10:55:51.325612+0000","last_peered":"2024-03-24T10:55:51.325612+0000","last_clean":"2024-03-24T10:55:51.325612+0000","last_became_active":"2024-03-24T10:45:00.626557+0000","last_became_peered":"2024-03-24T10:45:00.626557+0000","last_unstale":"2024-03-24T10:55:51.325612+0000","last_undegraded":"2024-03-24T10:55:51.325612+0000","last_fullsized":"2024-03-24T10:55:51.325612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:43.263796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.204517+0000","last_change":"2024-03-24T10:45:01.893192+0000","last_active":"2024-03-24T10:56:16.204517+0000","last_peered":"2024-03-24T10:56:16.204517+0000","last_clean":"2024-03-24T10:56:16.204517+0000","last_became_active":"2024-03-24T10:45:01.892934+0000","last_became_peered":"2024-03-24T10:45:01.892934+0000","last_unstale":"2024-03-24T10:56:16.204517+0000","last_undegraded":"2024-03-24T10:56:16.204517+0000","last_fullsized":"2024-03-24T10:56:16.204517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.235371+0000","last_change":"2024-03-24T10:45:00.630147+0000","last_active":"2024-03-24T10:56:16.235371+0000","last_peered":"2024-03-24T10:56:16.235371+0000","last_clean":"2024-03-24T10:56:16.235371+0000","last_became_active":"2024-03-24T10:45:00.629850+0000","last_became_peered":"2024-03-24T10:45:00.629850+0000","last_unstale":"2024-03-24T10:56:16.235371+0000","last_undegraded":"2024-03-24T10:56:16.235371+0000","last_fullsized":"2024-03-24T10:56:16.235371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:01:12.843222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:54.295704+0000","last_change":"2024-03-24T10:45:01.897333+0000","last_active":"2024-03-24T10:55:54.295704+0000","last_peered":"2024-03-24T10:55:54.295704+0000","last_clean":"2024-03-24T10:55:54.295704+0000","last_became_active":"2024-03-24T10:45:01.897080+0000","last_became_peered":"2024-03-24T10:45:01.897080+0000","last_unstale":"2024-03-24T10:55:54.295704+0000","last_undegraded":"2024-03-24T10:55:54.295704+0000","last_fullsized":"2024-03-24T10:55:54.295704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:13.887947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.233488+0000","last_change":"2024-03-24T10:45:01.956041+0000","last_active":"2024-03-24T10:55:01.233488+0000","last_peered":"2024-03-24T10:55:01.233488+0000","last_clean":"2024-03-24T10:55:01.233488+0000","last_became_active":"2024-03-24T10:45:01.955766+0000","last_became_peered":"2024-03-24T10:45:01.955766+0000","last_unstale":"2024-03-24T10:55:01.233488+0000","last_undegraded":"2024-03-24T10:55:01.233488+0000","last_fullsized":"2024-03-24T10:55:01.233488+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:16.341360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.228149+0000","last_change":"2024-03-24T10:45:00.639913+0000","last_active":"2024-03-24T10:55:01.228149+0000","last_peered":"2024-03-24T10:55:01.228149+0000","last_clean":"2024-03-24T10:55:01.228149+0000","last_became_active":"2024-03-24T10:45:00.639653+0000","last_became_peered":"2024-03-24T10:45:00.639653+0000","last_unstale":"2024-03-24T10:55:01.228149+0000","last_undegraded":"2024-03-24T10:55:01.228149+0000","last_fullsized":"2024-03-24T10:55:01.228149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:15.749451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.233473+0000","last_change":"2024-03-24T10:45:01.952340+0000","last_active":"2024-03-24T10:55:01.233473+0000","last_peered":"2024-03-24T10:55:01.233473+0000","last_clean":"2024-03-24T10:55:01.233473+0000","last_became_active":"2024-03-24T10:45:01.952074+0000","last_became_peered":"2024-03-24T10:45:01.952074+0000","last_unstale":"2024-03-24T10:55:01.233473+0000","last_undegraded":"2024-03-24T10:55:01.233473+0000","last_fullsized":"2024-03-24T10:55:01.233473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"24'12","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:31.242486+0000","last_change":"2024-03-24T10:45:00.633790+0000","last_active":"2024-03-24T10:57:31.242486+0000","last_peered":"2024-03-24T10:57:31.242486+0000","last_clean":"2024-03-24T10:57:31.242486+0000","last_became_active":"2024-03-24T10:45:00.633494+0000","last_became_peered":"2024-03-24T10:45:00.633494+0000","last_unstale":"2024-03-24T10:57:31.242486+0000","last_undegraded":"2024-03-24T10:57:31.242486+0000","last_fullsized":"2024-03-24T10:57:31.242486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:14.049099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.215687+0000","last_change":"2024-03-24T10:45:01.957958+0000","last_active":"2024-03-24T10:55:29.215687+0000","last_peered":"2024-03-24T10:55:29.215687+0000","last_clean":"2024-03-24T10:55:29.215687+0000","last_became_active":"2024-03-24T10:45:01.957676+0000","last_became_peered":"2024-03-24T10:45:01.957676+0000","last_unstale":"2024-03-24T10:55:29.215687+0000","last_undegraded":"2024-03-24T10:55:29.215687+0000","last_fullsized":"2024-03-24T10:55:29.215687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:01.208031+0000","last_change":"2024-03-24T10:45:00.629120+0000","last_active":"2024-03-24T10:57:01.208031+0000","last_peered":"2024-03-24T10:57:01.208031+0000","last_clean":"2024-03-24T10:57:01.208031+0000","last_became_active":"2024-03-24T10:45:00.628712+0000","last_became_peered":"2024-03-24T10:45:00.628712+0000","last_unstale":"2024-03-24T10:57:01.208031+0000","last_undegraded":"2024-03-24T10:57:01.208031+0000","last_fullsized":"2024-03-24T10:57:01.208031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:01.208005+0000","last_change":"2024-03-24T10:45:01.894602+0000","last_active":"2024-03-24T10:57:01.208005+0000","last_peered":"2024-03-24T10:57:01.208005+0000","last_clean":"2024-03-24T10:57:01.208005+0000","last_became_active":"2024-03-24T10:45:01.894362+0000","last_became_peered":"2024-03-24T10:45:01.894362+0000","last_unstale":"2024-03-24T10:57:01.208005+0000","last_undegraded":"2024-03-24T10:57:01.208005+0000","last_fullsized":"2024-03-24T10:57:01.208005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'13","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.215696+0000","last_change":"2024-03-24T10:45:00.639171+0000","last_active":"2024-03-24T10:55:29.215696+0000","last_peered":"2024-03-24T10:55:29.215696+0000","last_clean":"2024-03-24T10:55:29.215696+0000","last_became_active":"2024-03-24T10:45:00.638871+0000","last_became_peered":"2024-03-24T10:45:00.638871+0000","last_unstale":"2024-03-24T10:55:29.215696+0000","last_undegraded":"2024-03-24T10:55:29.215696+0000","last_fullsized":"2024-03-24T10:55:29.215696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:52:46.322196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":204,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":184,"ondisk_log_size":184,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65430,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542320,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31524,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340316,"statfs":{"total":96636764160,"available":96604483584,"internally_reserved":0,"allocated":352256,"data_stored":685052,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542320,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":299008,"data_stored":96516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542319,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":95020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542320,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":94046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542320,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":278528,"data_stored":94116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575023,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":278528,"data_stored":94986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575023,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":274432,"data_stored":93914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575023,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":344064,"data_stored":683826,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:59:07.020 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:59:07.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.122+0000 7fab29ea4640 1 Processor -- start 2024-03-24T10:59:07.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.122+0000 7fab29ea4640 1 -- start start 2024-03-24T10:59:07.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.123+0000 7fab29ea4640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fab2415ac00 0x7fab240cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:07.123 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.123+0000 7fab29ea4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fab240ccaa0 0x7fab2406bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:07.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.123+0000 7fab29ea4640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab240cb2f0 con 0x7fab241543a0 2024-03-24T10:59:07.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.123+0000 7fab29ea4640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fab240c6750 con 0x7fab2415ac00 2024-03-24T10:59:07.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.123+0000 7fab29ea4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fab240cceb0 con 0x7fab240ccaa0 2024-03-24T10:59:07.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.124+0000 7fab22ffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fab2415ac00 0x7fab240cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:07.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.124+0000 7fab23fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fab240ccaa0 0x7fab2406bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:07.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.124+0000 7fab22ffd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fab2415ac00 0x7fab240cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44062/0 (socket says 172.21.15.5:44062) 2024-03-24T10:59:07.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.124+0000 7fab22ffd640 1 -- 172.21.15.5:0/1199455803 learned_addr learned my addr 172.21.15.5:0/1199455803 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:07.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.127+0000 7fab22ffd640 1 -- 172.21.15.5:0/1199455803 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fab240ccaa0 msgr2=0x7fab2406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:07.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.127+0000 7fab22ffd640 1 --2- 172.21.15.5:0/1199455803 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fab240ccaa0 0x7fab2406bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.127+0000 7fab22ffd640 1 -- 172.21.15.5:0/1199455803 >> v1:172.21.15.5:6789/0 conn(0x7fab241543a0 legacy=0x7fab2406d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:07.127 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.127+0000 7fab22ffd640 1 -- 172.21.15.5:0/1199455803 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab2415b070 con 0x7fab2415ac00 2024-03-24T10:59:07.128 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.129+0000 7fab22ffd640 1 --2- 172.21.15.5:0/1199455803 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab2415ac00 0x7fab240cc560 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fab18009870 tx=0x7fab1802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2545eeb78f95acf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:07.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.134+0000 7fab227fc640 1 -- 172.21.15.5:0/1199455803 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab18031640 con 0x7fab2415ac00 2024-03-24T10:59:07.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.134+0000 7fab227fc640 1 -- 172.21.15.5:0/1199455803 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab180317a0 con 0x7fab2415ac00 2024-03-24T10:59:07.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.134+0000 7fab227fc640 1 -- 172.21.15.5:0/1199455803 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab18031b30 con 0x7fab2415ac00 2024-03-24T10:59:07.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.135+0000 7fab29ea4640 1 -- 172.21.15.5:0/1199455803 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab2415ac00 msgr2=0x7fab240cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:07.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.135+0000 7fab29ea4640 1 --2- 172.21.15.5:0/1199455803 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab2415ac00 0x7fab240cc560 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fab18009870 tx=0x7fab1802b8f0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.135+0000 7fab29ea4640 1 -- 172.21.15.5:0/1199455803 shutdown_connections 2024-03-24T10:59:07.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.135+0000 7fab29ea4640 1 --2- 172.21.15.5:0/1199455803 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fab240ccaa0 0x7fab2406bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.135+0000 7fab29ea4640 1 --2- 172.21.15.5:0/1199455803 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab2415ac00 0x7fab240cc560 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.135+0000 7fab29ea4640 1 -- 172.21.15.5:0/1199455803 >> 172.21.15.5:0/1199455803 conn(0x7fab24072ef0 msgr2=0x7fab24061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:07.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.135+0000 7fab29ea4640 1 -- 172.21.15.5:0/1199455803 shutdown_connections 2024-03-24T10:59:07.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.135+0000 7fab29ea4640 1 -- 172.21.15.5:0/1199455803 wait complete. 2024-03-24T10:59:07.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.137+0000 7fab29ea4640 1 Processor -- start 2024-03-24T10:59:07.137 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.137+0000 7fab29ea4640 1 -- start start 2024-03-24T10:59:07.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.139+0000 7fab29ea4640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab240ccaa0 0x7fab240b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:07.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.139+0000 7fab29ea4640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fab241543a0 0x7fab240b3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:07.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.139+0000 7fab29ea4640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab2415ac00 0x7fab24159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:07.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.139+0000 7fab29ea4640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fab24152880 con 0x7fab2415ac00 2024-03-24T10:59:07.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.139+0000 7fab29ea4640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fab241f08c0 con 0x7fab240ccaa0 2024-03-24T10:59:07.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.139+0000 7fab29ea4640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fab24158bb0 con 0x7fab241543a0 2024-03-24T10:59:07.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.139+0000 7fab23fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab2415ac00 0x7fab24159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:07.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.139+0000 7fab22ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fab241543a0 0x7fab240b3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:07.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.139+0000 7fab237fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab240ccaa0 0x7fab240b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:07.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.140+0000 7fab22ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fab241543a0 0x7fab240b3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:48542/0 (socket says 172.21.15.5:48542) 2024-03-24T10:59:07.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.140+0000 7fab22ffd640 1 -- 172.21.15.5:0/63432032 learned_addr learned my addr 172.21.15.5:0/63432032 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:07.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.145+0000 7fab23fff640 1 -- 172.21.15.5:0/63432032 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fab241543a0 msgr2=0x7fab240b3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:07.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.145+0000 7fab23fff640 1 --2- 172.21.15.5:0/63432032 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fab241543a0 0x7fab240b3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.145+0000 7fab23fff640 1 -- 172.21.15.5:0/63432032 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab240ccaa0 msgr2=0x7fab240b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:07.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.145+0000 7fab23fff640 1 --2- 172.21.15.5:0/63432032 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab240ccaa0 0x7fab240b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.145 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.145+0000 7fab23fff640 1 -- 172.21.15.5:0/63432032 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab180094d0 con 0x7fab2415ac00 2024-03-24T10:59:07.148 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.148+0000 7fab23fff640 1 --2- 172.21.15.5:0/63432032 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab2415ac00 0x7fab24159040 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fab1c00efc0 tx=0x7fab1c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:07.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.152+0000 7fab0bfff640 1 -- 172.21.15.5:0/63432032 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1c009c00 con 0x7fab2415ac00 2024-03-24T10:59:07.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.152+0000 7fab0bfff640 1 -- 172.21.15.5:0/63432032 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab1c00f040 con 0x7fab2415ac00 2024-03-24T10:59:07.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.152+0000 7fab0bfff640 1 -- 172.21.15.5:0/63432032 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1c0146c0 con 0x7fab2415ac00 2024-03-24T10:59:07.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.152+0000 7fab29ea4640 1 -- 172.21.15.5:0/63432032 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab241f0dd0 con 0x7fab2415ac00 2024-03-24T10:59:07.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.152+0000 7fab29ea4640 1 -- 172.21.15.5:0/63432032 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab24158280 con 0x7fab2415ac00 2024-03-24T10:59:07.154 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.153+0000 7fab29ea4640 1 -- 172.21.15.5:0/63432032 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab240b5e70 con 0x7fab2415ac00 2024-03-24T10:59:07.429 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.429+0000 7fab0bfff640 1 -- 172.21.15.5:0/63432032 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fab1c014820 con 0x7fab2415ac00 2024-03-24T10:59:07.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.430+0000 7fab0bfff640 1 --2- 172.21.15.5:0/63432032 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fab1006b890 0x7fab1006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:07.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.430+0000 7fab237fe640 1 --2- 172.21.15.5:0/63432032 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fab1006b890 0x7fab1006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:07.430 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.430+0000 7fab237fe640 1 --2- 172.21.15.5:0/63432032 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fab1006b890 0x7fab1006dd50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fab140017f0 tx=0x7fab14009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:07.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.452+0000 7fab0bfff640 1 -- 172.21.15.5:0/63432032 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fab1c059d60 con 0x7fab2415ac00 2024-03-24T10:59:07.452 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.452+0000 7fab0bfff640 1 -- 172.21.15.5:0/63432032 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fab1c056000 con 0x7fab2415ac00 2024-03-24T10:59:07.634 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.633+0000 7fab29ea4640 1 -- 172.21.15.5:0/63432032 --> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fab240cb800 con 0x7fab1006b890 2024-03-24T10:59:07.640 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.639+0000 7fab0bfff640 1 -- 172.21.15.5:0/63432032 <== mgr.4120 v2:172.21.15.5:6832/1102439181 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251997 (secure 0 0 0) 0x7fab14001440 con 0x7fab1006b890 2024-03-24T10:59:07.643 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:59:07.644 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-03-24T10:59:07.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.642+0000 7fab29ea4640 1 -- 172.21.15.5:0/63432032 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fab1006b890 msgr2=0x7fab1006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:07.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.642+0000 7fab29ea4640 1 --2- 172.21.15.5:0/63432032 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fab1006b890 0x7fab1006dd50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fab140017f0 tx=0x7fab14009040 comp rx=0 tx=0).stop 2024-03-24T10:59:07.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.642+0000 7fab29ea4640 1 -- 172.21.15.5:0/63432032 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab2415ac00 msgr2=0x7fab24159040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:07.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.642+0000 7fab29ea4640 1 --2- 172.21.15.5:0/63432032 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab2415ac00 0x7fab24159040 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fab1c00efc0 tx=0x7fab1c00c490 comp rx=0 tx=0).stop 2024-03-24T10:59:07.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.642+0000 7fab29ea4640 1 -- 172.21.15.5:0/63432032 shutdown_connections 2024-03-24T10:59:07.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.642+0000 7fab29ea4640 1 --2- 172.21.15.5:0/63432032 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fab1006b890 0x7fab1006dd50 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.642+0000 7fab29ea4640 1 --2- 172.21.15.5:0/63432032 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fab2415ac00 0x7fab24159040 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.642+0000 7fab29ea4640 1 --2- 172.21.15.5:0/63432032 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fab241543a0 0x7fab240b3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.642+0000 7fab29ea4640 1 --2- 172.21.15.5:0/63432032 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fab240ccaa0 0x7fab240b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.642+0000 7fab29ea4640 1 -- 172.21.15.5:0/63432032 >> 172.21.15.5:0/63432032 conn(0x7fab24072ef0 msgr2=0x7fab241507f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:07.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.643+0000 7fab29ea4640 1 -- 172.21.15.5:0/63432032 shutdown_connections 2024-03-24T10:59:07.646 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.643+0000 7fab29ea4640 1 -- 172.21.15.5:0/63432032 wait complete. 2024-03-24T10:59:07.655 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-03-24T10:59:07.545200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":363,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":301,"ondisk_log_size":301,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239296,"kb_used_data":2336,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754735424,"statfs":{"total":773094113280,"available":772849074176,"internally_reserved":0,"allocated":2392064,"data_stored":1937476,"data_compressed":65430,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":242535986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002318"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.230562+0000","last_change":"2024-03-24T10:45:01.894006+0000","last_active":"2024-03-24T10:55:26.230562+0000","last_peered":"2024-03-24T10:55:26.230562+0000","last_clean":"2024-03-24T10:55:26.230562+0000","last_became_active":"2024-03-24T10:45:01.893734+0000","last_became_peered":"2024-03-24T10:45:01.893734+0000","last_unstale":"2024-03-24T10:55:26.230562+0000","last_undegraded":"2024-03-24T10:55:26.230562+0000","last_fullsized":"2024-03-24T10:55:26.230562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:05:51.579388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.323823+0000","last_change":"2024-03-24T10:45:01.931707+0000","last_active":"2024-03-24T10:55:26.323823+0000","last_peered":"2024-03-24T10:55:26.323823+0000","last_clean":"2024-03-24T10:55:26.323823+0000","last_became_active":"2024-03-24T10:45:01.931449+0000","last_became_peered":"2024-03-24T10:45:01.931449+0000","last_unstale":"2024-03-24T10:55:26.323823+0000","last_undegraded":"2024-03-24T10:55:26.323823+0000","last_fullsized":"2024-03-24T10:55:26.323823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.215675+0000","last_change":"2024-03-24T10:45:01.895929+0000","last_active":"2024-03-24T10:55:29.215675+0000","last_peered":"2024-03-24T10:55:29.215675+0000","last_clean":"2024-03-24T10:55:29.215675+0000","last_became_active":"2024-03-24T10:45:01.895654+0000","last_became_peered":"2024-03-24T10:45:01.895654+0000","last_unstale":"2024-03-24T10:55:29.215675+0000","last_undegraded":"2024-03-24T10:55:29.215675+0000","last_fullsized":"2024-03-24T10:55:29.215675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:01.243946+0000","last_change":"2024-03-24T10:45:01.952468+0000","last_active":"2024-03-24T10:57:01.243946+0000","last_peered":"2024-03-24T10:57:01.243946+0000","last_clean":"2024-03-24T10:57:01.243946+0000","last_became_active":"2024-03-24T10:45:01.952219+0000","last_became_peered":"2024-03-24T10:45:01.952219+0000","last_unstale":"2024-03-24T10:57:01.243946+0000","last_undegraded":"2024-03-24T10:57:01.243946+0000","last_fullsized":"2024-03-24T10:57:01.243946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:27:12.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.228132+0000","last_change":"2024-03-24T10:45:01.895885+0000","last_active":"2024-03-24T10:55:01.228132+0000","last_peered":"2024-03-24T10:55:01.228132+0000","last_clean":"2024-03-24T10:55:01.228132+0000","last_became_active":"2024-03-24T10:45:01.895622+0000","last_became_peered":"2024-03-24T10:45:01.895622+0000","last_unstale":"2024-03-24T10:55:01.228132+0000","last_undegraded":"2024-03-24T10:55:01.228132+0000","last_fullsized":"2024-03-24T10:55:01.228132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:16.523177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:41.320408+0000","last_change":"2024-03-24T10:45:40.169714+0000","last_active":"2024-03-24T10:55:41.320408+0000","last_peered":"2024-03-24T10:55:41.320408+0000","last_clean":"2024-03-24T10:55:41.320408+0000","last_became_active":"2024-03-24T10:45:40.169447+0000","last_became_peered":"2024-03-24T10:45:40.169447+0000","last_unstale":"2024-03-24T10:55:41.320408+0000","last_undegraded":"2024-03-24T10:55:41.320408+0000","last_fullsized":"2024-03-24T10:55:41.320408+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:27:12.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.228112+0000","last_change":"2024-03-24T10:45:01.892745+0000","last_active":"2024-03-24T10:55:01.228112+0000","last_peered":"2024-03-24T10:55:01.228112+0000","last_clean":"2024-03-24T10:55:01.228112+0000","last_became_active":"2024-03-24T10:45:01.892452+0000","last_became_peered":"2024-03-24T10:45:01.892452+0000","last_unstale":"2024-03-24T10:55:01.228112+0000","last_undegraded":"2024-03-24T10:55:01.228112+0000","last_fullsized":"2024-03-24T10:55:01.228112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.198373+0000","last_change":"2024-03-24T10:45:01.953432+0000","last_active":"2024-03-24T10:55:01.198373+0000","last_peered":"2024-03-24T10:55:01.198373+0000","last_clean":"2024-03-24T10:55:01.198373+0000","last_became_active":"2024-03-24T10:45:01.953149+0000","last_became_peered":"2024-03-24T10:45:01.953149+0000","last_unstale":"2024-03-24T10:55:01.198373+0000","last_undegraded":"2024-03-24T10:55:01.198373+0000","last_fullsized":"2024-03-24T10:55:01.198373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.226282+0000","last_change":"2024-03-24T10:45:01.956488+0000","last_active":"2024-03-24T10:54:41.226282+0000","last_peered":"2024-03-24T10:54:41.226282+0000","last_clean":"2024-03-24T10:54:41.226282+0000","last_became_active":"2024-03-24T10:45:01.956207+0000","last_became_peered":"2024-03-24T10:45:01.956207+0000","last_unstale":"2024-03-24T10:54:41.226282+0000","last_undegraded":"2024-03-24T10:54:41.226282+0000","last_fullsized":"2024-03-24T10:54:41.226282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:23:26.523809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:44.289850+0000","last_change":"2024-03-24T10:45:01.956484+0000","last_active":"2024-03-24T10:54:44.289850+0000","last_peered":"2024-03-24T10:54:44.289850+0000","last_clean":"2024-03-24T10:54:44.289850+0000","last_became_active":"2024-03-24T10:45:01.956209+0000","last_became_peered":"2024-03-24T10:45:01.956209+0000","last_unstale":"2024-03-24T10:54:44.289850+0000","last_undegraded":"2024-03-24T10:54:44.289850+0000","last_fullsized":"2024-03-24T10:54:44.289850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:44.289823+0000","last_change":"2024-03-24T10:45:01.957103+0000","last_active":"2024-03-24T10:54:44.289823+0000","last_peered":"2024-03-24T10:54:44.289823+0000","last_clean":"2024-03-24T10:54:44.289823+0000","last_became_active":"2024-03-24T10:45:01.956849+0000","last_became_peered":"2024-03-24T10:45:01.956849+0000","last_unstale":"2024-03-24T10:54:44.289823+0000","last_undegraded":"2024-03-24T10:54:44.289823+0000","last_fullsized":"2024-03-24T10:54:44.289823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:44.262492+0000","last_change":"2024-03-24T10:45:01.896701+0000","last_active":"2024-03-24T10:54:44.262492+0000","last_peered":"2024-03-24T10:54:44.262492+0000","last_clean":"2024-03-24T10:54:44.262492+0000","last_became_active":"2024-03-24T10:45:01.896400+0000","last_became_peered":"2024-03-24T10:45:01.896400+0000","last_unstale":"2024-03-24T10:54:44.262492+0000","last_undegraded":"2024-03-24T10:54:44.262492+0000","last_fullsized":"2024-03-24T10:54:44.262492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:44.380352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.196593+0000","last_change":"2024-03-24T10:45:01.894954+0000","last_active":"2024-03-24T10:54:41.196593+0000","last_peered":"2024-03-24T10:54:41.196593+0000","last_clean":"2024-03-24T10:54:41.196593+0000","last_became_active":"2024-03-24T10:45:01.894632+0000","last_became_peered":"2024-03-24T10:45:01.894632+0000","last_unstale":"2024-03-24T10:54:41.196593+0000","last_undegraded":"2024-03-24T10:54:41.196593+0000","last_fullsized":"2024-03-24T10:54:41.196593+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.221985+0000","last_change":"2024-03-24T10:45:01.929588+0000","last_active":"2024-03-24T10:56:44.221985+0000","last_peered":"2024-03-24T10:56:44.221985+0000","last_clean":"2024-03-24T10:56:44.221985+0000","last_became_active":"2024-03-24T10:45:01.928460+0000","last_became_peered":"2024-03-24T10:45:01.928460+0000","last_unstale":"2024-03-24T10:56:44.221985+0000","last_undegraded":"2024-03-24T10:56:44.221985+0000","last_fullsized":"2024-03-24T10:56:44.221985+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:47:48.896914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.329359+0000","last_change":"2024-03-24T10:45:01.928721+0000","last_active":"2024-03-24T10:56:41.329359+0000","last_peered":"2024-03-24T10:56:41.329359+0000","last_clean":"2024-03-24T10:56:41.329359+0000","last_became_active":"2024-03-24T10:45:01.928467+0000","last_became_peered":"2024-03-24T10:45:01.928467+0000","last_unstale":"2024-03-24T10:56:41.329359+0000","last_undegraded":"2024-03-24T10:56:41.329359+0000","last_fullsized":"2024-03-24T10:56:41.329359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.242233+0000","last_change":"2024-03-24T10:45:01.956030+0000","last_active":"2024-03-24T10:56:41.242233+0000","last_peered":"2024-03-24T10:56:41.242233+0000","last_clean":"2024-03-24T10:56:41.242233+0000","last_became_active":"2024-03-24T10:45:01.955768+0000","last_became_peered":"2024-03-24T10:45:01.955768+0000","last_unstale":"2024-03-24T10:56:41.242233+0000","last_undegraded":"2024-03-24T10:56:41.242233+0000","last_fullsized":"2024-03-24T10:56:41.242233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.242276+0000","last_change":"2024-03-24T10:45:01.956181+0000","last_active":"2024-03-24T10:56:41.242276+0000","last_peered":"2024-03-24T10:56:41.242276+0000","last_clean":"2024-03-24T10:56:41.242276+0000","last_became_active":"2024-03-24T10:45:01.955890+0000","last_became_peered":"2024-03-24T10:45:01.955890+0000","last_unstale":"2024-03-24T10:56:41.242276+0000","last_undegraded":"2024-03-24T10:56:41.242276+0000","last_fullsized":"2024-03-24T10:56:41.242276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:15:11.107915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.325044+0000","last_change":"2024-03-24T10:45:01.896663+0000","last_active":"2024-03-24T10:56:41.325044+0000","last_peered":"2024-03-24T10:56:41.325044+0000","last_clean":"2024-03-24T10:56:41.325044+0000","last_became_active":"2024-03-24T10:45:01.896391+0000","last_became_peered":"2024-03-24T10:45:01.896391+0000","last_unstale":"2024-03-24T10:56:41.325044+0000","last_undegraded":"2024-03-24T10:56:41.325044+0000","last_fullsized":"2024-03-24T10:56:41.325044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:30.942758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.235423+0000","last_change":"2024-03-24T10:45:01.895951+0000","last_active":"2024-03-24T10:56:16.235423+0000","last_peered":"2024-03-24T10:56:16.235423+0000","last_clean":"2024-03-24T10:56:16.235423+0000","last_became_active":"2024-03-24T10:45:01.895668+0000","last_became_peered":"2024-03-24T10:45:01.895668+0000","last_unstale":"2024-03-24T10:56:16.235423+0000","last_undegraded":"2024-03-24T10:56:16.235423+0000","last_fullsized":"2024-03-24T10:56:16.235423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:56:43.462458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:19.297955+0000","last_change":"2024-03-24T10:45:01.898126+0000","last_active":"2024-03-24T10:56:19.297955+0000","last_peered":"2024-03-24T10:56:19.297955+0000","last_clean":"2024-03-24T10:56:19.297955+0000","last_became_active":"2024-03-24T10:45:01.897878+0000","last_became_peered":"2024-03-24T10:45:01.897878+0000","last_unstale":"2024-03-24T10:56:19.297955+0000","last_undegraded":"2024-03-24T10:56:19.297955+0000","last_fullsized":"2024-03-24T10:56:19.297955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:19.220025+0000","last_change":"2024-03-24T10:45:01.896109+0000","last_active":"2024-03-24T10:56:19.220025+0000","last_peered":"2024-03-24T10:56:19.220025+0000","last_clean":"2024-03-24T10:56:19.220025+0000","last_became_active":"2024-03-24T10:45:01.895696+0000","last_became_peered":"2024-03-24T10:45:01.895696+0000","last_unstale":"2024-03-24T10:56:19.220025+0000","last_undegraded":"2024-03-24T10:56:19.220025+0000","last_fullsized":"2024-03-24T10:56:19.220025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.240130+0000","last_change":"2024-03-24T10:45:01.958119+0000","last_active":"2024-03-24T10:56:16.240130+0000","last_peered":"2024-03-24T10:56:16.240130+0000","last_clean":"2024-03-24T10:56:16.240130+0000","last_became_active":"2024-03-24T10:45:01.957832+0000","last_became_peered":"2024-03-24T10:45:01.957832+0000","last_unstale":"2024-03-24T10:56:16.240130+0000","last_undegraded":"2024-03-24T10:56:16.240130+0000","last_fullsized":"2024-03-24T10:56:16.240130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:12.476490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.204495+0000","last_change":"2024-03-24T10:45:01.842442+0000","last_active":"2024-03-24T10:56:16.204495+0000","last_peered":"2024-03-24T10:56:16.204495+0000","last_clean":"2024-03-24T10:56:16.204495+0000","last_became_active":"2024-03-24T10:45:01.842199+0000","last_became_peered":"2024-03-24T10:45:01.842199+0000","last_unstale":"2024-03-24T10:56:16.204495+0000","last_undegraded":"2024-03-24T10:56:16.204495+0000","last_fullsized":"2024-03-24T10:56:16.204495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.237949+0000","last_change":"2024-03-24T10:45:01.952378+0000","last_active":"2024-03-24T10:55:51.237949+0000","last_peered":"2024-03-24T10:55:51.237949+0000","last_clean":"2024-03-24T10:55:51.237949+0000","last_became_active":"2024-03-24T10:45:01.952123+0000","last_became_peered":"2024-03-24T10:45:01.952123+0000","last_unstale":"2024-03-24T10:55:51.237949+0000","last_undegraded":"2024-03-24T10:55:51.237949+0000","last_fullsized":"2024-03-24T10:55:51.237949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.202466+0000","last_change":"2024-03-24T10:45:01.842299+0000","last_active":"2024-03-24T10:55:51.202466+0000","last_peered":"2024-03-24T10:55:51.202466+0000","last_clean":"2024-03-24T10:55:51.202466+0000","last_became_active":"2024-03-24T10:45:01.842097+0000","last_became_peered":"2024-03-24T10:45:01.842097+0000","last_unstale":"2024-03-24T10:55:51.202466+0000","last_undegraded":"2024-03-24T10:55:51.202466+0000","last_fullsized":"2024-03-24T10:55:51.202466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:16.341360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:54.295744+0000","last_change":"2024-03-24T10:45:01.893579+0000","last_active":"2024-03-24T10:55:54.295744+0000","last_peered":"2024-03-24T10:55:54.295744+0000","last_clean":"2024-03-24T10:55:54.295744+0000","last_became_active":"2024-03-24T10:45:01.893273+0000","last_became_peered":"2024-03-24T10:45:01.893273+0000","last_unstale":"2024-03-24T10:55:54.295744+0000","last_undegraded":"2024-03-24T10:55:54.295744+0000","last_fullsized":"2024-03-24T10:55:54.295744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.325622+0000","last_change":"2024-03-24T10:45:01.930032+0000","last_active":"2024-03-24T10:55:51.325622+0000","last_peered":"2024-03-24T10:55:51.325622+0000","last_clean":"2024-03-24T10:55:51.325622+0000","last_became_active":"2024-03-24T10:45:01.929770+0000","last_became_peered":"2024-03-24T10:45:01.929770+0000","last_unstale":"2024-03-24T10:55:51.325622+0000","last_undegraded":"2024-03-24T10:55:51.325622+0000","last_fullsized":"2024-03-24T10:55:51.325622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.202455+0000","last_change":"2024-03-24T10:45:01.842460+0000","last_active":"2024-03-24T10:55:51.202455+0000","last_peered":"2024-03-24T10:55:51.202455+0000","last_clean":"2024-03-24T10:55:51.202455+0000","last_became_active":"2024-03-24T10:45:01.842165+0000","last_became_peered":"2024-03-24T10:45:01.842165+0000","last_unstale":"2024-03-24T10:55:51.202455+0000","last_undegraded":"2024-03-24T10:55:51.202455+0000","last_fullsized":"2024-03-24T10:55:51.202455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.215656+0000","last_change":"2024-03-24T10:45:01.895868+0000","last_active":"2024-03-24T10:55:29.215656+0000","last_peered":"2024-03-24T10:55:29.215656+0000","last_clean":"2024-03-24T10:55:29.215656+0000","last_became_active":"2024-03-24T10:45:01.895290+0000","last_became_peered":"2024-03-24T10:45:01.895290+0000","last_unstale":"2024-03-24T10:55:29.215656+0000","last_undegraded":"2024-03-24T10:55:29.215656+0000","last_fullsized":"2024-03-24T10:55:29.215656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.235695+0000","last_change":"2024-03-24T10:45:01.957724+0000","last_active":"2024-03-24T10:55:26.235695+0000","last_peered":"2024-03-24T10:55:26.235695+0000","last_clean":"2024-03-24T10:55:26.235695+0000","last_became_active":"2024-03-24T10:45:01.957429+0000","last_became_peered":"2024-03-24T10:45:01.957429+0000","last_unstale":"2024-03-24T10:55:26.235695+0000","last_undegraded":"2024-03-24T10:55:26.235695+0000","last_fullsized":"2024-03-24T10:55:26.235695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:22.089070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.319191+0000","last_change":"2024-03-24T10:45:01.952159+0000","last_active":"2024-03-24T10:55:26.319191+0000","last_peered":"2024-03-24T10:55:26.319191+0000","last_clean":"2024-03-24T10:55:26.319191+0000","last_became_active":"2024-03-24T10:45:01.951961+0000","last_became_peered":"2024-03-24T10:45:01.951961+0000","last_unstale":"2024-03-24T10:55:26.319191+0000","last_undegraded":"2024-03-24T10:55:26.319191+0000","last_fullsized":"2024-03-24T10:55:26.319191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:44.380352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.293413+0000","last_change":"2024-03-24T10:45:01.893452+0000","last_active":"2024-03-24T10:55:29.293413+0000","last_peered":"2024-03-24T10:55:29.293413+0000","last_clean":"2024-03-24T10:55:29.293413+0000","last_became_active":"2024-03-24T10:45:01.893139+0000","last_became_peered":"2024-03-24T10:45:01.893139+0000","last_unstale":"2024-03-24T10:55:29.293413+0000","last_undegraded":"2024-03-24T10:55:29.293413+0000","last_fullsized":"2024-03-24T10:55:29.293413+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:04.264076+0000","last_change":"2024-03-24T10:45:01.928827+0000","last_active":"2024-03-24T10:55:04.264076+0000","last_peered":"2024-03-24T10:55:04.264076+0000","last_clean":"2024-03-24T10:55:04.264076+0000","last_became_active":"2024-03-24T10:45:01.928521+0000","last_became_peered":"2024-03-24T10:45:01.928521+0000","last_unstale":"2024-03-24T10:55:04.264076+0000","last_undegraded":"2024-03-24T10:55:04.264076+0000","last_fullsized":"2024-03-24T10:55:04.264076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:59.643246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"24'7","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.198403+0000","last_change":"2024-03-24T10:45:00.637993+0000","last_active":"2024-03-24T10:55:01.198403+0000","last_peered":"2024-03-24T10:55:01.198403+0000","last_clean":"2024-03-24T10:55:01.198403+0000","last_became_active":"2024-03-24T10:45:00.637803+0000","last_became_peered":"2024-03-24T10:45:00.637803+0000","last_unstale":"2024-03-24T10:55:01.198403+0000","last_undegraded":"2024-03-24T10:55:01.198403+0000","last_fullsized":"2024-03-24T10:55:01.198403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:28:28.854541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:04.213682+0000","last_change":"2024-03-24T10:45:01.897686+0000","last_active":"2024-03-24T10:55:04.213682+0000","last_peered":"2024-03-24T10:55:04.213682+0000","last_clean":"2024-03-24T10:55:04.213682+0000","last_became_active":"2024-03-24T10:45:01.897320+0000","last_became_peered":"2024-03-24T10:45:01.897320+0000","last_unstale":"2024-03-24T10:55:04.213682+0000","last_undegraded":"2024-03-24T10:55:04.213682+0000","last_fullsized":"2024-03-24T10:55:04.213682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:04.291481+0000","last_change":"2024-03-24T10:45:00.638329+0000","last_active":"2024-03-24T10:55:04.291481+0000","last_peered":"2024-03-24T10:55:04.291481+0000","last_clean":"2024-03-24T10:55:04.291481+0000","last_became_active":"2024-03-24T10:45:00.638006+0000","last_became_peered":"2024-03-24T10:45:00.638006+0000","last_unstale":"2024-03-24T10:55:04.291481+0000","last_undegraded":"2024-03-24T10:55:04.291481+0000","last_fullsized":"2024-03-24T10:55:04.291481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.196573+0000","last_change":"2024-03-24T10:45:00.633703+0000","last_active":"2024-03-24T10:54:41.196573+0000","last_peered":"2024-03-24T10:54:41.196573+0000","last_clean":"2024-03-24T10:54:41.196573+0000","last_became_active":"2024-03-24T10:45:00.633427+0000","last_became_peered":"2024-03-24T10:45:00.633427+0000","last_unstale":"2024-03-24T10:54:41.196573+0000","last_undegraded":"2024-03-24T10:54:41.196573+0000","last_fullsized":"2024-03-24T10:54:41.196573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:30.350191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.196539+0000","last_change":"2024-03-24T10:45:01.893902+0000","last_active":"2024-03-24T10:54:41.196539+0000","last_peered":"2024-03-24T10:54:41.196539+0000","last_clean":"2024-03-24T10:54:41.196539+0000","last_became_active":"2024-03-24T10:45:01.893617+0000","last_became_peered":"2024-03-24T10:45:01.893617+0000","last_unstale":"2024-03-24T10:54:41.196539+0000","last_undegraded":"2024-03-24T10:54:41.196539+0000","last_fullsized":"2024-03-24T10:54:41.196539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"24'4","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.231681+0000","last_change":"2024-03-24T10:45:00.638778+0000","last_active":"2024-03-24T10:54:41.231681+0000","last_peered":"2024-03-24T10:54:41.231681+0000","last_clean":"2024-03-24T10:54:41.231681+0000","last_became_active":"2024-03-24T10:45:00.638089+0000","last_became_peered":"2024-03-24T10:45:00.638089+0000","last_unstale":"2024-03-24T10:54:41.231681+0000","last_undegraded":"2024-03-24T10:54:41.231681+0000","last_fullsized":"2024-03-24T10:54:41.231681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.320204+0000","last_change":"2024-03-24T10:45:39.901726+0000","last_active":"2024-03-24T10:54:41.320204+0000","last_peered":"2024-03-24T10:54:41.320204+0000","last_clean":"2024-03-24T10:54:41.320204+0000","last_became_active":"2024-03-24T10:45:39.901416+0000","last_became_peered":"2024-03-24T10:45:39.901416+0000","last_unstale":"2024-03-24T10:54:41.320204+0000","last_undegraded":"2024-03-24T10:54:41.320204+0000","last_fullsized":"2024-03-24T10:54:41.320204+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:23:26.523809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'4","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:44.289869+0000","last_change":"2024-03-24T10:45:00.639822+0000","last_active":"2024-03-24T10:54:44.289869+0000","last_peered":"2024-03-24T10:54:44.289869+0000","last_clean":"2024-03-24T10:54:44.289869+0000","last_became_active":"2024-03-24T10:45:00.639600+0000","last_became_peered":"2024-03-24T10:45:00.639600+0000","last_unstale":"2024-03-24T10:54:44.289869+0000","last_undegraded":"2024-03-24T10:54:44.289869+0000","last_fullsized":"2024-03-24T10:54:44.289869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.231724+0000","last_change":"2024-03-24T10:45:01.954015+0000","last_active":"2024-03-24T10:54:41.231724+0000","last_peered":"2024-03-24T10:54:41.231724+0000","last_clean":"2024-03-24T10:54:41.231724+0000","last_became_active":"2024-03-24T10:45:01.953767+0000","last_became_peered":"2024-03-24T10:45:01.953767+0000","last_unstale":"2024-03-24T10:54:41.231724+0000","last_undegraded":"2024-03-24T10:54:41.231724+0000","last_fullsized":"2024-03-24T10:54:41.231724+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.231756+0000","last_change":"2024-03-24T10:45:00.636553+0000","last_active":"2024-03-24T10:54:41.231756+0000","last_peered":"2024-03-24T10:54:41.231756+0000","last_clean":"2024-03-24T10:54:41.231756+0000","last_became_active":"2024-03-24T10:45:00.636282+0000","last_became_peered":"2024-03-24T10:45:00.636282+0000","last_unstale":"2024-03-24T10:54:41.231756+0000","last_undegraded":"2024-03-24T10:54:41.231756+0000","last_fullsized":"2024-03-24T10:54:41.231756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:44.212223+0000","last_change":"2024-03-24T10:45:01.957905+0000","last_active":"2024-03-24T10:54:44.212223+0000","last_peered":"2024-03-24T10:54:44.212223+0000","last_clean":"2024-03-24T10:54:44.212223+0000","last_became_active":"2024-03-24T10:45:01.957676+0000","last_became_peered":"2024-03-24T10:45:01.957676+0000","last_unstale":"2024-03-24T10:54:44.212223+0000","last_undegraded":"2024-03-24T10:54:44.212223+0000","last_fullsized":"2024-03-24T10:54:44.212223+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.320240+0000","last_change":"2024-03-24T10:45:00.635197+0000","last_active":"2024-03-24T10:54:41.320240+0000","last_peered":"2024-03-24T10:54:41.320240+0000","last_clean":"2024-03-24T10:54:41.320240+0000","last_became_active":"2024-03-24T10:45:00.634989+0000","last_became_peered":"2024-03-24T10:45:00.634989+0000","last_unstale":"2024-03-24T10:54:41.320240+0000","last_undegraded":"2024-03-24T10:54:41.320240+0000","last_fullsized":"2024-03-24T10:54:41.320240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.272421+0000","last_change":"2024-03-24T10:45:01.853785+0000","last_active":"2024-03-24T10:56:44.272421+0000","last_peered":"2024-03-24T10:56:44.272421+0000","last_clean":"2024-03-24T10:56:44.272421+0000","last_became_active":"2024-03-24T10:45:01.853504+0000","last_became_peered":"2024-03-24T10:45:01.853504+0000","last_unstale":"2024-03-24T10:56:44.272421+0000","last_undegraded":"2024-03-24T10:56:44.272421+0000","last_fullsized":"2024-03-24T10:56:44.272421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:30.942758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":36,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.300179+0000","last_change":"2024-03-24T10:45:00.632381+0000","last_active":"2024-03-24T10:56:44.300179+0000","last_peered":"2024-03-24T10:56:44.300179+0000","last_clean":"2024-03-24T10:56:44.300179+0000","last_became_active":"2024-03-24T10:45:00.632108+0000","last_became_peered":"2024-03-24T10:45:00.632108+0000","last_unstale":"2024-03-24T10:56:44.300179+0000","last_undegraded":"2024-03-24T10:56:44.300179+0000","last_fullsized":"2024-03-24T10:56:44.300179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:06.142451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.226243+0000","last_change":"2024-03-24T10:45:01.853591+0000","last_active":"2024-03-24T10:54:41.226243+0000","last_peered":"2024-03-24T10:54:41.226243+0000","last_clean":"2024-03-24T10:54:41.226243+0000","last_became_active":"2024-03-24T10:45:01.853279+0000","last_became_peered":"2024-03-24T10:45:01.853279+0000","last_unstale":"2024-03-24T10:54:41.226243+0000","last_undegraded":"2024-03-24T10:54:41.226243+0000","last_fullsized":"2024-03-24T10:54:41.226243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:07:24.149961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'9","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.222018+0000","last_change":"2024-03-24T10:45:00.634192+0000","last_active":"2024-03-24T10:56:44.222018+0000","last_peered":"2024-03-24T10:56:44.222018+0000","last_clean":"2024-03-24T10:56:44.222018+0000","last_became_active":"2024-03-24T10:45:00.633893+0000","last_became_peered":"2024-03-24T10:45:00.633893+0000","last_unstale":"2024-03-24T10:56:44.222018+0000","last_undegraded":"2024-03-24T10:56:44.222018+0000","last_fullsized":"2024-03-24T10:56:44.222018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:30.350191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.222005+0000","last_change":"2024-03-24T10:45:01.893426+0000","last_active":"2024-03-24T10:56:44.222005+0000","last_peered":"2024-03-24T10:56:44.222005+0000","last_clean":"2024-03-24T10:56:44.222005+0000","last_became_active":"2024-03-24T10:45:01.893178+0000","last_became_peered":"2024-03-24T10:45:01.893178+0000","last_unstale":"2024-03-24T10:56:44.222005+0000","last_undegraded":"2024-03-24T10:56:44.222005+0000","last_fullsized":"2024-03-24T10:56:44.222005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:13.887947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"24'6","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.242255+0000","last_change":"2024-03-24T10:45:00.638214+0000","last_active":"2024-03-24T10:56:41.242255+0000","last_peered":"2024-03-24T10:56:41.242255+0000","last_clean":"2024-03-24T10:56:41.242255+0000","last_became_active":"2024-03-24T10:45:00.637973+0000","last_became_peered":"2024-03-24T10:45:00.637973+0000","last_unstale":"2024-03-24T10:56:41.242255+0000","last_undegraded":"2024-03-24T10:56:41.242255+0000","last_fullsized":"2024-03-24T10:56:41.242255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:15.749451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.237815+0000","last_change":"2024-03-24T10:45:01.856756+0000","last_active":"2024-03-24T10:56:41.237815+0000","last_peered":"2024-03-24T10:56:41.237815+0000","last_clean":"2024-03-24T10:56:41.237815+0000","last_became_active":"2024-03-24T10:45:01.856482+0000","last_became_peered":"2024-03-24T10:45:01.856482+0000","last_unstale":"2024-03-24T10:56:41.237815+0000","last_undegraded":"2024-03-24T10:56:41.237815+0000","last_fullsized":"2024-03-24T10:56:41.237815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:12.476490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.300155+0000","last_change":"2024-03-24T10:45:00.636481+0000","last_active":"2024-03-24T10:56:44.300155+0000","last_peered":"2024-03-24T10:56:44.300155+0000","last_clean":"2024-03-24T10:56:44.300155+0000","last_became_active":"2024-03-24T10:45:00.636206+0000","last_became_peered":"2024-03-24T10:45:00.636206+0000","last_unstale":"2024-03-24T10:56:44.300155+0000","last_undegraded":"2024-03-24T10:56:44.300155+0000","last_fullsized":"2024-03-24T10:56:44.300155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.300128+0000","last_change":"2024-03-24T10:45:01.897279+0000","last_active":"2024-03-24T10:56:44.300128+0000","last_peered":"2024-03-24T10:56:44.300128+0000","last_clean":"2024-03-24T10:56:44.300128+0000","last_became_active":"2024-03-24T10:45:01.896736+0000","last_became_peered":"2024-03-24T10:45:01.896736+0000","last_unstale":"2024-03-24T10:56:44.300128+0000","last_undegraded":"2024-03-24T10:56:44.300128+0000","last_fullsized":"2024-03-24T10:56:44.300128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.325004+0000","last_change":"2024-03-24T10:45:00.564066+0000","last_active":"2024-03-24T10:56:41.325004+0000","last_peered":"2024-03-24T10:56:41.325004+0000","last_clean":"2024-03-24T10:56:41.325004+0000","last_became_active":"2024-03-24T10:45:00.563812+0000","last_became_peered":"2024-03-24T10:45:00.563812+0000","last_unstale":"2024-03-24T10:56:41.325004+0000","last_undegraded":"2024-03-24T10:56:41.325004+0000","last_fullsized":"2024-03-24T10:56:41.325004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:28:28.854541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.237784+0000","last_change":"2024-03-24T10:45:01.854429+0000","last_active":"2024-03-24T10:56:41.237784+0000","last_peered":"2024-03-24T10:56:41.237784+0000","last_clean":"2024-03-24T10:56:41.237784+0000","last_became_active":"2024-03-24T10:45:01.854179+0000","last_became_peered":"2024-03-24T10:45:01.854179+0000","last_unstale":"2024-03-24T10:56:41.237784+0000","last_undegraded":"2024-03-24T10:56:41.237784+0000","last_fullsized":"2024-03-24T10:56:41.237784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:19.270242+0000","last_change":"2024-03-24T10:45:00.634223+0000","last_active":"2024-03-24T10:56:19.270242+0000","last_peered":"2024-03-24T10:56:19.270242+0000","last_clean":"2024-03-24T10:56:19.270242+0000","last_became_active":"2024-03-24T10:45:00.633355+0000","last_became_peered":"2024-03-24T10:45:00.633355+0000","last_unstale":"2024-03-24T10:56:19.270242+0000","last_undegraded":"2024-03-24T10:56:19.270242+0000","last_fullsized":"2024-03-24T10:56:19.270242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:04.930115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.320259+0000","last_change":"2024-03-24T10:45:40.518338+0000","last_active":"2024-03-24T10:54:41.320259+0000","last_peered":"2024-03-24T10:54:41.320259+0000","last_clean":"2024-03-24T10:54:41.320259+0000","last_became_active":"2024-03-24T10:45:40.518035+0000","last_became_peered":"2024-03-24T10:45:40.518035+0000","last_unstale":"2024-03-24T10:54:41.320259+0000","last_undegraded":"2024-03-24T10:54:41.320259+0000","last_fullsized":"2024-03-24T10:54:41.320259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:29:31.508911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.323256+0000","last_change":"2024-03-24T10:45:00.626484+0000","last_active":"2024-03-24T10:56:16.323256+0000","last_peered":"2024-03-24T10:56:16.323256+0000","last_clean":"2024-03-24T10:56:16.323256+0000","last_became_active":"2024-03-24T10:45:00.626195+0000","last_became_peered":"2024-03-24T10:45:00.626195+0000","last_unstale":"2024-03-24T10:56:16.323256+0000","last_undegraded":"2024-03-24T10:56:16.323256+0000","last_fullsized":"2024-03-24T10:56:16.323256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.235444+0000","last_change":"2024-03-24T10:45:01.856995+0000","last_active":"2024-03-24T10:56:16.235444+0000","last_peered":"2024-03-24T10:56:16.235444+0000","last_clean":"2024-03-24T10:56:16.235444+0000","last_became_active":"2024-03-24T10:45:01.856531+0000","last_became_peered":"2024-03-24T10:45:01.856531+0000","last_unstale":"2024-03-24T10:56:16.235444+0000","last_undegraded":"2024-03-24T10:56:16.235444+0000","last_fullsized":"2024-03-24T10:56:16.235444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:29:31.508911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.327524+0000","last_change":"2024-03-24T10:45:00.628438+0000","last_active":"2024-03-24T10:56:16.327524+0000","last_peered":"2024-03-24T10:56:16.327524+0000","last_clean":"2024-03-24T10:56:16.327524+0000","last_became_active":"2024-03-24T10:45:00.628158+0000","last_became_peered":"2024-03-24T10:45:00.628158+0000","last_unstale":"2024-03-24T10:56:16.327524+0000","last_undegraded":"2024-03-24T10:56:16.327524+0000","last_fullsized":"2024-03-24T10:56:16.327524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:29.826202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.315968+0000","last_change":"2024-03-24T10:45:39.901453+0000","last_active":"2024-03-24T10:54:41.315968+0000","last_peered":"2024-03-24T10:54:41.315968+0000","last_clean":"2024-03-24T10:54:41.315968+0000","last_became_active":"2024-03-24T10:45:39.901119+0000","last_became_peered":"2024-03-24T10:45:39.901119+0000","last_unstale":"2024-03-24T10:54:41.315968+0000","last_undegraded":"2024-03-24T10:54:41.315968+0000","last_fullsized":"2024-03-24T10:54:41.315968+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:16.341360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.323237+0000","last_change":"2024-03-24T10:45:00.639093+0000","last_active":"2024-03-24T10:56:16.323237+0000","last_peered":"2024-03-24T10:56:16.323237+0000","last_clean":"2024-03-24T10:56:16.323237+0000","last_became_active":"2024-03-24T10:45:00.638785+0000","last_became_peered":"2024-03-24T10:45:00.638785+0000","last_unstale":"2024-03-24T10:56:16.323237+0000","last_undegraded":"2024-03-24T10:56:16.323237+0000","last_fullsized":"2024-03-24T10:56:16.323237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.235406+0000","last_change":"2024-03-24T10:45:01.953772+0000","last_active":"2024-03-24T10:56:16.235406+0000","last_peered":"2024-03-24T10:56:16.235406+0000","last_clean":"2024-03-24T10:56:16.235406+0000","last_became_active":"2024-03-24T10:45:01.953402+0000","last_became_peered":"2024-03-24T10:45:01.953402+0000","last_unstale":"2024-03-24T10:56:16.235406+0000","last_undegraded":"2024-03-24T10:56:16.235406+0000","last_fullsized":"2024-03-24T10:56:16.235406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:22.089070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":183,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:51.238749+0000","last_change":"2024-03-24T10:44:48.241074+0000","last_active":"2024-03-24T10:56:51.238749+0000","last_peered":"2024-03-24T10:56:51.238749+0000","last_clean":"2024-03-24T10:56:51.238749+0000","last_became_active":"2024-03-24T10:44:48.239986+0000","last_became_peered":"2024-03-24T10:44:48.239986+0000","last_unstale":"2024-03-24T10:56:51.238749+0000","last_undegraded":"2024-03-24T10:56:51.238749+0000","last_fullsized":"2024-03-24T10:56:51.238749+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:46.142247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:46.142247+0000","last_clean_scrub_stamp":"2024-03-24T10:44:46.142247+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:05:50.110100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.235730+0000","last_change":"2024-03-24T10:45:00.637276+0000","last_active":"2024-03-24T10:55:26.235730+0000","last_peered":"2024-03-24T10:55:26.235730+0000","last_clean":"2024-03-24T10:55:26.235730+0000","last_became_active":"2024-03-24T10:45:00.636734+0000","last_became_peered":"2024-03-24T10:45:00.636734+0000","last_unstale":"2024-03-24T10:55:26.235730+0000","last_undegraded":"2024-03-24T10:55:26.235730+0000","last_fullsized":"2024-03-24T10:55:26.235730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.200440+0000","last_change":"2024-03-24T10:45:01.958505+0000","last_active":"2024-03-24T10:55:26.200440+0000","last_peered":"2024-03-24T10:55:26.200440+0000","last_clean":"2024-03-24T10:55:26.200440+0000","last_became_active":"2024-03-24T10:45:01.958268+0000","last_became_peered":"2024-03-24T10:45:01.958268+0000","last_unstale":"2024-03-24T10:55:26.200440+0000","last_undegraded":"2024-03-24T10:55:26.200440+0000","last_fullsized":"2024-03-24T10:55:26.200440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:27:12.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"24'5","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:01.239735+0000","last_change":"2024-03-24T10:45:00.597856+0000","last_active":"2024-03-24T10:57:01.239735+0000","last_peered":"2024-03-24T10:57:01.239735+0000","last_clean":"2024-03-24T10:57:01.239735+0000","last_became_active":"2024-03-24T10:45:00.597640+0000","last_became_peered":"2024-03-24T10:45:00.597640+0000","last_unstale":"2024-03-24T10:57:01.239735+0000","last_undegraded":"2024-03-24T10:57:01.239735+0000","last_fullsized":"2024-03-24T10:57:01.239735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:29.826202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.215622+0000","last_change":"2024-03-24T10:45:01.897502+0000","last_active":"2024-03-24T10:55:29.215622+0000","last_peered":"2024-03-24T10:55:29.215622+0000","last_clean":"2024-03-24T10:55:29.215622+0000","last_became_active":"2024-03-24T10:45:01.897250+0000","last_became_peered":"2024-03-24T10:45:01.897250+0000","last_unstale":"2024-03-24T10:55:29.215622+0000","last_undegraded":"2024-03-24T10:55:29.215622+0000","last_fullsized":"2024-03-24T10:55:29.215622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:09:46.341672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"24'6","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.319220+0000","last_change":"2024-03-24T10:45:00.627166+0000","last_active":"2024-03-24T10:55:26.319220+0000","last_peered":"2024-03-24T10:55:26.319220+0000","last_clean":"2024-03-24T10:55:26.319220+0000","last_became_active":"2024-03-24T10:45:00.626881+0000","last_became_peered":"2024-03-24T10:45:00.626881+0000","last_unstale":"2024-03-24T10:55:26.319220+0000","last_undegraded":"2024-03-24T10:55:26.319220+0000","last_fullsized":"2024-03-24T10:55:26.319220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:01.330987+0000","last_change":"2024-03-24T10:45:01.930271+0000","last_active":"2024-03-24T10:57:01.330987+0000","last_peered":"2024-03-24T10:57:01.330987+0000","last_clean":"2024-03-24T10:57:01.330987+0000","last_became_active":"2024-03-24T10:45:01.929998+0000","last_became_peered":"2024-03-24T10:45:01.929998+0000","last_unstale":"2024-03-24T10:57:01.330987+0000","last_undegraded":"2024-03-24T10:57:01.330987+0000","last_fullsized":"2024-03-24T10:57:01.330987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.235722+0000","last_change":"2024-03-24T10:45:00.638309+0000","last_active":"2024-03-24T10:55:26.235722+0000","last_peered":"2024-03-24T10:55:26.235722+0000","last_clean":"2024-03-24T10:55:26.235722+0000","last_became_active":"2024-03-24T10:45:00.638041+0000","last_became_peered":"2024-03-24T10:45:00.638041+0000","last_unstale":"2024-03-24T10:55:26.235722+0000","last_undegraded":"2024-03-24T10:55:26.235722+0000","last_fullsized":"2024-03-24T10:55:26.235722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:14.049099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.266124+0000","last_change":"2024-03-24T10:45:01.897722+0000","last_active":"2024-03-24T10:55:29.266124+0000","last_peered":"2024-03-24T10:55:29.266124+0000","last_clean":"2024-03-24T10:55:29.266124+0000","last_became_active":"2024-03-24T10:45:01.897159+0000","last_became_peered":"2024-03-24T10:45:01.897159+0000","last_unstale":"2024-03-24T10:55:29.266124+0000","last_undegraded":"2024-03-24T10:55:29.266124+0000","last_fullsized":"2024-03-24T10:55:29.266124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:42.293342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.323798+0000","last_change":"2024-03-24T10:45:00.629267+0000","last_active":"2024-03-24T10:55:26.323798+0000","last_peered":"2024-03-24T10:55:26.323798+0000","last_clean":"2024-03-24T10:55:26.323798+0000","last_became_active":"2024-03-24T10:45:00.628971+0000","last_became_peered":"2024-03-24T10:45:00.628971+0000","last_unstale":"2024-03-24T10:55:26.323798+0000","last_undegraded":"2024-03-24T10:55:26.323798+0000","last_fullsized":"2024-03-24T10:55:26.323798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:41.176786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.235742+0000","last_change":"2024-03-24T10:45:01.952527+0000","last_active":"2024-03-24T10:55:26.235742+0000","last_peered":"2024-03-24T10:55:26.235742+0000","last_clean":"2024-03-24T10:55:26.235742+0000","last_became_active":"2024-03-24T10:45:01.952192+0000","last_became_peered":"2024-03-24T10:45:01.952192+0000","last_unstale":"2024-03-24T10:55:26.235742+0000","last_undegraded":"2024-03-24T10:55:26.235742+0000","last_fullsized":"2024-03-24T10:55:26.235742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.230535+0000","last_change":"2024-03-24T10:45:00.631941+0000","last_active":"2024-03-24T10:55:26.230535+0000","last_peered":"2024-03-24T10:55:26.230535+0000","last_clean":"2024-03-24T10:55:26.230535+0000","last_became_active":"2024-03-24T10:45:00.631670+0000","last_became_peered":"2024-03-24T10:45:00.631670+0000","last_unstale":"2024-03-24T10:55:26.230535+0000","last_undegraded":"2024-03-24T10:55:26.230535+0000","last_fullsized":"2024-03-24T10:55:26.230535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:09:45.225115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.323774+0000","last_change":"2024-03-24T10:45:01.930483+0000","last_active":"2024-03-24T10:55:26.323774+0000","last_peered":"2024-03-24T10:55:26.323774+0000","last_clean":"2024-03-24T10:55:26.323774+0000","last_became_active":"2024-03-24T10:45:01.930220+0000","last_became_peered":"2024-03-24T10:45:01.930220+0000","last_unstale":"2024-03-24T10:55:26.323774+0000","last_undegraded":"2024-03-24T10:55:26.323774+0000","last_fullsized":"2024-03-24T10:55:26.323774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.325636+0000","last_change":"2024-03-24T10:45:00.633650+0000","last_active":"2024-03-24T10:55:51.325636+0000","last_peered":"2024-03-24T10:55:51.325636+0000","last_clean":"2024-03-24T10:55:51.325636+0000","last_became_active":"2024-03-24T10:45:00.633329+0000","last_became_peered":"2024-03-24T10:45:00.633329+0000","last_unstale":"2024-03-24T10:55:51.325636+0000","last_undegraded":"2024-03-24T10:55:51.325636+0000","last_fullsized":"2024-03-24T10:55:51.325636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.202473+0000","last_change":"2024-03-24T10:45:01.853487+0000","last_active":"2024-03-24T10:55:51.202473+0000","last_peered":"2024-03-24T10:55:51.202473+0000","last_clean":"2024-03-24T10:55:51.202473+0000","last_became_active":"2024-03-24T10:45:01.853257+0000","last_became_peered":"2024-03-24T10:45:01.853257+0000","last_unstale":"2024-03-24T10:55:51.202473+0000","last_undegraded":"2024-03-24T10:55:51.202473+0000","last_fullsized":"2024-03-24T10:55:51.202473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:54.295762+0000","last_change":"2024-03-24T10:45:00.639282+0000","last_active":"2024-03-24T10:55:54.295762+0000","last_peered":"2024-03-24T10:55:54.295762+0000","last_clean":"2024-03-24T10:55:54.295762+0000","last_became_active":"2024-03-24T10:45:00.638999+0000","last_became_peered":"2024-03-24T10:45:00.638999+0000","last_unstale":"2024-03-24T10:55:54.295762+0000","last_undegraded":"2024-03-24T10:55:54.295762+0000","last_fullsized":"2024-03-24T10:55:54.295762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:41.176786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:54.217821+0000","last_change":"2024-03-24T10:45:01.928900+0000","last_active":"2024-03-24T10:55:54.217821+0000","last_peered":"2024-03-24T10:55:54.217821+0000","last_clean":"2024-03-24T10:55:54.217821+0000","last_became_active":"2024-03-24T10:45:01.928610+0000","last_became_peered":"2024-03-24T10:45:01.928610+0000","last_unstale":"2024-03-24T10:55:54.217821+0000","last_undegraded":"2024-03-24T10:55:54.217821+0000","last_fullsized":"2024-03-24T10:55:54.217821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.232929+0000","last_change":"2024-03-24T10:45:00.631327+0000","last_active":"2024-03-24T10:55:51.232929+0000","last_peered":"2024-03-24T10:55:51.232929+0000","last_clean":"2024-03-24T10:55:51.232929+0000","last_became_active":"2024-03-24T10:45:00.631078+0000","last_became_peered":"2024-03-24T10:45:00.631078+0000","last_unstale":"2024-03-24T10:55:51.232929+0000","last_undegraded":"2024-03-24T10:55:51.232929+0000","last_fullsized":"2024-03-24T10:55:51.232929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:43.263796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:54.268364+0000","last_change":"2024-03-24T10:45:01.897329+0000","last_active":"2024-03-24T10:55:54.268364+0000","last_peered":"2024-03-24T10:55:54.268364+0000","last_clean":"2024-03-24T10:55:54.268364+0000","last_became_active":"2024-03-24T10:45:01.897063+0000","last_became_peered":"2024-03-24T10:45:01.897063+0000","last_unstale":"2024-03-24T10:55:54.268364+0000","last_undegraded":"2024-03-24T10:55:54.268364+0000","last_fullsized":"2024-03-24T10:55:54.268364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:07.259007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.325593+0000","last_change":"2024-03-24T10:45:00.597948+0000","last_active":"2024-03-24T10:55:51.325593+0000","last_peered":"2024-03-24T10:55:51.325593+0000","last_clean":"2024-03-24T10:55:51.325593+0000","last_became_active":"2024-03-24T10:45:00.597671+0000","last_became_peered":"2024-03-24T10:45:00.597671+0000","last_unstale":"2024-03-24T10:55:51.325593+0000","last_undegraded":"2024-03-24T10:55:51.325593+0000","last_fullsized":"2024-03-24T10:55:51.325593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:06.142451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.202427+0000","last_change":"2024-03-24T10:45:01.927822+0000","last_active":"2024-03-24T10:55:51.202427+0000","last_peered":"2024-03-24T10:55:51.202427+0000","last_clean":"2024-03-24T10:55:51.202427+0000","last_became_active":"2024-03-24T10:45:01.927538+0000","last_became_peered":"2024-03-24T10:45:01.927538+0000","last_unstale":"2024-03-24T10:55:51.202427+0000","last_undegraded":"2024-03-24T10:55:51.202427+0000","last_fullsized":"2024-03-24T10:55:51.202427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'3","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.325612+0000","last_change":"2024-03-24T10:45:00.626818+0000","last_active":"2024-03-24T10:55:51.325612+0000","last_peered":"2024-03-24T10:55:51.325612+0000","last_clean":"2024-03-24T10:55:51.325612+0000","last_became_active":"2024-03-24T10:45:00.626557+0000","last_became_peered":"2024-03-24T10:45:00.626557+0000","last_unstale":"2024-03-24T10:55:51.325612+0000","last_undegraded":"2024-03-24T10:55:51.325612+0000","last_fullsized":"2024-03-24T10:55:51.325612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:43.263796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.204517+0000","last_change":"2024-03-24T10:45:01.893192+0000","last_active":"2024-03-24T10:56:16.204517+0000","last_peered":"2024-03-24T10:56:16.204517+0000","last_clean":"2024-03-24T10:56:16.204517+0000","last_became_active":"2024-03-24T10:45:01.892934+0000","last_became_peered":"2024-03-24T10:45:01.892934+0000","last_unstale":"2024-03-24T10:56:16.204517+0000","last_undegraded":"2024-03-24T10:56:16.204517+0000","last_fullsized":"2024-03-24T10:56:16.204517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.235371+0000","last_change":"2024-03-24T10:45:00.630147+0000","last_active":"2024-03-24T10:56:16.235371+0000","last_peered":"2024-03-24T10:56:16.235371+0000","last_clean":"2024-03-24T10:56:16.235371+0000","last_became_active":"2024-03-24T10:45:00.629850+0000","last_became_peered":"2024-03-24T10:45:00.629850+0000","last_unstale":"2024-03-24T10:56:16.235371+0000","last_undegraded":"2024-03-24T10:56:16.235371+0000","last_fullsized":"2024-03-24T10:56:16.235371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:01:12.843222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:54.295704+0000","last_change":"2024-03-24T10:45:01.897333+0000","last_active":"2024-03-24T10:55:54.295704+0000","last_peered":"2024-03-24T10:55:54.295704+0000","last_clean":"2024-03-24T10:55:54.295704+0000","last_became_active":"2024-03-24T10:45:01.897080+0000","last_became_peered":"2024-03-24T10:45:01.897080+0000","last_unstale":"2024-03-24T10:55:54.295704+0000","last_undegraded":"2024-03-24T10:55:54.295704+0000","last_fullsized":"2024-03-24T10:55:54.295704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:13.887947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.233488+0000","last_change":"2024-03-24T10:45:01.956041+0000","last_active":"2024-03-24T10:55:01.233488+0000","last_peered":"2024-03-24T10:55:01.233488+0000","last_clean":"2024-03-24T10:55:01.233488+0000","last_became_active":"2024-03-24T10:45:01.955766+0000","last_became_peered":"2024-03-24T10:45:01.955766+0000","last_unstale":"2024-03-24T10:55:01.233488+0000","last_undegraded":"2024-03-24T10:55:01.233488+0000","last_fullsized":"2024-03-24T10:55:01.233488+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:16.341360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.228149+0000","last_change":"2024-03-24T10:45:00.639913+0000","last_active":"2024-03-24T10:55:01.228149+0000","last_peered":"2024-03-24T10:55:01.228149+0000","last_clean":"2024-03-24T10:55:01.228149+0000","last_became_active":"2024-03-24T10:45:00.639653+0000","last_became_peered":"2024-03-24T10:45:00.639653+0000","last_unstale":"2024-03-24T10:55:01.228149+0000","last_undegraded":"2024-03-24T10:55:01.228149+0000","last_fullsized":"2024-03-24T10:55:01.228149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:15.749451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.233473+0000","last_change":"2024-03-24T10:45:01.952340+0000","last_active":"2024-03-24T10:55:01.233473+0000","last_peered":"2024-03-24T10:55:01.233473+0000","last_clean":"2024-03-24T10:55:01.233473+0000","last_became_active":"2024-03-24T10:45:01.952074+0000","last_became_peered":"2024-03-24T10:45:01.952074+0000","last_unstale":"2024-03-24T10:55:01.233473+0000","last_undegraded":"2024-03-24T10:55:01.233473+0000","last_fullsized":"2024-03-24T10:55:01.233473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"24'12","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:31.242486+0000","last_change":"2024-03-24T10:45:00.633790+0000","last_active":"2024-03-24T10:57:31.242486+0000","last_peered":"2024-03-24T10:57:31.242486+0000","last_clean":"2024-03-24T10:57:31.242486+0000","last_became_active":"2024-03-24T10:45:00.633494+0000","last_became_peered":"2024-03-24T10:45:00.633494+0000","last_unstale":"2024-03-24T10:57:31.242486+0000","last_undegraded":"2024-03-24T10:57:31.242486+0000","last_fullsized":"2024-03-24T10:57:31.242486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:14.049099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.215687+0000","last_change":"2024-03-24T10:45:01.957958+0000","last_active":"2024-03-24T10:55:29.215687+0000","last_peered":"2024-03-24T10:55:29.215687+0000","last_clean":"2024-03-24T10:55:29.215687+0000","last_became_active":"2024-03-24T10:45:01.957676+0000","last_became_peered":"2024-03-24T10:45:01.957676+0000","last_unstale":"2024-03-24T10:55:29.215687+0000","last_undegraded":"2024-03-24T10:55:29.215687+0000","last_fullsized":"2024-03-24T10:55:29.215687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:01.208031+0000","last_change":"2024-03-24T10:45:00.629120+0000","last_active":"2024-03-24T10:57:01.208031+0000","last_peered":"2024-03-24T10:57:01.208031+0000","last_clean":"2024-03-24T10:57:01.208031+0000","last_became_active":"2024-03-24T10:45:00.628712+0000","last_became_peered":"2024-03-24T10:45:00.628712+0000","last_unstale":"2024-03-24T10:57:01.208031+0000","last_undegraded":"2024-03-24T10:57:01.208031+0000","last_fullsized":"2024-03-24T10:57:01.208031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:01.208005+0000","last_change":"2024-03-24T10:45:01.894602+0000","last_active":"2024-03-24T10:57:01.208005+0000","last_peered":"2024-03-24T10:57:01.208005+0000","last_clean":"2024-03-24T10:57:01.208005+0000","last_became_active":"2024-03-24T10:45:01.894362+0000","last_became_peered":"2024-03-24T10:45:01.894362+0000","last_unstale":"2024-03-24T10:57:01.208005+0000","last_undegraded":"2024-03-24T10:57:01.208005+0000","last_fullsized":"2024-03-24T10:57:01.208005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'13","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.215696+0000","last_change":"2024-03-24T10:45:00.639171+0000","last_active":"2024-03-24T10:55:29.215696+0000","last_peered":"2024-03-24T10:55:29.215696+0000","last_clean":"2024-03-24T10:55:29.215696+0000","last_became_active":"2024-03-24T10:45:00.638871+0000","last_became_peered":"2024-03-24T10:45:00.638871+0000","last_unstale":"2024-03-24T10:55:29.215696+0000","last_undegraded":"2024-03-24T10:55:29.215696+0000","last_fullsized":"2024-03-24T10:55:29.215696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:52:46.322196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":204,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":184,"ondisk_log_size":184,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65430,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542321,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31524,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340316,"statfs":{"total":96636764160,"available":96604483584,"internally_reserved":0,"allocated":352256,"data_stored":685052,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542321,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":299008,"data_stored":96516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542320,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":95020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542321,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":94046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542321,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":278528,"data_stored":94116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575023,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":278528,"data_stored":94986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575023,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":274432,"data_stored":93914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575023,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":344064,"data_stored":683826,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:59:07.659 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-24T10:59:07.659 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:59:07.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.796+0000 7f06e91b6640 1 Processor -- start 2024-03-24T10:59:07.797 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.796+0000 7f06e91b6640 1 -- start start 2024-03-24T10:59:07.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.797+0000 7f06e91b6640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f06e40cace0 0x7f06e41432d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:07.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.797+0000 7f06e91b6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06e415a310 0x7f06e4143810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:07.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.797+0000 7f06e91b6640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f06e406c240 0x7f06e406c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:07.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.797+0000 7f06e91b6640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f06e406f110 con 0x7f06e40cace0 2024-03-24T10:59:07.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.797+0000 7f06e91b6640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f06e415b320 con 0x7f06e406c240 2024-03-24T10:59:07.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.797+0000 7f06e91b6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f06e40cb0c0 con 0x7f06e415a310 2024-03-24T10:59:07.799 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.798+0000 7f06e2d9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f06e40cace0 0x7f06e41432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:07.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.798+0000 7f06e359e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f06e406c240 0x7f06e406c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:07.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.798+0000 7f06e2d9d640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f06e40cace0 0x7f06e41432d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44076/0 (socket says 172.21.15.5:44076) 2024-03-24T10:59:07.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.798+0000 7f06e2d9d640 1 -- 172.21.15.5:0/379406996 learned_addr learned my addr 172.21.15.5:0/379406996 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:07.800 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.798+0000 7f06e259c640 1 --2- 172.21.15.5:0/379406996 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06e415a310 0x7f06e4143810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:07.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.800+0000 7f06e2d9d640 1 -- 172.21.15.5:0/379406996 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06e415a310 msgr2=0x7f06e4143810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:07.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.800+0000 7f06e2d9d640 1 --2- 172.21.15.5:0/379406996 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06e415a310 0x7f06e4143810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.801 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.800+0000 7f06e2d9d640 1 -- 172.21.15.5:0/379406996 >> v2:172.21.15.57:3300/0 conn(0x7f06e406c240 msgr2=0x7f06e406c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:07.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.800+0000 7f06e2d9d640 1 --2- 172.21.15.5:0/379406996 >> v2:172.21.15.57:3300/0 conn(0x7f06e406c240 0x7f06e406c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.802 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.800+0000 7f06e2d9d640 1 -- 172.21.15.5:0/379406996 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e406cf70 con 0x7f06e40cace0 2024-03-24T10:59:07.804 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.803+0000 7f06e2d9d640 1 --2- 172.21.15.5:0/379406996 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06e40cace0 0x7f06e41432d0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f06e40c8840 tx=0x7f06d402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4236e6f53c0fefb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:07.809 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.808+0000 7f06e1d9b640 1 -- 172.21.15.5:0/379406996 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d4030860 con 0x7f06e40cace0 2024-03-24T10:59:07.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.808+0000 7f06e1d9b640 1 -- 172.21.15.5:0/379406996 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06d40309c0 con 0x7f06e40cace0 2024-03-24T10:59:07.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.808+0000 7f06e1d9b640 1 -- 172.21.15.5:0/379406996 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d4030d50 con 0x7f06e40cace0 2024-03-24T10:59:07.810 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.809+0000 7f06e91b6640 1 -- 172.21.15.5:0/379406996 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06e40cace0 msgr2=0x7f06e41432d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:07.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.809+0000 7f06e91b6640 1 --2- 172.21.15.5:0/379406996 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06e40cace0 0x7f06e41432d0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f06e40c8840 tx=0x7f06d402b9a0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.809+0000 7f06e91b6640 1 -- 172.21.15.5:0/379406996 shutdown_connections 2024-03-24T10:59:07.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.809+0000 7f06e91b6640 1 --2- 172.21.15.5:0/379406996 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06e415a310 0x7f06e4143810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.809+0000 7f06e91b6640 1 --2- 172.21.15.5:0/379406996 >> v2:172.21.15.57:3300/0 conn(0x7f06e406c240 0x7f06e406c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.809+0000 7f06e91b6640 1 --2- 172.21.15.5:0/379406996 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06e40cace0 0x7f06e41432d0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.809+0000 7f06e91b6640 1 -- 172.21.15.5:0/379406996 >> 172.21.15.5:0/379406996 conn(0x7f06e4071240 msgr2=0x7f06e4071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:07.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.809+0000 7f06e91b6640 1 -- 172.21.15.5:0/379406996 shutdown_connections 2024-03-24T10:59:07.811 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.809+0000 7f06e91b6640 1 -- 172.21.15.5:0/379406996 wait complete. 2024-03-24T10:59:07.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.811+0000 7f06e91b6640 1 Processor -- start 2024-03-24T10:59:07.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.812+0000 7f06e91b6640 1 -- start start 2024-03-24T10:59:07.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.813+0000 7f06e91b6640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06e406c240 0x7f06e41f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:07.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.813+0000 7f06e91b6640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06e40cace0 0x7f06e40ae000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:07.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.813+0000 7f06e91b6640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06e40ce6d0 0x7f06e41f07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:07.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.813+0000 7f06e91b6640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f06e41577c0 con 0x7f06e40ce6d0 2024-03-24T10:59:07.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.813+0000 7f06e91b6640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f06e4157aa0 con 0x7f06e40cace0 2024-03-24T10:59:07.814 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.813+0000 7f06e91b6640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f06e41effc0 con 0x7f06e406c240 2024-03-24T10:59:07.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.813+0000 7f06e359e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06e40ce6d0 0x7f06e41f07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:07.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.813+0000 7f06e2d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06e406c240 0x7f06e41f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:07.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.814+0000 7f06e259c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06e40cace0 0x7f06e40ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:07.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.814+0000 7f06e2d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06e406c240 0x7f06e41f3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:48570/0 (socket says 172.21.15.5:48570) 2024-03-24T10:59:07.815 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.814+0000 7f06e2d9d640 1 -- 172.21.15.5:0/3650774516 learned_addr learned my addr 172.21.15.5:0/3650774516 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:07.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.819+0000 7f06e2d9d640 1 -- 172.21.15.5:0/3650774516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06e40cace0 msgr2=0x7f06e40ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:07.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.819+0000 7f06e2d9d640 1 --2- 172.21.15.5:0/3650774516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06e40cace0 0x7f06e40ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.819+0000 7f06e2d9d640 1 -- 172.21.15.5:0/3650774516 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06e40ce6d0 msgr2=0x7f06e41f07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:07.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.819+0000 7f06e2d9d640 1 --2- 172.21.15.5:0/3650774516 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06e40ce6d0 0x7f06e41f07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:07.820 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.819+0000 7f06e2d9d640 1 -- 172.21.15.5:0/3650774516 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d40094d0 con 0x7f06e406c240 2024-03-24T10:59:07.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.822+0000 7f06e2d9d640 1 --2- 172.21.15.5:0/3650774516 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06e406c240 0x7f06e41f3bb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f06d4000c00 tx=0x7f06d4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:07.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.827+0000 7f06cb7fe640 1 -- 172.21.15.5:0/3650774516 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d4039c90 con 0x7f06e406c240 2024-03-24T10:59:07.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.827+0000 7f06cb7fe640 1 -- 172.21.15.5:0/3650774516 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06d4039df0 con 0x7f06e406c240 2024-03-24T10:59:07.828 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.827+0000 7f06e91b6640 1 -- 172.21.15.5:0/3650774516 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06e41f02a0 con 0x7f06e406c240 2024-03-24T10:59:07.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.827+0000 7f06cb7fe640 1 -- 172.21.15.5:0/3650774516 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d4043640 con 0x7f06e406c240 2024-03-24T10:59:07.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.827+0000 7f06e91b6640 1 -- 172.21.15.5:0/3650774516 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f06e41efc90 con 0x7f06e406c240 2024-03-24T10:59:07.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:07.828+0000 7f06e91b6640 1 -- 172.21.15.5:0/3650774516 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06e4157c10 con 0x7f06e406c240 2024-03-24T10:59:08.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.109+0000 7f06cb7fe640 1 -- 172.21.15.5:0/3650774516 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f06d40437a0 con 0x7f06e406c240 2024-03-24T10:59:08.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.109+0000 7f06cb7fe640 1 --2- 172.21.15.5:0/3650774516 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f06d006b860 0x7f06d006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:08.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.109+0000 7f06e259c640 1 --2- 172.21.15.5:0/3650774516 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f06d006b860 0x7f06d006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:08.111 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.110+0000 7f06e259c640 1 --2- 172.21.15.5:0/3650774516 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f06d006b860 0x7f06d006dd20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f06e40ce380 tx=0x7f06d8009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:08.115 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.115+0000 7f06cb7fe640 1 -- 172.21.15.5:0/3650774516 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f06d403e070 con 0x7f06e406c240 2024-03-24T10:59:08.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.134+0000 7f06cb7fe640 1 -- 172.21.15.5:0/3650774516 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f06d4078720 con 0x7f06e406c240 2024-03-24T10:59:08.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.330+0000 7f06e91b6640 1 -- 172.21.15.5:0/3650774516 --> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f06e415ad60 con 0x7f06d006b860 2024-03-24T10:59:08.337 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.336+0000 7f06cb7fe640 1 -- 172.21.15.5:0/3650774516 <== mgr.4120 v2:172.21.15.5:6832/1102439181 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251997 (secure 0 0 0) 0x7f06d8011a30 con 0x7f06d006b860 2024-03-24T10:59:08.341 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:59:08.342 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-03-24T10:59:08.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.340+0000 7f06e91b6640 1 -- 172.21.15.5:0/3650774516 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f06d006b860 msgr2=0x7f06d006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:08.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.340+0000 7f06e91b6640 1 --2- 172.21.15.5:0/3650774516 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f06d006b860 0x7f06d006dd20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f06e40ce380 tx=0x7f06d8009190 comp rx=0 tx=0).stop 2024-03-24T10:59:08.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.340+0000 7f06e91b6640 1 -- 172.21.15.5:0/3650774516 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06e406c240 msgr2=0x7f06e41f3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:08.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.340+0000 7f06e91b6640 1 --2- 172.21.15.5:0/3650774516 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06e406c240 0x7f06e41f3bb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f06d4000c00 tx=0x7f06d4004730 comp rx=0 tx=0).stop 2024-03-24T10:59:08.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.340+0000 7f06e91b6640 1 -- 172.21.15.5:0/3650774516 shutdown_connections 2024-03-24T10:59:08.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.340+0000 7f06e91b6640 1 --2- 172.21.15.5:0/3650774516 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f06d006b860 0x7f06d006dd20 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:08.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.340+0000 7f06e91b6640 1 --2- 172.21.15.5:0/3650774516 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f06e40ce6d0 0x7f06e41f07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:08.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.340+0000 7f06e91b6640 1 --2- 172.21.15.5:0/3650774516 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f06e40cace0 0x7f06e40ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:08.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.340+0000 7f06e91b6640 1 --2- 172.21.15.5:0/3650774516 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f06e406c240 0x7f06e41f3bb0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:08.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.340+0000 7f06e91b6640 1 -- 172.21.15.5:0/3650774516 >> 172.21.15.5:0/3650774516 conn(0x7f06e4071240 msgr2=0x7f06e40b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:08.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.340+0000 7f06e91b6640 1 -- 172.21.15.5:0/3650774516 shutdown_connections 2024-03-24T10:59:08.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.340+0000 7f06e91b6640 1 -- 172.21.15.5:0/3650774516 wait complete. 2024-03-24T10:59:08.356 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-03-24T10:59:07.545200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":363,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":301,"ondisk_log_size":301,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239296,"kb_used_data":2336,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754735424,"statfs":{"total":773094113280,"available":772849074176,"internally_reserved":0,"allocated":2392064,"data_stored":1937476,"data_compressed":65430,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":242535986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002318"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.230562+0000","last_change":"2024-03-24T10:45:01.894006+0000","last_active":"2024-03-24T10:55:26.230562+0000","last_peered":"2024-03-24T10:55:26.230562+0000","last_clean":"2024-03-24T10:55:26.230562+0000","last_became_active":"2024-03-24T10:45:01.893734+0000","last_became_peered":"2024-03-24T10:45:01.893734+0000","last_unstale":"2024-03-24T10:55:26.230562+0000","last_undegraded":"2024-03-24T10:55:26.230562+0000","last_fullsized":"2024-03-24T10:55:26.230562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:05:51.579388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.323823+0000","last_change":"2024-03-24T10:45:01.931707+0000","last_active":"2024-03-24T10:55:26.323823+0000","last_peered":"2024-03-24T10:55:26.323823+0000","last_clean":"2024-03-24T10:55:26.323823+0000","last_became_active":"2024-03-24T10:45:01.931449+0000","last_became_peered":"2024-03-24T10:45:01.931449+0000","last_unstale":"2024-03-24T10:55:26.323823+0000","last_undegraded":"2024-03-24T10:55:26.323823+0000","last_fullsized":"2024-03-24T10:55:26.323823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.215675+0000","last_change":"2024-03-24T10:45:01.895929+0000","last_active":"2024-03-24T10:55:29.215675+0000","last_peered":"2024-03-24T10:55:29.215675+0000","last_clean":"2024-03-24T10:55:29.215675+0000","last_became_active":"2024-03-24T10:45:01.895654+0000","last_became_peered":"2024-03-24T10:45:01.895654+0000","last_unstale":"2024-03-24T10:55:29.215675+0000","last_undegraded":"2024-03-24T10:55:29.215675+0000","last_fullsized":"2024-03-24T10:55:29.215675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:01.243946+0000","last_change":"2024-03-24T10:45:01.952468+0000","last_active":"2024-03-24T10:57:01.243946+0000","last_peered":"2024-03-24T10:57:01.243946+0000","last_clean":"2024-03-24T10:57:01.243946+0000","last_became_active":"2024-03-24T10:45:01.952219+0000","last_became_peered":"2024-03-24T10:45:01.952219+0000","last_unstale":"2024-03-24T10:57:01.243946+0000","last_undegraded":"2024-03-24T10:57:01.243946+0000","last_fullsized":"2024-03-24T10:57:01.243946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:27:12.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.228132+0000","last_change":"2024-03-24T10:45:01.895885+0000","last_active":"2024-03-24T10:55:01.228132+0000","last_peered":"2024-03-24T10:55:01.228132+0000","last_clean":"2024-03-24T10:55:01.228132+0000","last_became_active":"2024-03-24T10:45:01.895622+0000","last_became_peered":"2024-03-24T10:45:01.895622+0000","last_unstale":"2024-03-24T10:55:01.228132+0000","last_undegraded":"2024-03-24T10:55:01.228132+0000","last_fullsized":"2024-03-24T10:55:01.228132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:16.523177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:41.320408+0000","last_change":"2024-03-24T10:45:40.169714+0000","last_active":"2024-03-24T10:55:41.320408+0000","last_peered":"2024-03-24T10:55:41.320408+0000","last_clean":"2024-03-24T10:55:41.320408+0000","last_became_active":"2024-03-24T10:45:40.169447+0000","last_became_peered":"2024-03-24T10:45:40.169447+0000","last_unstale":"2024-03-24T10:55:41.320408+0000","last_undegraded":"2024-03-24T10:55:41.320408+0000","last_fullsized":"2024-03-24T10:55:41.320408+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:27:12.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.228112+0000","last_change":"2024-03-24T10:45:01.892745+0000","last_active":"2024-03-24T10:55:01.228112+0000","last_peered":"2024-03-24T10:55:01.228112+0000","last_clean":"2024-03-24T10:55:01.228112+0000","last_became_active":"2024-03-24T10:45:01.892452+0000","last_became_peered":"2024-03-24T10:45:01.892452+0000","last_unstale":"2024-03-24T10:55:01.228112+0000","last_undegraded":"2024-03-24T10:55:01.228112+0000","last_fullsized":"2024-03-24T10:55:01.228112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.198373+0000","last_change":"2024-03-24T10:45:01.953432+0000","last_active":"2024-03-24T10:55:01.198373+0000","last_peered":"2024-03-24T10:55:01.198373+0000","last_clean":"2024-03-24T10:55:01.198373+0000","last_became_active":"2024-03-24T10:45:01.953149+0000","last_became_peered":"2024-03-24T10:45:01.953149+0000","last_unstale":"2024-03-24T10:55:01.198373+0000","last_undegraded":"2024-03-24T10:55:01.198373+0000","last_fullsized":"2024-03-24T10:55:01.198373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.226282+0000","last_change":"2024-03-24T10:45:01.956488+0000","last_active":"2024-03-24T10:54:41.226282+0000","last_peered":"2024-03-24T10:54:41.226282+0000","last_clean":"2024-03-24T10:54:41.226282+0000","last_became_active":"2024-03-24T10:45:01.956207+0000","last_became_peered":"2024-03-24T10:45:01.956207+0000","last_unstale":"2024-03-24T10:54:41.226282+0000","last_undegraded":"2024-03-24T10:54:41.226282+0000","last_fullsized":"2024-03-24T10:54:41.226282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:23:26.523809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:44.289850+0000","last_change":"2024-03-24T10:45:01.956484+0000","last_active":"2024-03-24T10:54:44.289850+0000","last_peered":"2024-03-24T10:54:44.289850+0000","last_clean":"2024-03-24T10:54:44.289850+0000","last_became_active":"2024-03-24T10:45:01.956209+0000","last_became_peered":"2024-03-24T10:45:01.956209+0000","last_unstale":"2024-03-24T10:54:44.289850+0000","last_undegraded":"2024-03-24T10:54:44.289850+0000","last_fullsized":"2024-03-24T10:54:44.289850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:44.289823+0000","last_change":"2024-03-24T10:45:01.957103+0000","last_active":"2024-03-24T10:54:44.289823+0000","last_peered":"2024-03-24T10:54:44.289823+0000","last_clean":"2024-03-24T10:54:44.289823+0000","last_became_active":"2024-03-24T10:45:01.956849+0000","last_became_peered":"2024-03-24T10:45:01.956849+0000","last_unstale":"2024-03-24T10:54:44.289823+0000","last_undegraded":"2024-03-24T10:54:44.289823+0000","last_fullsized":"2024-03-24T10:54:44.289823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:44.262492+0000","last_change":"2024-03-24T10:45:01.896701+0000","last_active":"2024-03-24T10:54:44.262492+0000","last_peered":"2024-03-24T10:54:44.262492+0000","last_clean":"2024-03-24T10:54:44.262492+0000","last_became_active":"2024-03-24T10:45:01.896400+0000","last_became_peered":"2024-03-24T10:45:01.896400+0000","last_unstale":"2024-03-24T10:54:44.262492+0000","last_undegraded":"2024-03-24T10:54:44.262492+0000","last_fullsized":"2024-03-24T10:54:44.262492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:44.380352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.196593+0000","last_change":"2024-03-24T10:45:01.894954+0000","last_active":"2024-03-24T10:54:41.196593+0000","last_peered":"2024-03-24T10:54:41.196593+0000","last_clean":"2024-03-24T10:54:41.196593+0000","last_became_active":"2024-03-24T10:45:01.894632+0000","last_became_peered":"2024-03-24T10:45:01.894632+0000","last_unstale":"2024-03-24T10:54:41.196593+0000","last_undegraded":"2024-03-24T10:54:41.196593+0000","last_fullsized":"2024-03-24T10:54:41.196593+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.221985+0000","last_change":"2024-03-24T10:45:01.929588+0000","last_active":"2024-03-24T10:56:44.221985+0000","last_peered":"2024-03-24T10:56:44.221985+0000","last_clean":"2024-03-24T10:56:44.221985+0000","last_became_active":"2024-03-24T10:45:01.928460+0000","last_became_peered":"2024-03-24T10:45:01.928460+0000","last_unstale":"2024-03-24T10:56:44.221985+0000","last_undegraded":"2024-03-24T10:56:44.221985+0000","last_fullsized":"2024-03-24T10:56:44.221985+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:47:48.896914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.329359+0000","last_change":"2024-03-24T10:45:01.928721+0000","last_active":"2024-03-24T10:56:41.329359+0000","last_peered":"2024-03-24T10:56:41.329359+0000","last_clean":"2024-03-24T10:56:41.329359+0000","last_became_active":"2024-03-24T10:45:01.928467+0000","last_became_peered":"2024-03-24T10:45:01.928467+0000","last_unstale":"2024-03-24T10:56:41.329359+0000","last_undegraded":"2024-03-24T10:56:41.329359+0000","last_fullsized":"2024-03-24T10:56:41.329359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.242233+0000","last_change":"2024-03-24T10:45:01.956030+0000","last_active":"2024-03-24T10:56:41.242233+0000","last_peered":"2024-03-24T10:56:41.242233+0000","last_clean":"2024-03-24T10:56:41.242233+0000","last_became_active":"2024-03-24T10:45:01.955768+0000","last_became_peered":"2024-03-24T10:45:01.955768+0000","last_unstale":"2024-03-24T10:56:41.242233+0000","last_undegraded":"2024-03-24T10:56:41.242233+0000","last_fullsized":"2024-03-24T10:56:41.242233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.242276+0000","last_change":"2024-03-24T10:45:01.956181+0000","last_active":"2024-03-24T10:56:41.242276+0000","last_peered":"2024-03-24T10:56:41.242276+0000","last_clean":"2024-03-24T10:56:41.242276+0000","last_became_active":"2024-03-24T10:45:01.955890+0000","last_became_peered":"2024-03-24T10:45:01.955890+0000","last_unstale":"2024-03-24T10:56:41.242276+0000","last_undegraded":"2024-03-24T10:56:41.242276+0000","last_fullsized":"2024-03-24T10:56:41.242276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:15:11.107915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.325044+0000","last_change":"2024-03-24T10:45:01.896663+0000","last_active":"2024-03-24T10:56:41.325044+0000","last_peered":"2024-03-24T10:56:41.325044+0000","last_clean":"2024-03-24T10:56:41.325044+0000","last_became_active":"2024-03-24T10:45:01.896391+0000","last_became_peered":"2024-03-24T10:45:01.896391+0000","last_unstale":"2024-03-24T10:56:41.325044+0000","last_undegraded":"2024-03-24T10:56:41.325044+0000","last_fullsized":"2024-03-24T10:56:41.325044+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:30.942758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.235423+0000","last_change":"2024-03-24T10:45:01.895951+0000","last_active":"2024-03-24T10:56:16.235423+0000","last_peered":"2024-03-24T10:56:16.235423+0000","last_clean":"2024-03-24T10:56:16.235423+0000","last_became_active":"2024-03-24T10:45:01.895668+0000","last_became_peered":"2024-03-24T10:45:01.895668+0000","last_unstale":"2024-03-24T10:56:16.235423+0000","last_undegraded":"2024-03-24T10:56:16.235423+0000","last_fullsized":"2024-03-24T10:56:16.235423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:56:43.462458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:19.297955+0000","last_change":"2024-03-24T10:45:01.898126+0000","last_active":"2024-03-24T10:56:19.297955+0000","last_peered":"2024-03-24T10:56:19.297955+0000","last_clean":"2024-03-24T10:56:19.297955+0000","last_became_active":"2024-03-24T10:45:01.897878+0000","last_became_peered":"2024-03-24T10:45:01.897878+0000","last_unstale":"2024-03-24T10:56:19.297955+0000","last_undegraded":"2024-03-24T10:56:19.297955+0000","last_fullsized":"2024-03-24T10:56:19.297955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:19.220025+0000","last_change":"2024-03-24T10:45:01.896109+0000","last_active":"2024-03-24T10:56:19.220025+0000","last_peered":"2024-03-24T10:56:19.220025+0000","last_clean":"2024-03-24T10:56:19.220025+0000","last_became_active":"2024-03-24T10:45:01.895696+0000","last_became_peered":"2024-03-24T10:45:01.895696+0000","last_unstale":"2024-03-24T10:56:19.220025+0000","last_undegraded":"2024-03-24T10:56:19.220025+0000","last_fullsized":"2024-03-24T10:56:19.220025+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.240130+0000","last_change":"2024-03-24T10:45:01.958119+0000","last_active":"2024-03-24T10:56:16.240130+0000","last_peered":"2024-03-24T10:56:16.240130+0000","last_clean":"2024-03-24T10:56:16.240130+0000","last_became_active":"2024-03-24T10:45:01.957832+0000","last_became_peered":"2024-03-24T10:45:01.957832+0000","last_unstale":"2024-03-24T10:56:16.240130+0000","last_undegraded":"2024-03-24T10:56:16.240130+0000","last_fullsized":"2024-03-24T10:56:16.240130+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:12.476490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.204495+0000","last_change":"2024-03-24T10:45:01.842442+0000","last_active":"2024-03-24T10:56:16.204495+0000","last_peered":"2024-03-24T10:56:16.204495+0000","last_clean":"2024-03-24T10:56:16.204495+0000","last_became_active":"2024-03-24T10:45:01.842199+0000","last_became_peered":"2024-03-24T10:45:01.842199+0000","last_unstale":"2024-03-24T10:56:16.204495+0000","last_undegraded":"2024-03-24T10:56:16.204495+0000","last_fullsized":"2024-03-24T10:56:16.204495+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.237949+0000","last_change":"2024-03-24T10:45:01.952378+0000","last_active":"2024-03-24T10:55:51.237949+0000","last_peered":"2024-03-24T10:55:51.237949+0000","last_clean":"2024-03-24T10:55:51.237949+0000","last_became_active":"2024-03-24T10:45:01.952123+0000","last_became_peered":"2024-03-24T10:45:01.952123+0000","last_unstale":"2024-03-24T10:55:51.237949+0000","last_undegraded":"2024-03-24T10:55:51.237949+0000","last_fullsized":"2024-03-24T10:55:51.237949+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.202466+0000","last_change":"2024-03-24T10:45:01.842299+0000","last_active":"2024-03-24T10:55:51.202466+0000","last_peered":"2024-03-24T10:55:51.202466+0000","last_clean":"2024-03-24T10:55:51.202466+0000","last_became_active":"2024-03-24T10:45:01.842097+0000","last_became_peered":"2024-03-24T10:45:01.842097+0000","last_unstale":"2024-03-24T10:55:51.202466+0000","last_undegraded":"2024-03-24T10:55:51.202466+0000","last_fullsized":"2024-03-24T10:55:51.202466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:16.341360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:54.295744+0000","last_change":"2024-03-24T10:45:01.893579+0000","last_active":"2024-03-24T10:55:54.295744+0000","last_peered":"2024-03-24T10:55:54.295744+0000","last_clean":"2024-03-24T10:55:54.295744+0000","last_became_active":"2024-03-24T10:45:01.893273+0000","last_became_peered":"2024-03-24T10:45:01.893273+0000","last_unstale":"2024-03-24T10:55:54.295744+0000","last_undegraded":"2024-03-24T10:55:54.295744+0000","last_fullsized":"2024-03-24T10:55:54.295744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.325622+0000","last_change":"2024-03-24T10:45:01.930032+0000","last_active":"2024-03-24T10:55:51.325622+0000","last_peered":"2024-03-24T10:55:51.325622+0000","last_clean":"2024-03-24T10:55:51.325622+0000","last_became_active":"2024-03-24T10:45:01.929770+0000","last_became_peered":"2024-03-24T10:45:01.929770+0000","last_unstale":"2024-03-24T10:55:51.325622+0000","last_undegraded":"2024-03-24T10:55:51.325622+0000","last_fullsized":"2024-03-24T10:55:51.325622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.202455+0000","last_change":"2024-03-24T10:45:01.842460+0000","last_active":"2024-03-24T10:55:51.202455+0000","last_peered":"2024-03-24T10:55:51.202455+0000","last_clean":"2024-03-24T10:55:51.202455+0000","last_became_active":"2024-03-24T10:45:01.842165+0000","last_became_peered":"2024-03-24T10:45:01.842165+0000","last_unstale":"2024-03-24T10:55:51.202455+0000","last_undegraded":"2024-03-24T10:55:51.202455+0000","last_fullsized":"2024-03-24T10:55:51.202455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.215656+0000","last_change":"2024-03-24T10:45:01.895868+0000","last_active":"2024-03-24T10:55:29.215656+0000","last_peered":"2024-03-24T10:55:29.215656+0000","last_clean":"2024-03-24T10:55:29.215656+0000","last_became_active":"2024-03-24T10:45:01.895290+0000","last_became_peered":"2024-03-24T10:45:01.895290+0000","last_unstale":"2024-03-24T10:55:29.215656+0000","last_undegraded":"2024-03-24T10:55:29.215656+0000","last_fullsized":"2024-03-24T10:55:29.215656+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.235695+0000","last_change":"2024-03-24T10:45:01.957724+0000","last_active":"2024-03-24T10:55:26.235695+0000","last_peered":"2024-03-24T10:55:26.235695+0000","last_clean":"2024-03-24T10:55:26.235695+0000","last_became_active":"2024-03-24T10:45:01.957429+0000","last_became_peered":"2024-03-24T10:45:01.957429+0000","last_unstale":"2024-03-24T10:55:26.235695+0000","last_undegraded":"2024-03-24T10:55:26.235695+0000","last_fullsized":"2024-03-24T10:55:26.235695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:22.089070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.319191+0000","last_change":"2024-03-24T10:45:01.952159+0000","last_active":"2024-03-24T10:55:26.319191+0000","last_peered":"2024-03-24T10:55:26.319191+0000","last_clean":"2024-03-24T10:55:26.319191+0000","last_became_active":"2024-03-24T10:45:01.951961+0000","last_became_peered":"2024-03-24T10:45:01.951961+0000","last_unstale":"2024-03-24T10:55:26.319191+0000","last_undegraded":"2024-03-24T10:55:26.319191+0000","last_fullsized":"2024-03-24T10:55:26.319191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:44.380352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.293413+0000","last_change":"2024-03-24T10:45:01.893452+0000","last_active":"2024-03-24T10:55:29.293413+0000","last_peered":"2024-03-24T10:55:29.293413+0000","last_clean":"2024-03-24T10:55:29.293413+0000","last_became_active":"2024-03-24T10:45:01.893139+0000","last_became_peered":"2024-03-24T10:45:01.893139+0000","last_unstale":"2024-03-24T10:55:29.293413+0000","last_undegraded":"2024-03-24T10:55:29.293413+0000","last_fullsized":"2024-03-24T10:55:29.293413+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:04.264076+0000","last_change":"2024-03-24T10:45:01.928827+0000","last_active":"2024-03-24T10:55:04.264076+0000","last_peered":"2024-03-24T10:55:04.264076+0000","last_clean":"2024-03-24T10:55:04.264076+0000","last_became_active":"2024-03-24T10:45:01.928521+0000","last_became_peered":"2024-03-24T10:45:01.928521+0000","last_unstale":"2024-03-24T10:55:04.264076+0000","last_undegraded":"2024-03-24T10:55:04.264076+0000","last_fullsized":"2024-03-24T10:55:04.264076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:59.643246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"24'7","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.198403+0000","last_change":"2024-03-24T10:45:00.637993+0000","last_active":"2024-03-24T10:55:01.198403+0000","last_peered":"2024-03-24T10:55:01.198403+0000","last_clean":"2024-03-24T10:55:01.198403+0000","last_became_active":"2024-03-24T10:45:00.637803+0000","last_became_peered":"2024-03-24T10:45:00.637803+0000","last_unstale":"2024-03-24T10:55:01.198403+0000","last_undegraded":"2024-03-24T10:55:01.198403+0000","last_fullsized":"2024-03-24T10:55:01.198403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:28:28.854541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:04.213682+0000","last_change":"2024-03-24T10:45:01.897686+0000","last_active":"2024-03-24T10:55:04.213682+0000","last_peered":"2024-03-24T10:55:04.213682+0000","last_clean":"2024-03-24T10:55:04.213682+0000","last_became_active":"2024-03-24T10:45:01.897320+0000","last_became_peered":"2024-03-24T10:45:01.897320+0000","last_unstale":"2024-03-24T10:55:04.213682+0000","last_undegraded":"2024-03-24T10:55:04.213682+0000","last_fullsized":"2024-03-24T10:55:04.213682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:04.291481+0000","last_change":"2024-03-24T10:45:00.638329+0000","last_active":"2024-03-24T10:55:04.291481+0000","last_peered":"2024-03-24T10:55:04.291481+0000","last_clean":"2024-03-24T10:55:04.291481+0000","last_became_active":"2024-03-24T10:45:00.638006+0000","last_became_peered":"2024-03-24T10:45:00.638006+0000","last_unstale":"2024-03-24T10:55:04.291481+0000","last_undegraded":"2024-03-24T10:55:04.291481+0000","last_fullsized":"2024-03-24T10:55:04.291481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.196573+0000","last_change":"2024-03-24T10:45:00.633703+0000","last_active":"2024-03-24T10:54:41.196573+0000","last_peered":"2024-03-24T10:54:41.196573+0000","last_clean":"2024-03-24T10:54:41.196573+0000","last_became_active":"2024-03-24T10:45:00.633427+0000","last_became_peered":"2024-03-24T10:45:00.633427+0000","last_unstale":"2024-03-24T10:54:41.196573+0000","last_undegraded":"2024-03-24T10:54:41.196573+0000","last_fullsized":"2024-03-24T10:54:41.196573+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:30.350191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.196539+0000","last_change":"2024-03-24T10:45:01.893902+0000","last_active":"2024-03-24T10:54:41.196539+0000","last_peered":"2024-03-24T10:54:41.196539+0000","last_clean":"2024-03-24T10:54:41.196539+0000","last_became_active":"2024-03-24T10:45:01.893617+0000","last_became_peered":"2024-03-24T10:45:01.893617+0000","last_unstale":"2024-03-24T10:54:41.196539+0000","last_undegraded":"2024-03-24T10:54:41.196539+0000","last_fullsized":"2024-03-24T10:54:41.196539+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"24'4","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.231681+0000","last_change":"2024-03-24T10:45:00.638778+0000","last_active":"2024-03-24T10:54:41.231681+0000","last_peered":"2024-03-24T10:54:41.231681+0000","last_clean":"2024-03-24T10:54:41.231681+0000","last_became_active":"2024-03-24T10:45:00.638089+0000","last_became_peered":"2024-03-24T10:45:00.638089+0000","last_unstale":"2024-03-24T10:54:41.231681+0000","last_undegraded":"2024-03-24T10:54:41.231681+0000","last_fullsized":"2024-03-24T10:54:41.231681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.320204+0000","last_change":"2024-03-24T10:45:39.901726+0000","last_active":"2024-03-24T10:54:41.320204+0000","last_peered":"2024-03-24T10:54:41.320204+0000","last_clean":"2024-03-24T10:54:41.320204+0000","last_became_active":"2024-03-24T10:45:39.901416+0000","last_became_peered":"2024-03-24T10:45:39.901416+0000","last_unstale":"2024-03-24T10:54:41.320204+0000","last_undegraded":"2024-03-24T10:54:41.320204+0000","last_fullsized":"2024-03-24T10:54:41.320204+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:23:26.523809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'4","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:44.289869+0000","last_change":"2024-03-24T10:45:00.639822+0000","last_active":"2024-03-24T10:54:44.289869+0000","last_peered":"2024-03-24T10:54:44.289869+0000","last_clean":"2024-03-24T10:54:44.289869+0000","last_became_active":"2024-03-24T10:45:00.639600+0000","last_became_peered":"2024-03-24T10:45:00.639600+0000","last_unstale":"2024-03-24T10:54:44.289869+0000","last_undegraded":"2024-03-24T10:54:44.289869+0000","last_fullsized":"2024-03-24T10:54:44.289869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.231724+0000","last_change":"2024-03-24T10:45:01.954015+0000","last_active":"2024-03-24T10:54:41.231724+0000","last_peered":"2024-03-24T10:54:41.231724+0000","last_clean":"2024-03-24T10:54:41.231724+0000","last_became_active":"2024-03-24T10:45:01.953767+0000","last_became_peered":"2024-03-24T10:45:01.953767+0000","last_unstale":"2024-03-24T10:54:41.231724+0000","last_undegraded":"2024-03-24T10:54:41.231724+0000","last_fullsized":"2024-03-24T10:54:41.231724+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.231756+0000","last_change":"2024-03-24T10:45:00.636553+0000","last_active":"2024-03-24T10:54:41.231756+0000","last_peered":"2024-03-24T10:54:41.231756+0000","last_clean":"2024-03-24T10:54:41.231756+0000","last_became_active":"2024-03-24T10:45:00.636282+0000","last_became_peered":"2024-03-24T10:45:00.636282+0000","last_unstale":"2024-03-24T10:54:41.231756+0000","last_undegraded":"2024-03-24T10:54:41.231756+0000","last_fullsized":"2024-03-24T10:54:41.231756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:44.212223+0000","last_change":"2024-03-24T10:45:01.957905+0000","last_active":"2024-03-24T10:54:44.212223+0000","last_peered":"2024-03-24T10:54:44.212223+0000","last_clean":"2024-03-24T10:54:44.212223+0000","last_became_active":"2024-03-24T10:45:01.957676+0000","last_became_peered":"2024-03-24T10:45:01.957676+0000","last_unstale":"2024-03-24T10:54:44.212223+0000","last_undegraded":"2024-03-24T10:54:44.212223+0000","last_fullsized":"2024-03-24T10:54:44.212223+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.320240+0000","last_change":"2024-03-24T10:45:00.635197+0000","last_active":"2024-03-24T10:54:41.320240+0000","last_peered":"2024-03-24T10:54:41.320240+0000","last_clean":"2024-03-24T10:54:41.320240+0000","last_became_active":"2024-03-24T10:45:00.634989+0000","last_became_peered":"2024-03-24T10:45:00.634989+0000","last_unstale":"2024-03-24T10:54:41.320240+0000","last_undegraded":"2024-03-24T10:54:41.320240+0000","last_fullsized":"2024-03-24T10:54:41.320240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.272421+0000","last_change":"2024-03-24T10:45:01.853785+0000","last_active":"2024-03-24T10:56:44.272421+0000","last_peered":"2024-03-24T10:56:44.272421+0000","last_clean":"2024-03-24T10:56:44.272421+0000","last_became_active":"2024-03-24T10:45:01.853504+0000","last_became_peered":"2024-03-24T10:45:01.853504+0000","last_unstale":"2024-03-24T10:56:44.272421+0000","last_undegraded":"2024-03-24T10:56:44.272421+0000","last_fullsized":"2024-03-24T10:56:44.272421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:30.942758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":36,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.300179+0000","last_change":"2024-03-24T10:45:00.632381+0000","last_active":"2024-03-24T10:56:44.300179+0000","last_peered":"2024-03-24T10:56:44.300179+0000","last_clean":"2024-03-24T10:56:44.300179+0000","last_became_active":"2024-03-24T10:45:00.632108+0000","last_became_peered":"2024-03-24T10:45:00.632108+0000","last_unstale":"2024-03-24T10:56:44.300179+0000","last_undegraded":"2024-03-24T10:56:44.300179+0000","last_fullsized":"2024-03-24T10:56:44.300179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:06.142451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.226243+0000","last_change":"2024-03-24T10:45:01.853591+0000","last_active":"2024-03-24T10:54:41.226243+0000","last_peered":"2024-03-24T10:54:41.226243+0000","last_clean":"2024-03-24T10:54:41.226243+0000","last_became_active":"2024-03-24T10:45:01.853279+0000","last_became_peered":"2024-03-24T10:45:01.853279+0000","last_unstale":"2024-03-24T10:54:41.226243+0000","last_undegraded":"2024-03-24T10:54:41.226243+0000","last_fullsized":"2024-03-24T10:54:41.226243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:07:24.149961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'9","reported_seq":35,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.222018+0000","last_change":"2024-03-24T10:45:00.634192+0000","last_active":"2024-03-24T10:56:44.222018+0000","last_peered":"2024-03-24T10:56:44.222018+0000","last_clean":"2024-03-24T10:56:44.222018+0000","last_became_active":"2024-03-24T10:45:00.633893+0000","last_became_peered":"2024-03-24T10:45:00.633893+0000","last_unstale":"2024-03-24T10:56:44.222018+0000","last_undegraded":"2024-03-24T10:56:44.222018+0000","last_fullsized":"2024-03-24T10:56:44.222018+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:03:30.350191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.222005+0000","last_change":"2024-03-24T10:45:01.893426+0000","last_active":"2024-03-24T10:56:44.222005+0000","last_peered":"2024-03-24T10:56:44.222005+0000","last_clean":"2024-03-24T10:56:44.222005+0000","last_became_active":"2024-03-24T10:45:01.893178+0000","last_became_peered":"2024-03-24T10:45:01.893178+0000","last_unstale":"2024-03-24T10:56:44.222005+0000","last_undegraded":"2024-03-24T10:56:44.222005+0000","last_fullsized":"2024-03-24T10:56:44.222005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:13.887947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"24'6","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.242255+0000","last_change":"2024-03-24T10:45:00.638214+0000","last_active":"2024-03-24T10:56:41.242255+0000","last_peered":"2024-03-24T10:56:41.242255+0000","last_clean":"2024-03-24T10:56:41.242255+0000","last_became_active":"2024-03-24T10:45:00.637973+0000","last_became_peered":"2024-03-24T10:45:00.637973+0000","last_unstale":"2024-03-24T10:56:41.242255+0000","last_undegraded":"2024-03-24T10:56:41.242255+0000","last_fullsized":"2024-03-24T10:56:41.242255+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:15.749451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.237815+0000","last_change":"2024-03-24T10:45:01.856756+0000","last_active":"2024-03-24T10:56:41.237815+0000","last_peered":"2024-03-24T10:56:41.237815+0000","last_clean":"2024-03-24T10:56:41.237815+0000","last_became_active":"2024-03-24T10:45:01.856482+0000","last_became_peered":"2024-03-24T10:45:01.856482+0000","last_unstale":"2024-03-24T10:56:41.237815+0000","last_undegraded":"2024-03-24T10:56:41.237815+0000","last_fullsized":"2024-03-24T10:56:41.237815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:12.476490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.300155+0000","last_change":"2024-03-24T10:45:00.636481+0000","last_active":"2024-03-24T10:56:44.300155+0000","last_peered":"2024-03-24T10:56:44.300155+0000","last_clean":"2024-03-24T10:56:44.300155+0000","last_became_active":"2024-03-24T10:45:00.636206+0000","last_became_peered":"2024-03-24T10:45:00.636206+0000","last_unstale":"2024-03-24T10:56:44.300155+0000","last_undegraded":"2024-03-24T10:56:44.300155+0000","last_fullsized":"2024-03-24T10:56:44.300155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.300128+0000","last_change":"2024-03-24T10:45:01.897279+0000","last_active":"2024-03-24T10:56:44.300128+0000","last_peered":"2024-03-24T10:56:44.300128+0000","last_clean":"2024-03-24T10:56:44.300128+0000","last_became_active":"2024-03-24T10:45:01.896736+0000","last_became_peered":"2024-03-24T10:45:01.896736+0000","last_unstale":"2024-03-24T10:56:44.300128+0000","last_undegraded":"2024-03-24T10:56:44.300128+0000","last_fullsized":"2024-03-24T10:56:44.300128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.325004+0000","last_change":"2024-03-24T10:45:00.564066+0000","last_active":"2024-03-24T10:56:41.325004+0000","last_peered":"2024-03-24T10:56:41.325004+0000","last_clean":"2024-03-24T10:56:41.325004+0000","last_became_active":"2024-03-24T10:45:00.563812+0000","last_became_peered":"2024-03-24T10:45:00.563812+0000","last_unstale":"2024-03-24T10:56:41.325004+0000","last_undegraded":"2024-03-24T10:56:41.325004+0000","last_fullsized":"2024-03-24T10:56:41.325004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:28:28.854541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.237784+0000","last_change":"2024-03-24T10:45:01.854429+0000","last_active":"2024-03-24T10:56:41.237784+0000","last_peered":"2024-03-24T10:56:41.237784+0000","last_clean":"2024-03-24T10:56:41.237784+0000","last_became_active":"2024-03-24T10:45:01.854179+0000","last_became_peered":"2024-03-24T10:45:01.854179+0000","last_unstale":"2024-03-24T10:56:41.237784+0000","last_undegraded":"2024-03-24T10:56:41.237784+0000","last_fullsized":"2024-03-24T10:56:41.237784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:19.270242+0000","last_change":"2024-03-24T10:45:00.634223+0000","last_active":"2024-03-24T10:56:19.270242+0000","last_peered":"2024-03-24T10:56:19.270242+0000","last_clean":"2024-03-24T10:56:19.270242+0000","last_became_active":"2024-03-24T10:45:00.633355+0000","last_became_peered":"2024-03-24T10:45:00.633355+0000","last_unstale":"2024-03-24T10:56:19.270242+0000","last_undegraded":"2024-03-24T10:56:19.270242+0000","last_fullsized":"2024-03-24T10:56:19.270242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:04.930115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.320259+0000","last_change":"2024-03-24T10:45:40.518338+0000","last_active":"2024-03-24T10:54:41.320259+0000","last_peered":"2024-03-24T10:54:41.320259+0000","last_clean":"2024-03-24T10:54:41.320259+0000","last_became_active":"2024-03-24T10:45:40.518035+0000","last_became_peered":"2024-03-24T10:45:40.518035+0000","last_unstale":"2024-03-24T10:54:41.320259+0000","last_undegraded":"2024-03-24T10:54:41.320259+0000","last_fullsized":"2024-03-24T10:54:41.320259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:29:31.508911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.323256+0000","last_change":"2024-03-24T10:45:00.626484+0000","last_active":"2024-03-24T10:56:16.323256+0000","last_peered":"2024-03-24T10:56:16.323256+0000","last_clean":"2024-03-24T10:56:16.323256+0000","last_became_active":"2024-03-24T10:45:00.626195+0000","last_became_peered":"2024-03-24T10:45:00.626195+0000","last_unstale":"2024-03-24T10:56:16.323256+0000","last_undegraded":"2024-03-24T10:56:16.323256+0000","last_fullsized":"2024-03-24T10:56:16.323256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.235444+0000","last_change":"2024-03-24T10:45:01.856995+0000","last_active":"2024-03-24T10:56:16.235444+0000","last_peered":"2024-03-24T10:56:16.235444+0000","last_clean":"2024-03-24T10:56:16.235444+0000","last_became_active":"2024-03-24T10:45:01.856531+0000","last_became_peered":"2024-03-24T10:45:01.856531+0000","last_unstale":"2024-03-24T10:56:16.235444+0000","last_undegraded":"2024-03-24T10:56:16.235444+0000","last_fullsized":"2024-03-24T10:56:16.235444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:29:31.508911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.327524+0000","last_change":"2024-03-24T10:45:00.628438+0000","last_active":"2024-03-24T10:56:16.327524+0000","last_peered":"2024-03-24T10:56:16.327524+0000","last_clean":"2024-03-24T10:56:16.327524+0000","last_became_active":"2024-03-24T10:45:00.628158+0000","last_became_peered":"2024-03-24T10:45:00.628158+0000","last_unstale":"2024-03-24T10:56:16.327524+0000","last_undegraded":"2024-03-24T10:56:16.327524+0000","last_fullsized":"2024-03-24T10:56:16.327524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:29.826202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":17,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.315968+0000","last_change":"2024-03-24T10:45:39.901453+0000","last_active":"2024-03-24T10:54:41.315968+0000","last_peered":"2024-03-24T10:54:41.315968+0000","last_clean":"2024-03-24T10:54:41.315968+0000","last_became_active":"2024-03-24T10:45:39.901119+0000","last_became_peered":"2024-03-24T10:45:39.901119+0000","last_unstale":"2024-03-24T10:54:41.315968+0000","last_undegraded":"2024-03-24T10:54:41.315968+0000","last_fullsized":"2024-03-24T10:54:41.315968+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:16.341360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.323237+0000","last_change":"2024-03-24T10:45:00.639093+0000","last_active":"2024-03-24T10:56:16.323237+0000","last_peered":"2024-03-24T10:56:16.323237+0000","last_clean":"2024-03-24T10:56:16.323237+0000","last_became_active":"2024-03-24T10:45:00.638785+0000","last_became_peered":"2024-03-24T10:45:00.638785+0000","last_unstale":"2024-03-24T10:56:16.323237+0000","last_undegraded":"2024-03-24T10:56:16.323237+0000","last_fullsized":"2024-03-24T10:56:16.323237+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.235406+0000","last_change":"2024-03-24T10:45:01.953772+0000","last_active":"2024-03-24T10:56:16.235406+0000","last_peered":"2024-03-24T10:56:16.235406+0000","last_clean":"2024-03-24T10:56:16.235406+0000","last_became_active":"2024-03-24T10:45:01.953402+0000","last_became_peered":"2024-03-24T10:45:01.953402+0000","last_unstale":"2024-03-24T10:56:16.235406+0000","last_undegraded":"2024-03-24T10:56:16.235406+0000","last_fullsized":"2024-03-24T10:56:16.235406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:22.089070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":183,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:51.238749+0000","last_change":"2024-03-24T10:44:48.241074+0000","last_active":"2024-03-24T10:56:51.238749+0000","last_peered":"2024-03-24T10:56:51.238749+0000","last_clean":"2024-03-24T10:56:51.238749+0000","last_became_active":"2024-03-24T10:44:48.239986+0000","last_became_peered":"2024-03-24T10:44:48.239986+0000","last_unstale":"2024-03-24T10:56:51.238749+0000","last_undegraded":"2024-03-24T10:56:51.238749+0000","last_fullsized":"2024-03-24T10:56:51.238749+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:46.142247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:46.142247+0000","last_clean_scrub_stamp":"2024-03-24T10:44:46.142247+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:05:50.110100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.235730+0000","last_change":"2024-03-24T10:45:00.637276+0000","last_active":"2024-03-24T10:55:26.235730+0000","last_peered":"2024-03-24T10:55:26.235730+0000","last_clean":"2024-03-24T10:55:26.235730+0000","last_became_active":"2024-03-24T10:45:00.636734+0000","last_became_peered":"2024-03-24T10:45:00.636734+0000","last_unstale":"2024-03-24T10:55:26.235730+0000","last_undegraded":"2024-03-24T10:55:26.235730+0000","last_fullsized":"2024-03-24T10:55:26.235730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.200440+0000","last_change":"2024-03-24T10:45:01.958505+0000","last_active":"2024-03-24T10:55:26.200440+0000","last_peered":"2024-03-24T10:55:26.200440+0000","last_clean":"2024-03-24T10:55:26.200440+0000","last_became_active":"2024-03-24T10:45:01.958268+0000","last_became_peered":"2024-03-24T10:45:01.958268+0000","last_unstale":"2024-03-24T10:55:26.200440+0000","last_undegraded":"2024-03-24T10:55:26.200440+0000","last_fullsized":"2024-03-24T10:55:26.200440+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:27:12.555345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"24'5","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:01.239735+0000","last_change":"2024-03-24T10:45:00.597856+0000","last_active":"2024-03-24T10:57:01.239735+0000","last_peered":"2024-03-24T10:57:01.239735+0000","last_clean":"2024-03-24T10:57:01.239735+0000","last_became_active":"2024-03-24T10:45:00.597640+0000","last_became_peered":"2024-03-24T10:45:00.597640+0000","last_unstale":"2024-03-24T10:57:01.239735+0000","last_undegraded":"2024-03-24T10:57:01.239735+0000","last_fullsized":"2024-03-24T10:57:01.239735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:29.826202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.215622+0000","last_change":"2024-03-24T10:45:01.897502+0000","last_active":"2024-03-24T10:55:29.215622+0000","last_peered":"2024-03-24T10:55:29.215622+0000","last_clean":"2024-03-24T10:55:29.215622+0000","last_became_active":"2024-03-24T10:45:01.897250+0000","last_became_peered":"2024-03-24T10:45:01.897250+0000","last_unstale":"2024-03-24T10:55:29.215622+0000","last_undegraded":"2024-03-24T10:55:29.215622+0000","last_fullsized":"2024-03-24T10:55:29.215622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:09:46.341672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"24'6","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.319220+0000","last_change":"2024-03-24T10:45:00.627166+0000","last_active":"2024-03-24T10:55:26.319220+0000","last_peered":"2024-03-24T10:55:26.319220+0000","last_clean":"2024-03-24T10:55:26.319220+0000","last_became_active":"2024-03-24T10:45:00.626881+0000","last_became_peered":"2024-03-24T10:45:00.626881+0000","last_unstale":"2024-03-24T10:55:26.319220+0000","last_undegraded":"2024-03-24T10:55:26.319220+0000","last_fullsized":"2024-03-24T10:55:26.319220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:01.330987+0000","last_change":"2024-03-24T10:45:01.930271+0000","last_active":"2024-03-24T10:57:01.330987+0000","last_peered":"2024-03-24T10:57:01.330987+0000","last_clean":"2024-03-24T10:57:01.330987+0000","last_became_active":"2024-03-24T10:45:01.929998+0000","last_became_peered":"2024-03-24T10:45:01.929998+0000","last_unstale":"2024-03-24T10:57:01.330987+0000","last_undegraded":"2024-03-24T10:57:01.330987+0000","last_fullsized":"2024-03-24T10:57:01.330987+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.235722+0000","last_change":"2024-03-24T10:45:00.638309+0000","last_active":"2024-03-24T10:55:26.235722+0000","last_peered":"2024-03-24T10:55:26.235722+0000","last_clean":"2024-03-24T10:55:26.235722+0000","last_became_active":"2024-03-24T10:45:00.638041+0000","last_became_peered":"2024-03-24T10:45:00.638041+0000","last_unstale":"2024-03-24T10:55:26.235722+0000","last_undegraded":"2024-03-24T10:55:26.235722+0000","last_fullsized":"2024-03-24T10:55:26.235722+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:14.049099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.266124+0000","last_change":"2024-03-24T10:45:01.897722+0000","last_active":"2024-03-24T10:55:29.266124+0000","last_peered":"2024-03-24T10:55:29.266124+0000","last_clean":"2024-03-24T10:55:29.266124+0000","last_became_active":"2024-03-24T10:45:01.897159+0000","last_became_peered":"2024-03-24T10:45:01.897159+0000","last_unstale":"2024-03-24T10:55:29.266124+0000","last_undegraded":"2024-03-24T10:55:29.266124+0000","last_fullsized":"2024-03-24T10:55:29.266124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:42.293342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.323798+0000","last_change":"2024-03-24T10:45:00.629267+0000","last_active":"2024-03-24T10:55:26.323798+0000","last_peered":"2024-03-24T10:55:26.323798+0000","last_clean":"2024-03-24T10:55:26.323798+0000","last_became_active":"2024-03-24T10:45:00.628971+0000","last_became_peered":"2024-03-24T10:45:00.628971+0000","last_unstale":"2024-03-24T10:55:26.323798+0000","last_undegraded":"2024-03-24T10:55:26.323798+0000","last_fullsized":"2024-03-24T10:55:26.323798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:41.176786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.235742+0000","last_change":"2024-03-24T10:45:01.952527+0000","last_active":"2024-03-24T10:55:26.235742+0000","last_peered":"2024-03-24T10:55:26.235742+0000","last_clean":"2024-03-24T10:55:26.235742+0000","last_became_active":"2024-03-24T10:45:01.952192+0000","last_became_peered":"2024-03-24T10:45:01.952192+0000","last_unstale":"2024-03-24T10:55:26.235742+0000","last_undegraded":"2024-03-24T10:55:26.235742+0000","last_fullsized":"2024-03-24T10:55:26.235742+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.230535+0000","last_change":"2024-03-24T10:45:00.631941+0000","last_active":"2024-03-24T10:55:26.230535+0000","last_peered":"2024-03-24T10:55:26.230535+0000","last_clean":"2024-03-24T10:55:26.230535+0000","last_became_active":"2024-03-24T10:45:00.631670+0000","last_became_peered":"2024-03-24T10:45:00.631670+0000","last_unstale":"2024-03-24T10:55:26.230535+0000","last_undegraded":"2024-03-24T10:55:26.230535+0000","last_fullsized":"2024-03-24T10:55:26.230535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:09:45.225115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.323774+0000","last_change":"2024-03-24T10:45:01.930483+0000","last_active":"2024-03-24T10:55:26.323774+0000","last_peered":"2024-03-24T10:55:26.323774+0000","last_clean":"2024-03-24T10:55:26.323774+0000","last_became_active":"2024-03-24T10:45:01.930220+0000","last_became_peered":"2024-03-24T10:45:01.930220+0000","last_unstale":"2024-03-24T10:55:26.323774+0000","last_undegraded":"2024-03-24T10:55:26.323774+0000","last_fullsized":"2024-03-24T10:55:26.323774+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.325636+0000","last_change":"2024-03-24T10:45:00.633650+0000","last_active":"2024-03-24T10:55:51.325636+0000","last_peered":"2024-03-24T10:55:51.325636+0000","last_clean":"2024-03-24T10:55:51.325636+0000","last_became_active":"2024-03-24T10:45:00.633329+0000","last_became_peered":"2024-03-24T10:45:00.633329+0000","last_unstale":"2024-03-24T10:55:51.325636+0000","last_undegraded":"2024-03-24T10:55:51.325636+0000","last_fullsized":"2024-03-24T10:55:51.325636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.202473+0000","last_change":"2024-03-24T10:45:01.853487+0000","last_active":"2024-03-24T10:55:51.202473+0000","last_peered":"2024-03-24T10:55:51.202473+0000","last_clean":"2024-03-24T10:55:51.202473+0000","last_became_active":"2024-03-24T10:45:01.853257+0000","last_became_peered":"2024-03-24T10:45:01.853257+0000","last_unstale":"2024-03-24T10:55:51.202473+0000","last_undegraded":"2024-03-24T10:55:51.202473+0000","last_fullsized":"2024-03-24T10:55:51.202473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:54.295762+0000","last_change":"2024-03-24T10:45:00.639282+0000","last_active":"2024-03-24T10:55:54.295762+0000","last_peered":"2024-03-24T10:55:54.295762+0000","last_clean":"2024-03-24T10:55:54.295762+0000","last_became_active":"2024-03-24T10:45:00.638999+0000","last_became_peered":"2024-03-24T10:45:00.638999+0000","last_unstale":"2024-03-24T10:55:54.295762+0000","last_undegraded":"2024-03-24T10:55:54.295762+0000","last_fullsized":"2024-03-24T10:55:54.295762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:41.176786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:54.217821+0000","last_change":"2024-03-24T10:45:01.928900+0000","last_active":"2024-03-24T10:55:54.217821+0000","last_peered":"2024-03-24T10:55:54.217821+0000","last_clean":"2024-03-24T10:55:54.217821+0000","last_became_active":"2024-03-24T10:45:01.928610+0000","last_became_peered":"2024-03-24T10:45:01.928610+0000","last_unstale":"2024-03-24T10:55:54.217821+0000","last_undegraded":"2024-03-24T10:55:54.217821+0000","last_fullsized":"2024-03-24T10:55:54.217821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.232929+0000","last_change":"2024-03-24T10:45:00.631327+0000","last_active":"2024-03-24T10:55:51.232929+0000","last_peered":"2024-03-24T10:55:51.232929+0000","last_clean":"2024-03-24T10:55:51.232929+0000","last_became_active":"2024-03-24T10:45:00.631078+0000","last_became_peered":"2024-03-24T10:45:00.631078+0000","last_unstale":"2024-03-24T10:55:51.232929+0000","last_undegraded":"2024-03-24T10:55:51.232929+0000","last_fullsized":"2024-03-24T10:55:51.232929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:43.263796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:54.268364+0000","last_change":"2024-03-24T10:45:01.897329+0000","last_active":"2024-03-24T10:55:54.268364+0000","last_peered":"2024-03-24T10:55:54.268364+0000","last_clean":"2024-03-24T10:55:54.268364+0000","last_became_active":"2024-03-24T10:45:01.897063+0000","last_became_peered":"2024-03-24T10:45:01.897063+0000","last_unstale":"2024-03-24T10:55:54.268364+0000","last_undegraded":"2024-03-24T10:55:54.268364+0000","last_fullsized":"2024-03-24T10:55:54.268364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:07.259007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.325593+0000","last_change":"2024-03-24T10:45:00.597948+0000","last_active":"2024-03-24T10:55:51.325593+0000","last_peered":"2024-03-24T10:55:51.325593+0000","last_clean":"2024-03-24T10:55:51.325593+0000","last_became_active":"2024-03-24T10:45:00.597671+0000","last_became_peered":"2024-03-24T10:45:00.597671+0000","last_unstale":"2024-03-24T10:55:51.325593+0000","last_undegraded":"2024-03-24T10:55:51.325593+0000","last_fullsized":"2024-03-24T10:55:51.325593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:06.142451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.202427+0000","last_change":"2024-03-24T10:45:01.927822+0000","last_active":"2024-03-24T10:55:51.202427+0000","last_peered":"2024-03-24T10:55:51.202427+0000","last_clean":"2024-03-24T10:55:51.202427+0000","last_became_active":"2024-03-24T10:45:01.927538+0000","last_became_peered":"2024-03-24T10:45:01.927538+0000","last_unstale":"2024-03-24T10:55:51.202427+0000","last_undegraded":"2024-03-24T10:55:51.202427+0000","last_fullsized":"2024-03-24T10:55:51.202427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:50.760503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'3","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.325612+0000","last_change":"2024-03-24T10:45:00.626818+0000","last_active":"2024-03-24T10:55:51.325612+0000","last_peered":"2024-03-24T10:55:51.325612+0000","last_clean":"2024-03-24T10:55:51.325612+0000","last_became_active":"2024-03-24T10:45:00.626557+0000","last_became_peered":"2024-03-24T10:45:00.626557+0000","last_unstale":"2024-03-24T10:55:51.325612+0000","last_undegraded":"2024-03-24T10:55:51.325612+0000","last_fullsized":"2024-03-24T10:55:51.325612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:43.263796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.204517+0000","last_change":"2024-03-24T10:45:01.893192+0000","last_active":"2024-03-24T10:56:16.204517+0000","last_peered":"2024-03-24T10:56:16.204517+0000","last_clean":"2024-03-24T10:56:16.204517+0000","last_became_active":"2024-03-24T10:45:01.892934+0000","last_became_peered":"2024-03-24T10:45:01.892934+0000","last_unstale":"2024-03-24T10:56:16.204517+0000","last_undegraded":"2024-03-24T10:56:16.204517+0000","last_fullsized":"2024-03-24T10:56:16.204517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.235371+0000","last_change":"2024-03-24T10:45:00.630147+0000","last_active":"2024-03-24T10:56:16.235371+0000","last_peered":"2024-03-24T10:56:16.235371+0000","last_clean":"2024-03-24T10:56:16.235371+0000","last_became_active":"2024-03-24T10:45:00.629850+0000","last_became_peered":"2024-03-24T10:45:00.629850+0000","last_unstale":"2024-03-24T10:56:16.235371+0000","last_undegraded":"2024-03-24T10:56:16.235371+0000","last_fullsized":"2024-03-24T10:56:16.235371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:01:12.843222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:54.295704+0000","last_change":"2024-03-24T10:45:01.897333+0000","last_active":"2024-03-24T10:55:54.295704+0000","last_peered":"2024-03-24T10:55:54.295704+0000","last_clean":"2024-03-24T10:55:54.295704+0000","last_became_active":"2024-03-24T10:45:01.897080+0000","last_became_peered":"2024-03-24T10:45:01.897080+0000","last_unstale":"2024-03-24T10:55:54.295704+0000","last_undegraded":"2024-03-24T10:55:54.295704+0000","last_fullsized":"2024-03-24T10:55:54.295704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:13.887947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.233488+0000","last_change":"2024-03-24T10:45:01.956041+0000","last_active":"2024-03-24T10:55:01.233488+0000","last_peered":"2024-03-24T10:55:01.233488+0000","last_clean":"2024-03-24T10:55:01.233488+0000","last_became_active":"2024-03-24T10:45:01.955766+0000","last_became_peered":"2024-03-24T10:45:01.955766+0000","last_unstale":"2024-03-24T10:55:01.233488+0000","last_undegraded":"2024-03-24T10:55:01.233488+0000","last_fullsized":"2024-03-24T10:55:01.233488+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:41:16.341360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.228149+0000","last_change":"2024-03-24T10:45:00.639913+0000","last_active":"2024-03-24T10:55:01.228149+0000","last_peered":"2024-03-24T10:55:01.228149+0000","last_clean":"2024-03-24T10:55:01.228149+0000","last_became_active":"2024-03-24T10:45:00.639653+0000","last_became_peered":"2024-03-24T10:45:00.639653+0000","last_unstale":"2024-03-24T10:55:01.228149+0000","last_undegraded":"2024-03-24T10:55:01.228149+0000","last_fullsized":"2024-03-24T10:55:01.228149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:15.749451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.233473+0000","last_change":"2024-03-24T10:45:01.952340+0000","last_active":"2024-03-24T10:55:01.233473+0000","last_peered":"2024-03-24T10:55:01.233473+0000","last_clean":"2024-03-24T10:55:01.233473+0000","last_became_active":"2024-03-24T10:45:01.952074+0000","last_became_peered":"2024-03-24T10:45:01.952074+0000","last_unstale":"2024-03-24T10:55:01.233473+0000","last_undegraded":"2024-03-24T10:55:01.233473+0000","last_fullsized":"2024-03-24T10:55:01.233473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"24'12","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:31.242486+0000","last_change":"2024-03-24T10:45:00.633790+0000","last_active":"2024-03-24T10:57:31.242486+0000","last_peered":"2024-03-24T10:57:31.242486+0000","last_clean":"2024-03-24T10:57:31.242486+0000","last_became_active":"2024-03-24T10:45:00.633494+0000","last_became_peered":"2024-03-24T10:45:00.633494+0000","last_unstale":"2024-03-24T10:57:31.242486+0000","last_undegraded":"2024-03-24T10:57:31.242486+0000","last_fullsized":"2024-03-24T10:57:31.242486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:14.049099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.215687+0000","last_change":"2024-03-24T10:45:01.957958+0000","last_active":"2024-03-24T10:55:29.215687+0000","last_peered":"2024-03-24T10:55:29.215687+0000","last_clean":"2024-03-24T10:55:29.215687+0000","last_became_active":"2024-03-24T10:45:01.957676+0000","last_became_peered":"2024-03-24T10:45:01.957676+0000","last_unstale":"2024-03-24T10:55:29.215687+0000","last_undegraded":"2024-03-24T10:55:29.215687+0000","last_fullsized":"2024-03-24T10:55:29.215687+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:01.208031+0000","last_change":"2024-03-24T10:45:00.629120+0000","last_active":"2024-03-24T10:57:01.208031+0000","last_peered":"2024-03-24T10:57:01.208031+0000","last_clean":"2024-03-24T10:57:01.208031+0000","last_became_active":"2024-03-24T10:45:00.628712+0000","last_became_peered":"2024-03-24T10:45:00.628712+0000","last_unstale":"2024-03-24T10:57:01.208031+0000","last_undegraded":"2024-03-24T10:57:01.208031+0000","last_fullsized":"2024-03-24T10:57:01.208031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:01.208005+0000","last_change":"2024-03-24T10:45:01.894602+0000","last_active":"2024-03-24T10:57:01.208005+0000","last_peered":"2024-03-24T10:57:01.208005+0000","last_clean":"2024-03-24T10:57:01.208005+0000","last_became_active":"2024-03-24T10:45:01.894362+0000","last_became_peered":"2024-03-24T10:45:01.894362+0000","last_unstale":"2024-03-24T10:57:01.208005+0000","last_undegraded":"2024-03-24T10:57:01.208005+0000","last_fullsized":"2024-03-24T10:57:01.208005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'13","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.215696+0000","last_change":"2024-03-24T10:45:00.639171+0000","last_active":"2024-03-24T10:55:29.215696+0000","last_peered":"2024-03-24T10:55:29.215696+0000","last_clean":"2024-03-24T10:55:29.215696+0000","last_became_active":"2024-03-24T10:45:00.638871+0000","last_became_peered":"2024-03-24T10:45:00.638871+0000","last_unstale":"2024-03-24T10:55:29.215696+0000","last_undegraded":"2024-03-24T10:55:29.215696+0000","last_fullsized":"2024-03-24T10:55:29.215696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:52:46.322196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":204,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":184,"ondisk_log_size":184,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65430,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542321,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31524,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340316,"statfs":{"total":96636764160,"available":96604483584,"internally_reserved":0,"allocated":352256,"data_stored":685052,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542321,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":299008,"data_stored":96516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542320,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":95020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542321,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":282624,"data_stored":94046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542321,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":278528,"data_stored":94116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575023,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":278528,"data_stored":94986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575023,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":274432,"data_stored":93914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575023,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":344064,"data_stored":683826,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:59:08.360 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T10:59:08.498 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.497+0000 7f8ffafa1640 1 Processor -- start 2024-03-24T10:59:08.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.497+0000 7f8ffafa1640 1 -- start start 2024-03-24T10:59:08.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.498+0000 7f8ffafa1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ff40c58c0 0x7f8ff40caab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:08.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.498+0000 7f8ffafa1640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8ff40c9610 0x7f8ff40c9a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:08.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.498+0000 7f8ffafa1640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ff40c9fb0 con 0x7f8ff40c2ea0 2024-03-24T10:59:08.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.498+0000 7f8ffafa1640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f8ff40cb210 con 0x7f8ff40c9610 2024-03-24T10:59:08.499 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.498+0000 7f8ffafa1640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f8ff4156db0 con 0x7f8ff40c58c0 2024-03-24T10:59:08.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.499+0000 7f8ff3fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ff40c58c0 0x7f8ff40caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:08.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.499+0000 7f8ff953e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f8ff40c9610 0x7f8ff40c9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:08.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.499+0000 7f8ff3fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ff40c58c0 0x7f8ff40caab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:48586/0 (socket says 172.21.15.5:48586) 2024-03-24T10:59:08.500 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.499+0000 7f8ff3fff640 1 -- 172.21.15.5:0/3674944920 learned_addr learned my addr 172.21.15.5:0/3674944920 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:08.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.502+0000 7f8ff3fff640 1 -- 172.21.15.5:0/3674944920 >> v2:172.21.15.5:3300/0 conn(0x7f8ff40c9610 msgr2=0x7f8ff40c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:08.502 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.502+0000 7f8ff3fff640 1 --2- 172.21.15.5:0/3674944920 >> v2:172.21.15.5:3300/0 conn(0x7f8ff40c9610 0x7f8ff40c9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:08.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.502+0000 7f8ff3fff640 1 -- 172.21.15.5:0/3674944920 >> v1:172.21.15.5:6789/0 conn(0x7f8ff40c2ea0 legacy=0x7f8ff40c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:08.503 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.502+0000 7f8ff3fff640 1 -- 172.21.15.5:0/3674944920 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ff4156fc0 con 0x7f8ff40c58c0 2024-03-24T10:59:08.505 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.504+0000 7f8ff3fff640 1 --2- 172.21.15.5:0/3674944920 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ff40c58c0 0x7f8ff40caab0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0009870 tx=0x7f8fe002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2239ab7a676ba5ab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:08.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.508+0000 7f8ff37fe640 1 -- 172.21.15.5:0/3674944920 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0030840 con 0x7f8ff40c58c0 2024-03-24T10:59:08.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.508+0000 7f8ff37fe640 1 -- 172.21.15.5:0/3674944920 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fe00309a0 con 0x7f8ff40c58c0 2024-03-24T10:59:08.509 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.508+0000 7f8ff37fe640 1 -- 172.21.15.5:0/3674944920 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0030d30 con 0x7f8ff40c58c0 2024-03-24T10:59:08.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.509+0000 7f8ffafa1640 1 -- 172.21.15.5:0/3674944920 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ff40c58c0 msgr2=0x7f8ff40caab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:08.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.509+0000 7f8ffafa1640 1 --2- 172.21.15.5:0/3674944920 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ff40c58c0 0x7f8ff40caab0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0009870 tx=0x7f8fe002b980 comp rx=0 tx=0).stop 2024-03-24T10:59:08.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.509+0000 7f8ffafa1640 1 -- 172.21.15.5:0/3674944920 shutdown_connections 2024-03-24T10:59:08.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.509+0000 7f8ffafa1640 1 --2- 172.21.15.5:0/3674944920 >> v2:172.21.15.5:3300/0 conn(0x7f8ff40c9610 0x7f8ff40c9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:08.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.509+0000 7f8ffafa1640 1 --2- 172.21.15.5:0/3674944920 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ff40c58c0 0x7f8ff40caab0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:08.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.509+0000 7f8ffafa1640 1 -- 172.21.15.5:0/3674944920 >> 172.21.15.5:0/3674944920 conn(0x7f8ff406b130 msgr2=0x7f8ff406b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:08.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.509+0000 7f8ffafa1640 1 -- 172.21.15.5:0/3674944920 shutdown_connections 2024-03-24T10:59:08.510 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.509+0000 7f8ffafa1640 1 -- 172.21.15.5:0/3674944920 wait complete. 2024-03-24T10:59:08.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.510+0000 7f8ffafa1640 1 Processor -- start 2024-03-24T10:59:08.511 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.510+0000 7f8ffafa1640 1 -- start start 2024-03-24T10:59:08.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.511+0000 7f8ffafa1640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ff40c2ea0 0x7f8ff415abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:08.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.511+0000 7f8ffafa1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ff40c58c0 0x7f8ff41f3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:08.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.511+0000 7f8ffafa1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ff40c9610 0x7f8ff41e67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:08.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.511+0000 7f8ffafa1640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff41f4160 con 0x7f8ff40c2ea0 2024-03-24T10:59:08.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.511+0000 7f8ffafa1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f8ff41efe20 con 0x7f8ff40c58c0 2024-03-24T10:59:08.512 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.511+0000 7f8ffafa1640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f8ff41f00c0 con 0x7f8ff40c9610 2024-03-24T10:59:08.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.512+0000 7f8ff953e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ff40c9610 0x7f8ff41e67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:08.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.512+0000 7f8ff8d3d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ff40c2ea0 0x7f8ff415abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:08.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.512+0000 7f8ff953e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ff40c9610 0x7f8ff41e67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:48588/0 (socket says 172.21.15.5:48588) 2024-03-24T10:59:08.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.512+0000 7f8ff953e640 1 -- 172.21.15.5:0/2129274652 learned_addr learned my addr 172.21.15.5:0/2129274652 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:08.513 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.512+0000 7f8ff3fff640 1 --2- 172.21.15.5:0/2129274652 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ff40c58c0 0x7f8ff41f3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:08.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.515+0000 7f8ff953e640 1 -- 172.21.15.5:0/2129274652 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ff40c58c0 msgr2=0x7f8ff41f3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:08.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.515+0000 7f8ff953e640 1 --2- 172.21.15.5:0/2129274652 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ff40c58c0 0x7f8ff41f3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:08.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.515+0000 7f8ff953e640 1 -- 172.21.15.5:0/2129274652 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ff40c2ea0 msgr2=0x7f8ff415abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:08.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.515+0000 7f8ff953e640 1 --2- 172.21.15.5:0/2129274652 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ff40c2ea0 0x7f8ff415abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:08.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.515+0000 7f8ff953e640 1 -- 172.21.15.5:0/2129274652 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe00094d0 con 0x7f8ff40c9610 2024-03-24T10:59:08.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.518+0000 7f8ff953e640 1 --2- 172.21.15.5:0/2129274652 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ff40c9610 0x7f8ff41e67a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8fec00efc0 tx=0x7f8fec00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:08.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.522+0000 7f8ff17fa640 1 -- 172.21.15.5:0/2129274652 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fec014480 con 0x7f8ff40c9610 2024-03-24T10:59:08.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.522+0000 7f8ff17fa640 1 -- 172.21.15.5:0/2129274652 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fec00f040 con 0x7f8ff40c9610 2024-03-24T10:59:08.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.522+0000 7f8ff17fa640 1 -- 172.21.15.5:0/2129274652 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fec014820 con 0x7f8ff40c9610 2024-03-24T10:59:08.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.522+0000 7f8ffafa1640 1 -- 172.21.15.5:0/2129274652 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ff41f03a0 con 0x7f8ff40c9610 2024-03-24T10:59:08.524 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.522+0000 7f8ffafa1640 1 -- 172.21.15.5:0/2129274652 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ff41f0960 con 0x7f8ff40c9610 2024-03-24T10:59:08.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.524+0000 7f8ffafa1640 1 -- 172.21.15.5:0/2129274652 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ff4062a00 con 0x7f8ff40c9610 2024-03-24T10:59:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.804+0000 7f8ff17fa640 1 -- 172.21.15.5:0/2129274652 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8fec014980 con 0x7f8ff40c9610 2024-03-24T10:59:08.805 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.804+0000 7f8ff17fa640 1 --2- 172.21.15.5:0/2129274652 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8fd806b890 0x7f8fd806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:08.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.804+0000 7f8ff8d3d640 1 --2- 172.21.15.5:0/2129274652 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8fd806b890 0x7f8fd806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:08.806 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.805+0000 7f8ff8d3d640 1 --2- 172.21.15.5:0/2129274652 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8fd806b890 0x7f8fd806dd50 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8006fd0 tx=0x7f8fe8009060 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:08.812 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.811+0000 7f8ff17fa640 1 -- 172.21.15.5:0/2129274652 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f8fec059c30 con 0x7f8ff40c9610 2024-03-24T10:59:08.831 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:08.830+0000 7f8ff17fa640 1 -- 172.21.15.5:0/2129274652 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8fec055ed0 con 0x7f8ff40c9610 2024-03-24T10:59:09.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:09.036+0000 7f8ffafa1640 1 -- 172.21.15.5:0/2129274652 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8ff41f11a0 con 0x7f8ff40c9610 2024-03-24T10:59:09.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:09.107+0000 7f8ff17fa640 1 -- 172.21.15.5:0/2129274652 <== mon.2 v2:172.21.15.57:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+16862 (secure 0 0 0) 0x7f8ff41f11a0 con 0x7f8ff40c9610 2024-03-24T10:59:09.109 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:59:09.109 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":24,"fsid":"7d9848db-2fe7-4d95-a6e2-b04f4f639ebc","created":"2024-03-24T10:44:30.435176+0000","modified":"2024-03-24T10:46:07.878866+0000","last_up_change":"2024-03-24T10:44:44.634047+0000","last_in_change":"2024-03-24T10:44:35.986391+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T10:44:45.829320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T10:44:58.435543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T10:45:00.129313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"50f590b3-f5ec-4e1b-ac46-25cab307a27a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6825","nonce":1991336051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6827","nonce":1991336051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6831","nonce":1991336051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":1991336051},{"type":"v1","addr":"172.21.15.5:6829","nonce":1991336051}]},"public_addr":"172.21.15.5:6825/1991336051","cluster_addr":"172.21.15.5:6827/1991336051","heartbeat_back_addr":"172.21.15.5:6831/1991336051","heartbeat_front_addr":"172.21.15.5:6829/1991336051","state":["exists","up"]},{"osd":1,"uuid":"ccba9e93-c704-472b-a832-8d4eeb456c47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6801","nonce":2140667501}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6803","nonce":2140667501}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6807","nonce":2140667501}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":2140667501},{"type":"v1","addr":"172.21.15.5:6805","nonce":2140667501}]},"public_addr":"172.21.15.5:6801/2140667501","cluster_addr":"172.21.15.5:6803/2140667501","heartbeat_back_addr":"172.21.15.5:6807/2140667501","heartbeat_front_addr":"172.21.15.5:6805/2140667501","state":["exists","up"]},{"osd":2,"uuid":"e614c5b1-6504-452e-82e0-95c7ca271737","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6817","nonce":90414679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6819","nonce":90414679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6823","nonce":90414679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":90414679},{"type":"v1","addr":"172.21.15.5:6821","nonce":90414679}]},"public_addr":"172.21.15.5:6817/90414679","cluster_addr":"172.21.15.5:6819/90414679","heartbeat_back_addr":"172.21.15.5:6823/90414679","heartbeat_front_addr":"172.21.15.5:6821/90414679","state":["exists","up"]},{"osd":3,"uuid":"c60154d0-85e0-46a1-8273-4863bd8ad2dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6809","nonce":4212473119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6811","nonce":4212473119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6815","nonce":4212473119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":4212473119},{"type":"v1","addr":"172.21.15.5:6813","nonce":4212473119}]},"public_addr":"172.21.15.5:6809/4212473119","cluster_addr":"172.21.15.5:6811/4212473119","heartbeat_back_addr":"172.21.15.5:6815/4212473119","heartbeat_front_addr":"172.21.15.5:6813/4212473119","state":["exists","up"]},{"osd":4,"uuid":"21d74109-da3a-4fc0-b2d8-93e9ae3ed30c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6817","nonce":2193986505}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6818","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6819","nonce":2193986505}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6822","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6823","nonce":2193986505}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":2193986505},{"type":"v1","addr":"172.21.15.57:6821","nonce":2193986505}]},"public_addr":"172.21.15.57:6817/2193986505","cluster_addr":"172.21.15.57:6819/2193986505","heartbeat_back_addr":"172.21.15.57:6823/2193986505","heartbeat_front_addr":"172.21.15.57:6821/2193986505","state":["exists","up"]},{"osd":5,"uuid":"5cf19cc7-215f-4d69-a30a-80543c0293e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6809","nonce":3044987466}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6811","nonce":3044987466}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6815","nonce":3044987466}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":3044987466},{"type":"v1","addr":"172.21.15.57:6813","nonce":3044987466}]},"public_addr":"172.21.15.57:6809/3044987466","cluster_addr":"172.21.15.57:6811/3044987466","heartbeat_back_addr":"172.21.15.57:6815/3044987466","heartbeat_front_addr":"172.21.15.57:6813/3044987466","state":["exists","up"]},{"osd":6,"uuid":"a5e76c69-c3fc-47b2-8f92-e560cf503e45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6801","nonce":4290897445}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6803","nonce":4290897445}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6807","nonce":4290897445}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":4290897445},{"type":"v1","addr":"172.21.15.57:6805","nonce":4290897445}]},"public_addr":"172.21.15.57:6801/4290897445","cluster_addr":"172.21.15.57:6803/4290897445","heartbeat_back_addr":"172.21.15.57:6807/4290897445","heartbeat_front_addr":"172.21.15.57:6805/4290897445","state":["exists","up"]},{"osd":7,"uuid":"47d0b873-e3ac-4f40-a372-681436e45d0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6825","nonce":212605698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6826","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6827","nonce":212605698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6830","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6831","nonce":212605698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":212605698},{"type":"v1","addr":"172.21.15.57:6829","nonce":212605698}]},"public_addr":"172.21.15.57:6825/212605698","cluster_addr":"172.21.15.57:6827/212605698","heartbeat_back_addr":"172.21.15.57:6831/212605698","heartbeat_front_addr":"172.21.15.57:6829/212605698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.211237+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.192822+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.273429+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.199212+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.277082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.218768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.190915+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T10:44:41.225294+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.e","mappings":[{"from":7,"to":1}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.57:6837/1426147484":"2024-03-25T10:46:07.865481+0000","172.21.15.57:6835/1426147484":"2024-03-25T10:46:07.865481+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T10:59:09.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:09.110+0000 7f8ffafa1640 1 -- 172.21.15.5:0/2129274652 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8fd806b890 msgr2=0x7f8fd806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:09.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:09.110+0000 7f8ffafa1640 1 --2- 172.21.15.5:0/2129274652 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8fd806b890 0x7f8fd806dd50 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8fe8006fd0 tx=0x7f8fe8009060 comp rx=0 tx=0).stop 2024-03-24T10:59:09.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:09.110+0000 7f8ffafa1640 1 -- 172.21.15.5:0/2129274652 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ff40c9610 msgr2=0x7f8ff41e67a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:09.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:09.110+0000 7f8ffafa1640 1 --2- 172.21.15.5:0/2129274652 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ff40c9610 0x7f8ff41e67a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8fec00efc0 tx=0x7f8fec00c490 comp rx=0 tx=0).stop 2024-03-24T10:59:09.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:09.110+0000 7f8ffafa1640 1 -- 172.21.15.5:0/2129274652 shutdown_connections 2024-03-24T10:59:09.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:09.110+0000 7f8ffafa1640 1 --2- 172.21.15.5:0/2129274652 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f8fd806b890 0x7f8fd806dd50 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:09.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:09.110+0000 7f8ffafa1640 1 --2- 172.21.15.5:0/2129274652 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f8ff40c9610 0x7f8ff41e67a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:09.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:09.110+0000 7f8ffafa1640 1 --2- 172.21.15.5:0/2129274652 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8ff40c58c0 0x7f8ff41f3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:09.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:09.110+0000 7f8ffafa1640 1 --2- 172.21.15.5:0/2129274652 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f8ff40c2ea0 0x7f8ff415abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:09.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:09.110+0000 7f8ffafa1640 1 -- 172.21.15.5:0/2129274652 >> 172.21.15.5:0/2129274652 conn(0x7f8ff406b130 msgr2=0x7f8ff40c4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:09.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:09.110+0000 7f8ffafa1640 1 -- 172.21.15.5:0/2129274652 shutdown_connections 2024-03-24T10:59:09.112 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:09.110+0000 7f8ffafa1640 1 -- 172.21.15.5:0/2129274652 wait complete. 2024-03-24T10:59:10.124 INFO:tasks.ceph:Scrubbing osd.0 2024-03-24T10:59:10.124 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-24T10:59:10.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.229+0000 7f660d0c9640 1 Processor -- start 2024-03-24T10:59:10.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.230+0000 7f660d0c9640 1 -- start start 2024-03-24T10:59:10.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.230+0000 7f660d0c9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f660806bf50 0x7f660806c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:10.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.230+0000 7f660d0c9640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66080cc3e0 con 0x7f66080cc000 2024-03-24T10:59:10.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.230+0000 7f660d0c9640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f660806c8e0 con 0x7f66080cf340 2024-03-24T10:59:10.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.230+0000 7f660d0c9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f660815a400 con 0x7f660806bf50 2024-03-24T10:59:10.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.231+0000 7f6606d9d640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f66080cc000 0x7f660815e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:54536/0 (socket says 172.21.15.5:54536) 2024-03-24T10:59:10.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.231+0000 7f6606d9d640 1 -- 172.21.15.5:0/3012353636 learned_addr learned my addr 172.21.15.5:0/3012353636 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:59:10.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.232+0000 7f660759e640 1 --2- 172.21.15.5:0/3012353636 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f660806bf50 0x7f660806c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:10.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.234+0000 7f660659c640 1 -- 172.21.15.5:0/3012353636 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1959530293 0 0) 0x7f660806c8e0 con 0x7f66080cf340 2024-03-24T10:59:10.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.234+0000 7f660659c640 1 -- 172.21.15.5:0/3012353636 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65f4003520 con 0x7f66080cf340 2024-03-24T10:59:10.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.236+0000 7f660759e640 1 -- 172.21.15.5:0/3012353636 >> v1:172.21.15.57:6789/0 conn(0x7f66080cf340 legacy=0x7f66080cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:10.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.236+0000 7f660759e640 1 -- 172.21.15.5:0/3012353636 >> v1:172.21.15.5:6789/0 conn(0x7f66080cc000 legacy=0x7f660815e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:10.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.236+0000 7f660759e640 1 -- 172.21.15.5:0/3012353636 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f660815a5f0 con 0x7f660806bf50 2024-03-24T10:59:10.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.239+0000 7f660759e640 1 --2- 172.21.15.5:0/3012353636 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f660806bf50 0x7f660806c3a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f660815a7e0 tx=0x7f660002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5de07228fd4d61be server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:10.243 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.242+0000 7f660659c640 1 -- 172.21.15.5:0/3012353636 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660002ccd0 con 0x7f660806bf50 2024-03-24T10:59:10.245 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.244+0000 7f660659c640 1 -- 172.21.15.5:0/3012353636 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f660002ce30 con 0x7f660806bf50 2024-03-24T10:59:10.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.245+0000 7f660d0c9640 1 -- 172.21.15.5:0/3012353636 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f660806bf50 msgr2=0x7f660806c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:10.246 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.245+0000 7f660d0c9640 1 --2- 172.21.15.5:0/3012353636 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f660806bf50 0x7f660806c3a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f660815a7e0 tx=0x7f660002bf20 comp rx=0 tx=0).stop 2024-03-24T10:59:10.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.245+0000 7f660d0c9640 1 -- 172.21.15.5:0/3012353636 shutdown_connections 2024-03-24T10:59:10.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.245+0000 7f660d0c9640 1 --2- 172.21.15.5:0/3012353636 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f660806bf50 0x7f660806c3a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:10.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.245+0000 7f660d0c9640 1 -- 172.21.15.5:0/3012353636 >> 172.21.15.5:0/3012353636 conn(0x7f66080cfc30 msgr2=0x7f66080d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:10.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.245+0000 7f660d0c9640 1 -- 172.21.15.5:0/3012353636 shutdown_connections 2024-03-24T10:59:10.247 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.245+0000 7f660d0c9640 1 -- 172.21.15.5:0/3012353636 wait complete. 2024-03-24T10:59:10.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.246+0000 7f660d0c9640 1 Processor -- start 2024-03-24T10:59:10.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.246+0000 7f660d0c9640 1 -- start start 2024-03-24T10:59:10.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.247+0000 7f660d0c9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f660806bf50 0x7f66080b2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:10.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.247+0000 7f660d0c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66080cc000 0x7f6608155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:10.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.247+0000 7f660d0c9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66080cf340 0x7f6608159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:10.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.247+0000 7f660d0c9640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f6608155940 con 0x7f66080cf340 2024-03-24T10:59:10.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.247+0000 7f660d0c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f66081f4080 con 0x7f66080cc000 2024-03-24T10:59:10.248 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.247+0000 7f660d0c9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f6608155e40 con 0x7f660806bf50 2024-03-24T10:59:10.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.248+0000 7f65fffff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66080cc000 0x7f6608155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:10.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.248+0000 7f6606d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f660806bf50 0x7f66080b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:10.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.248+0000 7f6606d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f660806bf50 0x7f66080b2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:48596/0 (socket says 172.21.15.5:48596) 2024-03-24T10:59:10.249 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.248+0000 7f6606d9d640 1 -- 172.21.15.5:0/2494362195 learned_addr learned my addr 172.21.15.5:0/2494362195 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.253+0000 7f6606d9d640 1 -- 172.21.15.5:0/2494362195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66080cc000 msgr2=0x7f6608155400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.253+0000 7f6606d9d640 1 --2- 172.21.15.5:0/2494362195 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f66080cc000 0x7f6608155400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.253+0000 7f6606d9d640 1 -- 172.21.15.5:0/2494362195 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66080cf340 msgr2=0x7f6608159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:10.254 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.253+0000 7f6606d9d640 1 --2- 172.21.15.5:0/2494362195 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f66080cf340 0x7f6608159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:10.255 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.253+0000 7f6606d9d640 1 -- 172.21.15.5:0/2494362195 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6600009a70 con 0x7f660806bf50 2024-03-24T10:59:10.258 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.257+0000 7f6606d9d640 1 --2- 172.21.15.5:0/2494362195 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f660806bf50 0x7f66080b2030 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f65f800c430 tx=0x7f65f800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:10.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.263+0000 7f65ff7fe640 1 -- 172.21.15.5:0/2494362195 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f8005ce0 con 0x7f660806bf50 2024-03-24T10:59:10.264 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.263+0000 7f65ff7fe640 1 -- 172.21.15.5:0/2494362195 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65f8005e40 con 0x7f660806bf50 2024-03-24T10:59:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.263+0000 7f660d0c9640 1 -- 172.21.15.5:0/2494362195 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6608155b60 con 0x7f660806bf50 2024-03-24T10:59:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.263+0000 7f65ff7fe640 1 -- 172.21.15.5:0/2494362195 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f8007310 con 0x7f660806bf50 2024-03-24T10:59:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.263+0000 7f660d0c9640 1 -- 172.21.15.5:0/2494362195 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f66081f3a30 con 0x7f660806bf50 2024-03-24T10:59:10.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.264+0000 7f660d0c9640 1 -- 172.21.15.5:0/2494362195 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f660815ae70 con 0x7f660806bf50 2024-03-24T10:59:10.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.305+0000 7f65ff7fe640 1 -- 172.21.15.5:0/2494362195 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f65f8007470 con 0x7f660806bf50 2024-03-24T10:59:10.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.305+0000 7f65ff7fe640 1 --2- 172.21.15.5:0/2494362195 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f65e806b890 0x7f65e806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:10.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.305+0000 7f65fffff640 1 --2- 172.21.15.5:0/2494362195 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f65e806b890 0x7f65e806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:10.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.306+0000 7f65fffff640 1 --2- 172.21.15.5:0/2494362195 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f65e806b890 0x7f65e806dd50 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f66080b2620 tx=0x7f65f0009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:10.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.311+0000 7f65ff7fe640 1 -- 172.21.15.5:0/2494362195 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f65f8007720 con 0x7f660806bf50 2024-03-24T10:59:10.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.311+0000 7f65ff7fe640 1 --2- 172.21.15.5:0/2494362195 >> [v2:172.21.15.5:6824/1991336051,v1:172.21.15.5:6825/1991336051] conn(0x7f65e8073cc0 0x7f65e8076140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:10.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.311+0000 7f65ff7fe640 1 -- 172.21.15.5:0/2494362195 --> [v2:172.21.15.5:6824/1991336051,v1:172.21.15.5:6825/1991336051] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f65e8076810 con 0x7f65e8073cc0 2024-03-24T10:59:10.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.311+0000 7f660759e640 1 --2- 172.21.15.5:0/2494362195 >> [v2:172.21.15.5:6824/1991336051,v1:172.21.15.5:6825/1991336051] conn(0x7f65e8073cc0 0x7f65e8076140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:10.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.311+0000 7f65ff7fe640 1 -- 172.21.15.5:0/2494362195 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f65f8000ed0 con 0x7f660806bf50 2024-03-24T10:59:10.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.312+0000 7f660759e640 1 --2- 172.21.15.5:0/2494362195 >> [v2:172.21.15.5:6824/1991336051,v1:172.21.15.5:6825/1991336051] conn(0x7f65e8073cc0 0x7f65e8076140 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:10.313 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.312+0000 7f65ff7fe640 1 -- 172.21.15.5:0/2494362195 <== osd.0 v2:172.21.15.5:6824/1991336051 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f65e8076810 con 0x7f65e8073cc0 2024-03-24T10:59:10.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.326+0000 7f660d0c9640 1 -- 172.21.15.5:0/2494362195 --> [v2:172.21.15.5:6824/1991336051,v1:172.21.15.5:6825/1991336051] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f660815ab90 con 0x7f65e8073cc0 2024-03-24T10:59:10.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.326+0000 7f65ff7fe640 1 -- 172.21.15.5:0/2494362195 <== osd.0 v2:172.21.15.5:6824/1991336051 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f660002ccb0 con 0x7f65e8073cc0 2024-03-24T10:59:10.327 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-24T10:59:10.327 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T10:59:10.327 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-24T10:59:10.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.326+0000 7f660d0c9640 1 -- 172.21.15.5:0/2494362195 >> [v2:172.21.15.5:6824/1991336051,v1:172.21.15.5:6825/1991336051] conn(0x7f65e8073cc0 msgr2=0x7f65e8076140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:10.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.326+0000 7f660d0c9640 1 --2- 172.21.15.5:0/2494362195 >> [v2:172.21.15.5:6824/1991336051,v1:172.21.15.5:6825/1991336051] conn(0x7f65e8073cc0 0x7f65e8076140 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:10.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.326+0000 7f660d0c9640 1 -- 172.21.15.5:0/2494362195 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f65e806b890 msgr2=0x7f65e806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:10.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.326+0000 7f660d0c9640 1 --2- 172.21.15.5:0/2494362195 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f65e806b890 0x7f65e806dd50 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f66080b2620 tx=0x7f65f0009040 comp rx=0 tx=0).stop 2024-03-24T10:59:10.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.326+0000 7f660d0c9640 1 -- 172.21.15.5:0/2494362195 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f660806bf50 msgr2=0x7f66080b2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:10.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.326+0000 7f660d0c9640 1 --2- 172.21.15.5:0/2494362195 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f660806bf50 0x7f66080b2030 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f65f800c430 tx=0x7f65f800c8f0 comp rx=0 tx=0).stop 2024-03-24T10:59:10.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.326+0000 7f660759e640 1 -- 172.21.15.5:0/2494362195 reap_dead start 2024-03-24T10:59:10.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.326+0000 7f660d0c9640 1 -- 172.21.15.5:0/2494362195 shutdown_connections 2024-03-24T10:59:10.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.326+0000 7f660d0c9640 1 -- 172.21.15.5:0/2494362195 >> 172.21.15.5:0/2494362195 conn(0x7f66080cfc30 msgr2=0x7f66080c7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:10.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.326+0000 7f660d0c9640 1 -- 172.21.15.5:0/2494362195 shutdown_connections 2024-03-24T10:59:10.329 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.326+0000 7f660d0c9640 1 -- 172.21.15.5:0/2494362195 wait complete. 2024-03-24T10:59:10.339 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-24T10:59:10.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.442+0000 7f37940f1640 1 Processor -- start 2024-03-24T10:59:10.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.442+0000 7f37940f1640 1 -- start start 2024-03-24T10:59:10.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.443+0000 7f37940f1640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f378c0c7800 0x7f378c0c3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:10.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.443+0000 7f37940f1640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f378c0c9340 0x7f378c150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:10.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.443+0000 7f37940f1640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f378c0c6710 con 0x7f378c0c7420 2024-03-24T10:59:10.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.443+0000 7f37940f1640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f378c0c49b0 con 0x7f378c0c9340 2024-03-24T10:59:10.444 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.443+0000 7f37940f1640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f378c0c41f0 con 0x7f378c0c7800 2024-03-24T10:59:10.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.443+0000 7f379268e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f378c0c9340 0x7f378c150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:10.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.444+0000 7f379268e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f378c0c9340 0x7f378c150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44114/0 (socket says 172.21.15.5:44114) 2024-03-24T10:59:10.445 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.444+0000 7f379268e640 1 -- 172.21.15.5:0/935969012 learned_addr learned my addr 172.21.15.5:0/935969012 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:10.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.446+0000 7f379168c640 1 --2- 172.21.15.5:0/935969012 >> v2:172.21.15.57:3300/0 conn(0x7f378c0c7800 0x7f378c0c3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:10.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.446+0000 7f379268e640 1 -- 172.21.15.5:0/935969012 >> v2:172.21.15.57:3300/0 conn(0x7f378c0c7800 msgr2=0x7f378c0c3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:10.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.446+0000 7f379268e640 1 --2- 172.21.15.5:0/935969012 >> v2:172.21.15.57:3300/0 conn(0x7f378c0c7800 0x7f378c0c3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:10.447 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.446+0000 7f379268e640 1 -- 172.21.15.5:0/935969012 >> v1:172.21.15.57:6789/0 conn(0x7f378c0c7420 legacy=0x7f378c15a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:10.448 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.446+0000 7f379268e640 1 -- 172.21.15.5:0/935969012 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f378c15ab80 con 0x7f378c0c9340 2024-03-24T10:59:10.451 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.450+0000 7f379268e640 1 --2- 172.21.15.5:0/935969012 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f378c0c9340 0x7f378c150ad0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f378c0c9210 tx=0x7f378402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a70dc862d7890c84 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:10.515 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.514+0000 7f3790e8b640 1 -- 172.21.15.5:0/935969012 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378402cb20 con 0x7f378c0c9340 2024-03-24T10:59:10.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.514+0000 7f3790e8b640 1 -- 172.21.15.5:0/935969012 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f378402cc80 con 0x7f378c0c9340 2024-03-24T10:59:10.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.514+0000 7f3790e8b640 1 -- 172.21.15.5:0/935969012 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37840343f0 con 0x7f378c0c9340 2024-03-24T10:59:10.516 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.515+0000 7f37940f1640 1 -- 172.21.15.5:0/935969012 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f378c0c9340 msgr2=0x7f378c150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:10.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.515+0000 7f37940f1640 1 --2- 172.21.15.5:0/935969012 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f378c0c9340 0x7f378c150ad0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f378c0c9210 tx=0x7f378402bf80 comp rx=0 tx=0).stop 2024-03-24T10:59:10.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.515+0000 7f37940f1640 1 -- 172.21.15.5:0/935969012 shutdown_connections 2024-03-24T10:59:10.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.515+0000 7f37940f1640 1 --2- 172.21.15.5:0/935969012 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f378c0c9340 0x7f378c150ad0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:10.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.515+0000 7f37940f1640 1 --2- 172.21.15.5:0/935969012 >> v2:172.21.15.57:3300/0 conn(0x7f378c0c7800 0x7f378c0c3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:10.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.515+0000 7f37940f1640 1 -- 172.21.15.5:0/935969012 >> 172.21.15.5:0/935969012 conn(0x7f378c06c240 msgr2=0x7f378c06c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:10.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.515+0000 7f37940f1640 1 -- 172.21.15.5:0/935969012 shutdown_connections 2024-03-24T10:59:10.517 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.515+0000 7f37940f1640 1 -- 172.21.15.5:0/935969012 wait complete. 2024-03-24T10:59:10.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.517+0000 7f37940f1640 1 Processor -- start 2024-03-24T10:59:10.518 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.517+0000 7f37940f1640 1 -- start start 2024-03-24T10:59:10.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.517+0000 7f37940f1640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f378c0c7420 0x7f378c14e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:10.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.517+0000 7f37940f1640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f378c0c7800 0x7f378c14afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:10.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.517+0000 7f37940f1640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f378c0c9340 0x7f378c1dfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:10.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.517+0000 7f37940f1640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f378c14f0a0 con 0x7f378c0c7420 2024-03-24T10:59:10.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.517+0000 7f37940f1640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f378c1ed8c0 con 0x7f378c0c9340 2024-03-24T10:59:10.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.517+0000 7f37940f1640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f378c1ee850 con 0x7f378c0c7800 2024-03-24T10:59:10.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.518+0000 7f3791e8d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f378c0c7420 0x7f378c14e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:10.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.518+0000 7f3791e8d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f378c0c7420 0x7f378c14e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44118/0 (socket says 172.21.15.5:44118) 2024-03-24T10:59:10.519 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.518+0000 7f3791e8d640 1 -- 172.21.15.5:0/737005635 learned_addr learned my addr 172.21.15.5:0/737005635 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:10.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.522+0000 7f3791e8d640 1 -- 172.21.15.5:0/737005635 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f378c0c7800 msgr2=0x7f378c14afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:10.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.522+0000 7f3791e8d640 1 --2- 172.21.15.5:0/737005635 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f378c0c7800 0x7f378c14afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:10.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.522+0000 7f3791e8d640 1 -- 172.21.15.5:0/737005635 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f378c0c9340 msgr2=0x7f378c1dfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:10.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.522+0000 7f3791e8d640 1 --2- 172.21.15.5:0/737005635 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f378c0c9340 0x7f378c1dfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:10.523 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.522+0000 7f3791e8d640 1 -- 172.21.15.5:0/737005635 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3784009b30 con 0x7f378c0c7420 2024-03-24T10:59:10.525 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.524+0000 7f3791e8d640 1 --2- 172.21.15.5:0/737005635 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f378c0c7420 0x7f378c14e940 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f378800c440 tx=0x7f378800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:10.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.529+0000 7f37827fc640 1 -- 172.21.15.5:0/737005635 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3788005480 con 0x7f378c0c7420 2024-03-24T10:59:10.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.529+0000 7f37827fc640 1 -- 172.21.15.5:0/737005635 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37880035f0 con 0x7f378c0c7420 2024-03-24T10:59:10.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.529+0000 7f37940f1640 1 -- 172.21.15.5:0/737005635 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f378c1edad0 con 0x7f378c0c7420 2024-03-24T10:59:10.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.529+0000 7f37827fc640 1 -- 172.21.15.5:0/737005635 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37880039c0 con 0x7f378c0c7420 2024-03-24T10:59:10.530 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.529+0000 7f37940f1640 1 -- 172.21.15.5:0/737005635 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f378c1edd70 con 0x7f378c0c7420 2024-03-24T10:59:10.531 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.530+0000 7f37940f1640 1 -- 172.21.15.5:0/737005635 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f378c0688e0 con 0x7f378c0c7420 2024-03-24T10:59:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.823+0000 7f37827fc640 1 -- 172.21.15.5:0/737005635 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f378801b3b0 con 0x7f378c0c7420 2024-03-24T10:59:10.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.823+0000 7f37827fc640 1 --2- 172.21.15.5:0/737005635 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f377006b890 0x7f377006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:10.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.824+0000 7f379168c640 1 --2- 172.21.15.5:0/737005635 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f377006b890 0x7f377006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:10.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.824+0000 7f379168c640 1 --2- 172.21.15.5:0/737005635 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f377006b890 0x7f377006dd50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f377c006fd0 tx=0x7f377c008040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:10.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.848+0000 7f37827fc640 1 -- 172.21.15.5:0/737005635 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f3788058a10 con 0x7f378c0c7420 2024-03-24T10:59:10.849 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:10.848+0000 7f37827fc640 1 -- 172.21.15.5:0/737005635 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f37880545b0 con 0x7f378c0c7420 2024-03-24T10:59:11.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.036+0000 7f37940f1640 1 -- 172.21.15.5:0/737005635 --> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f378c0c46d0 con 0x7f377006b890 2024-03-24T10:59:11.037 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.036+0000 7f37827fc640 1 -- 172.21.15.5:0/737005635 <== mgr.4120 v2:172.21.15.5:6832/1102439181 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f377c00ae20 con 0x7f377006b890 2024-03-24T10:59:11.037 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 0 to deep-scrub 2024-03-24T10:59:11.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.038+0000 7f37940f1640 1 -- 172.21.15.5:0/737005635 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f377006b890 msgr2=0x7f377006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:11.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.038+0000 7f37940f1640 1 --2- 172.21.15.5:0/737005635 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f377006b890 0x7f377006dd50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f377c006fd0 tx=0x7f377c008040 comp rx=0 tx=0).stop 2024-03-24T10:59:11.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.038+0000 7f37940f1640 1 -- 172.21.15.5:0/737005635 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f378c0c7420 msgr2=0x7f378c14e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:11.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.038+0000 7f37940f1640 1 --2- 172.21.15.5:0/737005635 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f378c0c7420 0x7f378c14e940 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f378800c440 tx=0x7f378800c900 comp rx=0 tx=0).stop 2024-03-24T10:59:11.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.039+0000 7f37940f1640 1 -- 172.21.15.5:0/737005635 shutdown_connections 2024-03-24T10:59:11.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.039+0000 7f37940f1640 1 --2- 172.21.15.5:0/737005635 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f377006b890 0x7f377006dd50 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.039+0000 7f37940f1640 1 --2- 172.21.15.5:0/737005635 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f378c0c9340 0x7f378c1dfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.039+0000 7f37940f1640 1 --2- 172.21.15.5:0/737005635 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f378c0c7800 0x7f378c14afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.039+0000 7f37940f1640 1 --2- 172.21.15.5:0/737005635 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f378c0c7420 0x7f378c14e940 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.039+0000 7f37940f1640 1 -- 172.21.15.5:0/737005635 >> 172.21.15.5:0/737005635 conn(0x7f378c06c240 msgr2=0x7f378c1e3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:11.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.039+0000 7f37940f1640 1 -- 172.21.15.5:0/737005635 shutdown_connections 2024-03-24T10:59:11.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.039+0000 7f37940f1640 1 -- 172.21.15.5:0/737005635 wait complete. 2024-03-24T10:59:11.050 INFO:tasks.ceph:Scrubbing osd.1 2024-03-24T10:59:11.051 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-24T10:59:11.156 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.155+0000 7f0f5a20e640 1 Processor -- start 2024-03-24T10:59:11.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.155+0000 7f0f5a20e640 1 -- start start 2024-03-24T10:59:11.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.156+0000 7f0f5a20e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f540cf340 0x7f0f540cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:11.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.156+0000 7f0f5a20e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f540cc3e0 con 0x7f0f540cc000 2024-03-24T10:59:11.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.156+0000 7f0f5a20e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f5415a5f0 con 0x7f0f540ccb80 2024-03-24T10:59:11.157 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.156+0000 7f0f5a20e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0f5415a400 con 0x7f0f540cf340 2024-03-24T10:59:11.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.157+0000 7f0f52ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f540cf340 0x7f0f540cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:11.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.157+0000 7f0f537fe640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f0f540cc000 0x7f0f5415e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:54550/0 (socket says 172.21.15.5:54550) 2024-03-24T10:59:11.158 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.157+0000 7f0f537fe640 1 -- 172.21.15.5:0/1251534272 learned_addr learned my addr 172.21.15.5:0/1251534272 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:59:11.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.160+0000 7f0f52ffd640 1 -- 172.21.15.5:0/1251534272 >> v1:172.21.15.57:6789/0 conn(0x7f0f540ccb80 legacy=0x7f0f5406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:11.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.160+0000 7f0f52ffd640 1 -- 172.21.15.5:0/1251534272 >> v1:172.21.15.5:6789/0 conn(0x7f0f540cc000 legacy=0x7f0f5415e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:11.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.160+0000 7f0f52ffd640 1 -- 172.21.15.5:0/1251534272 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f5415a8b0 con 0x7f0f540cf340 2024-03-24T10:59:11.164 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.163+0000 7f0f52ffd640 1 --2- 172.21.15.5:0/1251534272 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f540cf340 0x7f0f540cc640 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0f480097d0 tx=0x7f0f4802b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d61a8032f34693d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:11.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.167+0000 7f0f527fc640 1 -- 172.21.15.5:0/1251534272 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f48030880 con 0x7f0f540cf340 2024-03-24T10:59:11.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.167+0000 7f0f527fc640 1 -- 172.21.15.5:0/1251534272 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f480309e0 con 0x7f0f540cf340 2024-03-24T10:59:11.168 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.167+0000 7f0f527fc640 1 -- 172.21.15.5:0/1251534272 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f48030d70 con 0x7f0f540cf340 2024-03-24T10:59:11.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.168+0000 7f0f5a20e640 1 -- 172.21.15.5:0/1251534272 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f540cf340 msgr2=0x7f0f540cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:11.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.168+0000 7f0f5a20e640 1 --2- 172.21.15.5:0/1251534272 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f540cf340 0x7f0f540cc640 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0f480097d0 tx=0x7f0f4802b9c0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.168+0000 7f0f5a20e640 1 -- 172.21.15.5:0/1251534272 shutdown_connections 2024-03-24T10:59:11.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.168+0000 7f0f5a20e640 1 --2- 172.21.15.5:0/1251534272 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f540cf340 0x7f0f540cc640 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.168+0000 7f0f5a20e640 1 -- 172.21.15.5:0/1251534272 >> 172.21.15.5:0/1251534272 conn(0x7f0f540cfc30 msgr2=0x7f0f540d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:11.169 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.168+0000 7f0f5a20e640 1 -- 172.21.15.5:0/1251534272 shutdown_connections 2024-03-24T10:59:11.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.168+0000 7f0f5a20e640 1 -- 172.21.15.5:0/1251534272 wait complete. 2024-03-24T10:59:11.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.169+0000 7f0f5a20e640 1 Processor -- start 2024-03-24T10:59:11.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.169+0000 7f0f5a20e640 1 -- start start 2024-03-24T10:59:11.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.170+0000 7f0f5a20e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f540cc000 0x7f0f54155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:11.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.170+0000 7f0f5a20e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f540ccb80 0x7f0f5415c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:11.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.170+0000 7f0f5a20e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f540cf340 0x7f0f541567a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:11.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.170+0000 7f0f5a20e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f0f540c6860 con 0x7f0f540ccb80 2024-03-24T10:59:11.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.170+0000 7f0f5a20e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f0f541f46c0 con 0x7f0f540cc000 2024-03-24T10:59:11.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.170+0000 7f0f5a20e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f0f541f3d40 con 0x7f0f540cf340 2024-03-24T10:59:11.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.170+0000 7f0f53fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f540cf340 0x7f0f541567a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:11.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.170+0000 7f0f52ffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f540ccb80 0x7f0f5415c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:11.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.171+0000 7f0f537fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f540cc000 0x7f0f54155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:11.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.171+0000 7f0f53fff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f540cf340 0x7f0f541567a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:48614/0 (socket says 172.21.15.5:48614) 2024-03-24T10:59:11.172 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.171+0000 7f0f53fff640 1 -- 172.21.15.5:0/2789698124 learned_addr learned my addr 172.21.15.5:0/2789698124 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:11.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.174+0000 7f0f52ffd640 1 -- 172.21.15.5:0/2789698124 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f540cf340 msgr2=0x7f0f541567a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:11.175 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.174+0000 7f0f52ffd640 1 --2- 172.21.15.5:0/2789698124 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f0f540cf340 0x7f0f541567a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.174+0000 7f0f52ffd640 1 -- 172.21.15.5:0/2789698124 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f540cc000 msgr2=0x7f0f54155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:11.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.174+0000 7f0f52ffd640 1 --2- 172.21.15.5:0/2789698124 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f0f540cc000 0x7f0f54155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.176 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.174+0000 7f0f52ffd640 1 -- 172.21.15.5:0/2789698124 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f48009400 con 0x7f0f540ccb80 2024-03-24T10:59:11.178 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.177+0000 7f0f52ffd640 1 --2- 172.21.15.5:0/2789698124 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f540ccb80 0x7f0f5415c590 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f0f540cc590 tx=0x7f0f480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:11.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.182+0000 7f0f3bfff640 1 -- 172.21.15.5:0/2789698124 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f4803e070 con 0x7f0f540ccb80 2024-03-24T10:59:11.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.182+0000 7f0f3bfff640 1 -- 172.21.15.5:0/2789698124 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f48030840 con 0x7f0f540ccb80 2024-03-24T10:59:11.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.182+0000 7f0f5a20e640 1 -- 172.21.15.5:0/2789698124 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f540c4860 con 0x7f0f540ccb80 2024-03-24T10:59:11.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.182+0000 7f0f3bfff640 1 -- 172.21.15.5:0/2789698124 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f480423f0 con 0x7f0f540ccb80 2024-03-24T10:59:11.183 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.182+0000 7f0f5a20e640 1 -- 172.21.15.5:0/2789698124 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f541f4000 con 0x7f0f540ccb80 2024-03-24T10:59:11.184 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.183+0000 7f0f5a20e640 1 -- 172.21.15.5:0/2789698124 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0f54158a40 con 0x7f0f540ccb80 2024-03-24T10:59:11.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.213+0000 7f0f3bfff640 1 -- 172.21.15.5:0/2789698124 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0f48042550 con 0x7f0f540ccb80 2024-03-24T10:59:11.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.213+0000 7f0f3bfff640 1 --2- 172.21.15.5:0/2789698124 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0f4006b810 0x7f0f4006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:11.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.213+0000 7f0f537fe640 1 --2- 172.21.15.5:0/2789698124 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0f4006b810 0x7f0f4006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:11.215 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.214+0000 7f0f537fe640 1 --2- 172.21.15.5:0/2789698124 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0f4006b810 0x7f0f4006dcd0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0f44000f60 tx=0x7f0f44009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:11.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.220+0000 7f0f3bfff640 1 -- 172.21.15.5:0/2789698124 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f0f4807b3c0 con 0x7f0f540ccb80 2024-03-24T10:59:11.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.220+0000 7f0f3bfff640 1 --2- 172.21.15.5:0/2789698124 >> [v2:172.21.15.5:6800/2140667501,v1:172.21.15.5:6801/2140667501] conn(0x7f0f40073c40 0x7f0f400760c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:11.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.220+0000 7f0f3bfff640 1 -- 172.21.15.5:0/2789698124 --> [v2:172.21.15.5:6800/2140667501,v1:172.21.15.5:6801/2140667501] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0f40076790 con 0x7f0f40073c40 2024-03-24T10:59:11.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.220+0000 7f0f53fff640 1 --2- 172.21.15.5:0/2789698124 >> [v2:172.21.15.5:6800/2140667501,v1:172.21.15.5:6801/2140667501] conn(0x7f0f40073c40 0x7f0f400760c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:11.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.220+0000 7f0f53fff640 1 --2- 172.21.15.5:0/2789698124 >> [v2:172.21.15.5:6800/2140667501,v1:172.21.15.5:6801/2140667501] conn(0x7f0f40073c40 0x7f0f400760c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:11.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.221+0000 7f0f3bfff640 1 -- 172.21.15.5:0/2789698124 <== osd.1 v2:172.21.15.5:6800/2140667501 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0f40076790 con 0x7f0f40073c40 2024-03-24T10:59:11.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.222+0000 7f0f3bfff640 1 -- 172.21.15.5:0/2789698124 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0f480776a0 con 0x7f0f540ccb80 2024-03-24T10:59:11.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.235+0000 7f0f5a20e640 1 -- 172.21.15.5:0/2789698124 --> [v2:172.21.15.5:6800/2140667501,v1:172.21.15.5:6801/2140667501] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f0f540c87c0 con 0x7f0f40073c40 2024-03-24T10:59:11.236 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.235+0000 7f0f3bfff640 1 -- 172.21.15.5:0/2789698124 <== osd.1 v2:172.21.15.5:6800/2140667501 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f0f540c87c0 con 0x7f0f40073c40 2024-03-24T10:59:11.236 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-24T10:59:11.236 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T10:59:11.236 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-24T10:59:11.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.235+0000 7f0f5a20e640 1 -- 172.21.15.5:0/2789698124 >> [v2:172.21.15.5:6800/2140667501,v1:172.21.15.5:6801/2140667501] conn(0x7f0f40073c40 msgr2=0x7f0f400760c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:11.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.235+0000 7f0f5a20e640 1 --2- 172.21.15.5:0/2789698124 >> [v2:172.21.15.5:6800/2140667501,v1:172.21.15.5:6801/2140667501] conn(0x7f0f40073c40 0x7f0f400760c0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.235+0000 7f0f5a20e640 1 -- 172.21.15.5:0/2789698124 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0f4006b810 msgr2=0x7f0f4006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:11.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.235+0000 7f0f5a20e640 1 --2- 172.21.15.5:0/2789698124 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f0f4006b810 0x7f0f4006dcd0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0f44000f60 tx=0x7f0f44009040 comp rx=0 tx=0).stop 2024-03-24T10:59:11.237 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.235+0000 7f0f5a20e640 1 -- 172.21.15.5:0/2789698124 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f540ccb80 msgr2=0x7f0f5415c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:11.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.235+0000 7f0f5a20e640 1 --2- 172.21.15.5:0/2789698124 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f0f540ccb80 0x7f0f5415c590 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f0f540cc590 tx=0x7f0f480046c0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.235+0000 7f0f53fff640 1 -- 172.21.15.5:0/2789698124 reap_dead start 2024-03-24T10:59:11.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.235+0000 7f0f5a20e640 1 -- 172.21.15.5:0/2789698124 shutdown_connections 2024-03-24T10:59:11.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.235+0000 7f0f5a20e640 1 -- 172.21.15.5:0/2789698124 >> 172.21.15.5:0/2789698124 conn(0x7f0f540cfc30 msgr2=0x7f0f540c71d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:11.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.235+0000 7f0f5a20e640 1 -- 172.21.15.5:0/2789698124 shutdown_connections 2024-03-24T10:59:11.238 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.235+0000 7f0f5a20e640 1 -- 172.21.15.5:0/2789698124 wait complete. 2024-03-24T10:59:11.247 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-24T10:59:11.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.349+0000 7f1b11f3e640 1 Processor -- start 2024-03-24T10:59:11.350 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.349+0000 7f1b11f3e640 1 -- start start 2024-03-24T10:59:11.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.350+0000 7f1b11f3e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1b0c152610 0x7f1b0c158b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:11.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.350+0000 7f1b11f3e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f1b0c0c9560 0x7f1b0c15bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:11.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.350+0000 7f1b11f3e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b0c0c9f60 0x7f1b0c0c8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:11.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.350+0000 7f1b11f3e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f1b0c1590c0 con 0x7f1b0c152610 2024-03-24T10:59:11.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.350+0000 7f1b11f3e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f1b0c15ba80 con 0x7f1b0c0c9560 2024-03-24T10:59:11.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.350+0000 7f1b11f3e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1b0c0ca530 con 0x7f1b0c0c9f60 2024-03-24T10:59:11.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.350+0000 7f1b0b7fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1b0c152610 0x7f1b0c158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:11.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.351+0000 7f1b0bfff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b0c0c9f60 0x7f1b0c0c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:11.352 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.351+0000 7f1b0b7fe640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f1b0c152610 0x7f1b0c158b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44142/0 (socket says 172.21.15.5:44142) 2024-03-24T10:59:11.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.351+0000 7f1b0b7fe640 1 -- 172.21.15.5:0/821335478 learned_addr learned my addr 172.21.15.5:0/821335478 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:11.353 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.351+0000 7f1b0affd640 1 --2- 172.21.15.5:0/821335478 >> v2:172.21.15.57:3300/0 conn(0x7f1b0c0c9560 0x7f1b0c15bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:11.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.353+0000 7f1b0b7fe640 1 -- 172.21.15.5:0/821335478 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b0c0c9f60 msgr2=0x7f1b0c0c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:11.354 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.353+0000 7f1b0b7fe640 1 --2- 172.21.15.5:0/821335478 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b0c0c9f60 0x7f1b0c0c8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.353+0000 7f1b0b7fe640 1 -- 172.21.15.5:0/821335478 >> v2:172.21.15.57:3300/0 conn(0x7f1b0c0c9560 msgr2=0x7f1b0c15bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:11.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.353+0000 7f1b0b7fe640 1 --2- 172.21.15.5:0/821335478 >> v2:172.21.15.57:3300/0 conn(0x7f1b0c0c9560 0x7f1b0c15bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.355 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.353+0000 7f1b0b7fe640 1 -- 172.21.15.5:0/821335478 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b0c0ca810 con 0x7f1b0c152610 2024-03-24T10:59:11.356 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.355+0000 7f1b0b7fe640 1 --2- 172.21.15.5:0/821335478 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b0c152610 0x7f1b0c158b80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f1afc009870 tx=0x7f1afc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2cd431738ad2fe4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:11.359 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.358+0000 7f1b0a7fc640 1 -- 172.21.15.5:0/821335478 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1afc030840 con 0x7f1b0c152610 2024-03-24T10:59:11.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.360+0000 7f1b0a7fc640 1 -- 172.21.15.5:0/821335478 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1afc0309a0 con 0x7f1b0c152610 2024-03-24T10:59:11.361 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.360+0000 7f1b0a7fc640 1 -- 172.21.15.5:0/821335478 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1afc030d30 con 0x7f1b0c152610 2024-03-24T10:59:11.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.361+0000 7f1b11f3e640 1 -- 172.21.15.5:0/821335478 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b0c152610 msgr2=0x7f1b0c158b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:11.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.361+0000 7f1b11f3e640 1 --2- 172.21.15.5:0/821335478 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b0c152610 0x7f1b0c158b80 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f1afc009870 tx=0x7f1afc02b980 comp rx=0 tx=0).stop 2024-03-24T10:59:11.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.361+0000 7f1b11f3e640 1 -- 172.21.15.5:0/821335478 shutdown_connections 2024-03-24T10:59:11.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.361+0000 7f1b11f3e640 1 --2- 172.21.15.5:0/821335478 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b0c0c9f60 0x7f1b0c0c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.362 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.361+0000 7f1b11f3e640 1 --2- 172.21.15.5:0/821335478 >> v2:172.21.15.57:3300/0 conn(0x7f1b0c0c9560 0x7f1b0c15bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.361+0000 7f1b11f3e640 1 --2- 172.21.15.5:0/821335478 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b0c152610 0x7f1b0c158b80 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.361+0000 7f1b11f3e640 1 -- 172.21.15.5:0/821335478 >> 172.21.15.5:0/821335478 conn(0x7f1b0c0711f0 msgr2=0x7f1b0c0715d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:11.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.361+0000 7f1b11f3e640 1 -- 172.21.15.5:0/821335478 shutdown_connections 2024-03-24T10:59:11.363 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.361+0000 7f1b11f3e640 1 -- 172.21.15.5:0/821335478 wait complete. 2024-03-24T10:59:11.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.362+0000 7f1b11f3e640 1 Processor -- start 2024-03-24T10:59:11.364 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.363+0000 7f1b11f3e640 1 -- start start 2024-03-24T10:59:11.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.363+0000 7f1b11f3e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b0c0c9560 0x7f1b0c1ef8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:11.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.363+0000 7f1b11f3e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b0c0c9f60 0x7f1b0c150e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:11.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.363+0000 7f1b11f3e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b0c152610 0x7f1b0c0af710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:11.365 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.363+0000 7f1b11f3e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0c155a10 con 0x7f1b0c0c9560 2024-03-24T10:59:11.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.363+0000 7f1b11f3e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0c154bb0 con 0x7f1b0c152610 2024-03-24T10:59:11.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.363+0000 7f1b11f3e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f1b0c154e90 con 0x7f1b0c0c9f60 2024-03-24T10:59:11.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.364+0000 7f1b0b7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b0c0c9560 0x7f1b0c1ef8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:11.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.364+0000 7f1b0affd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b0c0c9f60 0x7f1b0c150e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:11.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.364+0000 7f1b0bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b0c152610 0x7f1b0c0af710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:11.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.364+0000 7f1b0b7fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b0c0c9560 0x7f1b0c1ef8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44152/0 (socket says 172.21.15.5:44152) 2024-03-24T10:59:11.366 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.364+0000 7f1b0b7fe640 1 -- 172.21.15.5:0/3616263089 learned_addr learned my addr 172.21.15.5:0/3616263089 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:11.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.368+0000 7f1b0b7fe640 1 -- 172.21.15.5:0/3616263089 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b0c0c9f60 msgr2=0x7f1b0c150e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:11.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.368+0000 7f1b0b7fe640 1 --2- 172.21.15.5:0/3616263089 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b0c0c9f60 0x7f1b0c150e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.368+0000 7f1b0b7fe640 1 -- 172.21.15.5:0/3616263089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b0c152610 msgr2=0x7f1b0c0af710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:11.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.368+0000 7f1b0b7fe640 1 --2- 172.21.15.5:0/3616263089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b0c152610 0x7f1b0c0af710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.369 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.368+0000 7f1b0b7fe640 1 -- 172.21.15.5:0/3616263089 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1afc0094d0 con 0x7f1b0c0c9560 2024-03-24T10:59:11.371 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.370+0000 7f1b0b7fe640 1 --2- 172.21.15.5:0/3616263089 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b0c0c9560 0x7f1b0c1ef8b0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f1afc000c00 tx=0x7f1afc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:11.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.375+0000 7f1af3fff640 1 -- 172.21.15.5:0/3616263089 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1afc039c90 con 0x7f1b0c0c9560 2024-03-24T10:59:11.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.375+0000 7f1af3fff640 1 -- 172.21.15.5:0/3616263089 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1afc039df0 con 0x7f1b0c0c9560 2024-03-24T10:59:11.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.375+0000 7f1af3fff640 1 -- 172.21.15.5:0/3616263089 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1afc043640 con 0x7f1b0c0c9560 2024-03-24T10:59:11.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.375+0000 7f1b11f3e640 1 -- 172.21.15.5:0/3616263089 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b0c155170 con 0x7f1b0c0c9560 2024-03-24T10:59:11.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.375+0000 7f1b11f3e640 1 -- 172.21.15.5:0/3616263089 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b0c155730 con 0x7f1b0c0c9560 2024-03-24T10:59:11.377 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.376+0000 7f1b11f3e640 1 -- 172.21.15.5:0/3616263089 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b0c0688e0 con 0x7f1b0c0c9560 2024-03-24T10:59:11.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.664+0000 7f1af3fff640 1 -- 172.21.15.5:0/3616263089 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1afc0437a0 con 0x7f1b0c0c9560 2024-03-24T10:59:11.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.664+0000 7f1af3fff640 1 --2- 172.21.15.5:0/3616263089 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1af806b860 0x7f1af806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:11.666 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.664+0000 7f1b0affd640 1 --2- 172.21.15.5:0/3616263089 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1af806b860 0x7f1af806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:11.667 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.665+0000 7f1b0affd640 1 --2- 172.21.15.5:0/3616263089 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1af806b860 0x7f1af806dd20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c159440 tx=0x7f1b00009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:11.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.680+0000 7f1af3fff640 1 -- 172.21.15.5:0/3616263089 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f1afc03e070 con 0x7f1b0c0c9560 2024-03-24T10:59:11.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.680+0000 7f1af3fff640 1 -- 172.21.15.5:0/3616263089 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1afc078560 con 0x7f1b0c0c9560 2024-03-24T10:59:11.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.887+0000 7f1b11f3e640 1 -- 172.21.15.5:0/3616263089 --> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f1b0c0cadd0 con 0x7f1af806b860 2024-03-24T10:59:11.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.887+0000 7f1af3fff640 1 -- 172.21.15.5:0/3616263089 <== mgr.4120 v2:172.21.15.5:6832/1102439181 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1b00004020 con 0x7f1af806b860 2024-03-24T10:59:11.888 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 1 to deep-scrub 2024-03-24T10:59:11.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.890+0000 7f1b11f3e640 1 -- 172.21.15.5:0/3616263089 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1af806b860 msgr2=0x7f1af806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:11.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.890+0000 7f1b11f3e640 1 --2- 172.21.15.5:0/3616263089 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1af806b860 0x7f1af806dd20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c159440 tx=0x7f1b00009190 comp rx=0 tx=0).stop 2024-03-24T10:59:11.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.890+0000 7f1b11f3e640 1 -- 172.21.15.5:0/3616263089 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b0c0c9560 msgr2=0x7f1b0c1ef8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:11.891 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.890+0000 7f1b11f3e640 1 --2- 172.21.15.5:0/3616263089 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b0c0c9560 0x7f1b0c1ef8b0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f1afc000c00 tx=0x7f1afc004730 comp rx=0 tx=0).stop 2024-03-24T10:59:11.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.890+0000 7f1b11f3e640 1 -- 172.21.15.5:0/3616263089 shutdown_connections 2024-03-24T10:59:11.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.890+0000 7f1b11f3e640 1 --2- 172.21.15.5:0/3616263089 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f1af806b860 0x7f1af806dd20 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.890+0000 7f1b11f3e640 1 --2- 172.21.15.5:0/3616263089 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f1b0c152610 0x7f1b0c0af710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.890+0000 7f1b11f3e640 1 --2- 172.21.15.5:0/3616263089 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f1b0c0c9f60 0x7f1b0c150e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.890+0000 7f1b11f3e640 1 --2- 172.21.15.5:0/3616263089 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f1b0c0c9560 0x7f1b0c1ef8b0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:11.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.890+0000 7f1b11f3e640 1 -- 172.21.15.5:0/3616263089 >> 172.21.15.5:0/3616263089 conn(0x7f1b0c0711f0 msgr2=0x7f1b0c0c3940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:11.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.890+0000 7f1b11f3e640 1 -- 172.21.15.5:0/3616263089 shutdown_connections 2024-03-24T10:59:11.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:11.890+0000 7f1b11f3e640 1 -- 172.21.15.5:0/3616263089 wait complete. 2024-03-24T10:59:11.902 INFO:tasks.ceph:Scrubbing osd.2 2024-03-24T10:59:11.902 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-24T10:59:12.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.011+0000 7f3b76b56640 1 Processor -- start 2024-03-24T10:59:12.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.011+0000 7f3b76b56640 1 -- start start 2024-03-24T10:59:12.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.012+0000 7f3b76b56640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3b700cc000 0x7f3b7015e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:12.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.012+0000 7f3b76b56640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3b700cf340 0x7f3b700cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:12.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.012+0000 7f3b76b56640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b700cc3e0 con 0x7f3b700ccb80 2024-03-24T10:59:12.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.012+0000 7f3b76b56640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f3b7015a400 con 0x7f3b700cc000 2024-03-24T10:59:12.013 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.012+0000 7f3b76b56640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f3b70154bd0 con 0x7f3b700cf340 2024-03-24T10:59:12.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.012+0000 7f3b748f2640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3b700cc000 0x7f3b7015e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:12.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.012+0000 7f3b6ffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3b700cf340 0x7f3b700cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:12.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.013+0000 7f3b748f2640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f3b700cc000 0x7f3b7015e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44158/0 (socket says 172.21.15.5:44158) 2024-03-24T10:59:12.014 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.013+0000 7f3b748f2640 1 -- 172.21.15.5:0/961147067 learned_addr learned my addr 172.21.15.5:0/961147067 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:12.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.015+0000 7f3b748f2640 1 -- 172.21.15.5:0/961147067 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3b700cf340 msgr2=0x7f3b700cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:12.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.015+0000 7f3b748f2640 1 --2- 172.21.15.5:0/961147067 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3b700cf340 0x7f3b700cc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.015+0000 7f3b748f2640 1 -- 172.21.15.5:0/961147067 >> v1:172.21.15.5:6789/0 conn(0x7f3b700ccb80 legacy=0x7f3b7006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:12.016 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.015+0000 7f3b748f2640 1 -- 172.21.15.5:0/961147067 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b7015a5f0 con 0x7f3b700cc000 2024-03-24T10:59:12.018 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.017+0000 7f3b748f2640 1 --2- 172.21.15.5:0/961147067 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b700cc000 0x7f3b7015e3b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f3b700b27b0 tx=0x7f3b6402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f03f1d64be4cb51e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:12.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.022+0000 7f3b6f7fe640 1 -- 172.21.15.5:0/961147067 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b64030840 con 0x7f3b700cc000 2024-03-24T10:59:12.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.022+0000 7f3b6f7fe640 1 -- 172.21.15.5:0/961147067 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b640309a0 con 0x7f3b700cc000 2024-03-24T10:59:12.023 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.022+0000 7f3b6f7fe640 1 -- 172.21.15.5:0/961147067 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b64030d30 con 0x7f3b700cc000 2024-03-24T10:59:12.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.023+0000 7f3b76b56640 1 -- 172.21.15.5:0/961147067 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b700cc000 msgr2=0x7f3b7015e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:12.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.023+0000 7f3b76b56640 1 --2- 172.21.15.5:0/961147067 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b700cc000 0x7f3b7015e3b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f3b700b27b0 tx=0x7f3b6402b980 comp rx=0 tx=0).stop 2024-03-24T10:59:12.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.023+0000 7f3b76b56640 1 -- 172.21.15.5:0/961147067 shutdown_connections 2024-03-24T10:59:12.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.023+0000 7f3b76b56640 1 --2- 172.21.15.5:0/961147067 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3b700cf340 0x7f3b700cc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.023+0000 7f3b76b56640 1 --2- 172.21.15.5:0/961147067 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b700cc000 0x7f3b7015e3b0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.023+0000 7f3b76b56640 1 -- 172.21.15.5:0/961147067 >> 172.21.15.5:0/961147067 conn(0x7f3b700cfc30 msgr2=0x7f3b700d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:12.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.023+0000 7f3b76b56640 1 -- 172.21.15.5:0/961147067 shutdown_connections 2024-03-24T10:59:12.024 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.023+0000 7f3b76b56640 1 -- 172.21.15.5:0/961147067 wait complete. 2024-03-24T10:59:12.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.024+0000 7f3b76b56640 1 Processor -- start 2024-03-24T10:59:12.025 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.024+0000 7f3b76b56640 1 -- start start 2024-03-24T10:59:12.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.025+0000 7f3b76b56640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b700cc000 0x7f3b701f3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:12.026 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.025+0000 7f3b76b56640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3b700ccb80 0x7f3b701f0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:12.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.025+0000 7f3b76b56640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b700cf340 0x7f3b7014f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:12.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.025+0000 7f3b76b56640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f3b70156350 con 0x7f3b700cc000 2024-03-24T10:59:12.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.025+0000 7f3b76b56640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f3b701f03b0 con 0x7f3b700cf340 2024-03-24T10:59:12.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.025+0000 7f3b76b56640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f3b7014f810 con 0x7f3b700ccb80 2024-03-24T10:59:12.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.026+0000 7f3b748f2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b700cc000 0x7f3b701f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:12.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.026+0000 7f3b6ffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3b700ccb80 0x7f3b701f0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:12.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.026+0000 7f3b748f2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b700cc000 0x7f3b701f3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:44164/0 (socket says 172.21.15.5:44164) 2024-03-24T10:59:12.027 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.026+0000 7f3b748f2640 1 -- 172.21.15.5:0/4135382727 learned_addr learned my addr 172.21.15.5:0/4135382727 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:12.028 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.027+0000 7f3b750f3640 1 --2- 172.21.15.5:0/4135382727 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b700cf340 0x7f3b7014f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:12.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.030+0000 7f3b748f2640 1 -- 172.21.15.5:0/4135382727 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3b700ccb80 msgr2=0x7f3b701f0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:12.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.030+0000 7f3b748f2640 1 --2- 172.21.15.5:0/4135382727 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f3b700ccb80 0x7f3b701f0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.030+0000 7f3b748f2640 1 -- 172.21.15.5:0/4135382727 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b700cf340 msgr2=0x7f3b7014f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:12.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.030+0000 7f3b748f2640 1 --2- 172.21.15.5:0/4135382727 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f3b700cf340 0x7f3b7014f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.031 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.030+0000 7f3b748f2640 1 -- 172.21.15.5:0/4135382727 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b640094d0 con 0x7f3b700cc000 2024-03-24T10:59:12.035 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.034+0000 7f3b748f2640 1 --2- 172.21.15.5:0/4135382727 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b700cc000 0x7f3b701f3200 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f3b64000c00 tx=0x7f3b64004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:12.040 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.039+0000 7f3b6d7fa640 1 -- 172.21.15.5:0/4135382727 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b64039c90 con 0x7f3b700cc000 2024-03-24T10:59:12.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.039+0000 7f3b6d7fa640 1 -- 172.21.15.5:0/4135382727 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b64039df0 con 0x7f3b700cc000 2024-03-24T10:59:12.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.039+0000 7f3b76b56640 1 -- 172.21.15.5:0/4135382727 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b701efda0 con 0x7f3b700cc000 2024-03-24T10:59:12.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.039+0000 7f3b6d7fa640 1 -- 172.21.15.5:0/4135382727 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b64043640 con 0x7f3b700cc000 2024-03-24T10:59:12.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.039+0000 7f3b76b56640 1 -- 172.21.15.5:0/4135382727 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b70156f90 con 0x7f3b700cc000 2024-03-24T10:59:12.041 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.040+0000 7f3b76b56640 1 -- 172.21.15.5:0/4135382727 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3b7015ae70 con 0x7f3b700cc000 2024-03-24T10:59:12.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.071+0000 7f3b6d7fa640 1 -- 172.21.15.5:0/4135382727 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3b640437a0 con 0x7f3b700cc000 2024-03-24T10:59:12.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.072+0000 7f3b6d7fa640 1 --2- 172.21.15.5:0/4135382727 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3b5406b860 0x7f3b5406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:12.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.072+0000 7f3b6ffff640 1 --2- 172.21.15.5:0/4135382727 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3b5406b860 0x7f3b5406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:12.073 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.072+0000 7f3b6ffff640 1 --2- 172.21.15.5:0/4135382727 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3b5406b860 0x7f3b5406dd20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3b700c8390 tx=0x7f3b5c009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:12.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.079+0000 7f3b6d7fa640 1 -- 172.21.15.5:0/4135382727 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f3b6403e070 con 0x7f3b700cc000 2024-03-24T10:59:12.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.079+0000 7f3b6d7fa640 1 --2- 172.21.15.5:0/4135382727 >> [v2:172.21.15.5:6816/90414679,v1:172.21.15.5:6817/90414679] conn(0x7f3b54073c80 0x7f3b54076100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:12.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.079+0000 7f3b6d7fa640 1 -- 172.21.15.5:0/4135382727 --> [v2:172.21.15.5:6816/90414679,v1:172.21.15.5:6817/90414679] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3b540767d0 con 0x7f3b54073c80 2024-03-24T10:59:12.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.079+0000 7f3b6d7fa640 1 -- 172.21.15.5:0/4135382727 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3b640069c0 con 0x7f3b700cc000 2024-03-24T10:59:12.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.079+0000 7f3b750f3640 1 --2- 172.21.15.5:0/4135382727 >> [v2:172.21.15.5:6816/90414679,v1:172.21.15.5:6817/90414679] conn(0x7f3b54073c80 0x7f3b54076100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:12.081 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.080+0000 7f3b750f3640 1 --2- 172.21.15.5:0/4135382727 >> [v2:172.21.15.5:6816/90414679,v1:172.21.15.5:6817/90414679] conn(0x7f3b54073c80 0x7f3b54076100 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:12.082 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.080+0000 7f3b6d7fa640 1 -- 172.21.15.5:0/4135382727 <== osd.2 v2:172.21.15.5:6816/90414679 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3b540767d0 con 0x7f3b54073c80 2024-03-24T10:59:12.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.094+0000 7f3b76b56640 1 -- 172.21.15.5:0/4135382727 --> [v2:172.21.15.5:6816/90414679,v1:172.21.15.5:6817/90414679] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3b700c6b90 con 0x7f3b54073c80 2024-03-24T10:59:12.095 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.094+0000 7f3b6d7fa640 1 -- 172.21.15.5:0/4135382727 <== osd.2 v2:172.21.15.5:6816/90414679 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f3b700c6b90 con 0x7f3b54073c80 2024-03-24T10:59:12.095 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-24T10:59:12.095 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T10:59:12.095 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-24T10:59:12.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.094+0000 7f3b76b56640 1 -- 172.21.15.5:0/4135382727 >> [v2:172.21.15.5:6816/90414679,v1:172.21.15.5:6817/90414679] conn(0x7f3b54073c80 msgr2=0x7f3b54076100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:12.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.094+0000 7f3b76b56640 1 --2- 172.21.15.5:0/4135382727 >> [v2:172.21.15.5:6816/90414679,v1:172.21.15.5:6817/90414679] conn(0x7f3b54073c80 0x7f3b54076100 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.094+0000 7f3b76b56640 1 -- 172.21.15.5:0/4135382727 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3b5406b860 msgr2=0x7f3b5406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:12.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.094+0000 7f3b76b56640 1 --2- 172.21.15.5:0/4135382727 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f3b5406b860 0x7f3b5406dd20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3b700c8390 tx=0x7f3b5c009190 comp rx=0 tx=0).stop 2024-03-24T10:59:12.096 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.094+0000 7f3b76b56640 1 -- 172.21.15.5:0/4135382727 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b700cc000 msgr2=0x7f3b701f3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:12.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.094+0000 7f3b76b56640 1 --2- 172.21.15.5:0/4135382727 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f3b700cc000 0x7f3b701f3200 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f3b64000c00 tx=0x7f3b64004730 comp rx=0 tx=0).stop 2024-03-24T10:59:12.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.094+0000 7f3b750f3640 1 -- 172.21.15.5:0/4135382727 reap_dead start 2024-03-24T10:59:12.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.095+0000 7f3b76b56640 1 -- 172.21.15.5:0/4135382727 shutdown_connections 2024-03-24T10:59:12.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.095+0000 7f3b76b56640 1 -- 172.21.15.5:0/4135382727 >> 172.21.15.5:0/4135382727 conn(0x7f3b700cfc30 msgr2=0x7f3b70159a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:12.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.095+0000 7f3b76b56640 1 -- 172.21.15.5:0/4135382727 shutdown_connections 2024-03-24T10:59:12.097 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.095+0000 7f3b76b56640 1 -- 172.21.15.5:0/4135382727 wait complete. 2024-03-24T10:59:12.106 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-24T10:59:12.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.209+0000 7f2597de2640 1 Processor -- start 2024-03-24T10:59:12.210 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.209+0000 7f2597de2640 1 -- start start 2024-03-24T10:59:12.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.210+0000 7f2597de2640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2590154650 0x7f25900cea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:12.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.210+0000 7f2597de2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f259015abc0 0x7f25900cefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:12.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.210+0000 7f2597de2640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f259006c670 con 0x7f25900cae80 2024-03-24T10:59:12.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.210+0000 7f2597de2640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f259006cc30 con 0x7f2590154650 2024-03-24T10:59:12.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.210+0000 7f2597de2640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f25900cc180 con 0x7f259015abc0 2024-03-24T10:59:12.211 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.210+0000 7f2595b7e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f2590154650 0x7f25900cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:12.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.210+0000 7f259537d640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f25900cae80 0x7f259015a4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:54558/0 (socket says 172.21.15.5:54558) 2024-03-24T10:59:12.212 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.210+0000 7f259537d640 1 -- 172.21.15.5:0/345521696 learned_addr learned my addr 172.21.15.5:0/345521696 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:59:12.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.213+0000 7f2595b7e640 1 -- 172.21.15.5:0/345521696 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f259015abc0 msgr2=0x7f25900cefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:12.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.213+0000 7f2595b7e640 1 --2- 172.21.15.5:0/345521696 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f259015abc0 0x7f25900cefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.213+0000 7f2595b7e640 1 -- 172.21.15.5:0/345521696 >> v1:172.21.15.5:6789/0 conn(0x7f25900cae80 legacy=0x7f259015a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:12.214 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.213+0000 7f2595b7e640 1 -- 172.21.15.5:0/345521696 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f259006c950 con 0x7f2590154650 2024-03-24T10:59:12.216 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.215+0000 7f2595b7e640 1 --2- 172.21.15.5:0/345521696 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2590154650 0x7f25900cea90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f258c0097d0 tx=0x7f258c02b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a770337150c46a9a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:12.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.220+0000 7f2594b7c640 1 -- 172.21.15.5:0/345521696 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c031770 con 0x7f2590154650 2024-03-24T10:59:12.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.220+0000 7f2594b7c640 1 -- 172.21.15.5:0/345521696 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f258c0318d0 con 0x7f2590154650 2024-03-24T10:59:12.221 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.220+0000 7f2594b7c640 1 -- 172.21.15.5:0/345521696 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c031c60 con 0x7f2590154650 2024-03-24T10:59:12.222 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.221+0000 7f2597de2640 1 -- 172.21.15.5:0/345521696 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2590154650 msgr2=0x7f25900cea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:12.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.221+0000 7f2597de2640 1 --2- 172.21.15.5:0/345521696 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2590154650 0x7f25900cea90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f258c0097d0 tx=0x7f258c02b810 comp rx=0 tx=0).stop 2024-03-24T10:59:12.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.221+0000 7f2597de2640 1 -- 172.21.15.5:0/345521696 shutdown_connections 2024-03-24T10:59:12.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.221+0000 7f2597de2640 1 --2- 172.21.15.5:0/345521696 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f259015abc0 0x7f25900cefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.221+0000 7f2597de2640 1 --2- 172.21.15.5:0/345521696 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2590154650 0x7f25900cea90 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.221+0000 7f2597de2640 1 -- 172.21.15.5:0/345521696 >> 172.21.15.5:0/345521696 conn(0x7f2590062180 msgr2=0x7f25900b16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:12.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.221+0000 7f2597de2640 1 -- 172.21.15.5:0/345521696 shutdown_connections 2024-03-24T10:59:12.223 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.221+0000 7f2597de2640 1 -- 172.21.15.5:0/345521696 wait complete. 2024-03-24T10:59:12.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.223+0000 7f2597de2640 1 Processor -- start 2024-03-24T10:59:12.224 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.223+0000 7f2597de2640 1 -- start start 2024-03-24T10:59:12.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.224+0000 7f2597de2640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25900cae80 0x7f25901e5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:12.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.224+0000 7f2597de2640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2590154650 0x7f25901e4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:12.225 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.224+0000 7f2597de2640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f259015abc0 0x7f25901e48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:12.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.224+0000 7f2597de2640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f25900c73c0 con 0x7f25900cae80 2024-03-24T10:59:12.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.224+0000 7f2597de2640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f25901e3ed0 con 0x7f259015abc0 2024-03-24T10:59:12.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.224+0000 7f2597de2640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f259015ddf0 con 0x7f2590154650 2024-03-24T10:59:12.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.225+0000 7f2595b7e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25900cae80 0x7f25901e5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:12.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.225+0000 7f259637f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f259015abc0 0x7f25901e48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:12.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.225+0000 7f259637f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f259015abc0 0x7f25901e48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:44030/0 (socket says 172.21.15.5:44030) 2024-03-24T10:59:12.226 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.225+0000 7f259637f640 1 -- 172.21.15.5:0/1355075982 learned_addr learned my addr 172.21.15.5:0/1355075982 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:12.231 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.230+0000 7f2595b7e640 1 -- 172.21.15.5:0/1355075982 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2590154650 msgr2=0x7f25901e4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:12.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.230+0000 7f2595b7e640 1 --2- 172.21.15.5:0/1355075982 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2590154650 0x7f25901e4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.230+0000 7f2595b7e640 1 -- 172.21.15.5:0/1355075982 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f259015abc0 msgr2=0x7f25901e48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:12.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.230+0000 7f2595b7e640 1 --2- 172.21.15.5:0/1355075982 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f259015abc0 0x7f25901e48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.232 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.230+0000 7f2595b7e640 1 -- 172.21.15.5:0/1355075982 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f258c009400 con 0x7f25900cae80 2024-03-24T10:59:12.235 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.234+0000 7f2595b7e640 1 --2- 172.21.15.5:0/1355075982 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25900cae80 0x7f25901e5ef0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f258c033000 tx=0x7f258c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:12.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.239+0000 7f25867fc640 1 -- 172.21.15.5:0/1355075982 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c032890 con 0x7f25900cae80 2024-03-24T10:59:12.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.239+0000 7f25867fc640 1 -- 172.21.15.5:0/1355075982 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f258c0329f0 con 0x7f25900cae80 2024-03-24T10:59:12.240 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.239+0000 7f25867fc640 1 -- 172.21.15.5:0/1355075982 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c032da0 con 0x7f25900cae80 2024-03-24T10:59:12.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.239+0000 7f2597de2640 1 -- 172.21.15.5:0/1355075982 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f259015e090 con 0x7f25900cae80 2024-03-24T10:59:12.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.239+0000 7f2597de2640 1 -- 172.21.15.5:0/1355075982 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25901e3880 con 0x7f25900cae80 2024-03-24T10:59:12.241 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.240+0000 7f2597de2640 1 -- 172.21.15.5:0/1355075982 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25900681d0 con 0x7f25900cae80 2024-03-24T10:59:12.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.525+0000 7f25867fc640 1 -- 172.21.15.5:0/1355075982 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f258c04d3f0 con 0x7f25900cae80 2024-03-24T10:59:12.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.525+0000 7f25867fc640 1 --2- 172.21.15.5:0/1355075982 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f257406b860 0x7f257406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:12.526 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.525+0000 7f259537d640 1 --2- 172.21.15.5:0/1355075982 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f257406b860 0x7f257406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:12.527 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.526+0000 7f259537d640 1 --2- 172.21.15.5:0/1355075982 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f257406b860 0x7f257406dd20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2580005880 tx=0x7f2580007a20 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:12.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.545+0000 7f25867fc640 1 -- 172.21.15.5:0/1355075982 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f258c07aeb0 con 0x7f25900cae80 2024-03-24T10:59:12.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.545+0000 7f25867fc640 1 -- 172.21.15.5:0/1355075982 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f258c076a50 con 0x7f25900cae80 2024-03-24T10:59:12.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.729+0000 7f2597de2640 1 -- 172.21.15.5:0/1355075982 --> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f25901e3700 con 0x7f257406b860 2024-03-24T10:59:12.730 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.729+0000 7f25867fc640 1 -- 172.21.15.5:0/1355075982 <== mgr.4120 v2:172.21.15.5:6832/1102439181 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2580001200 con 0x7f257406b860 2024-03-24T10:59:12.731 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 2 to deep-scrub 2024-03-24T10:59:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.731+0000 7f2597de2640 1 -- 172.21.15.5:0/1355075982 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f257406b860 msgr2=0x7f257406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.731+0000 7f2597de2640 1 --2- 172.21.15.5:0/1355075982 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f257406b860 0x7f257406dd20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f2580005880 tx=0x7f2580007a20 comp rx=0 tx=0).stop 2024-03-24T10:59:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.731+0000 7f2597de2640 1 -- 172.21.15.5:0/1355075982 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25900cae80 msgr2=0x7f25901e5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.731+0000 7f2597de2640 1 --2- 172.21.15.5:0/1355075982 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25900cae80 0x7f25901e5ef0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f258c033000 tx=0x7f258c0046c0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.731+0000 7f2597de2640 1 -- 172.21.15.5:0/1355075982 shutdown_connections 2024-03-24T10:59:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.731+0000 7f2597de2640 1 --2- 172.21.15.5:0/1355075982 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f257406b860 0x7f257406dd20 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.731+0000 7f2597de2640 1 --2- 172.21.15.5:0/1355075982 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f259015abc0 0x7f25901e48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.731+0000 7f2597de2640 1 --2- 172.21.15.5:0/1355075982 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2590154650 0x7f25901e4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.731+0000 7f2597de2640 1 --2- 172.21.15.5:0/1355075982 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f25900cae80 0x7f25901e5ef0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.732+0000 7f2597de2640 1 -- 172.21.15.5:0/1355075982 >> 172.21.15.5:0/1355075982 conn(0x7f2590062180 msgr2=0x7f25900b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:12.733 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.732+0000 7f2597de2640 1 -- 172.21.15.5:0/1355075982 shutdown_connections 2024-03-24T10:59:12.734 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.732+0000 7f2597de2640 1 -- 172.21.15.5:0/1355075982 wait complete. 2024-03-24T10:59:12.745 INFO:tasks.ceph:Scrubbing osd.3 2024-03-24T10:59:12.745 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-24T10:59:12.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.849+0000 7f735206f640 1 Processor -- start 2024-03-24T10:59:12.851 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.849+0000 7f735206f640 1 -- start start 2024-03-24T10:59:12.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.851+0000 7f735206f640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f734c0cf340 0x7f734c0cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:12.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.851+0000 7f735206f640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f734c0ccb80 0x7f734c06bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:12.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.851+0000 7f735206f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f734c0cc3e0 con 0x7f734c0cc000 2024-03-24T10:59:12.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.851+0000 7f735206f640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f734c15a400 con 0x7f734c0ccb80 2024-03-24T10:59:12.852 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.851+0000 7f735206f640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f734c154bd0 con 0x7f734c0cf340 2024-03-24T10:59:12.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.851+0000 7f734affd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f734c0cf340 0x7f734c0cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:12.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.852+0000 7f734bfff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f734c0ccb80 0x7f734c06bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:12.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.852+0000 7f734affd640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f734c0cf340 0x7f734c0cc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:49336/0 (socket says 172.21.15.5:49336) 2024-03-24T10:59:12.853 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.852+0000 7f734affd640 1 -- 172.21.15.5:0/1028543909 learned_addr learned my addr 172.21.15.5:0/1028543909 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:12.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.854+0000 7f734bfff640 1 -- 172.21.15.5:0/1028543909 >> v2:172.21.15.57:3300/0 conn(0x7f734c0cf340 msgr2=0x7f734c0cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:12.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.854+0000 7f734bfff640 1 --2- 172.21.15.5:0/1028543909 >> v2:172.21.15.57:3300/0 conn(0x7f734c0cf340 0x7f734c0cc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.854+0000 7f734bfff640 1 -- 172.21.15.5:0/1028543909 >> v1:172.21.15.57:6789/0 conn(0x7f734c0cc000 legacy=0x7f734c15e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:12.856 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.854+0000 7f734bfff640 1 -- 172.21.15.5:0/1028543909 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f734c15a5f0 con 0x7f734c0ccb80 2024-03-24T10:59:12.857 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.856+0000 7f734bfff640 1 --2- 172.21.15.5:0/1028543909 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f734c0ccb80 0x7f734c06bf50 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f734c0cd790 tx=0x7f734402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb01eb380a11537 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:12.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.862+0000 7f734a7fc640 1 -- 172.21.15.5:0/1028543909 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734402cc80 con 0x7f734c0ccb80 2024-03-24T10:59:12.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.862+0000 7f734a7fc640 1 -- 172.21.15.5:0/1028543909 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f734402cde0 con 0x7f734c0ccb80 2024-03-24T10:59:12.863 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.862+0000 7f734a7fc640 1 -- 172.21.15.5:0/1028543909 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7344034660 con 0x7f734c0ccb80 2024-03-24T10:59:12.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.863+0000 7f735206f640 1 -- 172.21.15.5:0/1028543909 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f734c0ccb80 msgr2=0x7f734c06bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:12.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.863+0000 7f735206f640 1 --2- 172.21.15.5:0/1028543909 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f734c0ccb80 0x7f734c06bf50 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f734c0cd790 tx=0x7f734402bed0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.863+0000 7f735206f640 1 -- 172.21.15.5:0/1028543909 shutdown_connections 2024-03-24T10:59:12.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.863+0000 7f735206f640 1 --2- 172.21.15.5:0/1028543909 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f734c0ccb80 0x7f734c06bf50 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.863+0000 7f735206f640 1 --2- 172.21.15.5:0/1028543909 >> v2:172.21.15.57:3300/0 conn(0x7f734c0cf340 0x7f734c0cc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.863+0000 7f735206f640 1 -- 172.21.15.5:0/1028543909 >> 172.21.15.5:0/1028543909 conn(0x7f734c0cfc30 msgr2=0x7f734c0d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:12.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.863+0000 7f735206f640 1 -- 172.21.15.5:0/1028543909 shutdown_connections 2024-03-24T10:59:12.865 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.863+0000 7f735206f640 1 -- 172.21.15.5:0/1028543909 wait complete. 2024-03-24T10:59:12.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.865+0000 7f735206f640 1 Processor -- start 2024-03-24T10:59:12.866 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.865+0000 7f735206f640 1 -- start start 2024-03-24T10:59:12.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.866+0000 7f735206f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f734c0cc000 0x7f734c155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:12.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.866+0000 7f735206f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f734c0ccb80 0x7f734c14eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:12.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.866+0000 7f735206f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f734c0cf340 0x7f734c0b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:12.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.866+0000 7f735206f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f734c1f0740 con 0x7f734c0ccb80 2024-03-24T10:59:12.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.866+0000 7f735206f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f734c150840 con 0x7f734c0cc000 2024-03-24T10:59:12.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.866+0000 7f735206f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f734c1f0930 con 0x7f734c0cf340 2024-03-24T10:59:12.868 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.867+0000 7f734b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f734c0cc000 0x7f734c155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:12.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.867+0000 7f734b7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f734c0cc000 0x7f734c155190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:49348/0 (socket says 172.21.15.5:49348) 2024-03-24T10:59:12.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.867+0000 7f734b7fe640 1 -- 172.21.15.5:0/2042763835 learned_addr learned my addr 172.21.15.5:0/2042763835 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:12.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.867+0000 7f734bfff640 1 --2- 172.21.15.5:0/2042763835 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f734c0cf340 0x7f734c0b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:12.869 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.869+0000 7f734affd640 1 --2- 172.21.15.5:0/2042763835 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f734c0ccb80 0x7f734c14eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:12.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.871+0000 7f734b7fe640 1 -- 172.21.15.5:0/2042763835 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f734c0cf340 msgr2=0x7f734c0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:12.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.871+0000 7f734b7fe640 1 --2- 172.21.15.5:0/2042763835 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f734c0cf340 0x7f734c0b1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.871+0000 7f734b7fe640 1 -- 172.21.15.5:0/2042763835 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f734c0ccb80 msgr2=0x7f734c14eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:12.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.871+0000 7f734b7fe640 1 --2- 172.21.15.5:0/2042763835 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f734c0ccb80 0x7f734c14eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.872 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.871+0000 7f734b7fe640 1 -- 172.21.15.5:0/2042763835 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7344009a70 con 0x7f734c0cc000 2024-03-24T10:59:12.874 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.874+0000 7f734b7fe640 1 --2- 172.21.15.5:0/2042763835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f734c0cc000 0x7f734c155190 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f733c00c400 tx=0x7f733c00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:12.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.878+0000 7f7333fff640 1 -- 172.21.15.5:0/2042763835 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c0052d0 con 0x7f734c0cc000 2024-03-24T10:59:12.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.878+0000 7f7333fff640 1 -- 172.21.15.5:0/2042763835 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f733c005430 con 0x7f734c0cc000 2024-03-24T10:59:12.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.878+0000 7f7333fff640 1 -- 172.21.15.5:0/2042763835 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f733c003830 con 0x7f734c0cc000 2024-03-24T10:59:12.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.878+0000 7f735206f640 1 -- 172.21.15.5:0/2042763835 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f734c150ae0 con 0x7f734c0cc000 2024-03-24T10:59:12.879 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.878+0000 7f735206f640 1 -- 172.21.15.5:0/2042763835 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f734c1510a0 con 0x7f734c0cc000 2024-03-24T10:59:12.880 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.879+0000 7f735206f640 1 -- 172.21.15.5:0/2042763835 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f734c158b60 con 0x7f734c0cc000 2024-03-24T10:59:12.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.899+0000 7f7333fff640 1 -- 172.21.15.5:0/2042763835 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f733c003990 con 0x7f734c0cc000 2024-03-24T10:59:12.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.900+0000 7f7333fff640 1 --2- 172.21.15.5:0/2042763835 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f733806b860 0x7f733806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:12.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.900+0000 7f734affd640 1 --2- 172.21.15.5:0/2042763835 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f733806b860 0x7f733806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:12.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.900+0000 7f734affd640 1 --2- 172.21.15.5:0/2042763835 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f733806b860 0x7f733806dd20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f734c0c74d0 tx=0x7f7340009240 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:12.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.904+0000 7f7333fff640 1 -- 172.21.15.5:0/2042763835 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f733c058620 con 0x7f734c0cc000 2024-03-24T10:59:12.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.905+0000 7f7333fff640 1 --2- 172.21.15.5:0/2042763835 >> [v2:172.21.15.5:6808/4212473119,v1:172.21.15.5:6809/4212473119] conn(0x7f7338073c80 0x7f7338076100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:12.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.905+0000 7f7333fff640 1 -- 172.21.15.5:0/2042763835 --> [v2:172.21.15.5:6808/4212473119,v1:172.21.15.5:6809/4212473119] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f73380767d0 con 0x7f7338073c80 2024-03-24T10:59:12.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.905+0000 7f734bfff640 1 --2- 172.21.15.5:0/2042763835 >> [v2:172.21.15.5:6808/4212473119,v1:172.21.15.5:6809/4212473119] conn(0x7f7338073c80 0x7f7338076100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:12.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.905+0000 7f734bfff640 1 --2- 172.21.15.5:0/2042763835 >> [v2:172.21.15.5:6808/4212473119,v1:172.21.15.5:6809/4212473119] conn(0x7f7338073c80 0x7f7338076100 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:12.906 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.906+0000 7f7333fff640 1 -- 172.21.15.5:0/2042763835 <== osd.3 v2:172.21.15.5:6808/4212473119 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f73380767d0 con 0x7f7338073c80 2024-03-24T10:59:12.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.907+0000 7f7333fff640 1 -- 172.21.15.5:0/2042763835 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f733c001140 con 0x7f734c0cc000 2024-03-24T10:59:12.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.919+0000 7f735206f640 1 -- 172.21.15.5:0/2042763835 --> [v2:172.21.15.5:6808/4212473119,v1:172.21.15.5:6809/4212473119] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f734c15ae70 con 0x7f7338073c80 2024-03-24T10:59:12.920 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.919+0000 7f7333fff640 1 -- 172.21.15.5:0/2042763835 <== osd.3 v2:172.21.15.5:6808/4212473119 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7344009450 con 0x7f7338073c80 2024-03-24T10:59:12.920 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-24T10:59:12.921 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T10:59:12.921 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-24T10:59:12.921 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.919+0000 7f735206f640 1 -- 172.21.15.5:0/2042763835 >> [v2:172.21.15.5:6808/4212473119,v1:172.21.15.5:6809/4212473119] conn(0x7f7338073c80 msgr2=0x7f7338076100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:12.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.919+0000 7f735206f640 1 --2- 172.21.15.5:0/2042763835 >> [v2:172.21.15.5:6808/4212473119,v1:172.21.15.5:6809/4212473119] conn(0x7f7338073c80 0x7f7338076100 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.920+0000 7f735206f640 1 -- 172.21.15.5:0/2042763835 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f733806b860 msgr2=0x7f733806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:12.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.920+0000 7f735206f640 1 --2- 172.21.15.5:0/2042763835 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f733806b860 0x7f733806dd20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f734c0c74d0 tx=0x7f7340009240 comp rx=0 tx=0).stop 2024-03-24T10:59:12.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.920+0000 7f735206f640 1 -- 172.21.15.5:0/2042763835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f734c0cc000 msgr2=0x7f734c155190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:12.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.920+0000 7f735206f640 1 --2- 172.21.15.5:0/2042763835 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f734c0cc000 0x7f734c155190 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f733c00c400 tx=0x7f733c00c8c0 comp rx=0 tx=0).stop 2024-03-24T10:59:12.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.920+0000 7f734bfff640 1 -- 172.21.15.5:0/2042763835 reap_dead start 2024-03-24T10:59:12.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.920+0000 7f735206f640 1 -- 172.21.15.5:0/2042763835 shutdown_connections 2024-03-24T10:59:12.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.920+0000 7f735206f640 1 -- 172.21.15.5:0/2042763835 >> 172.21.15.5:0/2042763835 conn(0x7f734c0cfc30 msgr2=0x7f734c15e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:12.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.920+0000 7f735206f640 1 -- 172.21.15.5:0/2042763835 shutdown_connections 2024-03-24T10:59:12.922 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:12.920+0000 7f735206f640 1 -- 172.21.15.5:0/2042763835 wait complete. 2024-03-24T10:59:12.935 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-24T10:59:13.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.042+0000 7f77d606f640 1 Processor -- start 2024-03-24T10:59:13.043 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.042+0000 7f77d606f640 1 -- start start 2024-03-24T10:59:13.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.043+0000 7f77d606f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f77d006bb70 0x7f77d00ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:13.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.043+0000 7f77d606f640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77d006bf90 con 0x7f77d00ca9f0 2024-03-24T10:59:13.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.043+0000 7f77d606f640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77d00cd910 con 0x7f77d00cca90 2024-03-24T10:59:13.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.043+0000 7f77d606f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f77d00cdbf0 con 0x7f77d006bb70 2024-03-24T10:59:13.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.044+0000 7f77cf7fe640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f77d00cca90 0x7f77d015ac40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:47432/0 (socket says 172.21.15.5:47432) 2024-03-24T10:59:13.044 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.044+0000 7f77cf7fe640 1 -- 172.21.15.5:0/4072811885 learned_addr learned my addr 172.21.15.5:0/4072811885 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:59:13.047 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.046+0000 7f77ce7fc640 1 -- 172.21.15.5:0/4072811885 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 81859860 0 0) 0x7f77d00cd910 con 0x7f77d00cca90 2024-03-24T10:59:13.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.046+0000 7f77ce7fc640 1 -- 172.21.15.5:0/4072811885 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77b8003520 con 0x7f77d00cca90 2024-03-24T10:59:13.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.047+0000 7f77ce7fc640 1 -- 172.21.15.5:0/4072811885 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3456863335 0 0) 0x7f77d006bf90 con 0x7f77d00ca9f0 2024-03-24T10:59:13.048 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.047+0000 7f77ce7fc640 1 -- 172.21.15.5:0/4072811885 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77d00cd910 con 0x7f77d00ca9f0 2024-03-24T10:59:13.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.051+0000 7f77ce7fc640 1 -- 172.21.15.5:0/4072811885 <== mon.0 v1:172.21.15.5:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2830929130 0 0) 0x7f77d00cd910 con 0x7f77d00ca9f0 2024-03-24T10:59:13.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.051+0000 7f77ce7fc640 1 -- 172.21.15.5:0/4072811885 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f77d006bb70 msgr2=0x7f77d00ca4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:13.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.051+0000 7f77ce7fc640 1 --2- 172.21.15.5:0/4072811885 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f77d006bb70 0x7f77d00ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.052 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.051+0000 7f77ce7fc640 1 -- 172.21.15.5:0/4072811885 >> v1:172.21.15.57:6789/0 conn(0x7f77d00cca90 legacy=0x7f77d015ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:13.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.051+0000 7f77ce7fc640 1 -- 172.21.15.5:0/4072811885 --> v1:172.21.15.5:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d00cce70 con 0x7f77d00ca9f0 2024-03-24T10:59:13.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.051+0000 7f77ce7fc640 1 -- 172.21.15.5:0/4072811885 <== mon.0 v1:172.21.15.5:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 100681612 0 0) 0x7f77c8001670 con 0x7f77d00ca9f0 2024-03-24T10:59:13.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.051+0000 7f77ce7fc640 1 -- 172.21.15.5:0/4072811885 >> v1:172.21.15.5:6789/0 conn(0x7f77d00ca9f0 legacy=0x7f77d00ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:13.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.051+0000 7f77ce7fc640 1 --2- 172.21.15.5:0/4072811885 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77b8003b30 0x7f77b8003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:13.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.051+0000 7f77ce7fc640 1 --2- 172.21.15.5:0/4072811885 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f77d006bb70 0x7f77b80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:13.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.051+0000 7f77ce7fc640 1 --2- 172.21.15.5:0/4072811885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77b8004e20 0x7f77b800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:13.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.051+0000 7f77ce7fc640 1 -- 172.21.15.5:0/4072811885 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f77d00cdbf0 con 0x7f77b8003b30 2024-03-24T10:59:13.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.051+0000 7f77ce7fc640 1 -- 172.21.15.5:0/4072811885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f77b80055e0 con 0x7f77b8004e20 2024-03-24T10:59:13.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.051+0000 7f77ce7fc640 1 -- 172.21.15.5:0/4072811885 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f77b800bec0 con 0x7f77d006bb70 2024-03-24T10:59:13.053 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.052+0000 7f77cffff640 1 --2- 172.21.15.5:0/4072811885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77b8004e20 0x7f77b800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:13.055 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.054+0000 7f77cf7fe640 1 --2- 172.21.15.5:0/4072811885 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77b8003b30 0x7f77b8003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:13.057 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.056+0000 7f77ceffd640 1 --2- 172.21.15.5:0/4072811885 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f77d006bb70 0x7f77b80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:13.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.058+0000 7f77cffff640 1 -- 172.21.15.5:0/4072811885 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f77d006bb70 msgr2=0x7f77b80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:13.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.058+0000 7f77cffff640 1 --2- 172.21.15.5:0/4072811885 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f77d006bb70 0x7f77b80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.058+0000 7f77cffff640 1 -- 172.21.15.5:0/4072811885 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77b8003b30 msgr2=0x7f77b8003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:13.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.058+0000 7f77cffff640 1 --2- 172.21.15.5:0/4072811885 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77b8003b30 0x7f77b8003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.058+0000 7f77cffff640 1 -- 172.21.15.5:0/4072811885 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d00cd910 con 0x7f77b8004e20 2024-03-24T10:59:13.061 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.060+0000 7f77cffff640 1 --2- 172.21.15.5:0/4072811885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77b8004e20 0x7f77b800b980 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f77c8007e90 tx=0x7f77b80057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b08959e3d0b66a2e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:13.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.065+0000 7f77ce7fc640 1 -- 172.21.15.5:0/4072811885 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c802c890 con 0x7f77b8004e20 2024-03-24T10:59:13.066 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.066+0000 7f77ce7fc640 1 -- 172.21.15.5:0/4072811885 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77c802ca10 con 0x7f77b8004e20 2024-03-24T10:59:13.067 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.066+0000 7f77ce7fc640 1 -- 172.21.15.5:0/4072811885 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c802cdc0 con 0x7f77b8004e20 2024-03-24T10:59:13.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.066+0000 7f77d606f640 1 -- 172.21.15.5:0/4072811885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77b8004e20 msgr2=0x7f77b800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:13.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.066+0000 7f77d606f640 1 --2- 172.21.15.5:0/4072811885 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77b8004e20 0x7f77b800b980 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f77c8007e90 tx=0x7f77b80057f0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.066+0000 7f77cffff640 1 -- 172.21.15.5:0/4072811885 reap_dead start 2024-03-24T10:59:13.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.067+0000 7f77d606f640 1 -- 172.21.15.5:0/4072811885 shutdown_connections 2024-03-24T10:59:13.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.067+0000 7f77d606f640 1 -- 172.21.15.5:0/4072811885 >> 172.21.15.5:0/4072811885 conn(0x7f77d00b16c0 msgr2=0x7f77d00b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:13.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.067+0000 7f77d606f640 1 -- 172.21.15.5:0/4072811885 shutdown_connections 2024-03-24T10:59:13.068 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.067+0000 7f77d606f640 1 -- 172.21.15.5:0/4072811885 wait complete. 2024-03-24T10:59:13.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.068+0000 7f77d606f640 1 Processor -- start 2024-03-24T10:59:13.069 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.069+0000 7f77d606f640 1 -- start start 2024-03-24T10:59:13.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.070+0000 7f77d606f640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77b8004e20 0x7f77d014d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:13.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.070+0000 7f77d606f640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77d00b48a0 0x7f77d014e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:13.071 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.070+0000 7f77d606f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f77d00c70a0 0x7f77d00c7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.070+0000 7f77d606f640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f77d01505a0 con 0x7f77d00b48a0 2024-03-24T10:59:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.070+0000 7f77d606f640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f77d00c9e00 con 0x7f77b8004e20 2024-03-24T10:59:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.070+0000 7f77d606f640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f77d014f090 con 0x7f77d00c70a0 2024-03-24T10:59:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.070+0000 7f77ceffd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77d00b48a0 0x7f77d014e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.071+0000 7f77cf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77b8004e20 0x7f77d014d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.071+0000 7f77cffff640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f77d00c70a0 0x7f77d00c7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.071+0000 7f77cf7fe640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77b8004e20 0x7f77d014d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:49364/0 (socket says 172.21.15.5:49364) 2024-03-24T10:59:13.072 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.071+0000 7f77cf7fe640 1 -- 172.21.15.5:0/2825977586 learned_addr learned my addr 172.21.15.5:0/2825977586 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:13.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.075+0000 7f77cf7fe640 1 -- 172.21.15.5:0/2825977586 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f77d00c70a0 msgr2=0x7f77d00c7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:13.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.075+0000 7f77cf7fe640 1 --2- 172.21.15.5:0/2825977586 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f77d00c70a0 0x7f77d00c7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.075+0000 7f77cf7fe640 1 -- 172.21.15.5:0/2825977586 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77d00b48a0 msgr2=0x7f77d014e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:13.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.075+0000 7f77cf7fe640 1 --2- 172.21.15.5:0/2825977586 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77d00b48a0 0x7f77d014e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.076 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.075+0000 7f77cf7fe640 1 -- 172.21.15.5:0/2825977586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d00c7a60 con 0x7f77b8004e20 2024-03-24T10:59:13.078 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.078+0000 7f77cf7fe640 1 --2- 172.21.15.5:0/2825977586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77b8004e20 0x7f77d014d690 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f77c0007a20 tx=0x7f77c000f5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:13.083 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.082+0000 7f77b7fff640 1 -- 172.21.15.5:0/2825977586 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c0007b10 con 0x7f77b8004e20 2024-03-24T10:59:13.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.082+0000 7f77b7fff640 1 -- 172.21.15.5:0/2825977586 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77c000fe60 con 0x7f77b8004e20 2024-03-24T10:59:13.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.082+0000 7f77b7fff640 1 -- 172.21.15.5:0/2825977586 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c0013650 con 0x7f77b8004e20 2024-03-24T10:59:13.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.082+0000 7f77d606f640 1 -- 172.21.15.5:0/2825977586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77d014ff20 con 0x7f77b8004e20 2024-03-24T10:59:13.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.083+0000 7f77d606f640 1 -- 172.21.15.5:0/2825977586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77d00c9840 con 0x7f77b8004e20 2024-03-24T10:59:13.084 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.083+0000 7f77d606f640 1 -- 172.21.15.5:0/2825977586 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77d014fd80 con 0x7f77b8004e20 2024-03-24T10:59:13.107 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.106+0000 7f77b7fff640 1 -- 172.21.15.5:0/2825977586 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f77c00137b0 con 0x7f77b8004e20 2024-03-24T10:59:13.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.107+0000 7f77b7fff640 1 --2- 172.21.15.5:0/2825977586 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f77bc06b890 0x7f77bc06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:13.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.107+0000 7f77ceffd640 1 --2- 172.21.15.5:0/2825977586 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f77bc06b890 0x7f77bc06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:13.108 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.107+0000 7f77ceffd640 1 --2- 172.21.15.5:0/2825977586 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f77bc06b890 0x7f77bc06dd50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f77d00ca6d0 tx=0x7f77c4009060 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:13.110 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.109+0000 7f77b7fff640 1 -- 172.21.15.5:0/2825977586 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f77c0059880 con 0x7f77b8004e20 2024-03-24T10:59:13.388 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.387+0000 7f77b7fff640 1 -- 172.21.15.5:0/2825977586 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f77c0055b20 con 0x7f77b8004e20 2024-03-24T10:59:13.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.574+0000 7f77d606f640 1 -- 172.21.15.5:0/2825977586 --> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f77d01540b0 con 0x7f77bc06b890 2024-03-24T10:59:13.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.574+0000 7f77b7fff640 1 -- 172.21.15.5:0/2825977586 <== mgr.4120 v2:172.21.15.5:6832/1102439181 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f77d01540b0 con 0x7f77bc06b890 2024-03-24T10:59:13.575 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 3 to deep-scrub 2024-03-24T10:59:13.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.576+0000 7f77d606f640 1 -- 172.21.15.5:0/2825977586 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f77bc06b890 msgr2=0x7f77bc06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:13.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.576+0000 7f77d606f640 1 --2- 172.21.15.5:0/2825977586 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f77bc06b890 0x7f77bc06dd50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f77d00ca6d0 tx=0x7f77c4009060 comp rx=0 tx=0).stop 2024-03-24T10:59:13.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.576+0000 7f77d606f640 1 -- 172.21.15.5:0/2825977586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77b8004e20 msgr2=0x7f77d014d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:13.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.576+0000 7f77d606f640 1 --2- 172.21.15.5:0/2825977586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77b8004e20 0x7f77d014d690 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f77c0007a20 tx=0x7f77c000f5a0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.576+0000 7f77d606f640 1 -- 172.21.15.5:0/2825977586 shutdown_connections 2024-03-24T10:59:13.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.576+0000 7f77d606f640 1 --2- 172.21.15.5:0/2825977586 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f77bc06b890 0x7f77bc06dd50 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.576+0000 7f77d606f640 1 --2- 172.21.15.5:0/2825977586 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77d00b48a0 0x7f77d014e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.576+0000 7f77d606f640 1 --2- 172.21.15.5:0/2825977586 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f77d00c70a0 0x7f77d00c7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.578 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.576+0000 7f77d606f640 1 --2- 172.21.15.5:0/2825977586 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77b8004e20 0x7f77d014d690 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.576+0000 7f77d606f640 1 -- 172.21.15.5:0/2825977586 >> 172.21.15.5:0/2825977586 conn(0x7f77d00b16c0 msgr2=0x7f77d00cfac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:13.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.577+0000 7f77d606f640 1 -- 172.21.15.5:0/2825977586 shutdown_connections 2024-03-24T10:59:13.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.577+0000 7f77d606f640 1 -- 172.21.15.5:0/2825977586 wait complete. 2024-03-24T10:59:13.589 INFO:tasks.ceph:Scrubbing osd.4 2024-03-24T10:59:13.589 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-24T10:59:13.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.695+0000 7f2bbc82e640 1 Processor -- start 2024-03-24T10:59:13.697 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.696+0000 7f2bbc82e640 1 -- start start 2024-03-24T10:59:13.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.696+0000 7f2bbc82e640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2bb80cca90 0x7f2bb806bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:13.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.696+0000 7f2bbc82e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bb80cd910 con 0x7f2bb815ac40 2024-03-24T10:59:13.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.697+0000 7f2bbc82e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bb80cdbf0 con 0x7f2bb80ce080 2024-03-24T10:59:13.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.697+0000 7f2bbc82e640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f2bb80cce70 con 0x7f2bb80cca90 2024-03-24T10:59:13.698 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.697+0000 7f2bb659c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f2bb80cca90 0x7f2bb806bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:13.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.697+0000 7f2bb6d9d640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f2bb80ce080 0x7f2bb8164b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:47444/0 (socket says 172.21.15.5:47444) 2024-03-24T10:59:13.699 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.697+0000 7f2bb6d9d640 1 -- 172.21.15.5:0/3137675250 learned_addr learned my addr 172.21.15.5:0/3137675250 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:59:13.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.700+0000 7f2bb659c640 1 -- 172.21.15.5:0/3137675250 >> v1:172.21.15.57:6789/0 conn(0x7f2bb80ce080 legacy=0x7f2bb8164b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:13.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.700+0000 7f2bb659c640 1 -- 172.21.15.5:0/3137675250 >> v1:172.21.15.5:6789/0 conn(0x7f2bb815ac40 legacy=0x7f2bb80ca4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:13.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.700+0000 7f2bb659c640 1 -- 172.21.15.5:0/3137675250 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb806c1c0 con 0x7f2bb80cca90 2024-03-24T10:59:13.704 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.703+0000 7f2bb659c640 1 --2- 172.21.15.5:0/3137675250 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bb80cca90 0x7f2bb806bb70 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80097a0 tx=0x7f2ba802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa0bad8ec1342af5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:13.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.708+0000 7f2bb559a640 1 -- 172.21.15.5:0/3137675250 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba802e070 con 0x7f2bb80cca90 2024-03-24T10:59:13.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.708+0000 7f2bb559a640 1 -- 172.21.15.5:0/3137675250 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ba80315a0 con 0x7f2bb80cca90 2024-03-24T10:59:13.709 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.708+0000 7f2bb559a640 1 -- 172.21.15.5:0/3137675250 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8031930 con 0x7f2bb80cca90 2024-03-24T10:59:13.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.709+0000 7f2bbc82e640 1 -- 172.21.15.5:0/3137675250 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bb80cca90 msgr2=0x7f2bb806bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:13.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.709+0000 7f2bbc82e640 1 --2- 172.21.15.5:0/3137675250 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bb80cca90 0x7f2bb806bb70 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80097a0 tx=0x7f2ba802b840 comp rx=0 tx=0).stop 2024-03-24T10:59:13.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.709+0000 7f2bbc82e640 1 -- 172.21.15.5:0/3137675250 shutdown_connections 2024-03-24T10:59:13.710 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.709+0000 7f2bbc82e640 1 --2- 172.21.15.5:0/3137675250 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bb80cca90 0x7f2bb806bb70 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.709+0000 7f2bbc82e640 1 -- 172.21.15.5:0/3137675250 >> 172.21.15.5:0/3137675250 conn(0x7f2bb80b16c0 msgr2=0x7f2bb80b1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:13.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.709+0000 7f2bbc82e640 1 -- 172.21.15.5:0/3137675250 shutdown_connections 2024-03-24T10:59:13.711 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.709+0000 7f2bbc82e640 1 -- 172.21.15.5:0/3137675250 wait complete. 2024-03-24T10:59:13.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.711+0000 7f2bbc82e640 1 Processor -- start 2024-03-24T10:59:13.712 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.711+0000 7f2bbc82e640 1 -- start start 2024-03-24T10:59:13.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.713+0000 7f2bbc82e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bb80cca90 0x7f2bb80c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:13.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.713+0000 7f2bbc82e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2bb80ce080 0x7f2bb80c46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:13.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.713+0000 7f2bbc82e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2bb815ac40 0x7f2bb81542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:13.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.713+0000 7f2bbc82e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb81f00b0 con 0x7f2bb80cca90 2024-03-24T10:59:13.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.713+0000 7f2bbc82e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb80c6810 con 0x7f2bb80ce080 2024-03-24T10:59:13.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.713+0000 7f2bbc82e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f2bb81f08d0 con 0x7f2bb815ac40 2024-03-24T10:59:13.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.713+0000 7f2bb659c640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bb80cca90 0x7f2bb80c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:13.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.713+0000 7f2bb5d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2bb80ce080 0x7f2bb80c46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:13.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.713+0000 7f2bb6d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2bb815ac40 0x7f2bb81542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:13.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.714+0000 7f2bb5d9b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2bb80ce080 0x7f2bb80c46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:49374/0 (socket says 172.21.15.5:49374) 2024-03-24T10:59:13.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.714+0000 7f2bb5d9b640 1 -- 172.21.15.5:0/423505559 learned_addr learned my addr 172.21.15.5:0/423505559 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:13.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.717+0000 7f2bb659c640 1 -- 172.21.15.5:0/423505559 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2bb815ac40 msgr2=0x7f2bb81542b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:13.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.717+0000 7f2bb659c640 1 --2- 172.21.15.5:0/423505559 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f2bb815ac40 0x7f2bb81542b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.717+0000 7f2bb659c640 1 -- 172.21.15.5:0/423505559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2bb80ce080 msgr2=0x7f2bb80c46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:13.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.717+0000 7f2bb659c640 1 --2- 172.21.15.5:0/423505559 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f2bb80ce080 0x7f2bb80c46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.718 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.717+0000 7f2bb659c640 1 -- 172.21.15.5:0/423505559 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ba8009400 con 0x7f2bb80cca90 2024-03-24T10:59:13.720 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.719+0000 7f2bb659c640 1 --2- 172.21.15.5:0/423505559 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bb80cca90 0x7f2bb80c5ba0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f2bb808d160 tx=0x7f2ba80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:13.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.723+0000 7f2ba6ffd640 1 -- 172.21.15.5:0/423505559 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8032900 con 0x7f2bb80cca90 2024-03-24T10:59:13.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.723+0000 7f2ba6ffd640 1 -- 172.21.15.5:0/423505559 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ba8032aa0 con 0x7f2bb80cca90 2024-03-24T10:59:13.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.723+0000 7f2bbc82e640 1 -- 172.21.15.5:0/423505559 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb80c4bf0 con 0x7f2bb80cca90 2024-03-24T10:59:13.724 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.723+0000 7f2ba6ffd640 1 -- 172.21.15.5:0/423505559 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8032e50 con 0x7f2bb80cca90 2024-03-24T10:59:13.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.723+0000 7f2bbc82e640 1 -- 172.21.15.5:0/423505559 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bb81557d0 con 0x7f2bb80cca90 2024-03-24T10:59:13.725 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.724+0000 7f2bbc82e640 1 -- 172.21.15.5:0/423505559 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2bb80cc380 con 0x7f2bb80cca90 2024-03-24T10:59:13.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.750+0000 7f2ba6ffd640 1 -- 172.21.15.5:0/423505559 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2ba804e480 con 0x7f2bb80cca90 2024-03-24T10:59:13.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.750+0000 7f2ba6ffd640 1 --2- 172.21.15.5:0/423505559 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2b9806b860 0x7f2b9806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:13.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.750+0000 7f2bb5d9b640 1 --2- 172.21.15.5:0/423505559 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2b9806b860 0x7f2b9806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:13.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.751+0000 7f2bb5d9b640 1 --2- 172.21.15.5:0/423505559 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2b9806b860 0x7f2b9806dd20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2bb80b1fd0 tx=0x7f2bac009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.755+0000 7f2ba6ffd640 1 -- 172.21.15.5:0/423505559 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f2ba807a7f0 con 0x7f2bb80cca90 2024-03-24T10:59:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.755+0000 7f2ba6ffd640 1 --2- 172.21.15.5:0/423505559 >> [v2:172.21.15.57:6816/2193986505,v1:172.21.15.57:6817/2193986505] conn(0x7f2b98073c80 0x7f2b98076100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.755+0000 7f2ba6ffd640 1 -- 172.21.15.5:0/423505559 --> [v2:172.21.15.57:6816/2193986505,v1:172.21.15.57:6817/2193986505] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b980767d0 con 0x7f2b98073c80 2024-03-24T10:59:13.758 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.755+0000 7f2ba6ffd640 1 -- 172.21.15.5:0/423505559 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2ba8076a90 con 0x7f2bb80cca90 2024-03-24T10:59:13.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.756+0000 7f2bb6d9d640 1 --2- 172.21.15.5:0/423505559 >> [v2:172.21.15.57:6816/2193986505,v1:172.21.15.57:6817/2193986505] conn(0x7f2b98073c80 0x7f2b98076100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:13.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.756+0000 7f2bb6d9d640 1 --2- 172.21.15.5:0/423505559 >> [v2:172.21.15.57:6816/2193986505,v1:172.21.15.57:6817/2193986505] conn(0x7f2b98073c80 0x7f2b98076100 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:13.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.757+0000 7f2ba6ffd640 1 -- 172.21.15.5:0/423505559 <== osd.4 v2:172.21.15.57:6816/2193986505 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2b980767d0 con 0x7f2b98073c80 2024-03-24T10:59:13.772 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.771+0000 7f2bbc82e640 1 -- 172.21.15.5:0/423505559 --> [v2:172.21.15.57:6816/2193986505,v1:172.21.15.57:6817/2193986505] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2bb806ca40 con 0x7f2b98073c80 2024-03-24T10:59:13.773 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.772+0000 7f2ba6ffd640 1 -- 172.21.15.5:0/423505559 <== osd.4 v2:172.21.15.57:6816/2193986505 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f2bb0004e60 con 0x7f2b98073c80 2024-03-24T10:59:13.773 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-24T10:59:13.773 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T10:59:13.773 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-24T10:59:13.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.772+0000 7f2bbc82e640 1 -- 172.21.15.5:0/423505559 >> [v2:172.21.15.57:6816/2193986505,v1:172.21.15.57:6817/2193986505] conn(0x7f2b98073c80 msgr2=0x7f2b98076100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:13.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.772+0000 7f2bbc82e640 1 --2- 172.21.15.5:0/423505559 >> [v2:172.21.15.57:6816/2193986505,v1:172.21.15.57:6817/2193986505] conn(0x7f2b98073c80 0x7f2b98076100 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.772+0000 7f2bbc82e640 1 -- 172.21.15.5:0/423505559 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2b9806b860 msgr2=0x7f2b9806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:13.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.772+0000 7f2bbc82e640 1 --2- 172.21.15.5:0/423505559 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f2b9806b860 0x7f2b9806dd20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2bb80b1fd0 tx=0x7f2bac009040 comp rx=0 tx=0).stop 2024-03-24T10:59:13.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.772+0000 7f2bbc82e640 1 -- 172.21.15.5:0/423505559 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bb80cca90 msgr2=0x7f2bb80c5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:13.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.772+0000 7f2bbc82e640 1 --2- 172.21.15.5:0/423505559 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f2bb80cca90 0x7f2bb80c5ba0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f2bb808d160 tx=0x7f2ba80046c0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.774 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.772+0000 7f2bb6d9d640 1 -- 172.21.15.5:0/423505559 reap_dead start 2024-03-24T10:59:13.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.772+0000 7f2bbc82e640 1 -- 172.21.15.5:0/423505559 shutdown_connections 2024-03-24T10:59:13.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.772+0000 7f2bbc82e640 1 -- 172.21.15.5:0/423505559 >> 172.21.15.5:0/423505559 conn(0x7f2bb80b16c0 msgr2=0x7f2bb80af4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:13.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.772+0000 7f2bbc82e640 1 -- 172.21.15.5:0/423505559 shutdown_connections 2024-03-24T10:59:13.775 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.772+0000 7f2bbc82e640 1 -- 172.21.15.5:0/423505559 wait complete. 2024-03-24T10:59:13.784 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-24T10:59:13.887 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.886+0000 7f8690e84640 1 Processor -- start 2024-03-24T10:59:13.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.886+0000 7f8690e84640 1 -- start start 2024-03-24T10:59:13.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.887+0000 7f8690e84640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f868c0cc000 0x7f868c15e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:13.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.887+0000 7f8690e84640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f868c0cf340 0x7f868c0cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:13.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.887+0000 7f8690e84640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f868c0cc3e0 con 0x7f868c0ccb80 2024-03-24T10:59:13.888 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.887+0000 7f8690e84640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f868c15a400 con 0x7f868c0cc000 2024-03-24T10:59:13.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.887+0000 7f8690e84640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f868c154bd0 con 0x7f868c0cf340 2024-03-24T10:59:13.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.888+0000 7f8683fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f868c0cf340 0x7f868c0cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:13.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.888+0000 7f868a59c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f868c0cc000 0x7f868c15e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:13.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.888+0000 7f8683fff640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f868c0cf340 0x7f868c0cc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:49388/0 (socket says 172.21.15.5:49388) 2024-03-24T10:59:13.889 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.888+0000 7f8683fff640 1 -- 172.21.15.5:0/3328796296 learned_addr learned my addr 172.21.15.5:0/3328796296 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:13.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.891+0000 7f8683fff640 1 -- 172.21.15.5:0/3328796296 >> v2:172.21.15.5:3300/0 conn(0x7f868c0cc000 msgr2=0x7f868c15e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:13.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.891+0000 7f8683fff640 1 --2- 172.21.15.5:0/3328796296 >> v2:172.21.15.5:3300/0 conn(0x7f868c0cc000 0x7f868c15e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.891+0000 7f8683fff640 1 -- 172.21.15.5:0/3328796296 >> v1:172.21.15.5:6789/0 conn(0x7f868c0ccb80 legacy=0x7f868c06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:13.892 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.891+0000 7f8683fff640 1 -- 172.21.15.5:0/3328796296 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f868c15a5f0 con 0x7f868c0cf340 2024-03-24T10:59:13.894 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.893+0000 7f8683fff640 1 --2- 172.21.15.5:0/3328796296 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f868c0cf340 0x7f868c0cc640 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f868c0cd790 tx=0x7f867402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7135cfbbc2dcfcb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:13.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.898+0000 7f8689d9b640 1 -- 172.21.15.5:0/3328796296 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8674031870 con 0x7f868c0cf340 2024-03-24T10:59:13.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.898+0000 7f8689d9b640 1 -- 172.21.15.5:0/3328796296 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86740319d0 con 0x7f868c0cf340 2024-03-24T10:59:13.899 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.898+0000 7f8689d9b640 1 -- 172.21.15.5:0/3328796296 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8674031d60 con 0x7f868c0cf340 2024-03-24T10:59:13.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.899+0000 7f8690e84640 1 -- 172.21.15.5:0/3328796296 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f868c0cf340 msgr2=0x7f868c0cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:13.900 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.899+0000 7f8690e84640 1 --2- 172.21.15.5:0/3328796296 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f868c0cf340 0x7f868c0cc640 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f868c0cd790 tx=0x7f867402b910 comp rx=0 tx=0).stop 2024-03-24T10:59:13.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.899+0000 7f8690e84640 1 -- 172.21.15.5:0/3328796296 shutdown_connections 2024-03-24T10:59:13.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.899+0000 7f8690e84640 1 --2- 172.21.15.5:0/3328796296 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f868c0cf340 0x7f868c0cc640 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.899+0000 7f8690e84640 1 --2- 172.21.15.5:0/3328796296 >> v2:172.21.15.5:3300/0 conn(0x7f868c0cc000 0x7f868c15e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.899+0000 7f8690e84640 1 -- 172.21.15.5:0/3328796296 >> 172.21.15.5:0/3328796296 conn(0x7f868c0cfc30 msgr2=0x7f868c0d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:13.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.899+0000 7f8690e84640 1 -- 172.21.15.5:0/3328796296 shutdown_connections 2024-03-24T10:59:13.901 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.899+0000 7f8690e84640 1 -- 172.21.15.5:0/3328796296 wait complete. 2024-03-24T10:59:13.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.901+0000 7f8690e84640 1 Processor -- start 2024-03-24T10:59:13.902 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.902+0000 7f8690e84640 1 -- start start 2024-03-24T10:59:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.903+0000 7f8690e84640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f868c0cc000 0x7f868c151a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.903+0000 7f8690e84640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f868c0ccb80 0x7f868c1e4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.903+0000 7f8690e84640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f868c0cf340 0x7f868c1e54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.903+0000 7f8690e84640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f868c1555a0 con 0x7f868c0ccb80 2024-03-24T10:59:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.903+0000 7f8690e84640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f868c1efa00 con 0x7f868c0cc000 2024-03-24T10:59:13.904 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.903+0000 7f8690e84640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f868c1f3110 con 0x7f868c0cf340 2024-03-24T10:59:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.903+0000 7f868ad9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f868c0cf340 0x7f868c1e54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.903+0000 7f8683fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f868c0ccb80 0x7f868c1e4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.903+0000 7f868a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f868c0cc000 0x7f868c151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.904+0000 7f868ad9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f868c0cf340 0x7f868c1e54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:59770/0 (socket says 172.21.15.5:59770) 2024-03-24T10:59:13.905 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.904+0000 7f868ad9d640 1 -- 172.21.15.5:0/3684804101 learned_addr learned my addr 172.21.15.5:0/3684804101 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:13.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.907+0000 7f868ad9d640 1 -- 172.21.15.5:0/3684804101 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f868c0cc000 msgr2=0x7f868c151a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:13.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.907+0000 7f868ad9d640 1 --2- 172.21.15.5:0/3684804101 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f868c0cc000 0x7f868c151a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.907+0000 7f868ad9d640 1 -- 172.21.15.5:0/3684804101 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f868c0ccb80 msgr2=0x7f868c1e4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:13.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.907+0000 7f868ad9d640 1 --2- 172.21.15.5:0/3684804101 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f868c0ccb80 0x7f868c1e4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:13.908 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.907+0000 7f868ad9d640 1 -- 172.21.15.5:0/3684804101 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86740094d0 con 0x7f868c0cf340 2024-03-24T10:59:13.911 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.910+0000 7f868ad9d640 1 --2- 172.21.15.5:0/3684804101 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f868c0cf340 0x7f868c1e54d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f868400c780 tx=0x7f868400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:13.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.914+0000 7f8682ffd640 1 -- 172.21.15.5:0/3684804101 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8684003d80 con 0x7f868c0cf340 2024-03-24T10:59:13.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.914+0000 7f8682ffd640 1 -- 172.21.15.5:0/3684804101 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f868400ce90 con 0x7f868c0cf340 2024-03-24T10:59:13.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.914+0000 7f8682ffd640 1 -- 172.21.15.5:0/3684804101 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86840052b0 con 0x7f868c0cf340 2024-03-24T10:59:13.915 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.914+0000 7f8690e84640 1 -- 172.21.15.5:0/3684804101 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f868c1f3c70 con 0x7f868c0cf340 2024-03-24T10:59:13.916 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.914+0000 7f8690e84640 1 -- 172.21.15.5:0/3684804101 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f868c1f36b0 con 0x7f868c0cf340 2024-03-24T10:59:13.917 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.916+0000 7f8690e84640 1 -- 172.21.15.5:0/3684804101 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f868c1efb70 con 0x7f868c0cf340 2024-03-24T10:59:13.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.934+0000 7f8682ffd640 1 -- 172.21.15.5:0/3684804101 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8684005410 con 0x7f868c0cf340 2024-03-24T10:59:13.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.934+0000 7f8682ffd640 1 --2- 172.21.15.5:0/3684804101 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f866c06b860 0x7f866c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:13.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.934+0000 7f868a59c640 1 --2- 172.21.15.5:0/3684804101 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f866c06b860 0x7f866c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:13.935 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.934+0000 7f868a59c640 1 --2- 172.21.15.5:0/3684804101 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f866c06b860 0x7f866c06dd20 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f867c009b40 tx=0x7f867c009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:13.938 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:13.937+0000 7f8682ffd640 1 -- 172.21.15.5:0/3684804101 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f8684058660 con 0x7f868c0cf340 2024-03-24T10:59:14.206 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.205+0000 7f8682ffd640 1 -- 172.21.15.5:0/3684804101 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8684054900 con 0x7f868c0cf340 2024-03-24T10:59:14.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.439+0000 7f8690e84640 1 -- 172.21.15.5:0/3684804101 --> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f868c15ae70 con 0x7f866c06b860 2024-03-24T10:59:14.439 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.439+0000 7f8682ffd640 1 -- 172.21.15.5:0/3684804101 <== mgr.4120 v2:172.21.15.5:6832/1102439181 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f867c011a60 con 0x7f866c06b860 2024-03-24T10:59:14.440 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 4 to deep-scrub 2024-03-24T10:59:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.442+0000 7f8690e84640 1 -- 172.21.15.5:0/3684804101 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f866c06b860 msgr2=0x7f866c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.442+0000 7f8690e84640 1 --2- 172.21.15.5:0/3684804101 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f866c06b860 0x7f866c06dd20 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f867c009b40 tx=0x7f867c009190 comp rx=0 tx=0).stop 2024-03-24T10:59:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.442+0000 7f8690e84640 1 -- 172.21.15.5:0/3684804101 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f868c0cf340 msgr2=0x7f868c1e54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.442+0000 7f8690e84640 1 --2- 172.21.15.5:0/3684804101 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f868c0cf340 0x7f868c1e54d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f868400c780 tx=0x7f868400cc40 comp rx=0 tx=0).stop 2024-03-24T10:59:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.442+0000 7f8690e84640 1 -- 172.21.15.5:0/3684804101 shutdown_connections 2024-03-24T10:59:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.442+0000 7f8690e84640 1 --2- 172.21.15.5:0/3684804101 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f866c06b860 0x7f866c06dd20 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.442+0000 7f8690e84640 1 --2- 172.21.15.5:0/3684804101 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f868c0ccb80 0x7f868c1e4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.442+0000 7f8690e84640 1 --2- 172.21.15.5:0/3684804101 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f868c0cf340 0x7f868c1e54d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.442+0000 7f8690e84640 1 --2- 172.21.15.5:0/3684804101 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f868c0cc000 0x7f868c151a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.442+0000 7f8690e84640 1 -- 172.21.15.5:0/3684804101 >> 172.21.15.5:0/3684804101 conn(0x7f868c0cfc30 msgr2=0x7f868c154190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.442+0000 7f8690e84640 1 -- 172.21.15.5:0/3684804101 shutdown_connections 2024-03-24T10:59:14.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.442+0000 7f8690e84640 1 -- 172.21.15.5:0/3684804101 wait complete. 2024-03-24T10:59:14.454 INFO:tasks.ceph:Scrubbing osd.5 2024-03-24T10:59:14.454 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-24T10:59:14.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.561+0000 7fd1075d0640 1 Processor -- start 2024-03-24T10:59:14.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.562+0000 7fd1075d0640 1 -- start start 2024-03-24T10:59:14.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.562+0000 7fd1075d0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd1000cc000 0x7fd10015e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:14.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.562+0000 7fd1075d0640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd1000cf340 0x7fd1000cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:14.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.562+0000 7fd1075d0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd1000ccb80 0x7fd10006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:14.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.562+0000 7fd1075d0640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fd10015a400 con 0x7fd1000cf340 2024-03-24T10:59:14.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.562+0000 7fd1075d0640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fd100154bd0 con 0x7fd1000ccb80 2024-03-24T10:59:14.563 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.562+0000 7fd1075d0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd1000cc3e0 con 0x7fd1000cc000 2024-03-24T10:59:14.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.563+0000 7fd104b6b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd1000cf340 0x7fd1000cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:14.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.563+0000 7fd105b6d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fd1000ccb80 0x7fd10006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:14.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.563+0000 7fd10536c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd1000cc000 0x7fd10015e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:14.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.563+0000 7fd104b6b640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fd1000cf340 0x7fd1000cc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:40838/0 (socket says 172.21.15.5:40838) 2024-03-24T10:59:14.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.563+0000 7fd104b6b640 1 -- 172.21.15.5:0/2821039777 learned_addr learned my addr 172.21.15.5:0/2821039777 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:14.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.566+0000 7fd104b6b640 1 -- 172.21.15.5:0/2821039777 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd1000cc000 msgr2=0x7fd10015e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:14.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.566+0000 7fd104b6b640 1 --2- 172.21.15.5:0/2821039777 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd1000cc000 0x7fd10015e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:14.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.566+0000 7fd104b6b640 1 -- 172.21.15.5:0/2821039777 >> v2:172.21.15.57:3300/0 conn(0x7fd1000ccb80 msgr2=0x7fd10006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:14.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.566+0000 7fd104b6b640 1 --2- 172.21.15.5:0/2821039777 >> v2:172.21.15.57:3300/0 conn(0x7fd1000ccb80 0x7fd10006bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:14.566 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.566+0000 7fd104b6b640 1 -- 172.21.15.5:0/2821039777 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd10015a5f0 con 0x7fd1000cf340 2024-03-24T10:59:14.568 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.568+0000 7fd104b6b640 1 --2- 172.21.15.5:0/2821039777 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1000cf340 0x7fd1000cc640 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd10008dcc0 tx=0x7fd0f002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a4d3be31ffb2415 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:14.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.570+0000 7fd0f7fff640 1 -- 172.21.15.5:0/2821039777 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f0030860 con 0x7fd1000cf340 2024-03-24T10:59:14.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.572+0000 7fd0f7fff640 1 -- 172.21.15.5:0/2821039777 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0f00309c0 con 0x7fd1000cf340 2024-03-24T10:59:14.572 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.572+0000 7fd0f7fff640 1 -- 172.21.15.5:0/2821039777 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f0030d50 con 0x7fd1000cf340 2024-03-24T10:59:14.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.573+0000 7fd1075d0640 1 -- 172.21.15.5:0/2821039777 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1000cf340 msgr2=0x7fd1000cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:14.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.573+0000 7fd1075d0640 1 --2- 172.21.15.5:0/2821039777 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1000cf340 0x7fd1000cc640 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd10008dcc0 tx=0x7fd0f002b9a0 comp rx=0 tx=0).stop 2024-03-24T10:59:14.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.573+0000 7fd1075d0640 1 -- 172.21.15.5:0/2821039777 shutdown_connections 2024-03-24T10:59:14.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.573+0000 7fd1075d0640 1 --2- 172.21.15.5:0/2821039777 >> v2:172.21.15.57:3300/0 conn(0x7fd1000ccb80 0x7fd10006bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:14.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.573+0000 7fd1075d0640 1 --2- 172.21.15.5:0/2821039777 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1000cf340 0x7fd1000cc640 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:14.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.573+0000 7fd1075d0640 1 --2- 172.21.15.5:0/2821039777 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd1000cc000 0x7fd10015e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:14.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.573+0000 7fd1075d0640 1 -- 172.21.15.5:0/2821039777 >> 172.21.15.5:0/2821039777 conn(0x7fd1000cfc30 msgr2=0x7fd1000d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:14.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.573+0000 7fd1075d0640 1 -- 172.21.15.5:0/2821039777 shutdown_connections 2024-03-24T10:59:14.573 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.573+0000 7fd1075d0640 1 -- 172.21.15.5:0/2821039777 wait complete. 2024-03-24T10:59:14.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.574+0000 7fd1075d0640 1 Processor -- start 2024-03-24T10:59:14.574 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.574+0000 7fd1075d0640 1 -- start start 2024-03-24T10:59:14.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.575+0000 7fd1075d0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd1000cc000 0x7fd1001f0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:14.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.575+0000 7fd1075d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1000ccb80 0x7fd1001f2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:14.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.575+0000 7fd1075d0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1000cf340 0x7fd1001e54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:14.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.575+0000 7fd1075d0640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fd1001582c0 con 0x7fd1000cf340 2024-03-24T10:59:14.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.575+0000 7fd1075d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fd100154060 con 0x7fd1000ccb80 2024-03-24T10:59:14.575 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.575+0000 7fd1075d0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fd100157a40 con 0x7fd1000cc000 2024-03-24T10:59:14.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.575+0000 7fd105b6d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1000cf340 0x7fd1001e54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:14.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.575+0000 7fd10536c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd1000cc000 0x7fd1001f0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:14.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.576+0000 7fd104b6b640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1000ccb80 0x7fd1001f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:14.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.576+0000 7fd105b6d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1000cf340 0x7fd1001e54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:40854/0 (socket says 172.21.15.5:40854) 2024-03-24T10:59:14.576 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.576+0000 7fd105b6d640 1 -- 172.21.15.5:0/1354253341 learned_addr learned my addr 172.21.15.5:0/1354253341 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:14.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.579+0000 7fd105b6d640 1 -- 172.21.15.5:0/1354253341 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd1000cc000 msgr2=0x7fd1001f0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:14.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.579+0000 7fd105b6d640 1 --2- 172.21.15.5:0/1354253341 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fd1000cc000 0x7fd1001f0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:14.579 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.579+0000 7fd105b6d640 1 -- 172.21.15.5:0/1354253341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1000ccb80 msgr2=0x7fd1001f2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:14.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.579+0000 7fd105b6d640 1 --2- 172.21.15.5:0/1354253341 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd1000ccb80 0x7fd1001f2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:14.580 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.579+0000 7fd105b6d640 1 -- 172.21.15.5:0/1354253341 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f00094d0 con 0x7fd1000cf340 2024-03-24T10:59:14.582 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.582+0000 7fd105b6d640 1 --2- 172.21.15.5:0/1354253341 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1000cf340 0x7fd1001e54a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fd0f800efc0 tx=0x7fd0f800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:14.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.587+0000 7fd0f5ffb640 1 -- 172.21.15.5:0/1354253341 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f8014480 con 0x7fd1000cf340 2024-03-24T10:59:14.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.587+0000 7fd0f5ffb640 1 -- 172.21.15.5:0/1354253341 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0f800f040 con 0x7fd1000cf340 2024-03-24T10:59:14.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.587+0000 7fd0f5ffb640 1 -- 172.21.15.5:0/1354253341 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f8014820 con 0x7fd1000cf340 2024-03-24T10:59:14.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.587+0000 7fd1075d0640 1 -- 172.21.15.5:0/1354253341 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd100157d00 con 0x7fd1000cf340 2024-03-24T10:59:14.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.587+0000 7fd1075d0640 1 -- 172.21.15.5:0/1354253341 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1001550e0 con 0x7fd1000cf340 2024-03-24T10:59:14.588 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.588+0000 7fd1075d0640 1 -- 172.21.15.5:0/1354253341 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd10015ae70 con 0x7fd1000cf340 2024-03-24T10:59:14.612 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.612+0000 7fd0f5ffb640 1 -- 172.21.15.5:0/1354253341 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd0f8014980 con 0x7fd1000cf340 2024-03-24T10:59:14.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.613+0000 7fd0f5ffb640 1 --2- 172.21.15.5:0/1354253341 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd0e406b860 0x7fd0e406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:14.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.613+0000 7fd10536c640 1 --2- 172.21.15.5:0/1354253341 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd0e406b860 0x7fd0e406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:14.613 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.613+0000 7fd10536c640 1 --2- 172.21.15.5:0/1354253341 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd0e406b860 0x7fd0e406dd20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc009ac0 tx=0x7fd0fc009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.617+0000 7fd0f5ffb640 1 -- 172.21.15.5:0/1354253341 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fd0f805a0f0 con 0x7fd1000cf340 2024-03-24T10:59:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.617+0000 7fd0f5ffb640 1 --2- 172.21.15.5:0/1354253341 >> [v2:172.21.15.57:6808/3044987466,v1:172.21.15.57:6809/3044987466] conn(0x7fd0e4073c90 0x7fd0e4076110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.617+0000 7fd0f5ffb640 1 -- 172.21.15.5:0/1354253341 --> [v2:172.21.15.57:6808/3044987466,v1:172.21.15.57:6809/3044987466] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd0e40767e0 con 0x7fd0e4073c90 2024-03-24T10:59:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.617+0000 7fd0f5ffb640 1 -- 172.21.15.5:0/1354253341 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fd0f8056390 con 0x7fd1000cf340 2024-03-24T10:59:14.618 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.618+0000 7fd104b6b640 1 --2- 172.21.15.5:0/1354253341 >> [v2:172.21.15.57:6808/3044987466,v1:172.21.15.57:6809/3044987466] conn(0x7fd0e4073c90 0x7fd0e4076110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:14.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.618+0000 7fd104b6b640 1 --2- 172.21.15.5:0/1354253341 >> [v2:172.21.15.57:6808/3044987466,v1:172.21.15.57:6809/3044987466] conn(0x7fd0e4073c90 0x7fd0e4076110 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:14.619 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.619+0000 7fd0f5ffb640 1 -- 172.21.15.5:0/1354253341 <== osd.5 v2:172.21.15.57:6808/3044987466 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd0e40767e0 con 0x7fd0e4073c90 2024-03-24T10:59:14.632 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.633+0000 7fd1075d0640 1 -- 172.21.15.5:0/1354253341 --> [v2:172.21.15.57:6808/3044987466,v1:172.21.15.57:6809/3044987466] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd10015ab90 con 0x7fd0e4073c90 2024-03-24T10:59:14.633 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.633+0000 7fd0f5ffb640 1 -- 172.21.15.5:0/1354253341 <== osd.5 v2:172.21.15.57:6808/3044987466 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd0f003f040 con 0x7fd0e4073c90 2024-03-24T10:59:14.633 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-24T10:59:14.633 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T10:59:14.633 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-24T10:59:14.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.633+0000 7fd1075d0640 1 -- 172.21.15.5:0/1354253341 >> [v2:172.21.15.57:6808/3044987466,v1:172.21.15.57:6809/3044987466] conn(0x7fd0e4073c90 msgr2=0x7fd0e4076110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:14.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.633+0000 7fd1075d0640 1 --2- 172.21.15.5:0/1354253341 >> [v2:172.21.15.57:6808/3044987466,v1:172.21.15.57:6809/3044987466] conn(0x7fd0e4073c90 0x7fd0e4076110 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:14.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.633+0000 7fd1075d0640 1 -- 172.21.15.5:0/1354253341 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd0e406b860 msgr2=0x7fd0e406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:14.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.633+0000 7fd1075d0640 1 --2- 172.21.15.5:0/1354253341 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fd0e406b860 0x7fd0e406dd20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc009ac0 tx=0x7fd0fc009190 comp rx=0 tx=0).stop 2024-03-24T10:59:14.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.633+0000 7fd1075d0640 1 -- 172.21.15.5:0/1354253341 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1000cf340 msgr2=0x7fd1001e54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:14.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.633+0000 7fd1075d0640 1 --2- 172.21.15.5:0/1354253341 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fd1000cf340 0x7fd1001e54a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fd0f800efc0 tx=0x7fd0f800c490 comp rx=0 tx=0).stop 2024-03-24T10:59:14.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.633+0000 7fd105b6d640 1 -- 172.21.15.5:0/1354253341 reap_dead start 2024-03-24T10:59:14.635 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.633+0000 7fd1075d0640 1 -- 172.21.15.5:0/1354253341 shutdown_connections 2024-03-24T10:59:14.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.633+0000 7fd1075d0640 1 -- 172.21.15.5:0/1354253341 >> 172.21.15.5:0/1354253341 conn(0x7fd1000cfc30 msgr2=0x7fd1000c7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:14.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.634+0000 7fd1075d0640 1 -- 172.21.15.5:0/1354253341 shutdown_connections 2024-03-24T10:59:14.636 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.634+0000 7fd1075d0640 1 -- 172.21.15.5:0/1354253341 wait complete. 2024-03-24T10:59:14.644 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-24T10:59:14.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.747+0000 7f77f4fdd640 1 Processor -- start 2024-03-24T10:59:14.747 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.747+0000 7f77f4fdd640 1 -- start start 2024-03-24T10:59:14.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.748+0000 7f77f4fdd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f77f00cc000 0x7f77f015e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:14.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.748+0000 7f77f4fdd640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77f00cc3e0 con 0x7f77f006bf50 2024-03-24T10:59:14.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.748+0000 7f77f4fdd640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77f015a5f0 con 0x7f77f00cf340 2024-03-24T10:59:14.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.748+0000 7f77f4fdd640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f77f015a400 con 0x7f77f00cc000 2024-03-24T10:59:14.748 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.748+0000 7f77ee59c640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f77f00cc000 0x7f77f015e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:14.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.749+0000 7f77e7fff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f77f00cf340 0x7f77f00cc640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:47460/0 (socket says 172.21.15.5:47460) 2024-03-24T10:59:14.749 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.749+0000 7f77e7fff640 1 -- 172.21.15.5:0/2499957375 learned_addr learned my addr 172.21.15.5:0/2499957375 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:59:14.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.751+0000 7f77ee59c640 1 -- 172.21.15.5:0/2499957375 >> v1:172.21.15.57:6789/0 conn(0x7f77f00cf340 legacy=0x7f77f00cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:14.751 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.751+0000 7f77ee59c640 1 -- 172.21.15.5:0/2499957375 >> v1:172.21.15.5:6789/0 conn(0x7f77f006bf50 legacy=0x7f77f006c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:14.752 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.751+0000 7f77ee59c640 1 -- 172.21.15.5:0/2499957375 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77f015a8b0 con 0x7f77f00cc000 2024-03-24T10:59:14.753 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.754+0000 7f77ee59c640 1 --2- 172.21.15.5:0/2499957375 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77f00cc000 0x7f77f015e3b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f77e00097a0 tx=0x7f77e002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e4728478a2f920f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:14.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.759+0000 7f77edd9b640 1 -- 172.21.15.5:0/2499957375 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e0031830 con 0x7f77f00cc000 2024-03-24T10:59:14.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.759+0000 7f77edd9b640 1 -- 172.21.15.5:0/2499957375 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77e0031990 con 0x7f77f00cc000 2024-03-24T10:59:14.759 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.759+0000 7f77edd9b640 1 -- 172.21.15.5:0/2499957375 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e0031d20 con 0x7f77f00cc000 2024-03-24T10:59:14.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.760+0000 7f77f4fdd640 1 -- 172.21.15.5:0/2499957375 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77f00cc000 msgr2=0x7f77f015e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:14.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.760+0000 7f77f4fdd640 1 --2- 172.21.15.5:0/2499957375 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77f00cc000 0x7f77f015e3b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f77e00097a0 tx=0x7f77e002b8d0 comp rx=0 tx=0).stop 2024-03-24T10:59:14.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.760+0000 7f77f4fdd640 1 -- 172.21.15.5:0/2499957375 shutdown_connections 2024-03-24T10:59:14.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.760+0000 7f77f4fdd640 1 --2- 172.21.15.5:0/2499957375 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77f00cc000 0x7f77f015e3b0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:14.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.760+0000 7f77f4fdd640 1 -- 172.21.15.5:0/2499957375 >> 172.21.15.5:0/2499957375 conn(0x7f77f00cfc30 msgr2=0x7f77f00d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:14.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.760+0000 7f77f4fdd640 1 -- 172.21.15.5:0/2499957375 shutdown_connections 2024-03-24T10:59:14.760 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.760+0000 7f77f4fdd640 1 -- 172.21.15.5:0/2499957375 wait complete. 2024-03-24T10:59:14.761 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.762+0000 7f77f4fdd640 1 Processor -- start 2024-03-24T10:59:14.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.762+0000 7f77f4fdd640 1 -- start start 2024-03-24T10:59:14.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.762+0000 7f77f4fdd640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77f006bf50 0x7f77f00b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:14.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.762+0000 7f77f4fdd640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77f00cc000 0x7f77f00af590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:14.762 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.762+0000 7f77f4fdd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f77f00cf340 0x7f77f00c8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:14.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.762+0000 7f77f4fdd640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f77f00b01e0 con 0x7f77f00cc000 2024-03-24T10:59:14.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.762+0000 7f77f4fdd640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f77f00afc20 con 0x7f77f006bf50 2024-03-24T10:59:14.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.762+0000 7f77f4fdd640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f77f00aff00 con 0x7f77f00cf340 2024-03-24T10:59:14.763 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.763+0000 7f77e7fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77f00cc000 0x7f77f00af590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:14.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.763+0000 7f77ee59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77f006bf50 0x7f77f00b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:14.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.763+0000 7f77eed9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f77f00cf340 0x7f77f00c8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:14.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.764+0000 7f77e7fff640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77f00cc000 0x7f77f00af590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:40876/0 (socket says 172.21.15.5:40876) 2024-03-24T10:59:14.764 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.764+0000 7f77e7fff640 1 -- 172.21.15.5:0/2867792189 learned_addr learned my addr 172.21.15.5:0/2867792189 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:14.768 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.768+0000 7f77e7fff640 1 -- 172.21.15.5:0/2867792189 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f77f00cf340 msgr2=0x7f77f00c8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:14.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.768+0000 7f77e7fff640 1 --2- 172.21.15.5:0/2867792189 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f77f00cf340 0x7f77f00c8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:14.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.769+0000 7f77e7fff640 1 -- 172.21.15.5:0/2867792189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77f006bf50 msgr2=0x7f77f00b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:14.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.769+0000 7f77e7fff640 1 --2- 172.21.15.5:0/2867792189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77f006bf50 0x7f77f00b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:14.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.769+0000 7f77e7fff640 1 -- 172.21.15.5:0/2867792189 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77e0009400 con 0x7f77f00cc000 2024-03-24T10:59:14.769 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.769+0000 7f77eed9d640 1 --2- 172.21.15.5:0/2867792189 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f77f00cf340 0x7f77f00c8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:59:14.771 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.772+0000 7f77e7fff640 1 --2- 172.21.15.5:0/2867792189 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77f00cc000 0x7f77f00af590 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f77d800c4a0 tx=0x7f77d800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:14.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.777+0000 7f77e6ffd640 1 -- 172.21.15.5:0/2867792189 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d8005ce0 con 0x7f77f00cc000 2024-03-24T10:59:14.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.777+0000 7f77e6ffd640 1 -- 172.21.15.5:0/2867792189 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77d8005e40 con 0x7f77f00cc000 2024-03-24T10:59:14.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.777+0000 7f77f4fdd640 1 -- 172.21.15.5:0/2867792189 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77f00b18d0 con 0x7f77f00cc000 2024-03-24T10:59:14.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.777+0000 7f77e6ffd640 1 -- 172.21.15.5:0/2867792189 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d8007270 con 0x7f77f00cc000 2024-03-24T10:59:14.777 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.777+0000 7f77f4fdd640 1 -- 172.21.15.5:0/2867792189 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77f00b1e90 con 0x7f77f00cc000 2024-03-24T10:59:14.778 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:14.778+0000 7f77f4fdd640 1 -- 172.21.15.5:0/2867792189 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77f00b1710 con 0x7f77f00cc000 2024-03-24T10:59:15.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.058+0000 7f77e6ffd640 1 -- 172.21.15.5:0/2867792189 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f77d8007490 con 0x7f77f00cc000 2024-03-24T10:59:15.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.058+0000 7f77e6ffd640 1 --2- 172.21.15.5:0/2867792189 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f77d006b860 0x7f77d006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:15.058 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.058+0000 7f77ee59c640 1 --2- 172.21.15.5:0/2867792189 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f77d006b860 0x7f77d006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:15.059 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.059+0000 7f77ee59c640 1 --2- 172.21.15.5:0/2867792189 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f77d006b860 0x7f77d006dd20 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f77e0036000 tx=0x7f77e003c040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:15.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.080+0000 7f77e6ffd640 1 -- 172.21.15.5:0/2867792189 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f77d8058370 con 0x7f77f00cc000 2024-03-24T10:59:15.080 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.081+0000 7f77e6ffd640 1 -- 172.21.15.5:0/2867792189 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f77d8054610 con 0x7f77f00cc000 2024-03-24T10:59:15.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.265+0000 7f77f4fdd640 1 -- 172.21.15.5:0/2867792189 --> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f77f015ae70 con 0x7f77d006b860 2024-03-24T10:59:15.265 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.265+0000 7f77e6ffd640 1 -- 172.21.15.5:0/2867792189 <== mgr.4120 v2:172.21.15.5:6832/1102439181 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f77e003a420 con 0x7f77d006b860 2024-03-24T10:59:15.265 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 5 to deep-scrub 2024-03-24T10:59:15.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.267+0000 7f77f4fdd640 1 -- 172.21.15.5:0/2867792189 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f77d006b860 msgr2=0x7f77d006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:15.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.267+0000 7f77f4fdd640 1 --2- 172.21.15.5:0/2867792189 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f77d006b860 0x7f77d006dd20 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f77e0036000 tx=0x7f77e003c040 comp rx=0 tx=0).stop 2024-03-24T10:59:15.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.267+0000 7f77f4fdd640 1 -- 172.21.15.5:0/2867792189 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77f00cc000 msgr2=0x7f77f00af590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:15.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.267+0000 7f77f4fdd640 1 --2- 172.21.15.5:0/2867792189 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77f00cc000 0x7f77f00af590 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f77d800c4a0 tx=0x7f77d800c960 comp rx=0 tx=0).stop 2024-03-24T10:59:15.267 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.267+0000 7f77f4fdd640 1 -- 172.21.15.5:0/2867792189 shutdown_connections 2024-03-24T10:59:15.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.267+0000 7f77f4fdd640 1 --2- 172.21.15.5:0/2867792189 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f77d006b860 0x7f77d006dd20 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:15.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.267+0000 7f77f4fdd640 1 --2- 172.21.15.5:0/2867792189 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f77f00cc000 0x7f77f00af590 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:15.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.267+0000 7f77f4fdd640 1 --2- 172.21.15.5:0/2867792189 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f77f00cf340 0x7f77f00c8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:15.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.267+0000 7f77f4fdd640 1 --2- 172.21.15.5:0/2867792189 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f77f006bf50 0x7f77f00b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:15.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.267+0000 7f77f4fdd640 1 -- 172.21.15.5:0/2867792189 >> 172.21.15.5:0/2867792189 conn(0x7f77f00cfc30 msgr2=0x7f77f00c7660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:15.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.267+0000 7f77f4fdd640 1 -- 172.21.15.5:0/2867792189 shutdown_connections 2024-03-24T10:59:15.268 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.267+0000 7f77f4fdd640 1 -- 172.21.15.5:0/2867792189 wait complete. 2024-03-24T10:59:15.282 INFO:tasks.ceph:Scrubbing osd.6 2024-03-24T10:59:15.282 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-24T10:59:15.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.391+0000 7eff62b93640 1 Processor -- start 2024-03-24T10:59:15.391 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.391+0000 7eff62b93640 1 -- start start 2024-03-24T10:59:15.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.392+0000 7eff62b93640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7eff5c0cf340 0x7eff5c0cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:15.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.392+0000 7eff62b93640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7eff5c0ccb80 0x7eff5c06bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:15.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.392+0000 7eff62b93640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff5c0cc3e0 con 0x7eff5c0cc000 2024-03-24T10:59:15.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.392+0000 7eff62b93640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7eff5c15a400 con 0x7eff5c0cf340 2024-03-24T10:59:15.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.392+0000 7eff62b93640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7eff5c154bd0 con 0x7eff5c0ccb80 2024-03-24T10:59:15.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.393+0000 7eff61130640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7eff5c0ccb80 0x7eff5c06bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:15.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.393+0000 7eff5bfff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7eff5c0cf340 0x7eff5c0cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:15.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.393+0000 7eff5bfff640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7eff5c0cf340 0x7eff5c0cc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:40878/0 (socket says 172.21.15.5:40878) 2024-03-24T10:59:15.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.393+0000 7eff5bfff640 1 -- 172.21.15.5:0/444962119 learned_addr learned my addr 172.21.15.5:0/444962119 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:15.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.398+0000 7eff5b7fe640 1 -- 172.21.15.5:0/444962119 <== mon.1 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 531214878 0 0) 0x7eff5c0cc3e0 con 0x7eff5c0cc000 2024-03-24T10:59:15.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.398+0000 7eff5b7fe640 1 -- 172.21.15.5:0/444962119 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff4c003520 con 0x7eff5c0cc000 2024-03-24T10:59:15.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.399+0000 7eff5bfff640 1 -- 172.21.15.5:0/444962119 >> v2:172.21.15.57:3300/0 conn(0x7eff5c0ccb80 msgr2=0x7eff5c06bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:15.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.399+0000 7eff5bfff640 1 --2- 172.21.15.5:0/444962119 >> v2:172.21.15.57:3300/0 conn(0x7eff5c0ccb80 0x7eff5c06bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:15.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.399+0000 7eff5bfff640 1 -- 172.21.15.5:0/444962119 >> v1:172.21.15.57:6789/0 conn(0x7eff5c0cc000 legacy=0x7eff5c15e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:15.399 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.399+0000 7eff5bfff640 1 -- 172.21.15.5:0/444962119 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff5c15a5f0 con 0x7eff5c0cf340 2024-03-24T10:59:15.404 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.404+0000 7eff5bfff640 1 --2- 172.21.15.5:0/444962119 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff5c0cf340 0x7eff5c0cc640 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7eff5c0ca230 tx=0x7eff4802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c825cd2d3302312 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:15.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.411+0000 7eff5b7fe640 1 -- 172.21.15.5:0/444962119 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff48030860 con 0x7eff5c0cf340 2024-03-24T10:59:15.411 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.411+0000 7eff5b7fe640 1 -- 172.21.15.5:0/444962119 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff480309c0 con 0x7eff5c0cf340 2024-03-24T10:59:15.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.411+0000 7eff5b7fe640 1 -- 172.21.15.5:0/444962119 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff48030d50 con 0x7eff5c0cf340 2024-03-24T10:59:15.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.412+0000 7eff62b93640 1 -- 172.21.15.5:0/444962119 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff5c0cf340 msgr2=0x7eff5c0cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:15.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.412+0000 7eff62b93640 1 --2- 172.21.15.5:0/444962119 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff5c0cf340 0x7eff5c0cc640 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7eff5c0ca230 tx=0x7eff4802b9a0 comp rx=0 tx=0).stop 2024-03-24T10:59:15.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.412+0000 7eff62b93640 1 -- 172.21.15.5:0/444962119 shutdown_connections 2024-03-24T10:59:15.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.412+0000 7eff62b93640 1 --2- 172.21.15.5:0/444962119 >> v2:172.21.15.57:3300/0 conn(0x7eff5c0ccb80 0x7eff5c06bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:15.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.412+0000 7eff62b93640 1 --2- 172.21.15.5:0/444962119 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff5c0cf340 0x7eff5c0cc640 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:15.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.412+0000 7eff62b93640 1 -- 172.21.15.5:0/444962119 >> 172.21.15.5:0/444962119 conn(0x7eff5c0cfc30 msgr2=0x7eff5c0d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:15.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.413+0000 7eff62b93640 1 -- 172.21.15.5:0/444962119 shutdown_connections 2024-03-24T10:59:15.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.413+0000 7eff62b93640 1 -- 172.21.15.5:0/444962119 wait complete. 2024-03-24T10:59:15.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.414+0000 7eff62b93640 1 Processor -- start 2024-03-24T10:59:15.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.414+0000 7eff62b93640 1 -- start start 2024-03-24T10:59:15.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.415+0000 7eff62b93640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7eff5c0cc000 0x7eff5c1f0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:15.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.415+0000 7eff62b93640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff5c0ccb80 0x7eff5c154390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:15.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.415+0000 7eff62b93640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff5c0cf340 0x7eff5c0c5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:15.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.415+0000 7eff62b93640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c1f11a0 con 0x7eff5c0cf340 2024-03-24T10:59:15.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.415+0000 7eff62b93640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7eff5c157400 con 0x7eff5c0ccb80 2024-03-24T10:59:15.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.415+0000 7eff62b93640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7eff5c1548d0 con 0x7eff5c0cc000 2024-03-24T10:59:15.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.415+0000 7eff5bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff5c0ccb80 0x7eff5c154390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:15.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.415+0000 7eff6092f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7eff5c0cc000 0x7eff5c1f0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:15.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.416+0000 7eff5bfff640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff5c0ccb80 0x7eff5c154390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:49452/0 (socket says 172.21.15.5:49452) 2024-03-24T10:59:15.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.416+0000 7eff5bfff640 1 -- 172.21.15.5:0/3823313482 learned_addr learned my addr 172.21.15.5:0/3823313482 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:15.416 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.416+0000 7eff6092f640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7eff5c0cc000 0x7eff5c1f0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:59806/0 (socket says 172.21.15.5:59806) 2024-03-24T10:59:15.418 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.418+0000 7eff61130640 1 --2- 172.21.15.5:0/3823313482 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff5c0cf340 0x7eff5c0c5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:15.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.421+0000 7eff5bfff640 1 -- 172.21.15.5:0/3823313482 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7eff5c0cc000 msgr2=0x7eff5c1f0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:15.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.421+0000 7eff5bfff640 1 --2- 172.21.15.5:0/3823313482 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7eff5c0cc000 0x7eff5c1f0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:15.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.421+0000 7eff5bfff640 1 -- 172.21.15.5:0/3823313482 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff5c0cf340 msgr2=0x7eff5c0c5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:15.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.421+0000 7eff5bfff640 1 --2- 172.21.15.5:0/3823313482 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7eff5c0cf340 0x7eff5c0c5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:15.421 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.421+0000 7eff5bfff640 1 -- 172.21.15.5:0/3823313482 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff480094d0 con 0x7eff5c0ccb80 2024-03-24T10:59:15.426 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.426+0000 7eff5bfff640 1 --2- 172.21.15.5:0/3823313482 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff5c0ccb80 0x7eff5c154390 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7eff4802b9e0 tx=0x7eff48004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:15.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.538+0000 7eff597fa640 1 -- 172.21.15.5:0/3823313482 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff4803e040 con 0x7eff5c0ccb80 2024-03-24T10:59:15.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.538+0000 7eff597fa640 1 -- 172.21.15.5:0/3823313482 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff48039c30 con 0x7eff5c0ccb80 2024-03-24T10:59:15.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.538+0000 7eff597fa640 1 -- 172.21.15.5:0/3823313482 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff480443f0 con 0x7eff5c0ccb80 2024-03-24T10:59:15.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.538+0000 7eff62b93640 1 -- 172.21.15.5:0/3823313482 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff5c0c4420 con 0x7eff5c0ccb80 2024-03-24T10:59:15.538 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.538+0000 7eff62b93640 1 -- 172.21.15.5:0/3823313482 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff5c0c49c0 con 0x7eff5c0ccb80 2024-03-24T10:59:15.539 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.538+0000 7eff62b93640 1 -- 172.21.15.5:0/3823313482 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7eff5c0ca4b0 con 0x7eff5c0ccb80 2024-03-24T10:59:15.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.564+0000 7eff597fa640 1 -- 172.21.15.5:0/3823313482 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7eff4804d3b0 con 0x7eff5c0ccb80 2024-03-24T10:59:15.564 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.565+0000 7eff597fa640 1 --2- 172.21.15.5:0/3823313482 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7eff4006b860 0x7eff4006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:15.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.565+0000 7eff6092f640 1 --2- 172.21.15.5:0/3823313482 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7eff4006b860 0x7eff4006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:15.565 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.565+0000 7eff6092f640 1 --2- 172.21.15.5:0/3823313482 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7eff4006b860 0x7eff4006dd20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7eff500011e0 tx=0x7eff50009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:15.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.569+0000 7eff597fa640 1 -- 172.21.15.5:0/3823313482 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7eff4807b830 con 0x7eff5c0ccb80 2024-03-24T10:59:15.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.570+0000 7eff597fa640 1 --2- 172.21.15.5:0/3823313482 >> [v2:172.21.15.57:6800/4290897445,v1:172.21.15.57:6801/4290897445] conn(0x7eff40073c90 0x7eff40076110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:15.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.570+0000 7eff597fa640 1 -- 172.21.15.5:0/3823313482 --> [v2:172.21.15.57:6800/4290897445,v1:172.21.15.57:6801/4290897445] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7eff400767e0 con 0x7eff40073c90 2024-03-24T10:59:15.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.570+0000 7eff597fa640 1 -- 172.21.15.5:0/3823313482 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7eff48077ad0 con 0x7eff5c0ccb80 2024-03-24T10:59:15.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.570+0000 7eff61130640 1 --2- 172.21.15.5:0/3823313482 >> [v2:172.21.15.57:6800/4290897445,v1:172.21.15.57:6801/4290897445] conn(0x7eff40073c90 0x7eff40076110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:15.570 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.570+0000 7eff61130640 1 --2- 172.21.15.5:0/3823313482 >> [v2:172.21.15.57:6800/4290897445,v1:172.21.15.57:6801/4290897445] conn(0x7eff40073c90 0x7eff40076110 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:15.571 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.571+0000 7eff597fa640 1 -- 172.21.15.5:0/3823313482 <== osd.6 v2:172.21.15.57:6800/4290897445 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7eff400767e0 con 0x7eff40073c90 2024-03-24T10:59:15.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.585+0000 7eff62b93640 1 -- 172.21.15.5:0/3823313482 --> [v2:172.21.15.57:6800/4290897445,v1:172.21.15.57:6801/4290897445] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7eff5c075750 con 0x7eff40073c90 2024-03-24T10:59:15.585 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.585+0000 7eff597fa640 1 -- 172.21.15.5:0/3823313482 <== osd.6 v2:172.21.15.57:6800/4290897445 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7eff5c075750 con 0x7eff40073c90 2024-03-24T10:59:15.585 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-24T10:59:15.586 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T10:59:15.586 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-24T10:59:15.586 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.585+0000 7eff62b93640 1 -- 172.21.15.5:0/3823313482 >> [v2:172.21.15.57:6800/4290897445,v1:172.21.15.57:6801/4290897445] conn(0x7eff40073c90 msgr2=0x7eff40076110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:15.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.585+0000 7eff62b93640 1 --2- 172.21.15.5:0/3823313482 >> [v2:172.21.15.57:6800/4290897445,v1:172.21.15.57:6801/4290897445] conn(0x7eff40073c90 0x7eff40076110 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:15.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.586+0000 7eff62b93640 1 -- 172.21.15.5:0/3823313482 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7eff4006b860 msgr2=0x7eff4006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:15.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.586+0000 7eff62b93640 1 --2- 172.21.15.5:0/3823313482 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7eff4006b860 0x7eff4006dd20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7eff500011e0 tx=0x7eff50009040 comp rx=0 tx=0).stop 2024-03-24T10:59:15.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.586+0000 7eff62b93640 1 -- 172.21.15.5:0/3823313482 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff5c0ccb80 msgr2=0x7eff5c154390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:15.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.586+0000 7eff62b93640 1 --2- 172.21.15.5:0/3823313482 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7eff5c0ccb80 0x7eff5c154390 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7eff4802b9e0 tx=0x7eff48004730 comp rx=0 tx=0).stop 2024-03-24T10:59:15.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.586+0000 7eff61130640 1 -- 172.21.15.5:0/3823313482 reap_dead start 2024-03-24T10:59:15.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.586+0000 7eff62b93640 1 -- 172.21.15.5:0/3823313482 shutdown_connections 2024-03-24T10:59:15.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.586+0000 7eff62b93640 1 -- 172.21.15.5:0/3823313482 >> 172.21.15.5:0/3823313482 conn(0x7eff5c0cfc30 msgr2=0x7eff5c158cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:15.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.586+0000 7eff62b93640 1 -- 172.21.15.5:0/3823313482 shutdown_connections 2024-03-24T10:59:15.587 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.586+0000 7eff62b93640 1 -- 172.21.15.5:0/3823313482 wait complete. 2024-03-24T10:59:15.596 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-24T10:59:15.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.699+0000 7f69075c9640 1 Processor -- start 2024-03-24T10:59:15.700 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.700+0000 7f69075c9640 1 -- start start 2024-03-24T10:59:15.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.700+0000 7f69075c9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f69000cc000 0x7f690015e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:15.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.700+0000 7f69075c9640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f69000cf340 0x7f69000cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:15.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.700+0000 7f69075c9640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69000cc3e0 con 0x7f69000ccb80 2024-03-24T10:59:15.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.700+0000 7f69075c9640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7f690015a400 con 0x7f69000cf340 2024-03-24T10:59:15.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.700+0000 7f69075c9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f6900154bd0 con 0x7f69000cc000 2024-03-24T10:59:15.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.701+0000 7f6904b64640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f69000cf340 0x7f69000cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:15.701 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.701+0000 7f6905365640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f69000cc000 0x7f690015e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:15.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.702+0000 7f6904b64640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7f69000cf340 0x7f69000cc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:40906/0 (socket says 172.21.15.5:40906) 2024-03-24T10:59:15.702 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.702+0000 7f6904b64640 1 -- 172.21.15.5:0/870536846 learned_addr learned my addr 172.21.15.5:0/870536846 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:15.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.705+0000 7f6904b64640 1 -- 172.21.15.5:0/870536846 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f69000cc000 msgr2=0x7f690015e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:15.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.705+0000 7f6904b64640 1 --2- 172.21.15.5:0/870536846 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f69000cc000 0x7f690015e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:15.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.705+0000 7f6904b64640 1 -- 172.21.15.5:0/870536846 >> v1:172.21.15.5:6789/0 conn(0x7f69000ccb80 legacy=0x7f690006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:15.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.705+0000 7f6904b64640 1 -- 172.21.15.5:0/870536846 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f690015a5f0 con 0x7f69000cf340 2024-03-24T10:59:15.705 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.705+0000 7f6905365640 1 --2- 172.21.15.5:0/870536846 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f69000cc000 0x7f690015e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:59:15.707 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.707+0000 7f6904b64640 1 --2- 172.21.15.5:0/870536846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69000cf340 0x7f69000cc640 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f69000b27e0 tx=0x7f68f002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e2bc15a1c979443 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:15.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.713+0000 7f68f7fff640 1 -- 172.21.15.5:0/870536846 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f0030840 con 0x7f69000cf340 2024-03-24T10:59:15.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.713+0000 7f68f7fff640 1 -- 172.21.15.5:0/870536846 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68f00309a0 con 0x7f69000cf340 2024-03-24T10:59:15.713 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.713+0000 7f68f7fff640 1 -- 172.21.15.5:0/870536846 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f0030d30 con 0x7f69000cf340 2024-03-24T10:59:15.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.714+0000 7f69075c9640 1 -- 172.21.15.5:0/870536846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69000cf340 msgr2=0x7f69000cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:15.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.714+0000 7f69075c9640 1 --2- 172.21.15.5:0/870536846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69000cf340 0x7f69000cc640 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f69000b27e0 tx=0x7f68f002b980 comp rx=0 tx=0).stop 2024-03-24T10:59:15.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.714+0000 7f69075c9640 1 -- 172.21.15.5:0/870536846 shutdown_connections 2024-03-24T10:59:15.714 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.714+0000 7f69075c9640 1 --2- 172.21.15.5:0/870536846 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69000cf340 0x7f69000cc640 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:15.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.714+0000 7f69075c9640 1 --2- 172.21.15.5:0/870536846 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f69000cc000 0x7f690015e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:15.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.714+0000 7f69075c9640 1 -- 172.21.15.5:0/870536846 >> 172.21.15.5:0/870536846 conn(0x7f69000cfc30 msgr2=0x7f69000d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:15.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.714+0000 7f69075c9640 1 -- 172.21.15.5:0/870536846 shutdown_connections 2024-03-24T10:59:15.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.714+0000 7f69075c9640 1 -- 172.21.15.5:0/870536846 wait complete. 2024-03-24T10:59:15.715 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.715+0000 7f69075c9640 1 Processor -- start 2024-03-24T10:59:15.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.715+0000 7f69075c9640 1 -- start start 2024-03-24T10:59:15.716 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.716+0000 7f69075c9640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69000cc000 0x7f69000b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:15.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.716+0000 7f69075c9640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69000ccb80 0x7f69001f3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:15.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.716+0000 7f69075c9640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f69000cf340 0x7f6900151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:15.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.716+0000 7f69075c9640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f69001f0120 con 0x7f69000cc000 2024-03-24T10:59:15.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.716+0000 7f69075c9640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f69001f10f0 con 0x7f69000ccb80 2024-03-24T10:59:15.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.716+0000 7f69075c9640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f69001e4190 con 0x7f69000cf340 2024-03-24T10:59:15.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.717+0000 7f6905365640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69000cc000 0x7f69000b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:15.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.717+0000 7f6905b66640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f69000cf340 0x7f6900151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:15.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.717+0000 7f6904b64640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69000ccb80 0x7f69001f3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:15.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.717+0000 7f6905365640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69000cc000 0x7f69000b19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:40920/0 (socket says 172.21.15.5:40920) 2024-03-24T10:59:15.717 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.717+0000 7f6905365640 1 -- 172.21.15.5:0/914835585 learned_addr learned my addr 172.21.15.5:0/914835585 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:15.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.721+0000 7f6905365640 1 -- 172.21.15.5:0/914835585 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f69000cf340 msgr2=0x7f6900151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:15.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.721+0000 7f6905365640 1 --2- 172.21.15.5:0/914835585 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f69000cf340 0x7f6900151d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:15.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.721+0000 7f6905365640 1 -- 172.21.15.5:0/914835585 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69000ccb80 msgr2=0x7f69001f3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:15.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.721+0000 7f6905365640 1 --2- 172.21.15.5:0/914835585 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69000ccb80 0x7f69001f3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:15.721 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.721+0000 7f6905365640 1 -- 172.21.15.5:0/914835585 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68fc009400 con 0x7f69000cc000 2024-03-24T10:59:15.723 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.723+0000 7f6905365640 1 --2- 172.21.15.5:0/914835585 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69000cc000 0x7f69000b19d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f68fc00ebe0 tx=0x7f68fc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:15.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.727+0000 7f68f5ffb640 1 -- 172.21.15.5:0/914835585 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68fc009760 con 0x7f69000cc000 2024-03-24T10:59:15.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.727+0000 7f68f5ffb640 1 -- 172.21.15.5:0/914835585 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68fc00cc30 con 0x7f69000cc000 2024-03-24T10:59:15.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.727+0000 7f69075c9640 1 -- 172.21.15.5:0/914835585 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68f00094d0 con 0x7f69000cc000 2024-03-24T10:59:15.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.727+0000 7f68f5ffb640 1 -- 172.21.15.5:0/914835585 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68fc00eee0 con 0x7f69000cc000 2024-03-24T10:59:15.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.727+0000 7f69075c9640 1 -- 172.21.15.5:0/914835585 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69001f0610 con 0x7f69000cc000 2024-03-24T10:59:15.729 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.728+0000 7f69075c9640 1 -- 172.21.15.5:0/914835585 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69000ca280 con 0x7f69000cc000 2024-03-24T10:59:15.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.994+0000 7f68f5ffb640 1 -- 172.21.15.5:0/914835585 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f68fc013470 con 0x7f69000cc000 2024-03-24T10:59:15.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.995+0000 7f68f5ffb640 1 --2- 172.21.15.5:0/914835585 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f68e406b860 0x7f68e406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:15.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.995+0000 7f6904b64640 1 --2- 172.21.15.5:0/914835585 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f68e406b860 0x7f68e406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:15.995 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:15.995+0000 7f6904b64640 1 --2- 172.21.15.5:0/914835585 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f68e406b860 0x7f68e406dd20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f68f0036000 tx=0x7f68f003b040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:16.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.012+0000 7f68f5ffb640 1 -- 172.21.15.5:0/914835585 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f68fc059480 con 0x7f69000cc000 2024-03-24T10:59:16.012 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.012+0000 7f68f5ffb640 1 -- 172.21.15.5:0/914835585 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f68fc055720 con 0x7f69000cc000 2024-03-24T10:59:16.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.199+0000 7f69075c9640 1 -- 172.21.15.5:0/914835585 --> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f690015ae70 con 0x7f68e406b860 2024-03-24T10:59:16.200 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.200+0000 7f68f5ffb640 1 -- 172.21.15.5:0/914835585 <== mgr.4120 v2:172.21.15.5:6832/1102439181 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f68f003f040 con 0x7f68e406b860 2024-03-24T10:59:16.200 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 6 to deep-scrub 2024-03-24T10:59:16.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.202+0000 7f69075c9640 1 -- 172.21.15.5:0/914835585 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f68e406b860 msgr2=0x7f68e406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:16.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.202+0000 7f69075c9640 1 --2- 172.21.15.5:0/914835585 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f68e406b860 0x7f68e406dd20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f68f0036000 tx=0x7f68f003b040 comp rx=0 tx=0).stop 2024-03-24T10:59:16.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.202+0000 7f69075c9640 1 -- 172.21.15.5:0/914835585 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69000cc000 msgr2=0x7f69000b19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:16.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.202+0000 7f69075c9640 1 --2- 172.21.15.5:0/914835585 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69000cc000 0x7f69000b19d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f68fc00ebe0 tx=0x7f68fc00c520 comp rx=0 tx=0).stop 2024-03-24T10:59:16.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.202+0000 7f69075c9640 1 -- 172.21.15.5:0/914835585 shutdown_connections 2024-03-24T10:59:16.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.202+0000 7f69075c9640 1 --2- 172.21.15.5:0/914835585 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f68e406b860 0x7f68e406dd20 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:16.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.202+0000 7f69075c9640 1 --2- 172.21.15.5:0/914835585 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f69000cf340 0x7f6900151d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:16.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.202+0000 7f69075c9640 1 --2- 172.21.15.5:0/914835585 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f69000ccb80 0x7f69001f3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:16.203 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.202+0000 7f69075c9640 1 --2- 172.21.15.5:0/914835585 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f69000cc000 0x7f69000b19d0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:16.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.202+0000 7f69075c9640 1 -- 172.21.15.5:0/914835585 >> 172.21.15.5:0/914835585 conn(0x7f69000cfc30 msgr2=0x7f6900154020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:16.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.202+0000 7f69075c9640 1 -- 172.21.15.5:0/914835585 shutdown_connections 2024-03-24T10:59:16.204 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.202+0000 7f69075c9640 1 -- 172.21.15.5:0/914835585 wait complete. 2024-03-24T10:59:16.213 INFO:tasks.ceph:Scrubbing osd.7 2024-03-24T10:59:16.213 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-24T10:59:16.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.321+0000 7fa28cceb640 1 Processor -- start 2024-03-24T10:59:16.321 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.321+0000 7fa28cceb640 1 -- start start 2024-03-24T10:59:16.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.322+0000 7fa28cceb640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fa2880cc000 0x7fa28815e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:16.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.322+0000 7fa28cceb640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa28806bf50 0x7fa28806c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:16.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.322+0000 7fa28cceb640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2880cc3e0 con 0x7fa2880cf340 2024-03-24T10:59:16.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.322+0000 7fa28cceb640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fa28815a400 con 0x7fa2880cc000 2024-03-24T10:59:16.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.322+0000 7fa28cceb640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fa288154bd0 con 0x7fa28806bf50 2024-03-24T10:59:16.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.323+0000 7fa286d9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fa28806bf50 0x7fa28806c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:16.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.323+0000 7fa27ffff640 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fa2880cf340 0x7fa2880cc640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.5:47486/0 (socket says 172.21.15.5:47486) 2024-03-24T10:59:16.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.323+0000 7fa27ffff640 1 -- 172.21.15.5:0/2306974748 learned_addr learned my addr 172.21.15.5:0/2306974748 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:59:16.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.325+0000 7fa28659c640 1 --2- 172.21.15.5:0/2306974748 >> v2:172.21.15.5:3300/0 conn(0x7fa2880cc000 0x7fa28815e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:16.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.328+0000 7fa28659c640 1 -- 172.21.15.5:0/2306974748 >> v2:172.21.15.57:3300/0 conn(0x7fa28806bf50 msgr2=0x7fa28806c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:16.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.328+0000 7fa28659c640 1 --2- 172.21.15.5:0/2306974748 >> v2:172.21.15.57:3300/0 conn(0x7fa28806bf50 0x7fa28806c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:16.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.328+0000 7fa28659c640 1 -- 172.21.15.5:0/2306974748 >> v1:172.21.15.57:6789/0 conn(0x7fa2880cf340 legacy=0x7fa2880cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:16.328 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.328+0000 7fa28659c640 1 -- 172.21.15.5:0/2306974748 --> v2:172.21.15.5:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa28806c8e0 con 0x7fa2880cc000 2024-03-24T10:59:16.330 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.330+0000 7fa28659c640 1 --2- 172.21.15.5:0/2306974748 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa2880cc000 0x7fa28815e3b0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fa2880ca230 tx=0x7fa27802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb3472f3f3fee3bf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:16.334 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.334+0000 7fa285d9b640 1 -- 172.21.15.5:0/2306974748 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa278031830 con 0x7fa2880cc000 2024-03-24T10:59:16.339 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.340+0000 7fa285d9b640 1 -- 172.21.15.5:0/2306974748 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa278031990 con 0x7fa2880cc000 2024-03-24T10:59:16.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.340+0000 7fa285d9b640 1 -- 172.21.15.5:0/2306974748 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa278031d20 con 0x7fa2880cc000 2024-03-24T10:59:16.340 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.340+0000 7fa28cceb640 1 -- 172.21.15.5:0/2306974748 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa2880cc000 msgr2=0x7fa28815e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:16.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.340+0000 7fa28cceb640 1 --2- 172.21.15.5:0/2306974748 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa2880cc000 0x7fa28815e3b0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fa2880ca230 tx=0x7fa27802b8d0 comp rx=0 tx=0).stop 2024-03-24T10:59:16.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.340+0000 7fa28cceb640 1 -- 172.21.15.5:0/2306974748 shutdown_connections 2024-03-24T10:59:16.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.340+0000 7fa28cceb640 1 --2- 172.21.15.5:0/2306974748 >> v2:172.21.15.57:3300/0 conn(0x7fa28806bf50 0x7fa28806c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:16.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.340+0000 7fa28cceb640 1 --2- 172.21.15.5:0/2306974748 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa2880cc000 0x7fa28815e3b0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:16.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.341+0000 7fa28cceb640 1 -- 172.21.15.5:0/2306974748 >> 172.21.15.5:0/2306974748 conn(0x7fa2880cfc30 msgr2=0x7fa2880d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:16.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.341+0000 7fa28cceb640 1 -- 172.21.15.5:0/2306974748 shutdown_connections 2024-03-24T10:59:16.341 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.341+0000 7fa28cceb640 1 -- 172.21.15.5:0/2306974748 wait complete. 2024-03-24T10:59:16.342 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.343+0000 7fa28cceb640 1 Processor -- start 2024-03-24T10:59:16.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.343+0000 7fa28cceb640 1 -- start start 2024-03-24T10:59:16.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.344+0000 7fa28cceb640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa28806bf50 0x7fa2880c3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:16.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.344+0000 7fa28cceb640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2880cc000 0x7fa2881f0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:16.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.344+0000 7fa28cceb640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa2880cf340 0x7fa2880c8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:16.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.344+0000 7fa28cceb640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fa2881f46f0 con 0x7fa28806bf50 2024-03-24T10:59:16.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.344+0000 7fa28cceb640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fa2881f06d0 con 0x7fa2880cc000 2024-03-24T10:59:16.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.344+0000 7fa28cceb640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fa2880c4a20 con 0x7fa2880cf340 2024-03-24T10:59:16.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.345+0000 7fa286d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa2880cf340 0x7fa2880c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:16.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.345+0000 7fa286d9d640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa2880cf340 0x7fa2880c8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:59824/0 (socket says 172.21.15.5:59824) 2024-03-24T10:59:16.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.345+0000 7fa286d9d640 1 -- 172.21.15.5:0/434419546 learned_addr learned my addr 172.21.15.5:0/434419546 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:16.345 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.345+0000 7fa27ffff640 1 --2- 172.21.15.5:0/434419546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2880cc000 0x7fa2881f0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:16.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.348+0000 7fa286d9d640 1 -- 172.21.15.5:0/434419546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2880cc000 msgr2=0x7fa2881f0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:16.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.349+0000 7fa286d9d640 1 --2- 172.21.15.5:0/434419546 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fa2880cc000 0x7fa2881f0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:16.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.349+0000 7fa286d9d640 1 -- 172.21.15.5:0/434419546 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa28806bf50 msgr2=0x7fa2880c3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:16.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.349+0000 7fa286d9d640 1 --2- 172.21.15.5:0/434419546 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fa28806bf50 0x7fa2880c3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:16.349 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.349+0000 7fa286d9d640 1 -- 172.21.15.5:0/434419546 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa278009400 con 0x7fa2880cf340 2024-03-24T10:59:16.351 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.351+0000 7fa286d9d640 1 --2- 172.21.15.5:0/434419546 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa2880cf340 0x7fa2880c8880 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa28000efc0 tx=0x7fa28000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:16.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.356+0000 7fa27effd640 1 -- 172.21.15.5:0/434419546 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa280009ba0 con 0x7fa2880cf340 2024-03-24T10:59:16.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.357+0000 7fa27effd640 1 -- 172.21.15.5:0/434419546 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa28000f040 con 0x7fa2880cf340 2024-03-24T10:59:16.357 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.357+0000 7fa28cceb640 1 -- 172.21.15.5:0/434419546 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2881f4920 con 0x7fa2880cf340 2024-03-24T10:59:16.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.357+0000 7fa27effd640 1 -- 172.21.15.5:0/434419546 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2800146c0 con 0x7fa2880cf340 2024-03-24T10:59:16.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.357+0000 7fa28cceb640 1 -- 172.21.15.5:0/434419546 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa288157a10 con 0x7fa2880cf340 2024-03-24T10:59:16.358 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.358+0000 7fa28cceb640 1 -- 172.21.15.5:0/434419546 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa2880c83f0 con 0x7fa2880cf340 2024-03-24T10:59:16.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.392+0000 7fa27effd640 1 -- 172.21.15.5:0/434419546 <== mon.2 v2:172.21.15.57:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa280014820 con 0x7fa2880cf340 2024-03-24T10:59:16.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.392+0000 7fa27effd640 1 --2- 172.21.15.5:0/434419546 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa26806b890 0x7fa26806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:16.392 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.392+0000 7fa28659c640 1 --2- 172.21.15.5:0/434419546 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa26806b890 0x7fa26806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:16.393 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.393+0000 7fa28659c640 1 --2- 172.21.15.5:0/434419546 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa26806b890 0x7fa26806dd50 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa278036000 tx=0x7fa278030000 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:16.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.396+0000 7fa27effd640 1 -- 172.21.15.5:0/434419546 <== mon.2 v2:172.21.15.57:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fa28005a6f0 con 0x7fa2880cf340 2024-03-24T10:59:16.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.397+0000 7fa27effd640 1 --2- 172.21.15.5:0/434419546 >> [v2:172.21.15.57:6824/212605698,v1:172.21.15.57:6825/212605698] conn(0x7fa268073cb0 0x7fa268076130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:16.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.397+0000 7fa27effd640 1 -- 172.21.15.5:0/434419546 --> [v2:172.21.15.57:6824/212605698,v1:172.21.15.57:6825/212605698] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa268076800 con 0x7fa268073cb0 2024-03-24T10:59:16.397 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.397+0000 7fa27ffff640 1 --2- 172.21.15.5:0/434419546 >> [v2:172.21.15.57:6824/212605698,v1:172.21.15.57:6825/212605698] conn(0x7fa268073cb0 0x7fa268076130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:16.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.397+0000 7fa27effd640 1 -- 172.21.15.5:0/434419546 <== mon.2 v2:172.21.15.57:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa280056290 con 0x7fa2880cf340 2024-03-24T10:59:16.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.397+0000 7fa27ffff640 1 --2- 172.21.15.5:0/434419546 >> [v2:172.21.15.57:6824/212605698,v1:172.21.15.57:6825/212605698] conn(0x7fa268073cb0 0x7fa268076130 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:16.398 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.398+0000 7fa27effd640 1 -- 172.21.15.5:0/434419546 <== osd.7 v2:172.21.15.57:6824/212605698 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa268076800 con 0x7fa268073cb0 2024-03-24T10:59:16.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.412+0000 7fa28cceb640 1 -- 172.21.15.5:0/434419546 --> [v2:172.21.15.57:6824/212605698,v1:172.21.15.57:6825/212605698] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa28815ab90 con 0x7fa268073cb0 2024-03-24T10:59:16.412 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.412+0000 7fa27effd640 1 -- 172.21.15.5:0/434419546 <== osd.7 v2:172.21.15.57:6824/212605698 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa270004b00 con 0x7fa268073cb0 2024-03-24T10:59:16.412 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-24T10:59:16.412 INFO:teuthology.orchestra.run.smithi005.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-24T10:59:16.413 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-24T10:59:16.413 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.412+0000 7fa28cceb640 1 -- 172.21.15.5:0/434419546 >> [v2:172.21.15.57:6824/212605698,v1:172.21.15.57:6825/212605698] conn(0x7fa268073cb0 msgr2=0x7fa268076130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:16.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.412+0000 7fa28cceb640 1 --2- 172.21.15.5:0/434419546 >> [v2:172.21.15.57:6824/212605698,v1:172.21.15.57:6825/212605698] conn(0x7fa268073cb0 0x7fa268076130 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:16.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.412+0000 7fa28cceb640 1 -- 172.21.15.5:0/434419546 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa26806b890 msgr2=0x7fa26806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:16.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.412+0000 7fa28cceb640 1 --2- 172.21.15.5:0/434419546 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fa26806b890 0x7fa26806dd50 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa278036000 tx=0x7fa278030000 comp rx=0 tx=0).stop 2024-03-24T10:59:16.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.412+0000 7fa28cceb640 1 -- 172.21.15.5:0/434419546 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa2880cf340 msgr2=0x7fa2880c8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:16.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.412+0000 7fa28cceb640 1 --2- 172.21.15.5:0/434419546 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fa2880cf340 0x7fa2880c8880 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa28000efc0 tx=0x7fa28000c490 comp rx=0 tx=0).stop 2024-03-24T10:59:16.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.412+0000 7fa286d9d640 1 -- 172.21.15.5:0/434419546 reap_dead start 2024-03-24T10:59:16.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.413+0000 7fa28cceb640 1 -- 172.21.15.5:0/434419546 shutdown_connections 2024-03-24T10:59:16.414 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.413+0000 7fa28cceb640 1 -- 172.21.15.5:0/434419546 >> 172.21.15.5:0/434419546 conn(0x7fa2880cfc30 msgr2=0x7fa2881e5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:16.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.413+0000 7fa28cceb640 1 -- 172.21.15.5:0/434419546 shutdown_connections 2024-03-24T10:59:16.415 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.413+0000 7fa28cceb640 1 -- 172.21.15.5:0/434419546 wait complete. 2024-03-24T10:59:16.425 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-24T10:59:16.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.533+0000 7fb0353a7640 1 Processor -- start 2024-03-24T10:59:16.533 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.533+0000 7fb0353a7640 1 -- start start 2024-03-24T10:59:16.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.533+0000 7fb0353a7640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb0300cc000 0x7fb03015e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:16.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.533+0000 7fb0353a7640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb0300cf340 0x7fb0300cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:16.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.533+0000 7fb0353a7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0300ccb80 0x7fb03006bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:16.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.533+0000 7fb0353a7640 1 -- --> v2:172.21.15.5:3300/0 -- mon_getmap magic: 0 -- 0x7fb03015a400 con 0x7fb0300cc000 2024-03-24T10:59:16.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.533+0000 7fb0353a7640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fb030154bd0 con 0x7fb0300cf340 2024-03-24T10:59:16.534 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.533+0000 7fb0353a7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fb0300cc3e0 con 0x7fb0300ccb80 2024-03-24T10:59:16.535 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.535+0000 7fb02e7fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fb0300cf340 0x7fb0300cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:16.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.536+0000 7fb02effd640 1 --2- >> v2:172.21.15.5:3300/0 conn(0x7fb0300cc000 0x7fb03015e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:16.536 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.537+0000 7fb02f7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0300ccb80 0x7fb03006bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:16.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.537+0000 7fb02f7fe640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0300ccb80 0x7fb03006bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:59836/0 (socket says 172.21.15.5:59836) 2024-03-24T10:59:16.537 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.537+0000 7fb02f7fe640 1 -- 172.21.15.5:0/2605865047 learned_addr learned my addr 172.21.15.5:0/2605865047 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:16.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.541+0000 7fb02f7fe640 1 -- 172.21.15.5:0/2605865047 >> v2:172.21.15.57:3300/0 conn(0x7fb0300cf340 msgr2=0x7fb0300cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:16.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.541+0000 7fb02f7fe640 1 --2- 172.21.15.5:0/2605865047 >> v2:172.21.15.57:3300/0 conn(0x7fb0300cf340 0x7fb0300cc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:16.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.541+0000 7fb02f7fe640 1 -- 172.21.15.5:0/2605865047 >> v2:172.21.15.5:3300/0 conn(0x7fb0300cc000 msgr2=0x7fb03015e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:16.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.541+0000 7fb02f7fe640 1 --2- 172.21.15.5:0/2605865047 >> v2:172.21.15.5:3300/0 conn(0x7fb0300cc000 0x7fb03015e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:16.541 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.541+0000 7fb02f7fe640 1 -- 172.21.15.5:0/2605865047 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb03015a5f0 con 0x7fb0300ccb80 2024-03-24T10:59:16.546 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.547+0000 7fb02f7fe640 1 --2- 172.21.15.5:0/2605865047 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0300ccb80 0x7fb03006bf50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb0300cd790 tx=0x7fb02802bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e683ba318a5ee565 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:16.551 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.551+0000 7fb02dffb640 1 -- 172.21.15.5:0/2605865047 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02802ca80 con 0x7fb0300ccb80 2024-03-24T10:59:16.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.552+0000 7fb02dffb640 1 -- 172.21.15.5:0/2605865047 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb02802cbe0 con 0x7fb0300ccb80 2024-03-24T10:59:16.552 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.552+0000 7fb02dffb640 1 -- 172.21.15.5:0/2605865047 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0280343f0 con 0x7fb0300ccb80 2024-03-24T10:59:16.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.552+0000 7fb0353a7640 1 -- 172.21.15.5:0/2605865047 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0300ccb80 msgr2=0x7fb03006bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:16.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.552+0000 7fb0353a7640 1 --2- 172.21.15.5:0/2605865047 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0300ccb80 0x7fb03006bf50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fb0300cd790 tx=0x7fb02802bee0 comp rx=0 tx=0).stop 2024-03-24T10:59:16.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.553+0000 7fb0353a7640 1 -- 172.21.15.5:0/2605865047 shutdown_connections 2024-03-24T10:59:16.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.553+0000 7fb0353a7640 1 --2- 172.21.15.5:0/2605865047 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0300ccb80 0x7fb03006bf50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:16.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.553+0000 7fb0353a7640 1 --2- 172.21.15.5:0/2605865047 >> v2:172.21.15.57:3300/0 conn(0x7fb0300cf340 0x7fb0300cc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:16.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.553+0000 7fb0353a7640 1 --2- 172.21.15.5:0/2605865047 >> v2:172.21.15.5:3300/0 conn(0x7fb0300cc000 0x7fb03015e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:16.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.553+0000 7fb0353a7640 1 -- 172.21.15.5:0/2605865047 >> 172.21.15.5:0/2605865047 conn(0x7fb0300cfc30 msgr2=0x7fb0300d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:16.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.553+0000 7fb0353a7640 1 -- 172.21.15.5:0/2605865047 shutdown_connections 2024-03-24T10:59:16.553 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.553+0000 7fb0353a7640 1 -- 172.21.15.5:0/2605865047 wait complete. 2024-03-24T10:59:16.554 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.555+0000 7fb0353a7640 1 Processor -- start 2024-03-24T10:59:16.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.555+0000 7fb0353a7640 1 -- start start 2024-03-24T10:59:16.555 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.555+0000 7fb0353a7640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0300cc000 0x7fb0300af800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:16.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.555+0000 7fb0353a7640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0300ccb80 0x7fb0301efe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:16.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.555+0000 7fb0353a7640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0300cf340 0x7fb030152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:16.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.555+0000 7fb0353a7640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fb030157800 con 0x7fb0300cc000 2024-03-24T10:59:16.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.555+0000 7fb0353a7640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fb030157ae0 con 0x7fb0300ccb80 2024-03-24T10:59:16.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.555+0000 7fb0353a7640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fb0301f33b0 con 0x7fb0300cf340 2024-03-24T10:59:16.556 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.556+0000 7fb02e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0300ccb80 0x7fb0301efe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:16.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.557+0000 7fb02e7fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0300ccb80 0x7fb0301efe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:49496/0 (socket says 172.21.15.5:49496) 2024-03-24T10:59:16.557 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.557+0000 7fb02e7fc640 1 -- 172.21.15.5:0/361624313 learned_addr learned my addr 172.21.15.5:0/361624313 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:16.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.561+0000 7fb02e7fc640 1 -- 172.21.15.5:0/361624313 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0300cf340 msgr2=0x7fb030152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:16.561 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.561+0000 7fb02e7fc640 1 --2- 172.21.15.5:0/361624313 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0300cf340 0x7fb030152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:16.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.561+0000 7fb02e7fc640 1 -- 172.21.15.5:0/361624313 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0300cc000 msgr2=0x7fb0300af800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:16.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.561+0000 7fb02e7fc640 1 --2- 172.21.15.5:0/361624313 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0300cc000 0x7fb0300af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:16.562 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.561+0000 7fb02e7fc640 1 -- 172.21.15.5:0/361624313 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb028009b30 con 0x7fb0300ccb80 2024-03-24T10:59:16.567 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.568+0000 7fb02e7fc640 1 --2- 172.21.15.5:0/361624313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0300ccb80 0x7fb0301efe40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb02400e870 tx=0x7fb02400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:16.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.676+0000 7fb0177fe640 1 -- 172.21.15.5:0/361624313 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0240094f0 con 0x7fb0300ccb80 2024-03-24T10:59:16.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.676+0000 7fb0177fe640 1 -- 172.21.15.5:0/361624313 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb02400c940 con 0x7fb0300ccb80 2024-03-24T10:59:16.676 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.676+0000 7fb0177fe640 1 -- 172.21.15.5:0/361624313 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02400ccf0 con 0x7fb0300ccb80 2024-03-24T10:59:16.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.676+0000 7fb0353a7640 1 -- 172.21.15.5:0/361624313 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0301f3690 con 0x7fb0300ccb80 2024-03-24T10:59:16.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.676+0000 7fb0353a7640 1 -- 172.21.15.5:0/361624313 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0301f3c50 con 0x7fb0300ccb80 2024-03-24T10:59:16.677 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.677+0000 7fb0353a7640 1 -- 172.21.15.5:0/361624313 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0301f3de0 con 0x7fb0300ccb80 2024-03-24T10:59:16.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.946+0000 7fb0177fe640 1 -- 172.21.15.5:0/361624313 <== mon.1 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb024013470 con 0x7fb0300ccb80 2024-03-24T10:59:16.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.946+0000 7fb0177fe640 1 --2- 172.21.15.5:0/361624313 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb01c06b810 0x7fb01c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:16.946 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.946+0000 7fb02effd640 1 --2- 172.21.15.5:0/361624313 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb01c06b810 0x7fb01c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:16.947 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.947+0000 7fb02effd640 1 --2- 172.21.15.5:0/361624313 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb01c06b810 0x7fb01c06dcd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb020009870 tx=0x7fb020008040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:16.951 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.952+0000 7fb0177fe640 1 -- 172.21.15.5:0/361624313 <== mon.1 v2:172.21.15.57:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fb024059980 con 0x7fb0300ccb80 2024-03-24T10:59:16.963 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:16.963+0000 7fb0177fe640 1 -- 172.21.15.5:0/361624313 <== mon.1 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb024055520 con 0x7fb0300ccb80 2024-03-24T10:59:17.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.148+0000 7fb0353a7640 1 -- 172.21.15.5:0/361624313 --> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fb03015ae70 con 0x7fb01c06b810 2024-03-24T10:59:17.149 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.148+0000 7fb0177fe640 1 -- 172.21.15.5:0/361624313 <== mgr.4120 v2:172.21.15.5:6832/1102439181 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fb020009df0 con 0x7fb01c06b810 2024-03-24T10:59:17.149 INFO:teuthology.orchestra.run.smithi005.stderr:instructed osd(s) 7 to deep-scrub 2024-03-24T10:59:17.151 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.150+0000 7fb0353a7640 1 -- 172.21.15.5:0/361624313 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb01c06b810 msgr2=0x7fb01c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:17.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.150+0000 7fb0353a7640 1 --2- 172.21.15.5:0/361624313 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb01c06b810 0x7fb01c06dcd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb020009870 tx=0x7fb020008040 comp rx=0 tx=0).stop 2024-03-24T10:59:17.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.150+0000 7fb0353a7640 1 -- 172.21.15.5:0/361624313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0300ccb80 msgr2=0x7fb0301efe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:17.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.150+0000 7fb0353a7640 1 --2- 172.21.15.5:0/361624313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0300ccb80 0x7fb0301efe40 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb02400e870 tx=0x7fb02400ed30 comp rx=0 tx=0).stop 2024-03-24T10:59:17.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.150+0000 7fb0353a7640 1 -- 172.21.15.5:0/361624313 shutdown_connections 2024-03-24T10:59:17.152 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.150+0000 7fb0353a7640 1 --2- 172.21.15.5:0/361624313 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fb01c06b810 0x7fb01c06dcd0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:17.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.150+0000 7fb0353a7640 1 --2- 172.21.15.5:0/361624313 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fb0300cf340 0x7fb030152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:17.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.150+0000 7fb0353a7640 1 --2- 172.21.15.5:0/361624313 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fb0300ccb80 0x7fb0301efe40 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:17.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.150+0000 7fb0353a7640 1 --2- 172.21.15.5:0/361624313 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fb0300cc000 0x7fb0300af800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:17.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.150+0000 7fb0353a7640 1 -- 172.21.15.5:0/361624313 >> 172.21.15.5:0/361624313 conn(0x7fb0300cfc30 msgr2=0x7fb030153f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:17.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.150+0000 7fb0353a7640 1 -- 172.21.15.5:0/361624313 shutdown_connections 2024-03-24T10:59:17.153 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.150+0000 7fb0353a7640 1 -- 172.21.15.5:0/361624313 wait complete. 2024-03-24T10:59:17.162 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:59:17.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.305+0000 7f21f51d0640 1 Processor -- start 2024-03-24T10:59:17.305 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.305+0000 7f21f51d0640 1 -- start start 2024-03-24T10:59:17.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.306+0000 7f21f51d0640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f21f0150fd0 0x7f21f00c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:17.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.306+0000 7f21f51d0640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21f006ca40 con 0x7f21f01513b0 2024-03-24T10:59:17.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.306+0000 7f21f51d0640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21f015e100 con 0x7f21f015e590 2024-03-24T10:59:17.306 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.306+0000 7f21f51d0640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f21f006b240 con 0x7f21f0150fd0 2024-03-24T10:59:17.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.306+0000 7f21eed9d640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f21f0150fd0 0x7f21f00c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:17.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.307+0000 7f21ee59c640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7f21f01513b0 0x7f21f015aea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:58918/0 (socket says 172.21.15.5:58918) 2024-03-24T10:59:17.307 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.307+0000 7f21ee59c640 1 -- 172.21.15.5:0/1058347810 learned_addr learned my addr 172.21.15.5:0/1058347810 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:59:17.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.310+0000 7f21edd9b640 1 -- 172.21.15.5:0/1058347810 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2360092930 0 0) 0x7f21f006ca40 con 0x7f21f01513b0 2024-03-24T10:59:17.310 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.310+0000 7f21edd9b640 1 -- 172.21.15.5:0/1058347810 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21d8003520 con 0x7f21f01513b0 2024-03-24T10:59:17.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.311+0000 7f21eed9d640 1 -- 172.21.15.5:0/1058347810 >> v1:172.21.15.57:6789/0 conn(0x7f21f015e590 legacy=0x7f21f006bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:17.311 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.311+0000 7f21eed9d640 1 -- 172.21.15.5:0/1058347810 >> v1:172.21.15.5:6789/0 conn(0x7f21f01513b0 legacy=0x7f21f015aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:17.312 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.311+0000 7f21eed9d640 1 -- 172.21.15.5:0/1058347810 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f006b520 con 0x7f21f0150fd0 2024-03-24T10:59:17.315 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.315+0000 7f21eed9d640 1 --2- 172.21.15.5:0/1058347810 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21f0150fd0 0x7f21f00c70f0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f21e00097d0 tx=0x7f21e002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9bc490c6a0a6f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:17.320 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.320+0000 7f21edd9b640 1 -- 172.21.15.5:0/1058347810 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e0030610 con 0x7f21f0150fd0 2024-03-24T10:59:17.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.322+0000 7f21edd9b640 1 -- 172.21.15.5:0/1058347810 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21e0030770 con 0x7f21f0150fd0 2024-03-24T10:59:17.322 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.322+0000 7f21edd9b640 1 -- 172.21.15.5:0/1058347810 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e0030ae0 con 0x7f21f0150fd0 2024-03-24T10:59:17.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.322+0000 7f21f51d0640 1 -- 172.21.15.5:0/1058347810 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21f0150fd0 msgr2=0x7f21f00c70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:17.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.322+0000 7f21f51d0640 1 --2- 172.21.15.5:0/1058347810 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21f0150fd0 0x7f21f00c70f0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f21e00097d0 tx=0x7f21e002b960 comp rx=0 tx=0).stop 2024-03-24T10:59:17.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.322+0000 7f21f51d0640 1 -- 172.21.15.5:0/1058347810 shutdown_connections 2024-03-24T10:59:17.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.322+0000 7f21f51d0640 1 --2- 172.21.15.5:0/1058347810 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21f0150fd0 0x7f21f00c70f0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:17.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.323+0000 7f21f51d0640 1 -- 172.21.15.5:0/1058347810 >> 172.21.15.5:0/1058347810 conn(0x7f21f006f8b0 msgr2=0x7f21f006fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:17.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.323+0000 7f21f51d0640 1 -- 172.21.15.5:0/1058347810 shutdown_connections 2024-03-24T10:59:17.323 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.323+0000 7f21f51d0640 1 -- 172.21.15.5:0/1058347810 wait complete. 2024-03-24T10:59:17.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.325+0000 7f21f51d0640 1 Processor -- start 2024-03-24T10:59:17.325 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.325+0000 7f21f51d0640 1 -- start start 2024-03-24T10:59:17.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.326+0000 7f21f51d0640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21f0150fd0 0x7f21f015cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:17.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.326+0000 7f21f51d0640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21f01513b0 0x7f21f0145fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:17.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.326+0000 7f21f51d0640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21f015e590 0x7f21f0147af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:17.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.326+0000 7f21f51d0640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f21f0148bc0 con 0x7f21f015e590 2024-03-24T10:59:17.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.326+0000 7f21f51d0640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f21f015d060 con 0x7f21f0150fd0 2024-03-24T10:59:17.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.326+0000 7f21f51d0640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f21f00ca780 con 0x7f21f01513b0 2024-03-24T10:59:17.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.327+0000 7f21ef59e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21f015e590 0x7f21f0147af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:17.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.327+0000 7f21ee59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21f01513b0 0x7f21f0145fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:17.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.327+0000 7f21ee59c640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21f01513b0 0x7f21f0145fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:59854/0 (socket says 172.21.15.5:59854) 2024-03-24T10:59:17.327 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.327+0000 7f21ee59c640 1 -- 172.21.15.5:0/3083412591 learned_addr learned my addr 172.21.15.5:0/3083412591 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:17.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.331+0000 7f21ef59e640 1 -- 172.21.15.5:0/3083412591 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21f01513b0 msgr2=0x7f21f0145fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:17.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.331+0000 7f21ef59e640 1 --2- 172.21.15.5:0/3083412591 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21f01513b0 0x7f21f0145fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:17.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.331+0000 7f21ef59e640 1 -- 172.21.15.5:0/3083412591 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21f0150fd0 msgr2=0x7f21f015cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:17.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.331+0000 7f21ef59e640 1 --2- 172.21.15.5:0/3083412591 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21f0150fd0 0x7f21f015cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:17.331 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.331+0000 7f21ef59e640 1 -- 172.21.15.5:0/3083412591 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e0009400 con 0x7f21f015e590 2024-03-24T10:59:17.343 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.343+0000 7f21ef59e640 1 --2- 172.21.15.5:0/3083412591 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21f015e590 0x7f21f0147af0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f21e800c700 tx=0x7f21e800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:17.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.347+0000 7f21d77fe640 1 -- 172.21.15.5:0/3083412591 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e8003b70 con 0x7f21f015e590 2024-03-24T10:59:17.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.347+0000 7f21d77fe640 1 -- 172.21.15.5:0/3083412591 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21e8003cd0 con 0x7f21f015e590 2024-03-24T10:59:17.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.347+0000 7f21d77fe640 1 -- 172.21.15.5:0/3083412591 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e8005310 con 0x7f21f015e590 2024-03-24T10:59:17.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.347+0000 7f21f51d0640 1 -- 172.21.15.5:0/3083412591 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21f0147660 con 0x7f21f015e590 2024-03-24T10:59:17.347 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.347+0000 7f21f51d0640 1 -- 172.21.15.5:0/3083412591 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21f01f0d70 con 0x7f21f015e590 2024-03-24T10:59:17.348 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.348+0000 7f21f51d0640 1 -- 172.21.15.5:0/3083412591 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21f014eb70 con 0x7f21f015e590 2024-03-24T10:59:17.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.679+0000 7f21d77fe640 1 -- 172.21.15.5:0/3083412591 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f21e8005470 con 0x7f21f015e590 2024-03-24T10:59:17.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.680+0000 7f21d77fe640 1 --2- 172.21.15.5:0/3083412591 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f21dc06b810 0x7f21dc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:17.680 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.680+0000 7f21eed9d640 1 --2- 172.21.15.5:0/3083412591 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f21dc06b810 0x7f21dc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:17.681 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.680+0000 7f21eed9d640 1 --2- 172.21.15.5:0/3083412591 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f21dc06b810 0x7f21dc06dcd0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f21e0036000 tx=0x7f21e0030000 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:17.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.695+0000 7f21d77fe640 1 -- 172.21.15.5:0/3083412591 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f21e80597a0 con 0x7f21f015e590 2024-03-24T10:59:17.695 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.695+0000 7f21d77fe640 1 -- 172.21.15.5:0/3083412591 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f21e8055340 con 0x7f21f015e590 2024-03-24T10:59:17.977 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.977+0000 7f21f51d0640 1 -- 172.21.15.5:0/3083412591 --> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f21f006b800 con 0x7f21dc06b810 2024-03-24T10:59:17.983 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.983+0000 7f21d77fe640 1 -- 172.21.15.5:0/3083412591 <== mgr.4120 v2:172.21.15.5:6832/1102439181 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+250359 (secure 0 0 0) 0x7f21e00305b0 con 0x7f21dc06b810 2024-03-24T10:59:17.986 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:59:17.987 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-03-24T10:59:17.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.986+0000 7f21f51d0640 1 -- 172.21.15.5:0/3083412591 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f21dc06b810 msgr2=0x7f21dc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:17.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.986+0000 7f21f51d0640 1 --2- 172.21.15.5:0/3083412591 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f21dc06b810 0x7f21dc06dcd0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f21e0036000 tx=0x7f21e0030000 comp rx=0 tx=0).stop 2024-03-24T10:59:17.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.986+0000 7f21f51d0640 1 -- 172.21.15.5:0/3083412591 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21f015e590 msgr2=0x7f21f0147af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:17.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.986+0000 7f21f51d0640 1 --2- 172.21.15.5:0/3083412591 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21f015e590 0x7f21f0147af0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f21e800c700 tx=0x7f21e800cbc0 comp rx=0 tx=0).stop 2024-03-24T10:59:17.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.986+0000 7f21f51d0640 1 -- 172.21.15.5:0/3083412591 shutdown_connections 2024-03-24T10:59:17.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.986+0000 7f21f51d0640 1 --2- 172.21.15.5:0/3083412591 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f21dc06b810 0x7f21dc06dcd0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:17.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.986+0000 7f21f51d0640 1 --2- 172.21.15.5:0/3083412591 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f21f015e590 0x7f21f0147af0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:17.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.986+0000 7f21f51d0640 1 --2- 172.21.15.5:0/3083412591 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f21f01513b0 0x7f21f0145fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:17.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.986+0000 7f21f51d0640 1 --2- 172.21.15.5:0/3083412591 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f21f0150fd0 0x7f21f015cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:17.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.986+0000 7f21f51d0640 1 -- 172.21.15.5:0/3083412591 >> 172.21.15.5:0/3083412591 conn(0x7f21f006f8b0 msgr2=0x7f21f014d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:17.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.987+0000 7f21f51d0640 1 -- 172.21.15.5:0/3083412591 shutdown_connections 2024-03-24T10:59:17.989 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:17.987+0000 7f21f51d0640 1 -- 172.21.15.5:0/3083412591 wait complete. 2024-03-24T10:59:17.998 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":454,"stamp":"2024-03-24T10:59:17.547217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":363,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":342,"ondisk_log_size":342,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":243432,"kb_used_data":2336,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754731288,"statfs":{"total":773094113280,"available":772844838912,"internally_reserved":0,"allocated":2392064,"data_stored":1937476,"data_compressed":65430,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":246795826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002365"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.230562+0000","last_change":"2024-03-24T10:45:01.894006+0000","last_active":"2024-03-24T10:55:26.230562+0000","last_peered":"2024-03-24T10:55:26.230562+0000","last_clean":"2024-03-24T10:55:26.230562+0000","last_became_active":"2024-03-24T10:45:01.893734+0000","last_became_peered":"2024-03-24T10:45:01.893734+0000","last_unstale":"2024-03-24T10:55:26.230562+0000","last_undegraded":"2024-03-24T10:55:26.230562+0000","last_fullsized":"2024-03-24T10:55:26.230562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:05:51.579388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:31.333156+0000","last_change":"2024-03-24T10:45:01.931707+0000","last_active":"2024-03-24T10:57:31.333156+0000","last_peered":"2024-03-24T10:57:31.333156+0000","last_clean":"2024-03-24T10:57:31.333156+0000","last_became_active":"2024-03-24T10:45:01.931449+0000","last_became_peered":"2024-03-24T10:45:01.931449+0000","last_unstale":"2024-03-24T10:57:31.333156+0000","last_undegraded":"2024-03-24T10:57:31.333156+0000","last_fullsized":"2024-03-24T10:57:31.333156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:34.226046+0000","last_change":"2024-03-24T10:45:01.895929+0000","last_active":"2024-03-24T10:57:34.226046+0000","last_peered":"2024-03-24T10:57:34.226046+0000","last_clean":"2024-03-24T10:57:34.226046+0000","last_became_active":"2024-03-24T10:45:01.895654+0000","last_became_peered":"2024-03-24T10:45:01.895654+0000","last_unstale":"2024-03-24T10:57:34.226046+0000","last_undegraded":"2024-03-24T10:57:34.226046+0000","last_fullsized":"2024-03-24T10:57:34.226046+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:01.253737+0000","last_change":"2024-03-24T10:45:01.952468+0000","last_active":"2024-03-24T10:59:01.253737+0000","last_peered":"2024-03-24T10:59:01.253737+0000","last_clean":"2024-03-24T10:59:01.253737+0000","last_became_active":"2024-03-24T10:45:01.952219+0000","last_became_peered":"2024-03-24T10:45:01.952219+0000","last_unstale":"2024-03-24T10:59:01.253737+0000","last_undegraded":"2024-03-24T10:59:01.253737+0000","last_fullsized":"2024-03-24T10:59:01.253737+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.228132+0000","last_change":"2024-03-24T10:45:01.895885+0000","last_active":"2024-03-24T10:55:01.228132+0000","last_peered":"2024-03-24T10:55:01.228132+0000","last_clean":"2024-03-24T10:55:01.228132+0000","last_became_active":"2024-03-24T10:45:01.895622+0000","last_became_peered":"2024-03-24T10:45:01.895622+0000","last_unstale":"2024-03-24T10:55:01.228132+0000","last_undegraded":"2024-03-24T10:55:01.228132+0000","last_fullsized":"2024-03-24T10:55:01.228132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:48:16.523177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:58:11.331484+0000","last_change":"2024-03-24T10:45:40.169714+0000","last_active":"2024-03-24T10:58:11.331484+0000","last_peered":"2024-03-24T10:58:11.331484+0000","last_clean":"2024-03-24T10:58:11.331484+0000","last_became_active":"2024-03-24T10:45:40.169447+0000","last_became_peered":"2024-03-24T10:45:40.169447+0000","last_unstale":"2024-03-24T10:58:11.331484+0000","last_undegraded":"2024-03-24T10:58:11.331484+0000","last_fullsized":"2024-03-24T10:58:11.331484+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.228112+0000","last_change":"2024-03-24T10:45:01.892745+0000","last_active":"2024-03-24T10:55:01.228112+0000","last_peered":"2024-03-24T10:55:01.228112+0000","last_clean":"2024-03-24T10:55:01.228112+0000","last_became_active":"2024-03-24T10:45:01.892452+0000","last_became_peered":"2024-03-24T10:45:01.892452+0000","last_unstale":"2024-03-24T10:55:01.228112+0000","last_undegraded":"2024-03-24T10:55:01.228112+0000","last_fullsized":"2024-03-24T10:55:01.228112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:09.037705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:31.210358+0000","last_change":"2024-03-24T10:45:01.953432+0000","last_active":"2024-03-24T10:57:31.210358+0000","last_peered":"2024-03-24T10:57:31.210358+0000","last_clean":"2024-03-24T10:57:31.210358+0000","last_became_active":"2024-03-24T10:45:01.953149+0000","last_became_peered":"2024-03-24T10:45:01.953149+0000","last_unstale":"2024-03-24T10:57:31.210358+0000","last_undegraded":"2024-03-24T10:57:31.210358+0000","last_fullsized":"2024-03-24T10:57:31.210358+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.226282+0000","last_change":"2024-03-24T10:45:01.956488+0000","last_active":"2024-03-24T10:54:41.226282+0000","last_peered":"2024-03-24T10:54:41.226282+0000","last_clean":"2024-03-24T10:54:41.226282+0000","last_became_active":"2024-03-24T10:45:01.956207+0000","last_became_peered":"2024-03-24T10:45:01.956207+0000","last_unstale":"2024-03-24T10:54:41.226282+0000","last_undegraded":"2024-03-24T10:54:41.226282+0000","last_fullsized":"2024-03-24T10:54:41.226282+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:23:26.523809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:44.289850+0000","last_change":"2024-03-24T10:45:01.956484+0000","last_active":"2024-03-24T10:54:44.289850+0000","last_peered":"2024-03-24T10:54:44.289850+0000","last_clean":"2024-03-24T10:54:44.289850+0000","last_became_active":"2024-03-24T10:45:01.956209+0000","last_became_peered":"2024-03-24T10:45:01.956209+0000","last_unstale":"2024-03-24T10:54:44.289850+0000","last_undegraded":"2024-03-24T10:54:44.289850+0000","last_fullsized":"2024-03-24T10:54:44.289850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:45:01.184614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:44.289823+0000","last_change":"2024-03-24T10:45:01.957103+0000","last_active":"2024-03-24T10:54:44.289823+0000","last_peered":"2024-03-24T10:54:44.289823+0000","last_clean":"2024-03-24T10:54:44.289823+0000","last_became_active":"2024-03-24T10:45:01.956849+0000","last_became_peered":"2024-03-24T10:45:01.956849+0000","last_unstale":"2024-03-24T10:54:44.289823+0000","last_undegraded":"2024-03-24T10:54:44.289823+0000","last_fullsized":"2024-03-24T10:54:44.289823+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:03:58.229932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:44.262492+0000","last_change":"2024-03-24T10:45:01.896701+0000","last_active":"2024-03-24T10:54:44.262492+0000","last_peered":"2024-03-24T10:54:44.262492+0000","last_clean":"2024-03-24T10:54:44.262492+0000","last_became_active":"2024-03-24T10:45:01.896400+0000","last_became_peered":"2024-03-24T10:45:01.896400+0000","last_unstale":"2024-03-24T10:54:44.262492+0000","last_undegraded":"2024-03-24T10:54:44.262492+0000","last_fullsized":"2024-03-24T10:54:44.262492+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:44.380352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:06.208517+0000","last_change":"2024-03-24T10:45:01.894954+0000","last_active":"2024-03-24T10:57:06.208517+0000","last_peered":"2024-03-24T10:57:06.208517+0000","last_clean":"2024-03-24T10:57:06.208517+0000","last_became_active":"2024-03-24T10:45:01.894632+0000","last_became_peered":"2024-03-24T10:45:01.894632+0000","last_unstale":"2024-03-24T10:57:06.208517+0000","last_undegraded":"2024-03-24T10:57:06.208517+0000","last_fullsized":"2024-03-24T10:57:06.208517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:04.232928+0000","last_change":"2024-03-24T10:45:01.929588+0000","last_active":"2024-03-24T10:59:04.232928+0000","last_peered":"2024-03-24T10:59:04.232928+0000","last_clean":"2024-03-24T10:59:04.232928+0000","last_became_active":"2024-03-24T10:45:01.928460+0000","last_became_peered":"2024-03-24T10:45:01.928460+0000","last_unstale":"2024-03-24T10:59:04.232928+0000","last_undegraded":"2024-03-24T10:59:04.232928+0000","last_fullsized":"2024-03-24T10:59:04.232928+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:01.339821+0000","last_change":"2024-03-24T10:45:01.928721+0000","last_active":"2024-03-24T10:59:01.339821+0000","last_peered":"2024-03-24T10:59:01.339821+0000","last_clean":"2024-03-24T10:59:01.339821+0000","last_became_active":"2024-03-24T10:45:01.928467+0000","last_became_peered":"2024-03-24T10:45:01.928467+0000","last_unstale":"2024-03-24T10:59:01.339821+0000","last_undegraded":"2024-03-24T10:59:01.339821+0000","last_fullsized":"2024-03-24T10:59:01.339821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:01.253683+0000","last_change":"2024-03-24T10:45:01.956030+0000","last_active":"2024-03-24T10:59:01.253683+0000","last_peered":"2024-03-24T10:59:01.253683+0000","last_clean":"2024-03-24T10:59:01.253683+0000","last_became_active":"2024-03-24T10:45:01.955768+0000","last_became_peered":"2024-03-24T10:45:01.955768+0000","last_unstale":"2024-03-24T10:59:01.253683+0000","last_undegraded":"2024-03-24T10:59:01.253683+0000","last_fullsized":"2024-03-24T10:59:01.253683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:01.253762+0000","last_change":"2024-03-24T10:45:01.956181+0000","last_active":"2024-03-24T10:59:01.253762+0000","last_peered":"2024-03-24T10:59:01.253762+0000","last_clean":"2024-03-24T10:59:01.253762+0000","last_became_active":"2024-03-24T10:45:01.955890+0000","last_became_peered":"2024-03-24T10:45:01.955890+0000","last_unstale":"2024-03-24T10:59:01.253762+0000","last_undegraded":"2024-03-24T10:59:01.253762+0000","last_fullsized":"2024-03-24T10:59:01.253762+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:01.335020+0000","last_change":"2024-03-24T10:45:01.896663+0000","last_active":"2024-03-24T10:59:01.335020+0000","last_peered":"2024-03-24T10:59:01.335020+0000","last_clean":"2024-03-24T10:59:01.335020+0000","last_became_active":"2024-03-24T10:45:01.896391+0000","last_became_peered":"2024-03-24T10:45:01.896391+0000","last_unstale":"2024-03-24T10:59:01.335020+0000","last_undegraded":"2024-03-24T10:59:01.335020+0000","last_fullsized":"2024-03-24T10:59:01.335020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.235423+0000","last_change":"2024-03-24T10:45:01.895951+0000","last_active":"2024-03-24T10:56:16.235423+0000","last_peered":"2024-03-24T10:56:16.235423+0000","last_clean":"2024-03-24T10:56:16.235423+0000","last_became_active":"2024-03-24T10:45:01.895668+0000","last_became_peered":"2024-03-24T10:45:01.895668+0000","last_unstale":"2024-03-24T10:56:16.235423+0000","last_undegraded":"2024-03-24T10:56:16.235423+0000","last_fullsized":"2024-03-24T10:56:16.235423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:56:43.462458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:19.297955+0000","last_change":"2024-03-24T10:45:01.898126+0000","last_active":"2024-03-24T10:56:19.297955+0000","last_peered":"2024-03-24T10:56:19.297955+0000","last_clean":"2024-03-24T10:56:19.297955+0000","last_became_active":"2024-03-24T10:45:01.897878+0000","last_became_peered":"2024-03-24T10:45:01.897878+0000","last_unstale":"2024-03-24T10:56:19.297955+0000","last_undegraded":"2024-03-24T10:56:19.297955+0000","last_fullsized":"2024-03-24T10:56:19.297955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:32:43.162674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:58:34.230670+0000","last_change":"2024-03-24T10:45:01.896109+0000","last_active":"2024-03-24T10:58:34.230670+0000","last_peered":"2024-03-24T10:58:34.230670+0000","last_clean":"2024-03-24T10:58:34.230670+0000","last_became_active":"2024-03-24T10:45:01.895696+0000","last_became_peered":"2024-03-24T10:45:01.895696+0000","last_unstale":"2024-03-24T10:58:34.230670+0000","last_undegraded":"2024-03-24T10:58:34.230670+0000","last_fullsized":"2024-03-24T10:58:34.230670+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:58:31.251092+0000","last_change":"2024-03-24T10:45:01.958119+0000","last_active":"2024-03-24T10:58:31.251092+0000","last_peered":"2024-03-24T10:58:31.251092+0000","last_clean":"2024-03-24T10:58:31.251092+0000","last_became_active":"2024-03-24T10:45:01.957832+0000","last_became_peered":"2024-03-24T10:45:01.957832+0000","last_unstale":"2024-03-24T10:58:31.251092+0000","last_undegraded":"2024-03-24T10:58:31.251092+0000","last_fullsized":"2024-03-24T10:58:31.251092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:58:31.214915+0000","last_change":"2024-03-24T10:45:01.842442+0000","last_active":"2024-03-24T10:58:31.214915+0000","last_peered":"2024-03-24T10:58:31.214915+0000","last_clean":"2024-03-24T10:58:31.214915+0000","last_became_active":"2024-03-24T10:45:01.842199+0000","last_became_peered":"2024-03-24T10:45:01.842199+0000","last_unstale":"2024-03-24T10:58:31.214915+0000","last_undegraded":"2024-03-24T10:58:31.214915+0000","last_fullsized":"2024-03-24T10:58:31.214915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:58:01.248691+0000","last_change":"2024-03-24T10:45:01.952378+0000","last_active":"2024-03-24T10:58:01.248691+0000","last_peered":"2024-03-24T10:58:01.248691+0000","last_clean":"2024-03-24T10:58:01.248691+0000","last_became_active":"2024-03-24T10:45:01.952123+0000","last_became_peered":"2024-03-24T10:45:01.952123+0000","last_unstale":"2024-03-24T10:58:01.248691+0000","last_undegraded":"2024-03-24T10:58:01.248691+0000","last_fullsized":"2024-03-24T10:58:01.248691+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:58:01.212618+0000","last_change":"2024-03-24T10:45:01.842299+0000","last_active":"2024-03-24T10:58:01.212618+0000","last_peered":"2024-03-24T10:58:01.212618+0000","last_clean":"2024-03-24T10:58:01.212618+0000","last_became_active":"2024-03-24T10:45:01.842097+0000","last_became_peered":"2024-03-24T10:45:01.842097+0000","last_unstale":"2024-03-24T10:58:01.212618+0000","last_undegraded":"2024-03-24T10:58:01.212618+0000","last_fullsized":"2024-03-24T10:58:01.212618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:54.295744+0000","last_change":"2024-03-24T10:45:01.893579+0000","last_active":"2024-03-24T10:55:54.295744+0000","last_peered":"2024-03-24T10:55:54.295744+0000","last_clean":"2024-03-24T10:55:54.295744+0000","last_became_active":"2024-03-24T10:45:01.893273+0000","last_became_peered":"2024-03-24T10:45:01.893273+0000","last_unstale":"2024-03-24T10:55:54.295744+0000","last_undegraded":"2024-03-24T10:55:54.295744+0000","last_fullsized":"2024-03-24T10:55:54.295744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:37:05.593545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:58:01.335292+0000","last_change":"2024-03-24T10:45:01.930032+0000","last_active":"2024-03-24T10:58:01.335292+0000","last_peered":"2024-03-24T10:58:01.335292+0000","last_clean":"2024-03-24T10:58:01.335292+0000","last_became_active":"2024-03-24T10:45:01.929770+0000","last_became_peered":"2024-03-24T10:45:01.929770+0000","last_unstale":"2024-03-24T10:58:01.335292+0000","last_undegraded":"2024-03-24T10:58:01.335292+0000","last_fullsized":"2024-03-24T10:58:01.335292+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:58:01.212607+0000","last_change":"2024-03-24T10:45:01.842460+0000","last_active":"2024-03-24T10:58:01.212607+0000","last_peered":"2024-03-24T10:58:01.212607+0000","last_clean":"2024-03-24T10:58:01.212607+0000","last_became_active":"2024-03-24T10:45:01.842165+0000","last_became_peered":"2024-03-24T10:45:01.842165+0000","last_unstale":"2024-03-24T10:58:01.212607+0000","last_undegraded":"2024-03-24T10:58:01.212607+0000","last_fullsized":"2024-03-24T10:58:01.212607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:34.226031+0000","last_change":"2024-03-24T10:45:01.895868+0000","last_active":"2024-03-24T10:57:34.226031+0000","last_peered":"2024-03-24T10:57:34.226031+0000","last_clean":"2024-03-24T10:57:34.226031+0000","last_became_active":"2024-03-24T10:45:01.895290+0000","last_became_peered":"2024-03-24T10:45:01.895290+0000","last_unstale":"2024-03-24T10:57:34.226031+0000","last_undegraded":"2024-03-24T10:57:34.226031+0000","last_fullsized":"2024-03-24T10:57:34.226031+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:31.246382+0000","last_change":"2024-03-24T10:45:01.957724+0000","last_active":"2024-03-24T10:57:31.246382+0000","last_peered":"2024-03-24T10:57:31.246382+0000","last_clean":"2024-03-24T10:57:31.246382+0000","last_became_active":"2024-03-24T10:45:01.957429+0000","last_became_peered":"2024-03-24T10:45:01.957429+0000","last_unstale":"2024-03-24T10:57:31.246382+0000","last_undegraded":"2024-03-24T10:57:31.246382+0000","last_fullsized":"2024-03-24T10:57:31.246382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:31.328625+0000","last_change":"2024-03-24T10:45:01.952159+0000","last_active":"2024-03-24T10:57:31.328625+0000","last_peered":"2024-03-24T10:57:31.328625+0000","last_clean":"2024-03-24T10:57:31.328625+0000","last_became_active":"2024-03-24T10:45:01.951961+0000","last_became_peered":"2024-03-24T10:45:01.951961+0000","last_unstale":"2024-03-24T10:57:31.328625+0000","last_undegraded":"2024-03-24T10:57:31.328625+0000","last_fullsized":"2024-03-24T10:57:31.328625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.293413+0000","last_change":"2024-03-24T10:45:01.893452+0000","last_active":"2024-03-24T10:55:29.293413+0000","last_peered":"2024-03-24T10:55:29.293413+0000","last_clean":"2024-03-24T10:55:29.293413+0000","last_became_active":"2024-03-24T10:45:01.893139+0000","last_became_peered":"2024-03-24T10:45:01.893139+0000","last_unstale":"2024-03-24T10:55:29.293413+0000","last_undegraded":"2024-03-24T10:55:29.293413+0000","last_fullsized":"2024-03-24T10:55:29.293413+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:59:23.844877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:04.264076+0000","last_change":"2024-03-24T10:45:01.928827+0000","last_active":"2024-03-24T10:55:04.264076+0000","last_peered":"2024-03-24T10:55:04.264076+0000","last_clean":"2024-03-24T10:55:04.264076+0000","last_became_active":"2024-03-24T10:45:01.928521+0000","last_became_peered":"2024-03-24T10:45:01.928521+0000","last_unstale":"2024-03-24T10:55:04.264076+0000","last_undegraded":"2024-03-24T10:55:04.264076+0000","last_fullsized":"2024-03-24T10:55:04.264076+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:59.643246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"24'9","reported_seq":46,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:16.018282+0000","last_change":"2024-03-24T10:59:16.018329+0000","last_active":"2024-03-24T10:59:16.018282+0000","last_peered":"2024-03-24T10:59:16.018282+0000","last_clean":"2024-03-24T10:59:16.018282+0000","last_became_active":"2024-03-24T10:45:00.637803+0000","last_became_peered":"2024-03-24T10:45:00.637803+0000","last_unstale":"2024-03-24T10:59:16.018282+0000","last_undegraded":"2024-03-24T10:59:16.018282+0000","last_fullsized":"2024-03-24T10:59:16.018282+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'9","last_scrub_stamp":"2024-03-24T10:59:16.018180+0000","last_deep_scrub":"24'9","last_deep_scrub_stamp":"2024-03-24T10:59:16.018180+0000","last_clean_scrub_stamp":"2024-03-24T10:59:16.018180+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:42:19.679039+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:34.226038+0000","last_change":"2024-03-24T10:45:01.897686+0000","last_active":"2024-03-24T10:57:34.226038+0000","last_peered":"2024-03-24T10:57:34.226038+0000","last_clean":"2024-03-24T10:57:34.226038+0000","last_became_active":"2024-03-24T10:45:01.897320+0000","last_became_peered":"2024-03-24T10:45:01.897320+0000","last_unstale":"2024-03-24T10:57:34.226038+0000","last_undegraded":"2024-03-24T10:57:34.226038+0000","last_fullsized":"2024-03-24T10:57:34.226038+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:04.291481+0000","last_change":"2024-03-24T10:45:00.638329+0000","last_active":"2024-03-24T10:55:04.291481+0000","last_peered":"2024-03-24T10:55:04.291481+0000","last_clean":"2024-03-24T10:55:04.291481+0000","last_became_active":"2024-03-24T10:45:00.638006+0000","last_became_peered":"2024-03-24T10:45:00.638006+0000","last_unstale":"2024-03-24T10:55:04.291481+0000","last_undegraded":"2024-03-24T10:55:04.291481+0000","last_fullsized":"2024-03-24T10:55:04.291481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:17:47.410675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"24'5","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:15.054805+0000","last_change":"2024-03-24T10:59:15.054857+0000","last_active":"2024-03-24T10:59:15.054805+0000","last_peered":"2024-03-24T10:59:15.054805+0000","last_clean":"2024-03-24T10:59:15.054805+0000","last_became_active":"2024-03-24T10:45:00.633427+0000","last_became_peered":"2024-03-24T10:45:00.633427+0000","last_unstale":"2024-03-24T10:59:15.054805+0000","last_undegraded":"2024-03-24T10:59:15.054805+0000","last_fullsized":"2024-03-24T10:59:15.054805+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'5","last_scrub_stamp":"2024-03-24T10:59:15.054726+0000","last_deep_scrub":"24'5","last_deep_scrub_stamp":"2024-03-24T10:59:15.054726+0000","last_clean_scrub_stamp":"2024-03-24T10:59:15.054726+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:51:51.292565+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:06.208468+0000","last_change":"2024-03-24T10:45:01.893902+0000","last_active":"2024-03-24T10:57:06.208468+0000","last_peered":"2024-03-24T10:57:06.208468+0000","last_clean":"2024-03-24T10:57:06.208468+0000","last_became_active":"2024-03-24T10:45:01.893617+0000","last_became_peered":"2024-03-24T10:45:01.893617+0000","last_unstale":"2024-03-24T10:57:06.208468+0000","last_undegraded":"2024-03-24T10:57:06.208468+0000","last_fullsized":"2024-03-24T10:57:06.208468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"24'4","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:06.244381+0000","last_change":"2024-03-24T10:45:00.638778+0000","last_active":"2024-03-24T10:57:06.244381+0000","last_peered":"2024-03-24T10:57:06.244381+0000","last_clean":"2024-03-24T10:57:06.244381+0000","last_became_active":"2024-03-24T10:45:00.638089+0000","last_became_peered":"2024-03-24T10:45:00.638089+0000","last_unstale":"2024-03-24T10:57:06.244381+0000","last_undegraded":"2024-03-24T10:57:06.244381+0000","last_fullsized":"2024-03-24T10:57:06.244381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:06.331378+0000","last_change":"2024-03-24T10:45:39.901726+0000","last_active":"2024-03-24T10:57:06.331378+0000","last_peered":"2024-03-24T10:57:06.331378+0000","last_clean":"2024-03-24T10:57:06.331378+0000","last_became_active":"2024-03-24T10:45:39.901416+0000","last_became_peered":"2024-03-24T10:45:39.901416+0000","last_unstale":"2024-03-24T10:57:06.331378+0000","last_undegraded":"2024-03-24T10:57:06.331378+0000","last_fullsized":"2024-03-24T10:57:06.331378+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'4","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:44.289869+0000","last_change":"2024-03-24T10:45:00.639822+0000","last_active":"2024-03-24T10:54:44.289869+0000","last_peered":"2024-03-24T10:54:44.289869+0000","last_clean":"2024-03-24T10:54:44.289869+0000","last_became_active":"2024-03-24T10:45:00.639600+0000","last_became_peered":"2024-03-24T10:45:00.639600+0000","last_unstale":"2024-03-24T10:54:44.289869+0000","last_undegraded":"2024-03-24T10:54:44.289869+0000","last_fullsized":"2024-03-24T10:54:44.289869+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:22:58.526690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:06.244427+0000","last_change":"2024-03-24T10:45:01.954015+0000","last_active":"2024-03-24T10:57:06.244427+0000","last_peered":"2024-03-24T10:57:06.244427+0000","last_clean":"2024-03-24T10:57:06.244427+0000","last_became_active":"2024-03-24T10:45:01.953767+0000","last_became_peered":"2024-03-24T10:45:01.953767+0000","last_unstale":"2024-03-24T10:57:06.244427+0000","last_undegraded":"2024-03-24T10:57:06.244427+0000","last_fullsized":"2024-03-24T10:57:06.244427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"24'6","reported_seq":40,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:15.314996+0000","last_change":"2024-03-24T10:59:15.315042+0000","last_active":"2024-03-24T10:59:15.314996+0000","last_peered":"2024-03-24T10:59:15.314996+0000","last_clean":"2024-03-24T10:59:15.314996+0000","last_became_active":"2024-03-24T10:45:00.636282+0000","last_became_peered":"2024-03-24T10:45:00.636282+0000","last_unstale":"2024-03-24T10:59:15.314996+0000","last_undegraded":"2024-03-24T10:59:15.314996+0000","last_fullsized":"2024-03-24T10:59:15.314996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'6","last_scrub_stamp":"2024-03-24T10:59:15.314886+0000","last_deep_scrub":"24'6","last_deep_scrub_stamp":"2024-03-24T10:59:15.314886+0000","last_clean_scrub_stamp":"2024-03-24T10:59:15.314886+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:51:52.966239+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:09.224087+0000","last_change":"2024-03-24T10:45:01.957905+0000","last_active":"2024-03-24T10:57:09.224087+0000","last_peered":"2024-03-24T10:57:09.224087+0000","last_clean":"2024-03-24T10:57:09.224087+0000","last_became_active":"2024-03-24T10:45:01.957676+0000","last_became_peered":"2024-03-24T10:45:01.957676+0000","last_unstale":"2024-03-24T10:57:09.224087+0000","last_undegraded":"2024-03-24T10:57:09.224087+0000","last_fullsized":"2024-03-24T10:57:09.224087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:06.331405+0000","last_change":"2024-03-24T10:45:00.635197+0000","last_active":"2024-03-24T10:57:06.331405+0000","last_peered":"2024-03-24T10:57:06.331405+0000","last_clean":"2024-03-24T10:57:06.331405+0000","last_became_active":"2024-03-24T10:45:00.634989+0000","last_became_peered":"2024-03-24T10:45:00.634989+0000","last_unstale":"2024-03-24T10:57:06.331405+0000","last_undegraded":"2024-03-24T10:57:06.331405+0000","last_fullsized":"2024-03-24T10:57:06.331405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.272421+0000","last_change":"2024-03-24T10:45:01.853785+0000","last_active":"2024-03-24T10:56:44.272421+0000","last_peered":"2024-03-24T10:56:44.272421+0000","last_clean":"2024-03-24T10:56:44.272421+0000","last_became_active":"2024-03-24T10:45:01.853504+0000","last_became_peered":"2024-03-24T10:45:01.853504+0000","last_unstale":"2024-03-24T10:56:44.272421+0000","last_undegraded":"2024-03-24T10:56:44.272421+0000","last_fullsized":"2024-03-24T10:56:44.272421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:30.942758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":36,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.300179+0000","last_change":"2024-03-24T10:45:00.632381+0000","last_active":"2024-03-24T10:56:44.300179+0000","last_peered":"2024-03-24T10:56:44.300179+0000","last_clean":"2024-03-24T10:56:44.300179+0000","last_became_active":"2024-03-24T10:45:00.632108+0000","last_became_peered":"2024-03-24T10:45:00.632108+0000","last_unstale":"2024-03-24T10:56:44.300179+0000","last_undegraded":"2024-03-24T10:56:44.300179+0000","last_fullsized":"2024-03-24T10:56:44.300179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:06.142451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:54:41.226243+0000","last_change":"2024-03-24T10:45:01.853591+0000","last_active":"2024-03-24T10:54:41.226243+0000","last_peered":"2024-03-24T10:54:41.226243+0000","last_clean":"2024-03-24T10:54:41.226243+0000","last_became_active":"2024-03-24T10:45:01.853279+0000","last_became_peered":"2024-03-24T10:45:01.853279+0000","last_unstale":"2024-03-24T10:54:41.226243+0000","last_undegraded":"2024-03-24T10:54:41.226243+0000","last_fullsized":"2024-03-24T10:54:41.226243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:07:24.149961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"24'15","reported_seq":50,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:11.426620+0000","last_change":"2024-03-24T10:59:11.426671+0000","last_active":"2024-03-24T10:59:11.426620+0000","last_peered":"2024-03-24T10:59:11.426620+0000","last_clean":"2024-03-24T10:59:11.426620+0000","last_became_active":"2024-03-24T10:45:00.633893+0000","last_became_peered":"2024-03-24T10:45:00.633893+0000","last_unstale":"2024-03-24T10:59:11.426620+0000","last_undegraded":"2024-03-24T10:59:11.426620+0000","last_fullsized":"2024-03-24T10:59:11.426620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'15","last_scrub_stamp":"2024-03-24T10:59:11.426552+0000","last_deep_scrub":"24'15","last_deep_scrub_stamp":"2024-03-24T10:59:11.426552+0000","last_clean_scrub_stamp":"2024-03-24T10:59:11.426552+0000","objects_scrubbed":6,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:34:28.140694+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:13.389392+0000","last_change":"2024-03-24T10:59:13.389433+0000","last_active":"2024-03-24T10:59:13.389392+0000","last_peered":"2024-03-24T10:59:13.389392+0000","last_clean":"2024-03-24T10:59:13.389392+0000","last_became_active":"2024-03-24T10:45:01.893178+0000","last_became_peered":"2024-03-24T10:45:01.893178+0000","last_unstale":"2024-03-24T10:59:13.389392+0000","last_undegraded":"2024-03-24T10:59:13.389392+0000","last_fullsized":"2024-03-24T10:59:13.389392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:13.389283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:13.389283+0000","last_clean_scrub_stamp":"2024-03-24T10:59:13.389283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:50:12.156525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"24'6","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:01.253716+0000","last_change":"2024-03-24T10:45:00.638214+0000","last_active":"2024-03-24T10:59:01.253716+0000","last_peered":"2024-03-24T10:59:01.253716+0000","last_clean":"2024-03-24T10:59:01.253716+0000","last_became_active":"2024-03-24T10:45:00.637973+0000","last_became_peered":"2024-03-24T10:45:00.637973+0000","last_unstale":"2024-03-24T10:59:01.253716+0000","last_undegraded":"2024-03-24T10:59:01.253716+0000","last_fullsized":"2024-03-24T10:59:01.253716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.237815+0000","last_change":"2024-03-24T10:45:01.856756+0000","last_active":"2024-03-24T10:56:41.237815+0000","last_peered":"2024-03-24T10:56:41.237815+0000","last_clean":"2024-03-24T10:56:41.237815+0000","last_became_active":"2024-03-24T10:45:01.856482+0000","last_became_peered":"2024-03-24T10:45:01.856482+0000","last_unstale":"2024-03-24T10:56:41.237815+0000","last_undegraded":"2024-03-24T10:56:41.237815+0000","last_fullsized":"2024-03-24T10:56:41.237815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:38:12.476490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.300155+0000","last_change":"2024-03-24T10:45:00.636481+0000","last_active":"2024-03-24T10:56:44.300155+0000","last_peered":"2024-03-24T10:56:44.300155+0000","last_clean":"2024-03-24T10:56:44.300155+0000","last_became_active":"2024-03-24T10:45:00.636206+0000","last_became_peered":"2024-03-24T10:45:00.636206+0000","last_unstale":"2024-03-24T10:56:44.300155+0000","last_undegraded":"2024-03-24T10:56:44.300155+0000","last_fullsized":"2024-03-24T10:56:44.300155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:53:14.780117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:44.300128+0000","last_change":"2024-03-24T10:45:01.897279+0000","last_active":"2024-03-24T10:56:44.300128+0000","last_peered":"2024-03-24T10:56:44.300128+0000","last_clean":"2024-03-24T10:56:44.300128+0000","last_became_active":"2024-03-24T10:45:01.896736+0000","last_became_peered":"2024-03-24T10:45:01.896736+0000","last_unstale":"2024-03-24T10:56:44.300128+0000","last_undegraded":"2024-03-24T10:56:44.300128+0000","last_fullsized":"2024-03-24T10:56:44.300128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:10.504780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"24'25","reported_seq":60,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:13.034065+0000","last_change":"2024-03-24T10:59:13.034134+0000","last_active":"2024-03-24T10:59:13.034065+0000","last_peered":"2024-03-24T10:59:13.034065+0000","last_clean":"2024-03-24T10:59:13.034065+0000","last_became_active":"2024-03-24T10:45:00.563812+0000","last_became_peered":"2024-03-24T10:45:00.563812+0000","last_unstale":"2024-03-24T10:59:13.034065+0000","last_undegraded":"2024-03-24T10:59:13.034065+0000","last_fullsized":"2024-03-24T10:59:13.034065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'25","last_scrub_stamp":"2024-03-24T10:59:13.033967+0000","last_deep_scrub":"24'25","last_deep_scrub_stamp":"2024-03-24T10:59:13.033967+0000","last_clean_scrub_stamp":"2024-03-24T10:59:13.033967+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:57:34.971277+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:41.237784+0000","last_change":"2024-03-24T10:45:01.854429+0000","last_active":"2024-03-24T10:56:41.237784+0000","last_peered":"2024-03-24T10:56:41.237784+0000","last_clean":"2024-03-24T10:56:41.237784+0000","last_became_active":"2024-03-24T10:45:01.854179+0000","last_became_peered":"2024-03-24T10:45:01.854179+0000","last_unstale":"2024-03-24T10:56:41.237784+0000","last_undegraded":"2024-03-24T10:56:41.237784+0000","last_fullsized":"2024-03-24T10:56:41.237784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:44:41.796891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:19.270242+0000","last_change":"2024-03-24T10:45:00.634223+0000","last_active":"2024-03-24T10:56:19.270242+0000","last_peered":"2024-03-24T10:56:19.270242+0000","last_clean":"2024-03-24T10:56:19.270242+0000","last_became_active":"2024-03-24T10:45:00.633355+0000","last_became_peered":"2024-03-24T10:45:00.633355+0000","last_unstale":"2024-03-24T10:56:19.270242+0000","last_undegraded":"2024-03-24T10:56:19.270242+0000","last_fullsized":"2024-03-24T10:56:19.270242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:04.930115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:11.340607+0000","last_change":"2024-03-24T10:45:40.518338+0000","last_active":"2024-03-24T10:59:11.340607+0000","last_peered":"2024-03-24T10:59:11.340607+0000","last_clean":"2024-03-24T10:59:11.340607+0000","last_became_active":"2024-03-24T10:45:40.518035+0000","last_became_peered":"2024-03-24T10:45:40.518035+0000","last_unstale":"2024-03-24T10:59:11.340607+0000","last_undegraded":"2024-03-24T10:59:11.340607+0000","last_fullsized":"2024-03-24T10:59:11.340607+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"24'7","reported_seq":42,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:14.067847+0000","last_change":"2024-03-24T10:59:14.067880+0000","last_active":"2024-03-24T10:59:14.067847+0000","last_peered":"2024-03-24T10:59:14.067847+0000","last_clean":"2024-03-24T10:59:14.067847+0000","last_became_active":"2024-03-24T10:45:00.626195+0000","last_became_peered":"2024-03-24T10:45:00.626195+0000","last_unstale":"2024-03-24T10:59:14.067847+0000","last_undegraded":"2024-03-24T10:59:14.067847+0000","last_fullsized":"2024-03-24T10:59:14.067847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'7","last_scrub_stamp":"2024-03-24T10:59:14.067785+0000","last_deep_scrub":"24'7","last_deep_scrub_stamp":"2024-03-24T10:59:14.067785+0000","last_clean_scrub_stamp":"2024-03-24T10:59:14.067785+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:05:15.517970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.235444+0000","last_change":"2024-03-24T10:45:01.856995+0000","last_active":"2024-03-24T10:56:16.235444+0000","last_peered":"2024-03-24T10:56:16.235444+0000","last_clean":"2024-03-24T10:56:16.235444+0000","last_became_active":"2024-03-24T10:45:01.856531+0000","last_became_peered":"2024-03-24T10:45:01.856531+0000","last_unstale":"2024-03-24T10:56:16.235444+0000","last_undegraded":"2024-03-24T10:56:16.235444+0000","last_fullsized":"2024-03-24T10:56:16.235444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:29:31.508911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"24'4","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:14.469596+0000","last_change":"2024-03-24T10:59:14.469638+0000","last_active":"2024-03-24T10:59:14.469596+0000","last_peered":"2024-03-24T10:59:14.469596+0000","last_clean":"2024-03-24T10:59:14.469596+0000","last_became_active":"2024-03-24T10:45:00.628158+0000","last_became_peered":"2024-03-24T10:45:00.628158+0000","last_unstale":"2024-03-24T10:59:14.469596+0000","last_undegraded":"2024-03-24T10:59:14.469596+0000","last_fullsized":"2024-03-24T10:59:14.469596+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'4","last_scrub_stamp":"2024-03-24T10:59:14.469521+0000","last_deep_scrub":"24'4","last_deep_scrub_stamp":"2024-03-24T10:59:14.469521+0000","last_clean_scrub_stamp":"2024-03-24T10:59:14.469521+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:51:50.707360+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:11.335803+0000","last_change":"2024-03-24T10:45:39.901453+0000","last_active":"2024-03-24T10:59:11.335803+0000","last_peered":"2024-03-24T10:59:11.335803+0000","last_clean":"2024-03-24T10:59:11.335803+0000","last_became_active":"2024-03-24T10:45:39.901119+0000","last_became_peered":"2024-03-24T10:45:39.901119+0000","last_unstale":"2024-03-24T10:59:11.335803+0000","last_undegraded":"2024-03-24T10:59:11.335803+0000","last_fullsized":"2024-03-24T10:59:11.335803+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"24'8","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:16.055714+0000","last_change":"2024-03-24T10:59:16.055761+0000","last_active":"2024-03-24T10:59:16.055714+0000","last_peered":"2024-03-24T10:59:16.055714+0000","last_clean":"2024-03-24T10:59:16.055714+0000","last_became_active":"2024-03-24T10:45:00.638785+0000","last_became_peered":"2024-03-24T10:45:00.638785+0000","last_unstale":"2024-03-24T10:59:16.055714+0000","last_undegraded":"2024-03-24T10:59:16.055714+0000","last_fullsized":"2024-03-24T10:59:16.055714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'8","last_scrub_stamp":"2024-03-24T10:59:16.055597+0000","last_deep_scrub":"24'8","last_deep_scrub_stamp":"2024-03-24T10:59:16.055597+0000","last_clean_scrub_stamp":"2024-03-24T10:59:16.055597+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:36:00.915701+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.235406+0000","last_change":"2024-03-24T10:45:01.953772+0000","last_active":"2024-03-24T10:56:16.235406+0000","last_peered":"2024-03-24T10:56:16.235406+0000","last_clean":"2024-03-24T10:56:16.235406+0000","last_became_active":"2024-03-24T10:45:01.953402+0000","last_became_peered":"2024-03-24T10:45:01.953402+0000","last_unstale":"2024-03-24T10:56:16.235406+0000","last_undegraded":"2024-03-24T10:56:16.235406+0000","last_fullsized":"2024-03-24T10:56:16.235406+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:22.089070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":183,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:51.238749+0000","last_change":"2024-03-24T10:44:48.241074+0000","last_active":"2024-03-24T10:56:51.238749+0000","last_peered":"2024-03-24T10:56:51.238749+0000","last_clean":"2024-03-24T10:56:51.238749+0000","last_became_active":"2024-03-24T10:44:48.239986+0000","last_became_peered":"2024-03-24T10:44:48.239986+0000","last_unstale":"2024-03-24T10:56:51.238749+0000","last_undegraded":"2024-03-24T10:56:51.238749+0000","last_fullsized":"2024-03-24T10:56:51.238749+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:46.142247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:46.142247+0000","last_clean_scrub_stamp":"2024-03-24T10:44:46.142247+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:05:50.110100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:31.246418+0000","last_change":"2024-03-24T10:45:00.637276+0000","last_active":"2024-03-24T10:57:31.246418+0000","last_peered":"2024-03-24T10:57:31.246418+0000","last_clean":"2024-03-24T10:57:31.246418+0000","last_became_active":"2024-03-24T10:45:00.636734+0000","last_became_peered":"2024-03-24T10:45:00.636734+0000","last_unstale":"2024-03-24T10:57:31.246418+0000","last_undegraded":"2024-03-24T10:57:31.246418+0000","last_fullsized":"2024-03-24T10:57:31.246418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:31.210384+0000","last_change":"2024-03-24T10:45:01.958505+0000","last_active":"2024-03-24T10:57:31.210384+0000","last_peered":"2024-03-24T10:57:31.210384+0000","last_clean":"2024-03-24T10:57:31.210384+0000","last_became_active":"2024-03-24T10:45:01.958268+0000","last_became_peered":"2024-03-24T10:45:01.958268+0000","last_unstale":"2024-03-24T10:57:31.210384+0000","last_undegraded":"2024-03-24T10:57:31.210384+0000","last_fullsized":"2024-03-24T10:57:31.210384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"24'5","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:01.239735+0000","last_change":"2024-03-24T10:45:00.597856+0000","last_active":"2024-03-24T10:57:01.239735+0000","last_peered":"2024-03-24T10:57:01.239735+0000","last_clean":"2024-03-24T10:57:01.239735+0000","last_became_active":"2024-03-24T10:45:00.597640+0000","last_became_peered":"2024-03-24T10:45:00.597640+0000","last_unstale":"2024-03-24T10:57:01.239735+0000","last_undegraded":"2024-03-24T10:57:01.239735+0000","last_fullsized":"2024-03-24T10:57:01.239735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:31:29.826202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:34.226005+0000","last_change":"2024-03-24T10:45:01.897502+0000","last_active":"2024-03-24T10:57:34.226005+0000","last_peered":"2024-03-24T10:57:34.226005+0000","last_clean":"2024-03-24T10:57:34.226005+0000","last_became_active":"2024-03-24T10:45:01.897250+0000","last_became_peered":"2024-03-24T10:45:01.897250+0000","last_unstale":"2024-03-24T10:57:34.226005+0000","last_undegraded":"2024-03-24T10:57:34.226005+0000","last_fullsized":"2024-03-24T10:57:34.226005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"24'9","reported_seq":44,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:15.059407+0000","last_change":"2024-03-24T10:59:15.059452+0000","last_active":"2024-03-24T10:59:15.059407+0000","last_peered":"2024-03-24T10:59:15.059407+0000","last_clean":"2024-03-24T10:59:15.059407+0000","last_became_active":"2024-03-24T10:45:00.626881+0000","last_became_peered":"2024-03-24T10:45:00.626881+0000","last_unstale":"2024-03-24T10:59:15.059407+0000","last_undegraded":"2024-03-24T10:59:15.059407+0000","last_fullsized":"2024-03-24T10:59:15.059407+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'9","last_scrub_stamp":"2024-03-24T10:59:15.059339+0000","last_deep_scrub":"24'9","last_deep_scrub_stamp":"2024-03-24T10:59:15.059339+0000","last_clean_scrub_stamp":"2024-03-24T10:59:15.059339+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:26.175028+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:01.339797+0000","last_change":"2024-03-24T10:45:01.930271+0000","last_active":"2024-03-24T10:59:01.339797+0000","last_peered":"2024-03-24T10:59:01.339797+0000","last_clean":"2024-03-24T10:59:01.339797+0000","last_became_active":"2024-03-24T10:45:01.929998+0000","last_became_peered":"2024-03-24T10:45:01.929998+0000","last_unstale":"2024-03-24T10:59:01.339797+0000","last_undegraded":"2024-03-24T10:59:01.339797+0000","last_fullsized":"2024-03-24T10:59:01.339797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:31.246404+0000","last_change":"2024-03-24T10:45:00.638309+0000","last_active":"2024-03-24T10:57:31.246404+0000","last_peered":"2024-03-24T10:57:31.246404+0000","last_clean":"2024-03-24T10:57:31.246404+0000","last_became_active":"2024-03-24T10:45:00.638041+0000","last_became_peered":"2024-03-24T10:45:00.638041+0000","last_unstale":"2024-03-24T10:57:31.246404+0000","last_undegraded":"2024-03-24T10:57:31.246404+0000","last_fullsized":"2024-03-24T10:57:31.246404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:29.266124+0000","last_change":"2024-03-24T10:45:01.897722+0000","last_active":"2024-03-24T10:55:29.266124+0000","last_peered":"2024-03-24T10:55:29.266124+0000","last_clean":"2024-03-24T10:55:29.266124+0000","last_became_active":"2024-03-24T10:45:01.897159+0000","last_became_peered":"2024-03-24T10:45:01.897159+0000","last_unstale":"2024-03-24T10:55:29.266124+0000","last_undegraded":"2024-03-24T10:55:29.266124+0000","last_fullsized":"2024-03-24T10:55:29.266124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:42.293342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":36,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:15.494767+0000","last_change":"2024-03-24T10:59:15.494816+0000","last_active":"2024-03-24T10:59:15.494767+0000","last_peered":"2024-03-24T10:59:15.494767+0000","last_clean":"2024-03-24T10:59:15.494767+0000","last_became_active":"2024-03-24T10:45:00.628971+0000","last_became_peered":"2024-03-24T10:45:00.628971+0000","last_unstale":"2024-03-24T10:59:15.494767+0000","last_undegraded":"2024-03-24T10:59:15.494767+0000","last_fullsized":"2024-03-24T10:59:15.494767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-24T10:59:15.494650+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-24T10:59:15.494650+0000","last_clean_scrub_stamp":"2024-03-24T10:59:15.494650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:07:32.776779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:31.246453+0000","last_change":"2024-03-24T10:45:01.952527+0000","last_active":"2024-03-24T10:57:31.246453+0000","last_peered":"2024-03-24T10:57:31.246453+0000","last_clean":"2024-03-24T10:57:31.246453+0000","last_became_active":"2024-03-24T10:45:01.952192+0000","last_became_peered":"2024-03-24T10:45:01.952192+0000","last_unstale":"2024-03-24T10:57:31.246453+0000","last_undegraded":"2024-03-24T10:57:31.246453+0000","last_fullsized":"2024-03-24T10:57:31.246453+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:26.230535+0000","last_change":"2024-03-24T10:45:00.631941+0000","last_active":"2024-03-24T10:55:26.230535+0000","last_peered":"2024-03-24T10:55:26.230535+0000","last_clean":"2024-03-24T10:55:26.230535+0000","last_became_active":"2024-03-24T10:45:00.631670+0000","last_became_peered":"2024-03-24T10:45:00.631670+0000","last_unstale":"2024-03-24T10:55:26.230535+0000","last_undegraded":"2024-03-24T10:55:26.230535+0000","last_fullsized":"2024-03-24T10:55:26.230535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:09:45.225115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:31.333127+0000","last_change":"2024-03-24T10:45:01.930483+0000","last_active":"2024-03-24T10:57:31.333127+0000","last_peered":"2024-03-24T10:57:31.333127+0000","last_clean":"2024-03-24T10:57:31.333127+0000","last_became_active":"2024-03-24T10:45:01.930220+0000","last_became_peered":"2024-03-24T10:45:01.930220+0000","last_unstale":"2024-03-24T10:57:31.333127+0000","last_undegraded":"2024-03-24T10:57:31.333127+0000","last_fullsized":"2024-03-24T10:57:31.333127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:58:01.335305+0000","last_change":"2024-03-24T10:45:00.633650+0000","last_active":"2024-03-24T10:58:01.335305+0000","last_peered":"2024-03-24T10:58:01.335305+0000","last_clean":"2024-03-24T10:58:01.335305+0000","last_became_active":"2024-03-24T10:45:00.633329+0000","last_became_peered":"2024-03-24T10:45:00.633329+0000","last_unstale":"2024-03-24T10:58:01.335305+0000","last_undegraded":"2024-03-24T10:58:01.335305+0000","last_fullsized":"2024-03-24T10:58:01.335305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:58:01.212626+0000","last_change":"2024-03-24T10:45:01.853487+0000","last_active":"2024-03-24T10:58:01.212626+0000","last_peered":"2024-03-24T10:58:01.212626+0000","last_clean":"2024-03-24T10:58:01.212626+0000","last_became_active":"2024-03-24T10:45:01.853257+0000","last_became_peered":"2024-03-24T10:45:01.853257+0000","last_unstale":"2024-03-24T10:58:01.212626+0000","last_undegraded":"2024-03-24T10:58:01.212626+0000","last_fullsized":"2024-03-24T10:58:01.212626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:54.295762+0000","last_change":"2024-03-24T10:45:00.639282+0000","last_active":"2024-03-24T10:55:54.295762+0000","last_peered":"2024-03-24T10:55:54.295762+0000","last_clean":"2024-03-24T10:55:54.295762+0000","last_became_active":"2024-03-24T10:45:00.638999+0000","last_became_peered":"2024-03-24T10:45:00.638999+0000","last_unstale":"2024-03-24T10:55:54.295762+0000","last_undegraded":"2024-03-24T10:55:54.295762+0000","last_fullsized":"2024-03-24T10:55:54.295762+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:41.176786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:58:04.228283+0000","last_change":"2024-03-24T10:45:01.928900+0000","last_active":"2024-03-24T10:58:04.228283+0000","last_peered":"2024-03-24T10:58:04.228283+0000","last_clean":"2024-03-24T10:58:04.228283+0000","last_became_active":"2024-03-24T10:45:01.928610+0000","last_became_peered":"2024-03-24T10:45:01.928610+0000","last_unstale":"2024-03-24T10:58:04.228283+0000","last_undegraded":"2024-03-24T10:58:04.228283+0000","last_fullsized":"2024-03-24T10:58:04.228283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":28,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:51.232929+0000","last_change":"2024-03-24T10:45:00.631327+0000","last_active":"2024-03-24T10:55:51.232929+0000","last_peered":"2024-03-24T10:55:51.232929+0000","last_clean":"2024-03-24T10:55:51.232929+0000","last_became_active":"2024-03-24T10:45:00.631078+0000","last_became_peered":"2024-03-24T10:45:00.631078+0000","last_unstale":"2024-03-24T10:55:51.232929+0000","last_undegraded":"2024-03-24T10:55:51.232929+0000","last_fullsized":"2024-03-24T10:55:51.232929+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:51:43.263796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:54.268364+0000","last_change":"2024-03-24T10:45:01.897329+0000","last_active":"2024-03-24T10:55:54.268364+0000","last_peered":"2024-03-24T10:55:54.268364+0000","last_clean":"2024-03-24T10:55:54.268364+0000","last_became_active":"2024-03-24T10:45:01.897063+0000","last_became_peered":"2024-03-24T10:45:01.897063+0000","last_unstale":"2024-03-24T10:55:54.268364+0000","last_undegraded":"2024-03-24T10:55:54.268364+0000","last_fullsized":"2024-03-24T10:55:54.268364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:31:07.259007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"24'11","reported_seq":47,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:12.474191+0000","last_change":"2024-03-24T10:59:12.474217+0000","last_active":"2024-03-24T10:59:12.474191+0000","last_peered":"2024-03-24T10:59:12.474191+0000","last_clean":"2024-03-24T10:59:12.474191+0000","last_became_active":"2024-03-24T10:45:00.597671+0000","last_became_peered":"2024-03-24T10:45:00.597671+0000","last_unstale":"2024-03-24T10:59:12.474191+0000","last_undegraded":"2024-03-24T10:59:12.474191+0000","last_fullsized":"2024-03-24T10:59:12.474191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'11","last_scrub_stamp":"2024-03-24T10:59:12.474135+0000","last_deep_scrub":"24'11","last_deep_scrub_stamp":"2024-03-24T10:59:12.474135+0000","last_clean_scrub_stamp":"2024-03-24T10:59:12.474135+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:29:36.067679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:58:01.212580+0000","last_change":"2024-03-24T10:45:01.927822+0000","last_active":"2024-03-24T10:58:01.212580+0000","last_peered":"2024-03-24T10:58:01.212580+0000","last_clean":"2024-03-24T10:58:01.212580+0000","last_became_active":"2024-03-24T10:45:01.927538+0000","last_became_peered":"2024-03-24T10:45:01.927538+0000","last_unstale":"2024-03-24T10:58:01.212580+0000","last_undegraded":"2024-03-24T10:58:01.212580+0000","last_fullsized":"2024-03-24T10:58:01.212580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"24'5","reported_seq":41,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:13.474814+0000","last_change":"2024-03-24T10:59:13.474838+0000","last_active":"2024-03-24T10:59:13.474814+0000","last_peered":"2024-03-24T10:59:13.474814+0000","last_clean":"2024-03-24T10:59:13.474814+0000","last_became_active":"2024-03-24T10:45:00.626557+0000","last_became_peered":"2024-03-24T10:45:00.626557+0000","last_unstale":"2024-03-24T10:59:13.474814+0000","last_undegraded":"2024-03-24T10:59:13.474814+0000","last_fullsized":"2024-03-24T10:59:13.474814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'5","last_scrub_stamp":"2024-03-24T10:59:13.474755+0000","last_deep_scrub":"24'5","last_deep_scrub_stamp":"2024-03-24T10:59:13.474755+0000","last_clean_scrub_stamp":"2024-03-24T10:59:13.474755+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:34:39.478664+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:58:31.214937+0000","last_change":"2024-03-24T10:45:01.893192+0000","last_active":"2024-03-24T10:58:31.214937+0000","last_peered":"2024-03-24T10:58:31.214937+0000","last_clean":"2024-03-24T10:58:31.214937+0000","last_became_active":"2024-03-24T10:45:01.892934+0000","last_became_peered":"2024-03-24T10:45:01.892934+0000","last_unstale":"2024-03-24T10:58:31.214937+0000","last_undegraded":"2024-03-24T10:58:31.214937+0000","last_fullsized":"2024-03-24T10:58:31.214937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:56:16.235371+0000","last_change":"2024-03-24T10:45:00.630147+0000","last_active":"2024-03-24T10:56:16.235371+0000","last_peered":"2024-03-24T10:56:16.235371+0000","last_clean":"2024-03-24T10:56:16.235371+0000","last_became_active":"2024-03-24T10:45:00.629850+0000","last_became_peered":"2024-03-24T10:45:00.629850+0000","last_unstale":"2024-03-24T10:56:16.235371+0000","last_undegraded":"2024-03-24T10:56:16.235371+0000","last_fullsized":"2024-03-24T10:56:16.235371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:01:12.843222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:54.295704+0000","last_change":"2024-03-24T10:45:01.897333+0000","last_active":"2024-03-24T10:55:54.295704+0000","last_peered":"2024-03-24T10:55:54.295704+0000","last_clean":"2024-03-24T10:55:54.295704+0000","last_became_active":"2024-03-24T10:45:01.897080+0000","last_became_peered":"2024-03-24T10:45:01.897080+0000","last_unstale":"2024-03-24T10:55:54.295704+0000","last_undegraded":"2024-03-24T10:55:54.295704+0000","last_fullsized":"2024-03-24T10:55:54.295704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:13.887947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:31.246443+0000","last_change":"2024-03-24T10:45:01.956041+0000","last_active":"2024-03-24T10:57:31.246443+0000","last_peered":"2024-03-24T10:57:31.246443+0000","last_clean":"2024-03-24T10:57:31.246443+0000","last_became_active":"2024-03-24T10:45:01.955766+0000","last_became_peered":"2024-03-24T10:45:01.955766+0000","last_unstale":"2024-03-24T10:57:31.246443+0000","last_undegraded":"2024-03-24T10:57:31.246443+0000","last_fullsized":"2024-03-24T10:57:31.246443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:55:01.228149+0000","last_change":"2024-03-24T10:45:00.639913+0000","last_active":"2024-03-24T10:55:01.228149+0000","last_peered":"2024-03-24T10:55:01.228149+0000","last_clean":"2024-03-24T10:55:01.228149+0000","last_became_active":"2024-03-24T10:45:00.639653+0000","last_became_peered":"2024-03-24T10:45:00.639653+0000","last_unstale":"2024-03-24T10:55:01.228149+0000","last_undegraded":"2024-03-24T10:55:01.228149+0000","last_fullsized":"2024-03-24T10:55:01.228149+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:39:15.749451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:31.246433+0000","last_change":"2024-03-24T10:45:01.952340+0000","last_active":"2024-03-24T10:57:31.246433+0000","last_peered":"2024-03-24T10:57:31.246433+0000","last_clean":"2024-03-24T10:57:31.246433+0000","last_became_active":"2024-03-24T10:45:01.952074+0000","last_became_peered":"2024-03-24T10:45:01.952074+0000","last_unstale":"2024-03-24T10:57:31.246433+0000","last_undegraded":"2024-03-24T10:57:31.246433+0000","last_fullsized":"2024-03-24T10:57:31.246433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"24'12","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:31.242486+0000","last_change":"2024-03-24T10:45:00.633790+0000","last_active":"2024-03-24T10:57:31.242486+0000","last_peered":"2024-03-24T10:57:31.242486+0000","last_clean":"2024-03-24T10:57:31.242486+0000","last_became_active":"2024-03-24T10:45:00.633494+0000","last_became_peered":"2024-03-24T10:45:00.633494+0000","last_unstale":"2024-03-24T10:57:31.242486+0000","last_undegraded":"2024-03-24T10:57:31.242486+0000","last_fullsized":"2024-03-24T10:57:31.242486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:44:59.364223+0000","last_clean_scrub_stamp":"2024-03-24T10:44:59.364223+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:01:14.049099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:57:34.226055+0000","last_change":"2024-03-24T10:45:01.957958+0000","last_active":"2024-03-24T10:57:34.226055+0000","last_peered":"2024-03-24T10:57:34.226055+0000","last_clean":"2024-03-24T10:57:34.226055+0000","last_became_active":"2024-03-24T10:45:01.957676+0000","last_became_peered":"2024-03-24T10:45:01.957676+0000","last_unstale":"2024-03-24T10:57:34.226055+0000","last_undegraded":"2024-03-24T10:57:34.226055+0000","last_fullsized":"2024-03-24T10:57:34.226055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":38,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:14.078489+0000","last_change":"2024-03-24T10:59:14.078526+0000","last_active":"2024-03-24T10:59:14.078489+0000","last_peered":"2024-03-24T10:59:14.078489+0000","last_clean":"2024-03-24T10:59:14.078489+0000","last_became_active":"2024-03-24T10:45:00.628712+0000","last_became_peered":"2024-03-24T10:45:00.628712+0000","last_unstale":"2024-03-24T10:59:14.078489+0000","last_undegraded":"2024-03-24T10:59:14.078489+0000","last_fullsized":"2024-03-24T10:59:14.078489+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'2","last_scrub_stamp":"2024-03-24T10:59:14.078396+0000","last_deep_scrub":"21'2","last_deep_scrub_stamp":"2024-03-24T10:59:14.078396+0000","last_clean_scrub_stamp":"2024-03-24T10:59:14.078396+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:34:40.082305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:01.217079+0000","last_change":"2024-03-24T10:45:01.894602+0000","last_active":"2024-03-24T10:59:01.217079+0000","last_peered":"2024-03-24T10:59:01.217079+0000","last_clean":"2024-03-24T10:59:01.217079+0000","last_became_active":"2024-03-24T10:45:01.894362+0000","last_became_peered":"2024-03-24T10:45:01.894362+0000","last_unstale":"2024-03-24T10:59:01.217079+0000","last_undegraded":"2024-03-24T10:59:01.217079+0000","last_fullsized":"2024-03-24T10:59:01.217079+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:45:00.480779+0000","last_clean_scrub_stamp":"2024-03-24T10:45:00.480779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"24'19","reported_seq":54,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:12.424975+0000","last_change":"2024-03-24T10:59:12.425030+0000","last_active":"2024-03-24T10:59:12.424975+0000","last_peered":"2024-03-24T10:59:12.424975+0000","last_clean":"2024-03-24T10:59:12.424975+0000","last_became_active":"2024-03-24T10:45:00.638871+0000","last_became_peered":"2024-03-24T10:45:00.638871+0000","last_unstale":"2024-03-24T10:59:12.424975+0000","last_undegraded":"2024-03-24T10:59:12.424975+0000","last_fullsized":"2024-03-24T10:59:12.424975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'19","last_scrub_stamp":"2024-03-24T10:59:12.424858+0000","last_deep_scrub":"24'19","last_deep_scrub_stamp":"2024-03-24T10:59:12.424858+0000","last_clean_scrub_stamp":"2024-03-24T10:59:12.424858+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:14:26.074546+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":204,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":225,"ondisk_log_size":225,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65430,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542323,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":352256,"data_stored":685052,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542323,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":299008,"data_stored":96516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542322,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":282624,"data_stored":95020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542323,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":282624,"data_stored":94046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542323,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":94116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575025,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":278528,"data_stored":94986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575025,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31444,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340396,"statfs":{"total":96636764160,"available":96604565504,"internally_reserved":0,"allocated":274432,"data_stored":93914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575025,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31512,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340328,"statfs":{"total":96636764160,"available":96604495872,"internally_reserved":0,"allocated":344064,"data_stored":683826,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:59:18.005 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.005 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.005 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.006 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.006 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.006 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.006 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.006 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.006 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.006 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.007 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.007 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.007 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.007 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.007 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.007 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.007 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.007 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.008 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.008 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.008 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.008 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.008 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.008 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.008 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.008 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.008 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.008 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.009 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.009 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.009 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.009 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.009 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.009 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.009 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-03-24T10:44:59.364223+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=44, tm_sec=59, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.009 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.010 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-03-24T10:44:59.364223+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=44, tm_sec=59, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.010 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.010 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-03-24T10:44:59.364223+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=44, tm_sec=59, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.010 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.010 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.010 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-03-24T10:44:59.364223+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=44, tm_sec=59, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.010 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.010 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-03-24T10:44:59.364223+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=44, tm_sec=59, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.011 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.011 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-03-24T10:44:59.364223+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=44, tm_sec=59, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.011 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.011 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-03-24T10:44:59.364223+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=44, tm_sec=59, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.011 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.011 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.011 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-03-24T10:44:59.364223+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=44, tm_sec=59, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.012 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.012 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.012 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.012 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.012 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-24T10:44:46.142247+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=44, tm_sec=46, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.012 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-03-24T10:44:59.364223+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=44, tm_sec=59, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.012 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.013 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-03-24T10:44:59.364223+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=44, tm_sec=59, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.013 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.013 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.013 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-03-24T10:44:59.364223+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=44, tm_sec=59, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.013 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.013 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.013 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-03-24T10:44:59.364223+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=44, tm_sec=59, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.013 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.014 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-03-24T10:44:59.364223+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=44, tm_sec=59, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.014 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.014 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-03-24T10:44:59.364223+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=44, tm_sec=59, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.014 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.014 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-03-24T10:44:59.364223+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=44, tm_sec=59, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.014 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.014 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.015 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.015 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-03-24T10:44:59.364223+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=44, tm_sec=59, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.015 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.015 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.015 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-03-24T10:44:59.364223+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=44, tm_sec=59, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.015 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.015 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-03-24T10:44:59.364223+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=44, tm_sec=59, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.015 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.016 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-03-24T10:45:00.480779+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=45, tm_sec=0, tm_wday=6, tm_yday=84, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=24, tm_hour=10, tm_min=59, tm_sec=9, tm_wday=6, tm_yday=84, tm_isdst=0) 2024-03-24T10:59:18.016 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-24T10:59:33.264 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:59:33.267 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T10:59:38.017 DEBUG:teuthology.orchestra.run.smithi005:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T10:59:38.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.119+0000 7f5210d5e640 1 Processor -- start 2024-03-24T10:59:38.119 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.119+0000 7f5210d5e640 1 -- start start 2024-03-24T10:59:38.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.120+0000 7f5210d5e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f520c0cae90 0x7f520c0cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:38.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.120+0000 7f5210d5e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f520c0cb270 0x7f520c0cc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:38.120 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.120+0000 7f5210d5e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f520c0ccba0 con 0x7f520c0cd960 2024-03-24T10:59:38.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.120+0000 7f5210d5e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7f520c0cce40 con 0x7f520c0cae90 2024-03-24T10:59:38.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.120+0000 7f5210d5e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f520c15a590 con 0x7f520c0cb270 2024-03-24T10:59:38.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.121+0000 7f520a59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f520c0cae90 0x7f520c0cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:38.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.121+0000 7f5209d9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f520c0cb270 0x7f520c0cc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:38.121 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.121+0000 7f5209d9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f520c0cb270 0x7f520c0cc2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3301/0 says I am v2:172.21.15.5:42410/0 (socket says 172.21.15.5:42410) 2024-03-24T10:59:38.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.121+0000 7f520a59c640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f520c0cae90 0x7f520c0cf530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:41230/0 (socket says 172.21.15.5:41230) 2024-03-24T10:59:38.122 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.121+0000 7f5209d9b640 1 -- 172.21.15.5:0/2946846748 learned_addr learned my addr 172.21.15.5:0/2946846748 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:38.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.124+0000 7f5209d9b640 1 -- 172.21.15.5:0/2946846748 >> v2:172.21.15.57:3300/0 conn(0x7f520c0cae90 msgr2=0x7f520c0cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:38.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.124+0000 7f5209d9b640 1 --2- 172.21.15.5:0/2946846748 >> v2:172.21.15.57:3300/0 conn(0x7f520c0cae90 0x7f520c0cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:38.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.124+0000 7f5209d9b640 1 -- 172.21.15.5:0/2946846748 >> v1:172.21.15.57:6789/0 conn(0x7f520c0cd960 legacy=0x7f520c0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:38.124 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.124+0000 7f5209d9b640 1 -- 172.21.15.5:0/2946846748 --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f520c15a870 con 0x7f520c0cb270 2024-03-24T10:59:38.126 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.126+0000 7f5209d9b640 1 --2- 172.21.15.5:0/2946846748 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f520c0cb270 0x7f520c0cc2a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5200009870 tx=0x7f520002ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b4e259961d5cf20 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:38.130 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.131+0000 7f520959a640 1 -- 172.21.15.5:0/2946846748 <== mon.2 v2:172.21.15.57:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520002f840 con 0x7f520c0cb270 2024-03-24T10:59:38.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.131+0000 7f520959a640 1 -- 172.21.15.5:0/2946846748 <== mon.2 v2:172.21.15.57:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f520002f9a0 con 0x7f520c0cb270 2024-03-24T10:59:38.131 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.131+0000 7f520959a640 1 -- 172.21.15.5:0/2946846748 <== mon.2 v2:172.21.15.57:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520002fd30 con 0x7f520c0cb270 2024-03-24T10:59:38.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.131+0000 7f5210d5e640 1 -- 172.21.15.5:0/2946846748 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f520c0cb270 msgr2=0x7f520c0cc2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:38.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.131+0000 7f5210d5e640 1 --2- 172.21.15.5:0/2946846748 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f520c0cb270 0x7f520c0cc2a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5200009870 tx=0x7f520002ba20 comp rx=0 tx=0).stop 2024-03-24T10:59:38.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.132+0000 7f5210d5e640 1 -- 172.21.15.5:0/2946846748 shutdown_connections 2024-03-24T10:59:38.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.132+0000 7f5210d5e640 1 --2- 172.21.15.5:0/2946846748 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f520c0cb270 0x7f520c0cc2a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:38.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.132+0000 7f5210d5e640 1 --2- 172.21.15.5:0/2946846748 >> v2:172.21.15.57:3300/0 conn(0x7f520c0cae90 0x7f520c0cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:38.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.132+0000 7f5210d5e640 1 -- 172.21.15.5:0/2946846748 >> 172.21.15.5:0/2946846748 conn(0x7f520c0cfb80 msgr2=0x7f520c0cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:38.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.132+0000 7f5210d5e640 1 -- 172.21.15.5:0/2946846748 shutdown_connections 2024-03-24T10:59:38.132 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.132+0000 7f5210d5e640 1 -- 172.21.15.5:0/2946846748 wait complete. 2024-03-24T10:59:38.133 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.134+0000 7f5210d5e640 1 Processor -- start 2024-03-24T10:59:38.134 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.134+0000 7f5210d5e640 1 -- start start 2024-03-24T10:59:38.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.135+0000 7f5210d5e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f520c0cae90 0x7f520c1f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:38.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.135+0000 7f5210d5e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f520c0cb270 0x7f520c0c3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:38.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.135+0000 7f5210d5e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520c0cd960 0x7f520c15bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:38.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.135+0000 7f5210d5e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7f520c0b2fd0 con 0x7f520c0cd960 2024-03-24T10:59:38.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.135+0000 7f5210d5e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7f520c1f4240 con 0x7f520c0cae90 2024-03-24T10:59:38.135 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.135+0000 7f5210d5e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7f520c0c5140 con 0x7f520c0cb270 2024-03-24T10:59:38.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.135+0000 7f520ad9d640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520c0cd960 0x7f520c15bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:38.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.136+0000 7f5209d9b640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f520c0cb270 0x7f520c0c3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:38.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.136+0000 7f520a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f520c0cae90 0x7f520c1f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:38.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.136+0000 7f520a59c640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f520c0cae90 0x7f520c1f3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.5:41232/0 (socket says 172.21.15.5:41232) 2024-03-24T10:59:38.136 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.136+0000 7f520a59c640 1 -- 172.21.15.5:0/2513129996 learned_addr learned my addr 172.21.15.5:0/2513129996 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:38.139 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.139+0000 7f520ad9d640 1 -- 172.21.15.5:0/2513129996 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f520c0cb270 msgr2=0x7f520c0c3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:38.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.139+0000 7f520ad9d640 1 --2- 172.21.15.5:0/2513129996 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f520c0cb270 0x7f520c0c3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:38.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.139+0000 7f520ad9d640 1 -- 172.21.15.5:0/2513129996 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f520c0cae90 msgr2=0x7f520c1f3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:38.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.139+0000 7f520ad9d640 1 --2- 172.21.15.5:0/2513129996 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f520c0cae90 0x7f520c1f3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:38.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.140+0000 7f520ad9d640 1 -- 172.21.15.5:0/2513129996 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52000094d0 con 0x7f520c0cd960 2024-03-24T10:59:38.140 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.140+0000 7f520a59c640 1 --2- 172.21.15.5:0/2513129996 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f520c0cae90 0x7f520c1f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:59:38.142 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.142+0000 7f520ad9d640 1 --2- 172.21.15.5:0/2513129996 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520c0cd960 0x7f520c15bc10 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f520400c700 tx=0x7f520400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:38.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.145+0000 7f51faffd640 1 -- 172.21.15.5:0/2513129996 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204003b70 con 0x7f520c0cd960 2024-03-24T10:59:38.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.145+0000 7f51faffd640 1 -- 172.21.15.5:0/2513129996 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5204003cd0 con 0x7f520c0cd960 2024-03-24T10:59:38.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.145+0000 7f51faffd640 1 -- 172.21.15.5:0/2513129996 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5204005310 con 0x7f520c0cd960 2024-03-24T10:59:38.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.145+0000 7f5210d5e640 1 -- 172.21.15.5:0/2513129996 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f520c0c5420 con 0x7f520c0cd960 2024-03-24T10:59:38.146 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.146+0000 7f5210d5e640 1 -- 172.21.15.5:0/2513129996 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f520c0c59e0 con 0x7f520c0cd960 2024-03-24T10:59:38.150 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.147+0000 7f5210d5e640 1 -- 172.21.15.5:0/2513129996 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f520c0681d0 con 0x7f520c0cd960 2024-03-24T10:59:38.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.170+0000 7f51faffd640 1 -- 172.21.15.5:0/2513129996 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5204005470 con 0x7f520c0cd960 2024-03-24T10:59:38.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.170+0000 7f51faffd640 1 --2- 172.21.15.5:0/2513129996 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f51ec06b810 0x7f51ec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:38.170 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.170+0000 7f520a59c640 1 --2- 172.21.15.5:0/2513129996 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f51ec06b810 0x7f51ec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:38.171 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.171+0000 7f520a59c640 1 --2- 172.21.15.5:0/2513129996 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f51ec06b810 0x7f51ec06dcd0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f51fc0112c0 tx=0x7f51fc009190 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:38.177 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.177+0000 7f51faffd640 1 -- 172.21.15.5:0/2513129996 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7f5204059030 con 0x7f520c0cd960 2024-03-24T10:59:38.456 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.456+0000 7f51faffd640 1 -- 172.21.15.5:0/2513129996 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f52040552d0 con 0x7f520c0cd960 2024-03-24T10:59:38.652 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.652+0000 7f5210d5e640 1 -- 172.21.15.5:0/2513129996 --> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f520c15ae30 con 0x7f51ec06b810 2024-03-24T10:59:38.665 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.664+0000 7f51faffd640 1 -- 172.21.15.5:0/2513129996 <== mgr.4120 v2:172.21.15.5:6832/1102439181 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252140 (secure 0 0 0) 0x7f51fc009d20 con 0x7f51ec06b810 2024-03-24T10:59:38.668 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-24T10:59:38.669 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-03-24T10:59:38.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.668+0000 7f5210d5e640 1 -- 172.21.15.5:0/2513129996 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f51ec06b810 msgr2=0x7f51ec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:38.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.668+0000 7f5210d5e640 1 --2- 172.21.15.5:0/2513129996 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f51ec06b810 0x7f51ec06dcd0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f51fc0112c0 tx=0x7f51fc009190 comp rx=0 tx=0).stop 2024-03-24T10:59:38.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.668+0000 7f5210d5e640 1 -- 172.21.15.5:0/2513129996 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520c0cd960 msgr2=0x7f520c15bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:38.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.668+0000 7f5210d5e640 1 --2- 172.21.15.5:0/2513129996 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520c0cd960 0x7f520c15bc10 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f520400c700 tx=0x7f520400cbc0 comp rx=0 tx=0).stop 2024-03-24T10:59:38.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.668+0000 7f5210d5e640 1 -- 172.21.15.5:0/2513129996 shutdown_connections 2024-03-24T10:59:38.671 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.668+0000 7f5210d5e640 1 --2- 172.21.15.5:0/2513129996 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7f51ec06b810 0x7f51ec06dcd0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:38.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.668+0000 7f5210d5e640 1 --2- 172.21.15.5:0/2513129996 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7f520c0cd960 0x7f520c15bc10 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:38.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.668+0000 7f5210d5e640 1 --2- 172.21.15.5:0/2513129996 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7f520c0cb270 0x7f520c0c3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:38.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.668+0000 7f5210d5e640 1 --2- 172.21.15.5:0/2513129996 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f520c0cae90 0x7f520c1f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:38.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.668+0000 7f5210d5e640 1 -- 172.21.15.5:0/2513129996 >> 172.21.15.5:0/2513129996 conn(0x7f520c0cfb80 msgr2=0x7f520c0c8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:38.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.668+0000 7f5210d5e640 1 -- 172.21.15.5:0/2513129996 shutdown_connections 2024-03-24T10:59:38.672 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.668+0000 7f5210d5e640 1 -- 172.21.15.5:0/2513129996 wait complete. 2024-03-24T10:59:38.679 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":464,"stamp":"2024-03-24T10:59:37.551106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":127,"num_read_kb":110,"num_write":363,"num_write_kb":2478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":400,"ondisk_log_size":400,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":247584,"kb_used_data":2336,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754727136,"statfs":{"total":773094113280,"available":772840587264,"internally_reserved":0,"allocated":2392064,"data_stored":1937476,"data_compressed":65430,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":12750,"internal_metadata":251055666},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002357"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:29.179735+0000","last_change":"2024-03-24T10:59:29.179735+0000","last_active":"2024-03-24T10:59:29.179735+0000","last_peered":"2024-03-24T10:59:29.179735+0000","last_clean":"2024-03-24T10:59:29.179735+0000","last_became_active":"2024-03-24T10:45:01.893734+0000","last_became_peered":"2024-03-24T10:45:01.893734+0000","last_unstale":"2024-03-24T10:59:29.179735+0000","last_undegraded":"2024-03-24T10:59:29.179735+0000","last_fullsized":"2024-03-24T10:59:29.179735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:29.179629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:29.179629+0000","last_clean_scrub_stamp":"2024-03-24T10:59:29.179629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:47:08.274618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:22.381147+0000","last_change":"2024-03-24T10:59:22.381147+0000","last_active":"2024-03-24T10:59:22.381147+0000","last_peered":"2024-03-24T10:59:22.381147+0000","last_clean":"2024-03-24T10:59:22.381147+0000","last_became_active":"2024-03-24T10:45:01.931449+0000","last_became_peered":"2024-03-24T10:45:01.931449+0000","last_unstale":"2024-03-24T10:59:22.381147+0000","last_undegraded":"2024-03-24T10:59:22.381147+0000","last_fullsized":"2024-03-24T10:59:22.381147+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:22.381060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:22.381060+0000","last_clean_scrub_stamp":"2024-03-24T10:59:22.381060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:32:41.479857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:15.370590+0000","last_change":"2024-03-24T10:59:15.370609+0000","last_active":"2024-03-24T10:59:15.370590+0000","last_peered":"2024-03-24T10:59:15.370590+0000","last_clean":"2024-03-24T10:59:15.370590+0000","last_became_active":"2024-03-24T10:45:01.895654+0000","last_became_peered":"2024-03-24T10:45:01.895654+0000","last_unstale":"2024-03-24T10:59:15.370590+0000","last_undegraded":"2024-03-24T10:59:15.370590+0000","last_fullsized":"2024-03-24T10:59:15.370590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:15.370520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:15.370520+0000","last_clean_scrub_stamp":"2024-03-24T10:59:15.370520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:51:53.021873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:22.194374+0000","last_change":"2024-03-24T10:59:22.194410+0000","last_active":"2024-03-24T10:59:22.194374+0000","last_peered":"2024-03-24T10:59:22.194374+0000","last_clean":"2024-03-24T10:59:22.194374+0000","last_became_active":"2024-03-24T10:45:01.952219+0000","last_became_peered":"2024-03-24T10:45:01.952219+0000","last_unstale":"2024-03-24T10:59:22.194374+0000","last_undegraded":"2024-03-24T10:59:22.194374+0000","last_fullsized":"2024-03-24T10:59:22.194374+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:22.194257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:22.194257+0000","last_clean_scrub_stamp":"2024-03-24T10:59:22.194257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:30:32.015309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:30.139470+0000","last_change":"2024-03-24T10:59:30.139509+0000","last_active":"2024-03-24T10:59:30.139470+0000","last_peered":"2024-03-24T10:59:30.139470+0000","last_clean":"2024-03-24T10:59:30.139470+0000","last_became_active":"2024-03-24T10:45:01.895622+0000","last_became_peered":"2024-03-24T10:45:01.895622+0000","last_unstale":"2024-03-24T10:59:30.139470+0000","last_undegraded":"2024-03-24T10:59:30.139470+0000","last_fullsized":"2024-03-24T10:59:30.139470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:30.139352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:30.139352+0000","last_clean_scrub_stamp":"2024-03-24T10:59:30.139352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:01:35.219137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:19.991786+0000","last_change":"2024-03-24T10:59:19.991820+0000","last_active":"2024-03-24T10:59:19.991786+0000","last_peered":"2024-03-24T10:59:19.991786+0000","last_clean":"2024-03-24T10:59:19.991786+0000","last_became_active":"2024-03-24T10:45:40.169447+0000","last_became_peered":"2024-03-24T10:45:40.169447+0000","last_unstale":"2024-03-24T10:59:19.991786+0000","last_undegraded":"2024-03-24T10:59:19.991786+0000","last_fullsized":"2024-03-24T10:59:19.991786+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:19.991683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:19.991683+0000","last_clean_scrub_stamp":"2024-03-24T10:59:19.991683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:07:37.273813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:28.224198+0000","last_change":"2024-03-24T10:59:28.224217+0000","last_active":"2024-03-24T10:59:28.224198+0000","last_peered":"2024-03-24T10:59:28.224198+0000","last_clean":"2024-03-24T10:59:28.224198+0000","last_became_active":"2024-03-24T10:45:01.892452+0000","last_became_peered":"2024-03-24T10:45:01.892452+0000","last_unstale":"2024-03-24T10:59:28.224198+0000","last_undegraded":"2024-03-24T10:59:28.224198+0000","last_fullsized":"2024-03-24T10:59:28.224198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:28.224131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:28.224131+0000","last_clean_scrub_stamp":"2024-03-24T10:59:28.224131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:09:46.998509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:26.098693+0000","last_change":"2024-03-24T10:59:26.098711+0000","last_active":"2024-03-24T10:59:26.098693+0000","last_peered":"2024-03-24T10:59:26.098693+0000","last_clean":"2024-03-24T10:59:26.098693+0000","last_became_active":"2024-03-24T10:45:01.953149+0000","last_became_peered":"2024-03-24T10:45:01.953149+0000","last_unstale":"2024-03-24T10:59:26.098693+0000","last_undegraded":"2024-03-24T10:59:26.098693+0000","last_fullsized":"2024-03-24T10:59:26.098693+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:26.098615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:26.098615+0000","last_clean_scrub_stamp":"2024-03-24T10:59:26.098615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:42:37.493913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:33.189309+0000","last_change":"2024-03-24T10:59:33.189343+0000","last_active":"2024-03-24T10:59:33.189309+0000","last_peered":"2024-03-24T10:59:33.189309+0000","last_clean":"2024-03-24T10:59:33.189309+0000","last_became_active":"2024-03-24T10:45:01.956207+0000","last_became_peered":"2024-03-24T10:45:01.956207+0000","last_unstale":"2024-03-24T10:59:33.189309+0000","last_undegraded":"2024-03-24T10:59:33.189309+0000","last_fullsized":"2024-03-24T10:59:33.189309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:33.189210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:33.189210+0000","last_clean_scrub_stamp":"2024-03-24T10:59:33.189210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:08:16.924062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:26.406926+0000","last_change":"2024-03-24T10:59:26.406926+0000","last_active":"2024-03-24T10:59:26.406926+0000","last_peered":"2024-03-24T10:59:26.406926+0000","last_clean":"2024-03-24T10:59:26.406926+0000","last_became_active":"2024-03-24T10:45:01.956209+0000","last_became_peered":"2024-03-24T10:45:01.956209+0000","last_unstale":"2024-03-24T10:59:26.406926+0000","last_undegraded":"2024-03-24T10:59:26.406926+0000","last_fullsized":"2024-03-24T10:59:26.406926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:26.406843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:26.406843+0000","last_clean_scrub_stamp":"2024-03-24T10:59:26.406843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:30:36.227894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:27.405873+0000","last_change":"2024-03-24T10:59:27.405873+0000","last_active":"2024-03-24T10:59:27.405873+0000","last_peered":"2024-03-24T10:59:27.405873+0000","last_clean":"2024-03-24T10:59:27.405873+0000","last_became_active":"2024-03-24T10:45:01.956849+0000","last_became_peered":"2024-03-24T10:45:01.956849+0000","last_unstale":"2024-03-24T10:59:27.405873+0000","last_undegraded":"2024-03-24T10:59:27.405873+0000","last_fullsized":"2024-03-24T10:59:27.405873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:27.405789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:27.405789+0000","last_clean_scrub_stamp":"2024-03-24T10:59:27.405789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:55:10.271681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:16.993751+0000","last_change":"2024-03-24T10:59:16.993751+0000","last_active":"2024-03-24T10:59:16.993751+0000","last_peered":"2024-03-24T10:59:16.993751+0000","last_clean":"2024-03-24T10:59:16.993751+0000","last_became_active":"2024-03-24T10:45:01.896400+0000","last_became_peered":"2024-03-24T10:45:01.896400+0000","last_unstale":"2024-03-24T10:59:16.993751+0000","last_undegraded":"2024-03-24T10:59:16.993751+0000","last_fullsized":"2024-03-24T10:59:16.993751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:16.993638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:16.993638+0000","last_clean_scrub_stamp":"2024-03-24T10:59:16.993638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:08:54.247949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:24.101367+0000","last_change":"2024-03-24T10:59:24.101367+0000","last_active":"2024-03-24T10:59:24.101367+0000","last_peered":"2024-03-24T10:59:24.101367+0000","last_clean":"2024-03-24T10:59:24.101367+0000","last_became_active":"2024-03-24T10:45:01.894632+0000","last_became_peered":"2024-03-24T10:45:01.894632+0000","last_unstale":"2024-03-24T10:59:24.101367+0000","last_undegraded":"2024-03-24T10:59:24.101367+0000","last_fullsized":"2024-03-24T10:59:24.101367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:24.101282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:24.101282+0000","last_clean_scrub_stamp":"2024-03-24T10:59:24.101282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:55:06.967174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:20.310488+0000","last_change":"2024-03-24T10:59:20.310525+0000","last_active":"2024-03-24T10:59:20.310488+0000","last_peered":"2024-03-24T10:59:20.310488+0000","last_clean":"2024-03-24T10:59:20.310488+0000","last_became_active":"2024-03-24T10:45:01.928460+0000","last_became_peered":"2024-03-24T10:45:01.928460+0000","last_unstale":"2024-03-24T10:59:20.310488+0000","last_undegraded":"2024-03-24T10:59:20.310488+0000","last_fullsized":"2024-03-24T10:59:20.310488+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:20.310402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:20.310402+0000","last_clean_scrub_stamp":"2024-03-24T10:59:20.310402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:20:52.877123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:18.419006+0000","last_change":"2024-03-24T10:59:18.419041+0000","last_active":"2024-03-24T10:59:18.419006+0000","last_peered":"2024-03-24T10:59:18.419006+0000","last_clean":"2024-03-24T10:59:18.419006+0000","last_became_active":"2024-03-24T10:45:01.928467+0000","last_became_peered":"2024-03-24T10:45:01.928467+0000","last_unstale":"2024-03-24T10:59:18.419006+0000","last_undegraded":"2024-03-24T10:59:18.419006+0000","last_fullsized":"2024-03-24T10:59:18.419006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:18.418936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:18.418936+0000","last_clean_scrub_stamp":"2024-03-24T10:59:18.418936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:54:38.449087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:25.186727+0000","last_change":"2024-03-24T10:59:25.186746+0000","last_active":"2024-03-24T10:59:25.186727+0000","last_peered":"2024-03-24T10:59:25.186727+0000","last_clean":"2024-03-24T10:59:25.186727+0000","last_became_active":"2024-03-24T10:45:01.955768+0000","last_became_peered":"2024-03-24T10:45:01.955768+0000","last_unstale":"2024-03-24T10:59:25.186727+0000","last_undegraded":"2024-03-24T10:59:25.186727+0000","last_fullsized":"2024-03-24T10:59:25.186727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:25.186662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:25.186662+0000","last_clean_scrub_stamp":"2024-03-24T10:59:25.186662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:59:53.486512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:27.149385+0000","last_change":"2024-03-24T10:59:27.149420+0000","last_active":"2024-03-24T10:59:27.149385+0000","last_peered":"2024-03-24T10:59:27.149385+0000","last_clean":"2024-03-24T10:59:27.149385+0000","last_became_active":"2024-03-24T10:45:01.955890+0000","last_became_peered":"2024-03-24T10:45:01.955890+0000","last_unstale":"2024-03-24T10:59:27.149385+0000","last_undegraded":"2024-03-24T10:59:27.149385+0000","last_fullsized":"2024-03-24T10:59:27.149385+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:27.149275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:27.149275+0000","last_clean_scrub_stamp":"2024-03-24T10:59:27.149275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:41.711245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:17.040401+0000","last_change":"2024-03-24T10:59:17.040435+0000","last_active":"2024-03-24T10:59:17.040401+0000","last_peered":"2024-03-24T10:59:17.040401+0000","last_clean":"2024-03-24T10:59:17.040401+0000","last_became_active":"2024-03-24T10:45:01.896391+0000","last_became_peered":"2024-03-24T10:45:01.896391+0000","last_unstale":"2024-03-24T10:59:17.040401+0000","last_undegraded":"2024-03-24T10:59:17.040401+0000","last_fullsized":"2024-03-24T10:59:17.040401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:17.040332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:17.040332+0000","last_clean_scrub_stamp":"2024-03-24T10:59:17.040332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:07:59.822350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:31.172250+0000","last_change":"2024-03-24T10:59:31.172272+0000","last_active":"2024-03-24T10:59:31.172250+0000","last_peered":"2024-03-24T10:59:31.172250+0000","last_clean":"2024-03-24T10:59:31.172250+0000","last_became_active":"2024-03-24T10:45:01.895668+0000","last_became_peered":"2024-03-24T10:45:01.895668+0000","last_unstale":"2024-03-24T10:59:31.172250+0000","last_undegraded":"2024-03-24T10:59:31.172250+0000","last_fullsized":"2024-03-24T10:59:31.172250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:31.172186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:31.172186+0000","last_clean_scrub_stamp":"2024-03-24T10:59:31.172186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:14:24.174117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:25.403409+0000","last_change":"2024-03-24T10:59:25.403446+0000","last_active":"2024-03-24T10:59:25.403409+0000","last_peered":"2024-03-24T10:59:25.403409+0000","last_clean":"2024-03-24T10:59:25.403409+0000","last_became_active":"2024-03-24T10:45:01.897878+0000","last_became_peered":"2024-03-24T10:45:01.897878+0000","last_unstale":"2024-03-24T10:59:25.403409+0000","last_undegraded":"2024-03-24T10:59:25.403409+0000","last_fullsized":"2024-03-24T10:59:25.403409+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:25.403304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:25.403304+0000","last_clean_scrub_stamp":"2024-03-24T10:59:25.403304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:20:57.970024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:16.328664+0000","last_change":"2024-03-24T10:59:16.328699+0000","last_active":"2024-03-24T10:59:16.328664+0000","last_peered":"2024-03-24T10:59:16.328664+0000","last_clean":"2024-03-24T10:59:16.328664+0000","last_became_active":"2024-03-24T10:45:01.895696+0000","last_became_peered":"2024-03-24T10:45:01.895696+0000","last_unstale":"2024-03-24T10:59:16.328664+0000","last_undegraded":"2024-03-24T10:59:16.328664+0000","last_fullsized":"2024-03-24T10:59:16.328664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:16.328563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:16.328563+0000","last_clean_scrub_stamp":"2024-03-24T10:59:16.328563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:24:04.845257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:29.149535+0000","last_change":"2024-03-24T10:59:29.149572+0000","last_active":"2024-03-24T10:59:29.149535+0000","last_peered":"2024-03-24T10:59:29.149535+0000","last_clean":"2024-03-24T10:59:29.149535+0000","last_became_active":"2024-03-24T10:45:01.957832+0000","last_became_peered":"2024-03-24T10:45:01.957832+0000","last_unstale":"2024-03-24T10:59:29.149535+0000","last_undegraded":"2024-03-24T10:59:29.149535+0000","last_fullsized":"2024-03-24T10:59:29.149535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:29.149465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:29.149465+0000","last_clean_scrub_stamp":"2024-03-24T10:59:29.149465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:22:15.965248+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:18.019972+0000","last_change":"2024-03-24T10:59:18.019991+0000","last_active":"2024-03-24T10:59:18.019972+0000","last_peered":"2024-03-24T10:59:18.019972+0000","last_clean":"2024-03-24T10:59:18.019972+0000","last_became_active":"2024-03-24T10:45:01.842199+0000","last_became_peered":"2024-03-24T10:45:01.842199+0000","last_unstale":"2024-03-24T10:59:18.019972+0000","last_undegraded":"2024-03-24T10:59:18.019972+0000","last_fullsized":"2024-03-24T10:59:18.019972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:18.019906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:18.019906+0000","last_clean_scrub_stamp":"2024-03-24T10:59:18.019906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:30:44.871420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:21.237621+0000","last_change":"2024-03-24T10:59:21.237640+0000","last_active":"2024-03-24T10:59:21.237621+0000","last_peered":"2024-03-24T10:59:21.237621+0000","last_clean":"2024-03-24T10:59:21.237621+0000","last_became_active":"2024-03-24T10:45:01.952123+0000","last_became_peered":"2024-03-24T10:45:01.952123+0000","last_unstale":"2024-03-24T10:59:21.237621+0000","last_undegraded":"2024-03-24T10:59:21.237621+0000","last_fullsized":"2024-03-24T10:59:21.237621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:21.237553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:21.237553+0000","last_clean_scrub_stamp":"2024-03-24T10:59:21.237553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:57:29.220641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:17.050494+0000","last_change":"2024-03-24T10:59:17.050514+0000","last_active":"2024-03-24T10:59:17.050494+0000","last_peered":"2024-03-24T10:59:17.050494+0000","last_clean":"2024-03-24T10:59:17.050494+0000","last_became_active":"2024-03-24T10:45:01.842097+0000","last_became_peered":"2024-03-24T10:45:01.842097+0000","last_unstale":"2024-03-24T10:59:17.050494+0000","last_undegraded":"2024-03-24T10:59:17.050494+0000","last_fullsized":"2024-03-24T10:59:17.050494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:17.050431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:17.050431+0000","last_clean_scrub_stamp":"2024-03-24T10:59:17.050431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:22:19.517013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:22.411559+0000","last_change":"2024-03-24T10:59:22.411578+0000","last_active":"2024-03-24T10:59:22.411559+0000","last_peered":"2024-03-24T10:59:22.411559+0000","last_clean":"2024-03-24T10:59:22.411559+0000","last_became_active":"2024-03-24T10:45:01.893273+0000","last_became_peered":"2024-03-24T10:45:01.893273+0000","last_unstale":"2024-03-24T10:59:22.411559+0000","last_undegraded":"2024-03-24T10:59:22.411559+0000","last_fullsized":"2024-03-24T10:59:22.411559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:22.411498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:22.411498+0000","last_clean_scrub_stamp":"2024-03-24T10:59:22.411498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:46:54.370093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:19.419154+0000","last_change":"2024-03-24T10:59:19.419186+0000","last_active":"2024-03-24T10:59:19.419154+0000","last_peered":"2024-03-24T10:59:19.419154+0000","last_clean":"2024-03-24T10:59:19.419154+0000","last_became_active":"2024-03-24T10:45:01.929770+0000","last_became_peered":"2024-03-24T10:45:01.929770+0000","last_unstale":"2024-03-24T10:59:19.419154+0000","last_undegraded":"2024-03-24T10:59:19.419154+0000","last_fullsized":"2024-03-24T10:59:19.419154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:19.419078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:19.419078+0000","last_clean_scrub_stamp":"2024-03-24T10:59:19.419078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:13:14.580630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:18.978434+0000","last_change":"2024-03-24T10:59:18.978470+0000","last_active":"2024-03-24T10:59:18.978434+0000","last_peered":"2024-03-24T10:59:18.978434+0000","last_clean":"2024-03-24T10:59:18.978434+0000","last_became_active":"2024-03-24T10:45:01.842165+0000","last_became_peered":"2024-03-24T10:45:01.842165+0000","last_unstale":"2024-03-24T10:59:18.978434+0000","last_undegraded":"2024-03-24T10:59:18.978434+0000","last_fullsized":"2024-03-24T10:59:18.978434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:18.978316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:18.978316+0000","last_clean_scrub_stamp":"2024-03-24T10:59:18.978316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:54:39.008467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:14.393509+0000","last_change":"2024-03-24T10:59:14.393547+0000","last_active":"2024-03-24T10:59:14.393509+0000","last_peered":"2024-03-24T10:59:14.393509+0000","last_clean":"2024-03-24T10:59:14.393509+0000","last_became_active":"2024-03-24T10:45:01.895290+0000","last_became_peered":"2024-03-24T10:45:01.895290+0000","last_unstale":"2024-03-24T10:59:14.393509+0000","last_undegraded":"2024-03-24T10:59:14.393509+0000","last_fullsized":"2024-03-24T10:59:14.393509+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:14.393402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:14.393402+0000","last_clean_scrub_stamp":"2024-03-24T10:59:14.393402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:08:51.556206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:28.104244+0000","last_change":"2024-03-24T10:59:28.104244+0000","last_active":"2024-03-24T10:59:28.104244+0000","last_peered":"2024-03-24T10:59:28.104244+0000","last_clean":"2024-03-24T10:59:28.104244+0000","last_became_active":"2024-03-24T10:45:01.957429+0000","last_became_peered":"2024-03-24T10:45:01.957429+0000","last_unstale":"2024-03-24T10:59:28.104244+0000","last_undegraded":"2024-03-24T10:59:28.104244+0000","last_fullsized":"2024-03-24T10:59:28.104244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:28.104136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:28.104136+0000","last_clean_scrub_stamp":"2024-03-24T10:59:28.104136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:54:47.300393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:18.005162+0000","last_change":"2024-03-24T10:59:18.005162+0000","last_active":"2024-03-24T10:59:18.005162+0000","last_peered":"2024-03-24T10:59:18.005162+0000","last_clean":"2024-03-24T10:59:18.005162+0000","last_became_active":"2024-03-24T10:45:01.951961+0000","last_became_peered":"2024-03-24T10:45:01.951961+0000","last_unstale":"2024-03-24T10:59:18.005162+0000","last_undegraded":"2024-03-24T10:59:18.005162+0000","last_fullsized":"2024-03-24T10:59:18.005162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:18.005076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:18.005076+0000","last_clean_scrub_stamp":"2024-03-24T10:59:18.005076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:14:31.654764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:21.407043+0000","last_change":"2024-03-24T10:59:21.407063+0000","last_active":"2024-03-24T10:59:21.407043+0000","last_peered":"2024-03-24T10:59:21.407043+0000","last_clean":"2024-03-24T10:59:21.407043+0000","last_became_active":"2024-03-24T10:45:01.893139+0000","last_became_peered":"2024-03-24T10:45:01.893139+0000","last_unstale":"2024-03-24T10:59:21.407043+0000","last_undegraded":"2024-03-24T10:59:21.407043+0000","last_fullsized":"2024-03-24T10:59:21.407043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:21.406967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:21.406967+0000","last_clean_scrub_stamp":"2024-03-24T10:59:21.406967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:24:09.923661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:20.106564+0000","last_change":"2024-03-24T10:59:20.106583+0000","last_active":"2024-03-24T10:59:20.106564+0000","last_peered":"2024-03-24T10:59:20.106564+0000","last_clean":"2024-03-24T10:59:20.106564+0000","last_became_active":"2024-03-24T10:45:01.928521+0000","last_became_peered":"2024-03-24T10:45:01.928521+0000","last_unstale":"2024-03-24T10:59:20.106564+0000","last_undegraded":"2024-03-24T10:59:20.106564+0000","last_fullsized":"2024-03-24T10:59:20.106564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:20.106503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:20.106503+0000","last_clean_scrub_stamp":"2024-03-24T10:59:20.106503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:14:33.756191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"24'9","reported_seq":46,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:16.018282+0000","last_change":"2024-03-24T10:59:16.018329+0000","last_active":"2024-03-24T10:59:16.018282+0000","last_peered":"2024-03-24T10:59:16.018282+0000","last_clean":"2024-03-24T10:59:16.018282+0000","last_became_active":"2024-03-24T10:45:00.637803+0000","last_became_peered":"2024-03-24T10:45:00.637803+0000","last_unstale":"2024-03-24T10:59:16.018282+0000","last_undegraded":"2024-03-24T10:59:16.018282+0000","last_fullsized":"2024-03-24T10:59:16.018282+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'9","last_scrub_stamp":"2024-03-24T10:59:16.018180+0000","last_deep_scrub":"24'9","last_deep_scrub_stamp":"2024-03-24T10:59:16.018180+0000","last_clean_scrub_stamp":"2024-03-24T10:59:16.018180+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:42:19.679039+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:18.312113+0000","last_change":"2024-03-24T10:59:18.312148+0000","last_active":"2024-03-24T10:59:18.312113+0000","last_peered":"2024-03-24T10:59:18.312113+0000","last_clean":"2024-03-24T10:59:18.312113+0000","last_became_active":"2024-03-24T10:45:01.897320+0000","last_became_peered":"2024-03-24T10:45:01.897320+0000","last_unstale":"2024-03-24T10:59:18.312113+0000","last_undegraded":"2024-03-24T10:59:18.312113+0000","last_fullsized":"2024-03-24T10:59:18.312113+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:18.312003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:18.312003+0000","last_clean_scrub_stamp":"2024-03-24T10:59:18.312003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:53:55.738567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"24'10","reported_seq":44,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:18.412982+0000","last_change":"2024-03-24T10:59:18.413036+0000","last_active":"2024-03-24T10:59:18.412982+0000","last_peered":"2024-03-24T10:59:18.412982+0000","last_clean":"2024-03-24T10:59:18.412982+0000","last_became_active":"2024-03-24T10:45:00.638006+0000","last_became_peered":"2024-03-24T10:45:00.638006+0000","last_unstale":"2024-03-24T10:59:18.412982+0000","last_undegraded":"2024-03-24T10:59:18.412982+0000","last_fullsized":"2024-03-24T10:59:18.412982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'10","last_scrub_stamp":"2024-03-24T10:59:18.412869+0000","last_deep_scrub":"24'10","last_deep_scrub_stamp":"2024-03-24T10:59:18.412869+0000","last_clean_scrub_stamp":"2024-03-24T10:59:18.412869+0000","objects_scrubbed":3,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:50:17.180111+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"24'5","reported_seq":40,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:15.054857+0000","last_change":"2024-03-24T10:59:15.054857+0000","last_active":"2024-03-24T10:59:15.054857+0000","last_peered":"2024-03-24T10:59:15.054857+0000","last_clean":"2024-03-24T10:59:15.054857+0000","last_became_active":"2024-03-24T10:45:00.633427+0000","last_became_peered":"2024-03-24T10:45:00.633427+0000","last_unstale":"2024-03-24T10:59:15.054857+0000","last_undegraded":"2024-03-24T10:59:15.054857+0000","last_fullsized":"2024-03-24T10:59:15.054857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'5","last_scrub_stamp":"2024-03-24T10:59:15.054726+0000","last_deep_scrub":"24'5","last_deep_scrub_stamp":"2024-03-24T10:59:15.054726+0000","last_clean_scrub_stamp":"2024-03-24T10:59:15.054726+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:51:51.292565+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:22.068368+0000","last_change":"2024-03-24T10:59:22.068368+0000","last_active":"2024-03-24T10:59:22.068368+0000","last_peered":"2024-03-24T10:59:22.068368+0000","last_clean":"2024-03-24T10:59:22.068368+0000","last_became_active":"2024-03-24T10:45:01.893617+0000","last_became_peered":"2024-03-24T10:45:01.893617+0000","last_unstale":"2024-03-24T10:59:22.068368+0000","last_undegraded":"2024-03-24T10:59:22.068368+0000","last_fullsized":"2024-03-24T10:59:22.068368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:22.068227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:22.068227+0000","last_clean_scrub_stamp":"2024-03-24T10:59:22.068227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:57:30.051315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"24'5","reported_seq":40,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:19.241460+0000","last_change":"2024-03-24T10:59:19.241460+0000","last_active":"2024-03-24T10:59:19.241460+0000","last_peered":"2024-03-24T10:59:19.241460+0000","last_clean":"2024-03-24T10:59:19.241460+0000","last_became_active":"2024-03-24T10:45:00.638089+0000","last_became_peered":"2024-03-24T10:45:00.638089+0000","last_unstale":"2024-03-24T10:59:19.241460+0000","last_undegraded":"2024-03-24T10:59:19.241460+0000","last_fullsized":"2024-03-24T10:59:19.241460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'5","last_scrub_stamp":"2024-03-24T10:59:19.241378+0000","last_deep_scrub":"24'5","last_deep_scrub_stamp":"2024-03-24T10:59:19.241378+0000","last_clean_scrub_stamp":"2024-03-24T10:59:19.241378+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:36.931954+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":38,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:23.394757+0000","last_change":"2024-03-24T10:59:23.394757+0000","last_active":"2024-03-24T10:59:23.394757+0000","last_peered":"2024-03-24T10:59:23.394757+0000","last_clean":"2024-03-24T10:59:23.394757+0000","last_became_active":"2024-03-24T10:45:39.901416+0000","last_became_peered":"2024-03-24T10:45:39.901416+0000","last_unstale":"2024-03-24T10:59:23.394757+0000","last_undegraded":"2024-03-24T10:59:23.394757+0000","last_fullsized":"2024-03-24T10:59:23.394757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:23.394671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:23.394671+0000","last_clean_scrub_stamp":"2024-03-24T10:59:23.394671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:07:18.075807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"24'7","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:20.429828+0000","last_change":"2024-03-24T10:59:20.429828+0000","last_active":"2024-03-24T10:59:20.429828+0000","last_peered":"2024-03-24T10:59:20.429828+0000","last_clean":"2024-03-24T10:59:20.429828+0000","last_became_active":"2024-03-24T10:45:00.639600+0000","last_became_peered":"2024-03-24T10:45:00.639600+0000","last_unstale":"2024-03-24T10:59:20.429828+0000","last_undegraded":"2024-03-24T10:59:20.429828+0000","last_fullsized":"2024-03-24T10:59:20.429828+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'7","last_scrub_stamp":"2024-03-24T10:59:20.429731+0000","last_deep_scrub":"24'7","last_deep_scrub_stamp":"2024-03-24T10:59:20.429731+0000","last_clean_scrub_stamp":"2024-03-24T10:59:20.429731+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:51:58.081084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:24.216382+0000","last_change":"2024-03-24T10:59:24.216382+0000","last_active":"2024-03-24T10:59:24.216382+0000","last_peered":"2024-03-24T10:59:24.216382+0000","last_clean":"2024-03-24T10:59:24.216382+0000","last_became_active":"2024-03-24T10:45:01.953767+0000","last_became_peered":"2024-03-24T10:45:01.953767+0000","last_unstale":"2024-03-24T10:59:24.216382+0000","last_undegraded":"2024-03-24T10:59:24.216382+0000","last_fullsized":"2024-03-24T10:59:24.216382+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:24.216221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:24.216221+0000","last_clean_scrub_stamp":"2024-03-24T10:59:24.216221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:03:49.864111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"24'6","reported_seq":41,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:15.315042+0000","last_change":"2024-03-24T10:59:15.315042+0000","last_active":"2024-03-24T10:59:15.315042+0000","last_peered":"2024-03-24T10:59:15.315042+0000","last_clean":"2024-03-24T10:59:15.315042+0000","last_became_active":"2024-03-24T10:45:00.636282+0000","last_became_peered":"2024-03-24T10:45:00.636282+0000","last_unstale":"2024-03-24T10:59:15.315042+0000","last_undegraded":"2024-03-24T10:59:15.315042+0000","last_fullsized":"2024-03-24T10:59:15.315042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'6","last_scrub_stamp":"2024-03-24T10:59:15.314886+0000","last_deep_scrub":"24'6","last_deep_scrub_stamp":"2024-03-24T10:59:15.314886+0000","last_clean_scrub_stamp":"2024-03-24T10:59:15.314886+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:51:52.966239+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:21.343045+0000","last_change":"2024-03-24T10:59:21.343045+0000","last_active":"2024-03-24T10:59:21.343045+0000","last_peered":"2024-03-24T10:59:21.343045+0000","last_clean":"2024-03-24T10:59:21.343045+0000","last_became_active":"2024-03-24T10:45:01.957676+0000","last_became_peered":"2024-03-24T10:45:01.957676+0000","last_unstale":"2024-03-24T10:59:21.343045+0000","last_undegraded":"2024-03-24T10:59:21.343045+0000","last_fullsized":"2024-03-24T10:59:21.343045+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:21.342928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:21.342928+0000","last_clean_scrub_stamp":"2024-03-24T10:59:21.342928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:30:31.163979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"24'9","reported_seq":45,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:17.418271+0000","last_change":"2024-03-24T10:59:17.418271+0000","last_active":"2024-03-24T10:59:17.418271+0000","last_peered":"2024-03-24T10:59:17.418271+0000","last_clean":"2024-03-24T10:59:17.418271+0000","last_became_active":"2024-03-24T10:45:00.634989+0000","last_became_peered":"2024-03-24T10:45:00.634989+0000","last_unstale":"2024-03-24T10:59:17.418271+0000","last_undegraded":"2024-03-24T10:59:17.418271+0000","last_fullsized":"2024-03-24T10:59:17.418271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'9","last_scrub_stamp":"2024-03-24T10:59:17.418184+0000","last_deep_scrub":"24'9","last_deep_scrub_stamp":"2024-03-24T10:59:17.418184+0000","last_clean_scrub_stamp":"2024-03-24T10:59:17.418184+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:30:44.269698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:15.963034+0000","last_change":"2024-03-24T10:59:15.963068+0000","last_active":"2024-03-24T10:59:15.963034+0000","last_peered":"2024-03-24T10:59:15.963034+0000","last_clean":"2024-03-24T10:59:15.963034+0000","last_became_active":"2024-03-24T10:45:01.853504+0000","last_became_peered":"2024-03-24T10:45:01.853504+0000","last_unstale":"2024-03-24T10:59:15.963034+0000","last_undegraded":"2024-03-24T10:59:15.963034+0000","last_fullsized":"2024-03-24T10:59:15.963034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:15.962934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:15.962934+0000","last_clean_scrub_stamp":"2024-03-24T10:59:15.962934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:54:10.350012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"24'16","reported_seq":51,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:16.425144+0000","last_change":"2024-03-24T10:59:16.425196+0000","last_active":"2024-03-24T10:59:16.425144+0000","last_peered":"2024-03-24T10:59:16.425144+0000","last_clean":"2024-03-24T10:59:16.425144+0000","last_became_active":"2024-03-24T10:45:00.632108+0000","last_became_peered":"2024-03-24T10:45:00.632108+0000","last_unstale":"2024-03-24T10:59:16.425144+0000","last_undegraded":"2024-03-24T10:59:16.425144+0000","last_fullsized":"2024-03-24T10:59:16.425144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'16","last_scrub_stamp":"2024-03-24T10:59:16.425042+0000","last_deep_scrub":"24'16","last_deep_scrub_stamp":"2024-03-24T10:59:16.425042+0000","last_clean_scrub_stamp":"2024-03-24T10:59:16.425042+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:34:33.139183+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:25.227063+0000","last_change":"2024-03-24T10:59:25.227063+0000","last_active":"2024-03-24T10:59:25.227063+0000","last_peered":"2024-03-24T10:59:25.227063+0000","last_clean":"2024-03-24T10:59:25.227063+0000","last_became_active":"2024-03-24T10:45:01.853279+0000","last_became_peered":"2024-03-24T10:45:01.853279+0000","last_unstale":"2024-03-24T10:59:25.227063+0000","last_undegraded":"2024-03-24T10:59:25.227063+0000","last_fullsized":"2024-03-24T10:59:25.227063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:25.226922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:25.226922+0000","last_clean_scrub_stamp":"2024-03-24T10:59:25.226922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:59:09.576144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"24'15","reported_seq":50,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:11.426620+0000","last_change":"2024-03-24T10:59:11.426671+0000","last_active":"2024-03-24T10:59:11.426620+0000","last_peered":"2024-03-24T10:59:11.426620+0000","last_clean":"2024-03-24T10:59:11.426620+0000","last_became_active":"2024-03-24T10:45:00.633893+0000","last_became_peered":"2024-03-24T10:45:00.633893+0000","last_unstale":"2024-03-24T10:59:11.426620+0000","last_undegraded":"2024-03-24T10:59:11.426620+0000","last_fullsized":"2024-03-24T10:59:11.426620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'15","last_scrub_stamp":"2024-03-24T10:59:11.426552+0000","last_deep_scrub":"24'15","last_deep_scrub_stamp":"2024-03-24T10:59:11.426552+0000","last_clean_scrub_stamp":"2024-03-24T10:59:11.426552+0000","objects_scrubbed":6,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:34:28.140694+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:13.389392+0000","last_change":"2024-03-24T10:59:13.389433+0000","last_active":"2024-03-24T10:59:13.389392+0000","last_peered":"2024-03-24T10:59:13.389392+0000","last_clean":"2024-03-24T10:59:13.389392+0000","last_became_active":"2024-03-24T10:45:01.893178+0000","last_became_peered":"2024-03-24T10:45:01.893178+0000","last_unstale":"2024-03-24T10:59:13.389392+0000","last_undegraded":"2024-03-24T10:59:13.389392+0000","last_fullsized":"2024-03-24T10:59:13.389392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:13.389283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:13.389283+0000","last_clean_scrub_stamp":"2024-03-24T10:59:13.389283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:50:12.156525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"24'9","reported_seq":44,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:17.333818+0000","last_change":"2024-03-24T10:59:17.333867+0000","last_active":"2024-03-24T10:59:17.333818+0000","last_peered":"2024-03-24T10:59:17.333818+0000","last_clean":"2024-03-24T10:59:17.333818+0000","last_became_active":"2024-03-24T10:45:00.637973+0000","last_became_peered":"2024-03-24T10:45:00.637973+0000","last_unstale":"2024-03-24T10:59:17.333818+0000","last_undegraded":"2024-03-24T10:59:17.333818+0000","last_fullsized":"2024-03-24T10:59:17.333818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'9","last_scrub_stamp":"2024-03-24T10:59:17.333723+0000","last_deep_scrub":"24'9","last_deep_scrub_stamp":"2024-03-24T10:59:17.333723+0000","last_clean_scrub_stamp":"2024-03-24T10:59:17.333723+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:59:17.516197+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:26.217857+0000","last_change":"2024-03-24T10:59:26.217877+0000","last_active":"2024-03-24T10:59:26.217857+0000","last_peered":"2024-03-24T10:59:26.217857+0000","last_clean":"2024-03-24T10:59:26.217857+0000","last_became_active":"2024-03-24T10:45:01.856482+0000","last_became_peered":"2024-03-24T10:45:01.856482+0000","last_unstale":"2024-03-24T10:59:26.217857+0000","last_undegraded":"2024-03-24T10:59:26.217857+0000","last_fullsized":"2024-03-24T10:59:26.217857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:26.217788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:26.217788+0000","last_clean_scrub_stamp":"2024-03-24T10:59:26.217788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:20:18.431111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"24'8","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:17.406290+0000","last_change":"2024-03-24T10:59:17.406336+0000","last_active":"2024-03-24T10:59:17.406290+0000","last_peered":"2024-03-24T10:59:17.406290+0000","last_clean":"2024-03-24T10:59:17.406290+0000","last_became_active":"2024-03-24T10:45:00.636206+0000","last_became_peered":"2024-03-24T10:45:00.636206+0000","last_unstale":"2024-03-24T10:59:17.406290+0000","last_undegraded":"2024-03-24T10:59:17.406290+0000","last_fullsized":"2024-03-24T10:59:17.406290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'8","last_scrub_stamp":"2024-03-24T10:59:17.406191+0000","last_deep_scrub":"24'8","last_deep_scrub_stamp":"2024-03-24T10:59:17.406191+0000","last_clean_scrub_stamp":"2024-03-24T10:59:17.406191+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:14:31.055880+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:23.375534+0000","last_change":"2024-03-24T10:59:23.375555+0000","last_active":"2024-03-24T10:59:23.375534+0000","last_peered":"2024-03-24T10:59:23.375534+0000","last_clean":"2024-03-24T10:59:23.375534+0000","last_became_active":"2024-03-24T10:45:01.896736+0000","last_became_peered":"2024-03-24T10:45:01.896736+0000","last_unstale":"2024-03-24T10:59:23.375534+0000","last_undegraded":"2024-03-24T10:59:23.375534+0000","last_fullsized":"2024-03-24T10:59:23.375534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:23.375468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:23.375468+0000","last_clean_scrub_stamp":"2024-03-24T10:59:23.375468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:54:00.802032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"24'25","reported_seq":60,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:13.034065+0000","last_change":"2024-03-24T10:59:13.034134+0000","last_active":"2024-03-24T10:59:13.034065+0000","last_peered":"2024-03-24T10:59:13.034065+0000","last_clean":"2024-03-24T10:59:13.034065+0000","last_became_active":"2024-03-24T10:45:00.563812+0000","last_became_peered":"2024-03-24T10:45:00.563812+0000","last_unstale":"2024-03-24T10:59:13.034065+0000","last_undegraded":"2024-03-24T10:59:13.034065+0000","last_fullsized":"2024-03-24T10:59:13.034065+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'25","last_scrub_stamp":"2024-03-24T10:59:13.033967+0000","last_deep_scrub":"24'25","last_deep_scrub_stamp":"2024-03-24T10:59:13.033967+0000","last_clean_scrub_stamp":"2024-03-24T10:59:13.033967+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:57:34.971277+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:17.160668+0000","last_change":"2024-03-24T10:59:17.160709+0000","last_active":"2024-03-24T10:59:17.160668+0000","last_peered":"2024-03-24T10:59:17.160668+0000","last_clean":"2024-03-24T10:59:17.160668+0000","last_became_active":"2024-03-24T10:45:01.854179+0000","last_became_peered":"2024-03-24T10:45:01.854179+0000","last_unstale":"2024-03-24T10:59:17.160668+0000","last_undegraded":"2024-03-24T10:59:17.160668+0000","last_fullsized":"2024-03-24T10:59:17.160668+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:17.160539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:17.160539+0000","last_clean_scrub_stamp":"2024-03-24T10:59:17.160539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:53:54.587103+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"24'2","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:14.985913+0000","last_change":"2024-03-24T10:59:14.985938+0000","last_active":"2024-03-24T10:59:14.985913+0000","last_peered":"2024-03-24T10:59:14.985913+0000","last_clean":"2024-03-24T10:59:14.985913+0000","last_became_active":"2024-03-24T10:45:00.633355+0000","last_became_peered":"2024-03-24T10:45:00.633355+0000","last_unstale":"2024-03-24T10:59:14.985913+0000","last_undegraded":"2024-03-24T10:59:14.985913+0000","last_fullsized":"2024-03-24T10:59:14.985913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'2","last_scrub_stamp":"2024-03-24T10:59:14.985852+0000","last_deep_scrub":"24'2","last_deep_scrub_stamp":"2024-03-24T10:59:14.985852+0000","last_clean_scrub_stamp":"2024-03-24T10:59:14.985852+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:57:36.923162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:24.444364+0000","last_change":"2024-03-24T10:59:24.444408+0000","last_active":"2024-03-24T10:59:24.444364+0000","last_peered":"2024-03-24T10:59:24.444364+0000","last_clean":"2024-03-24T10:59:24.444364+0000","last_became_active":"2024-03-24T10:45:40.518035+0000","last_became_peered":"2024-03-24T10:45:40.518035+0000","last_unstale":"2024-03-24T10:59:24.444364+0000","last_undegraded":"2024-03-24T10:59:24.444364+0000","last_fullsized":"2024-03-24T10:59:24.444364+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:24.444287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:24.444287+0000","last_clean_scrub_stamp":"2024-03-24T10:59:24.444287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:03:50.092177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"24'7","reported_seq":42,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:14.067847+0000","last_change":"2024-03-24T10:59:14.067880+0000","last_active":"2024-03-24T10:59:14.067847+0000","last_peered":"2024-03-24T10:59:14.067847+0000","last_clean":"2024-03-24T10:59:14.067847+0000","last_became_active":"2024-03-24T10:45:00.626195+0000","last_became_peered":"2024-03-24T10:45:00.626195+0000","last_unstale":"2024-03-24T10:59:14.067847+0000","last_undegraded":"2024-03-24T10:59:14.067847+0000","last_fullsized":"2024-03-24T10:59:14.067847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'7","last_scrub_stamp":"2024-03-24T10:59:14.067785+0000","last_deep_scrub":"24'7","last_deep_scrub_stamp":"2024-03-24T10:59:14.067785+0000","last_clean_scrub_stamp":"2024-03-24T10:59:14.067785+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:05:15.517970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:27.181408+0000","last_change":"2024-03-24T10:59:27.181451+0000","last_active":"2024-03-24T10:59:27.181408+0000","last_peered":"2024-03-24T10:59:27.181408+0000","last_clean":"2024-03-24T10:59:27.181408+0000","last_became_active":"2024-03-24T10:45:01.856531+0000","last_became_peered":"2024-03-24T10:45:01.856531+0000","last_unstale":"2024-03-24T10:59:27.181408+0000","last_undegraded":"2024-03-24T10:59:27.181408+0000","last_fullsized":"2024-03-24T10:59:27.181408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:27.181279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:27.181279+0000","last_clean_scrub_stamp":"2024-03-24T10:59:27.181279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:19:34.894210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"24'4","reported_seq":39,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:14.469596+0000","last_change":"2024-03-24T10:59:14.469638+0000","last_active":"2024-03-24T10:59:14.469596+0000","last_peered":"2024-03-24T10:59:14.469596+0000","last_clean":"2024-03-24T10:59:14.469596+0000","last_became_active":"2024-03-24T10:45:00.628158+0000","last_became_peered":"2024-03-24T10:45:00.628158+0000","last_unstale":"2024-03-24T10:59:14.469596+0000","last_undegraded":"2024-03-24T10:59:14.469596+0000","last_fullsized":"2024-03-24T10:59:14.469596+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'4","last_scrub_stamp":"2024-03-24T10:59:14.469521+0000","last_deep_scrub":"24'4","last_deep_scrub_stamp":"2024-03-24T10:59:14.469521+0000","last_clean_scrub_stamp":"2024-03-24T10:59:14.469521+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:51:50.707360+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:19.039852+0000","last_change":"2024-03-24T10:59:19.039898+0000","last_active":"2024-03-24T10:59:19.039852+0000","last_peered":"2024-03-24T10:59:19.039852+0000","last_clean":"2024-03-24T10:59:19.039852+0000","last_became_active":"2024-03-24T10:45:39.901119+0000","last_became_peered":"2024-03-24T10:45:39.901119+0000","last_unstale":"2024-03-24T10:59:19.039852+0000","last_undegraded":"2024-03-24T10:59:19.039852+0000","last_fullsized":"2024-03-24T10:59:19.039852+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:19.039741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:19.039741+0000","last_clean_scrub_stamp":"2024-03-24T10:59:19.039741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:50:17.806983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"24'8","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:16.055714+0000","last_change":"2024-03-24T10:59:16.055761+0000","last_active":"2024-03-24T10:59:16.055714+0000","last_peered":"2024-03-24T10:59:16.055714+0000","last_clean":"2024-03-24T10:59:16.055714+0000","last_became_active":"2024-03-24T10:45:00.638785+0000","last_became_peered":"2024-03-24T10:45:00.638785+0000","last_unstale":"2024-03-24T10:59:16.055714+0000","last_undegraded":"2024-03-24T10:59:16.055714+0000","last_fullsized":"2024-03-24T10:59:16.055714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'8","last_scrub_stamp":"2024-03-24T10:59:16.055597+0000","last_deep_scrub":"24'8","last_deep_scrub_stamp":"2024-03-24T10:59:16.055597+0000","last_clean_scrub_stamp":"2024-03-24T10:59:16.055597+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:36:00.915701+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:32.149683+0000","last_change":"2024-03-24T10:59:32.149720+0000","last_active":"2024-03-24T10:59:32.149683+0000","last_peered":"2024-03-24T10:59:32.149683+0000","last_clean":"2024-03-24T10:59:32.149683+0000","last_became_active":"2024-03-24T10:45:01.953402+0000","last_became_peered":"2024-03-24T10:45:01.953402+0000","last_unstale":"2024-03-24T10:59:32.149683+0000","last_undegraded":"2024-03-24T10:59:32.149683+0000","last_fullsized":"2024-03-24T10:59:32.149683+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:32.149574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:32.149574+0000","last_clean_scrub_stamp":"2024-03-24T10:59:32.149574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:56.785648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"24'119","reported_seq":194,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:18.188112+0000","last_change":"2024-03-24T10:59:18.188162+0000","last_active":"2024-03-24T10:59:18.188112+0000","last_peered":"2024-03-24T10:59:18.188112+0000","last_clean":"2024-03-24T10:59:18.188112+0000","last_became_active":"2024-03-24T10:44:48.239986+0000","last_became_peered":"2024-03-24T10:44:48.239986+0000","last_unstale":"2024-03-24T10:59:18.188112+0000","last_undegraded":"2024-03-24T10:59:18.188112+0000","last_fullsized":"2024-03-24T10:59:18.188112+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'119","last_scrub_stamp":"2024-03-24T10:59:18.188000+0000","last_deep_scrub":"24'119","last_deep_scrub_stamp":"2024-03-24T10:59:18.188000+0000","last_clean_scrub_stamp":"2024-03-24T10:59:18.188000+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:02:31.784010+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"24'11","reported_seq":48,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:16.286865+0000","last_change":"2024-03-24T10:59:16.286865+0000","last_active":"2024-03-24T10:59:16.286865+0000","last_peered":"2024-03-24T10:59:16.286865+0000","last_clean":"2024-03-24T10:59:16.286865+0000","last_became_active":"2024-03-24T10:45:00.636734+0000","last_became_peered":"2024-03-24T10:45:00.636734+0000","last_unstale":"2024-03-24T10:59:16.286865+0000","last_undegraded":"2024-03-24T10:59:16.286865+0000","last_fullsized":"2024-03-24T10:59:16.286865+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'11","last_scrub_stamp":"2024-03-24T10:59:16.286735+0000","last_deep_scrub":"24'11","last_deep_scrub_stamp":"2024-03-24T10:59:16.286735+0000","last_clean_scrub_stamp":"2024-03-24T10:59:16.286735+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:24:04.803429+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:27.116934+0000","last_change":"2024-03-24T10:59:27.116934+0000","last_active":"2024-03-24T10:59:27.116934+0000","last_peered":"2024-03-24T10:59:27.116934+0000","last_clean":"2024-03-24T10:59:27.116934+0000","last_became_active":"2024-03-24T10:45:01.958268+0000","last_became_peered":"2024-03-24T10:45:01.958268+0000","last_unstale":"2024-03-24T10:59:27.116934+0000","last_undegraded":"2024-03-24T10:59:27.116934+0000","last_fullsized":"2024-03-24T10:59:27.116934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:27.116787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:27.116787+0000","last_clean_scrub_stamp":"2024-03-24T10:59:27.116787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:18:04.515974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"24'6","reported_seq":42,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:19.235596+0000","last_change":"2024-03-24T10:59:19.235635+0000","last_active":"2024-03-24T10:59:19.235596+0000","last_peered":"2024-03-24T10:59:19.235596+0000","last_clean":"2024-03-24T10:59:19.235596+0000","last_became_active":"2024-03-24T10:45:00.597640+0000","last_became_peered":"2024-03-24T10:45:00.597640+0000","last_unstale":"2024-03-24T10:59:19.235596+0000","last_undegraded":"2024-03-24T10:59:19.235596+0000","last_fullsized":"2024-03-24T10:59:19.235596+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'6","last_scrub_stamp":"2024-03-24T10:59:19.235522+0000","last_deep_scrub":"24'6","last_deep_scrub_stamp":"2024-03-24T10:59:19.235522+0000","last_clean_scrub_stamp":"2024-03-24T10:59:19.235522+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:57:27.218609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:17.317269+0000","last_change":"2024-03-24T10:59:17.317288+0000","last_active":"2024-03-24T10:59:17.317269+0000","last_peered":"2024-03-24T10:59:17.317269+0000","last_clean":"2024-03-24T10:59:17.317269+0000","last_became_active":"2024-03-24T10:45:01.897250+0000","last_became_peered":"2024-03-24T10:45:01.897250+0000","last_unstale":"2024-03-24T10:59:17.317269+0000","last_undegraded":"2024-03-24T10:59:17.317269+0000","last_fullsized":"2024-03-24T10:59:17.317269+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:17.317204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:17.317204+0000","last_clean_scrub_stamp":"2024-03-24T10:59:17.317204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:46:49.275799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"24'9","reported_seq":45,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:15.059452+0000","last_change":"2024-03-24T10:59:15.059452+0000","last_active":"2024-03-24T10:59:15.059452+0000","last_peered":"2024-03-24T10:59:15.059452+0000","last_clean":"2024-03-24T10:59:15.059452+0000","last_became_active":"2024-03-24T10:45:00.626881+0000","last_became_peered":"2024-03-24T10:45:00.626881+0000","last_unstale":"2024-03-24T10:59:15.059452+0000","last_undegraded":"2024-03-24T10:59:15.059452+0000","last_fullsized":"2024-03-24T10:59:15.059452+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'9","last_scrub_stamp":"2024-03-24T10:59:15.059339+0000","last_deep_scrub":"24'9","last_deep_scrub_stamp":"2024-03-24T10:59:15.059339+0000","last_clean_scrub_stamp":"2024-03-24T10:59:15.059339+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:34:26.175028+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:20.441955+0000","last_change":"2024-03-24T10:59:20.441991+0000","last_active":"2024-03-24T10:59:20.441955+0000","last_peered":"2024-03-24T10:59:20.441955+0000","last_clean":"2024-03-24T10:59:20.441955+0000","last_became_active":"2024-03-24T10:45:01.929998+0000","last_became_peered":"2024-03-24T10:45:01.929998+0000","last_unstale":"2024-03-24T10:59:20.441955+0000","last_undegraded":"2024-03-24T10:59:20.441955+0000","last_fullsized":"2024-03-24T10:59:20.441955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:20.441859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:20.441859+0000","last_clean_scrub_stamp":"2024-03-24T10:59:20.441859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:02:34.037869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"24'7","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:18.283740+0000","last_change":"2024-03-24T10:59:18.283740+0000","last_active":"2024-03-24T10:59:18.283740+0000","last_peered":"2024-03-24T10:59:18.283740+0000","last_clean":"2024-03-24T10:59:18.283740+0000","last_became_active":"2024-03-24T10:45:00.638041+0000","last_became_peered":"2024-03-24T10:45:00.638041+0000","last_unstale":"2024-03-24T10:59:18.283740+0000","last_undegraded":"2024-03-24T10:59:18.283740+0000","last_fullsized":"2024-03-24T10:59:18.283740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'7","last_scrub_stamp":"2024-03-24T10:59:18.283593+0000","last_deep_scrub":"24'7","last_deep_scrub_stamp":"2024-03-24T10:59:18.283593+0000","last_clean_scrub_stamp":"2024-03-24T10:59:18.283593+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:13:13.445145+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:19.073888+0000","last_change":"2024-03-24T10:59:19.073908+0000","last_active":"2024-03-24T10:59:19.073888+0000","last_peered":"2024-03-24T10:59:19.073888+0000","last_clean":"2024-03-24T10:59:19.073888+0000","last_became_active":"2024-03-24T10:45:01.897159+0000","last_became_peered":"2024-03-24T10:45:01.897159+0000","last_unstale":"2024-03-24T10:59:19.073888+0000","last_undegraded":"2024-03-24T10:59:19.073888+0000","last_fullsized":"2024-03-24T10:59:19.073888+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:19.073819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:19.073819+0000","last_clean_scrub_stamp":"2024-03-24T10:59:19.073819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:08:01.855838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":37,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:15.494816+0000","last_change":"2024-03-24T10:59:15.494816+0000","last_active":"2024-03-24T10:59:15.494816+0000","last_peered":"2024-03-24T10:59:15.494816+0000","last_clean":"2024-03-24T10:59:15.494816+0000","last_became_active":"2024-03-24T10:45:00.628971+0000","last_became_peered":"2024-03-24T10:45:00.628971+0000","last_unstale":"2024-03-24T10:59:15.494816+0000","last_undegraded":"2024-03-24T10:59:15.494816+0000","last_fullsized":"2024-03-24T10:59:15.494816+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'1","last_scrub_stamp":"2024-03-24T10:59:15.494650+0000","last_deep_scrub":"21'1","last_deep_scrub_stamp":"2024-03-24T10:59:15.494650+0000","last_clean_scrub_stamp":"2024-03-24T10:59:15.494650+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:07:32.776779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:23.178699+0000","last_change":"2024-03-24T10:59:23.178699+0000","last_active":"2024-03-24T10:59:23.178699+0000","last_peered":"2024-03-24T10:59:23.178699+0000","last_clean":"2024-03-24T10:59:23.178699+0000","last_became_active":"2024-03-24T10:45:01.952192+0000","last_became_peered":"2024-03-24T10:45:01.952192+0000","last_unstale":"2024-03-24T10:59:23.178699+0000","last_undegraded":"2024-03-24T10:59:23.178699+0000","last_fullsized":"2024-03-24T10:59:23.178699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:23.178553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:23.178553+0000","last_clean_scrub_stamp":"2024-03-24T10:59:23.178553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:07:17.859689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"24'7","reported_seq":44,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:22.242623+0000","last_change":"2024-03-24T10:59:22.242623+0000","last_active":"2024-03-24T10:59:22.242623+0000","last_peered":"2024-03-24T10:59:22.242623+0000","last_clean":"2024-03-24T10:59:22.242623+0000","last_became_active":"2024-03-24T10:45:00.631670+0000","last_became_peered":"2024-03-24T10:45:00.631670+0000","last_unstale":"2024-03-24T10:59:22.242623+0000","last_undegraded":"2024-03-24T10:59:22.242623+0000","last_fullsized":"2024-03-24T10:59:22.242623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'7","last_scrub_stamp":"2024-03-24T10:59:22.242536+0000","last_deep_scrub":"24'7","last_deep_scrub_stamp":"2024-03-24T10:59:22.242536+0000","last_clean_scrub_stamp":"2024-03-24T10:59:22.242536+0000","objects_scrubbed":3,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:03:47.890426+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:21.403220+0000","last_change":"2024-03-24T10:59:21.403220+0000","last_active":"2024-03-24T10:59:21.403220+0000","last_peered":"2024-03-24T10:59:21.403220+0000","last_clean":"2024-03-24T10:59:21.403220+0000","last_became_active":"2024-03-24T10:45:01.930220+0000","last_became_peered":"2024-03-24T10:45:01.930220+0000","last_unstale":"2024-03-24T10:59:21.403220+0000","last_undegraded":"2024-03-24T10:59:21.403220+0000","last_fullsized":"2024-03-24T10:59:21.403220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:21.403135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:21.403135+0000","last_clean_scrub_stamp":"2024-03-24T10:59:21.403135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:57:29.386223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"24'11","reported_seq":46,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:16.461696+0000","last_change":"2024-03-24T10:59:16.461752+0000","last_active":"2024-03-24T10:59:16.461696+0000","last_peered":"2024-03-24T10:59:16.461696+0000","last_clean":"2024-03-24T10:59:16.461696+0000","last_became_active":"2024-03-24T10:45:00.633329+0000","last_became_peered":"2024-03-24T10:45:00.633329+0000","last_unstale":"2024-03-24T10:59:16.461696+0000","last_undegraded":"2024-03-24T10:59:16.461696+0000","last_fullsized":"2024-03-24T10:59:16.461696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'11","last_scrub_stamp":"2024-03-24T10:59:16.461589+0000","last_deep_scrub":"24'11","last_deep_scrub_stamp":"2024-03-24T10:59:16.461589+0000","last_clean_scrub_stamp":"2024-03-24T10:59:16.461589+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T16:51:54.112942+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:20.010442+0000","last_change":"2024-03-24T10:59:20.010462+0000","last_active":"2024-03-24T10:59:20.010442+0000","last_peered":"2024-03-24T10:59:20.010442+0000","last_clean":"2024-03-24T10:59:20.010442+0000","last_became_active":"2024-03-24T10:45:01.853257+0000","last_became_peered":"2024-03-24T10:45:01.853257+0000","last_unstale":"2024-03-24T10:59:20.010442+0000","last_undegraded":"2024-03-24T10:59:20.010442+0000","last_fullsized":"2024-03-24T10:59:20.010442+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:20.010369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:20.010369+0000","last_clean_scrub_stamp":"2024-03-24T10:59:20.010369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:13:15.171921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"24'11","reported_seq":46,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:19.397249+0000","last_change":"2024-03-24T10:59:19.397297+0000","last_active":"2024-03-24T10:59:19.397249+0000","last_peered":"2024-03-24T10:59:19.397249+0000","last_clean":"2024-03-24T10:59:19.397249+0000","last_became_active":"2024-03-24T10:45:00.638999+0000","last_became_peered":"2024-03-24T10:45:00.638999+0000","last_unstale":"2024-03-24T10:59:19.397249+0000","last_undegraded":"2024-03-24T10:59:19.397249+0000","last_fullsized":"2024-03-24T10:59:19.397249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'11","last_scrub_stamp":"2024-03-24T10:59:19.397144+0000","last_deep_scrub":"24'11","last_deep_scrub_stamp":"2024-03-24T10:59:19.397144+0000","last_clean_scrub_stamp":"2024-03-24T10:59:19.397144+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:08:56.559947+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:19.312599+0000","last_change":"2024-03-24T10:59:19.312619+0000","last_active":"2024-03-24T10:59:19.312599+0000","last_peered":"2024-03-24T10:59:19.312599+0000","last_clean":"2024-03-24T10:59:19.312599+0000","last_became_active":"2024-03-24T10:45:01.928610+0000","last_became_peered":"2024-03-24T10:45:01.928610+0000","last_unstale":"2024-03-24T10:59:19.312599+0000","last_undegraded":"2024-03-24T10:59:19.312599+0000","last_fullsized":"2024-03-24T10:59:19.312599+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:19.312530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:19.312530+0000","last_clean_scrub_stamp":"2024-03-24T10:59:19.312530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:30:37.003106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"24'3","reported_seq":38,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:21.267373+0000","last_change":"2024-03-24T10:59:21.267416+0000","last_active":"2024-03-24T10:59:21.267373+0000","last_peered":"2024-03-24T10:59:21.267373+0000","last_clean":"2024-03-24T10:59:21.267373+0000","last_became_active":"2024-03-24T10:45:00.631078+0000","last_became_peered":"2024-03-24T10:45:00.631078+0000","last_unstale":"2024-03-24T10:59:21.267373+0000","last_undegraded":"2024-03-24T10:59:21.267373+0000","last_fullsized":"2024-03-24T10:59:21.267373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'3","last_scrub_stamp":"2024-03-24T10:59:21.267263+0000","last_deep_scrub":"24'3","last_deep_scrub_stamp":"2024-03-24T10:59:21.267263+0000","last_clean_scrub_stamp":"2024-03-24T10:59:21.267263+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T14:07:15.948399+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:18.038365+0000","last_change":"2024-03-24T10:59:18.038406+0000","last_active":"2024-03-24T10:59:18.038365+0000","last_peered":"2024-03-24T10:59:18.038365+0000","last_clean":"2024-03-24T10:59:18.038365+0000","last_became_active":"2024-03-24T10:45:01.897063+0000","last_became_peered":"2024-03-24T10:45:01.897063+0000","last_unstale":"2024-03-24T10:59:18.038365+0000","last_undegraded":"2024-03-24T10:59:18.038365+0000","last_fullsized":"2024-03-24T10:59:18.038365+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:18.038233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:18.038233+0000","last_clean_scrub_stamp":"2024-03-24T10:59:18.038233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:36:11.531814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"24'11","reported_seq":47,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:12.474191+0000","last_change":"2024-03-24T10:59:12.474217+0000","last_active":"2024-03-24T10:59:12.474191+0000","last_peered":"2024-03-24T10:59:12.474191+0000","last_clean":"2024-03-24T10:59:12.474191+0000","last_became_active":"2024-03-24T10:45:00.597671+0000","last_became_peered":"2024-03-24T10:45:00.597671+0000","last_unstale":"2024-03-24T10:59:12.474191+0000","last_undegraded":"2024-03-24T10:59:12.474191+0000","last_fullsized":"2024-03-24T10:59:12.474191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'11","last_scrub_stamp":"2024-03-24T10:59:12.474135+0000","last_deep_scrub":"24'11","last_deep_scrub_stamp":"2024-03-24T10:59:12.474135+0000","last_clean_scrub_stamp":"2024-03-24T10:59:12.474135+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:29:36.067679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:25.144688+0000","last_change":"2024-03-24T10:59:25.144706+0000","last_active":"2024-03-24T10:59:25.144688+0000","last_peered":"2024-03-24T10:59:25.144688+0000","last_clean":"2024-03-24T10:59:25.144688+0000","last_became_active":"2024-03-24T10:45:01.927538+0000","last_became_peered":"2024-03-24T10:45:01.927538+0000","last_unstale":"2024-03-24T10:59:25.144688+0000","last_undegraded":"2024-03-24T10:59:25.144688+0000","last_fullsized":"2024-03-24T10:59:25.144688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:25.144625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:25.144625+0000","last_clean_scrub_stamp":"2024-03-24T10:59:25.144625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:29:29.850930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"24'5","reported_seq":41,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:13.474814+0000","last_change":"2024-03-24T10:59:13.474838+0000","last_active":"2024-03-24T10:59:13.474814+0000","last_peered":"2024-03-24T10:59:13.474814+0000","last_clean":"2024-03-24T10:59:13.474814+0000","last_became_active":"2024-03-24T10:45:00.626557+0000","last_became_peered":"2024-03-24T10:45:00.626557+0000","last_unstale":"2024-03-24T10:59:13.474814+0000","last_undegraded":"2024-03-24T10:59:13.474814+0000","last_fullsized":"2024-03-24T10:59:13.474814+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'5","last_scrub_stamp":"2024-03-24T10:59:13.474755+0000","last_deep_scrub":"24'5","last_deep_scrub_stamp":"2024-03-24T10:59:13.474755+0000","last_clean_scrub_stamp":"2024-03-24T10:59:13.474755+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:34:39.478664+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:21.044022+0000","last_change":"2024-03-24T10:59:21.044063+0000","last_active":"2024-03-24T10:59:21.044022+0000","last_peered":"2024-03-24T10:59:21.044022+0000","last_clean":"2024-03-24T10:59:21.044022+0000","last_became_active":"2024-03-24T10:45:01.892934+0000","last_became_peered":"2024-03-24T10:45:01.892934+0000","last_unstale":"2024-03-24T10:59:21.044022+0000","last_undegraded":"2024-03-24T10:59:21.044022+0000","last_fullsized":"2024-03-24T10:59:21.044022+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:21.043920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:21.043920+0000","last_clean_scrub_stamp":"2024-03-24T10:59:21.043920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:02:34.639930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"24'6","reported_seq":41,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:20.242989+0000","last_change":"2024-03-24T10:59:20.243014+0000","last_active":"2024-03-24T10:59:20.242989+0000","last_peered":"2024-03-24T10:59:20.242989+0000","last_clean":"2024-03-24T10:59:20.242989+0000","last_became_active":"2024-03-24T10:45:00.629850+0000","last_became_peered":"2024-03-24T10:45:00.629850+0000","last_unstale":"2024-03-24T10:59:20.242989+0000","last_undegraded":"2024-03-24T10:59:20.242989+0000","last_fullsized":"2024-03-24T10:59:20.242989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'6","last_scrub_stamp":"2024-03-24T10:59:20.242932+0000","last_deep_scrub":"24'6","last_deep_scrub_stamp":"2024-03-24T10:59:20.242932+0000","last_clean_scrub_stamp":"2024-03-24T10:59:20.242932+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T21:32:39.341729+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:24.375291+0000","last_change":"2024-03-24T10:59:24.375329+0000","last_active":"2024-03-24T10:59:24.375291+0000","last_peered":"2024-03-24T10:59:24.375291+0000","last_clean":"2024-03-24T10:59:24.375291+0000","last_became_active":"2024-03-24T10:45:01.897080+0000","last_became_peered":"2024-03-24T10:45:01.897080+0000","last_unstale":"2024-03-24T10:59:24.375291+0000","last_undegraded":"2024-03-24T10:59:24.375291+0000","last_fullsized":"2024-03-24T10:59:24.375291+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:24.375186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:24.375186+0000","last_clean_scrub_stamp":"2024-03-24T10:59:24.375186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T15:13:50.360002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:26.194675+0000","last_change":"2024-03-24T10:59:26.194709+0000","last_active":"2024-03-24T10:59:26.194675+0000","last_peered":"2024-03-24T10:59:26.194675+0000","last_clean":"2024-03-24T10:59:26.194675+0000","last_became_active":"2024-03-24T10:45:01.955766+0000","last_became_peered":"2024-03-24T10:45:01.955766+0000","last_unstale":"2024-03-24T10:59:26.194675+0000","last_undegraded":"2024-03-24T10:59:26.194675+0000","last_fullsized":"2024-03-24T10:59:26.194675+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:26.194576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:26.194576+0000","last_clean_scrub_stamp":"2024-03-24T10:59:26.194576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T13:18:03.593763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"24'8","reported_seq":42,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:24.257702+0000","last_change":"2024-03-24T10:59:24.257727+0000","last_active":"2024-03-24T10:59:24.257702+0000","last_peered":"2024-03-24T10:59:24.257702+0000","last_clean":"2024-03-24T10:59:24.257702+0000","last_became_active":"2024-03-24T10:45:00.639653+0000","last_became_peered":"2024-03-24T10:45:00.639653+0000","last_unstale":"2024-03-24T10:59:24.257702+0000","last_undegraded":"2024-03-24T10:59:24.257702+0000","last_fullsized":"2024-03-24T10:59:24.257702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'8","last_scrub_stamp":"2024-03-24T10:59:24.257642+0000","last_deep_scrub":"24'8","last_deep_scrub_stamp":"2024-03-24T10:59:24.257642+0000","last_clean_scrub_stamp":"2024-03-24T10:59:24.257642+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:50:51.324734+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:20.224535+0000","last_change":"2024-03-24T10:59:20.224568+0000","last_active":"2024-03-24T10:59:20.224535+0000","last_peered":"2024-03-24T10:59:20.224535+0000","last_clean":"2024-03-24T10:59:20.224535+0000","last_became_active":"2024-03-24T10:45:01.952074+0000","last_became_peered":"2024-03-24T10:45:01.952074+0000","last_unstale":"2024-03-24T10:59:20.224535+0000","last_undegraded":"2024-03-24T10:59:20.224535+0000","last_fullsized":"2024-03-24T10:59:20.224535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:20.224432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:20.224432+0000","last_clean_scrub_stamp":"2024-03-24T10:59:20.224432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T18:12:08.146558+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"24'19","reported_seq":53,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:23.216258+0000","last_change":"2024-03-24T10:59:23.216311+0000","last_active":"2024-03-24T10:59:23.216258+0000","last_peered":"2024-03-24T10:59:23.216258+0000","last_clean":"2024-03-24T10:59:23.216258+0000","last_became_active":"2024-03-24T10:45:00.633494+0000","last_became_peered":"2024-03-24T10:45:00.633494+0000","last_unstale":"2024-03-24T10:59:23.216258+0000","last_undegraded":"2024-03-24T10:59:23.216258+0000","last_fullsized":"2024-03-24T10:59:23.216258+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'19","last_scrub_stamp":"2024-03-24T10:59:23.216184+0000","last_deep_scrub":"24'19","last_deep_scrub_stamp":"2024-03-24T10:59:23.216184+0000","last_clean_scrub_stamp":"2024-03-24T10:59:23.216184+0000","objects_scrubbed":7,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:19:02.513763+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:22.348198+0000","last_change":"2024-03-24T10:59:22.348217+0000","last_active":"2024-03-24T10:59:22.348198+0000","last_peered":"2024-03-24T10:59:22.348198+0000","last_clean":"2024-03-24T10:59:22.348198+0000","last_became_active":"2024-03-24T10:45:01.957676+0000","last_became_peered":"2024-03-24T10:45:01.957676+0000","last_unstale":"2024-03-24T10:59:22.348198+0000","last_undegraded":"2024-03-24T10:59:22.348198+0000","last_fullsized":"2024-03-24T10:59:22.348198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:22.348136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:22.348136+0000","last_clean_scrub_stamp":"2024-03-24T10:59:22.348136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T20:55:05.214028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":38,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:14.078489+0000","last_change":"2024-03-24T10:59:14.078526+0000","last_active":"2024-03-24T10:59:14.078489+0000","last_peered":"2024-03-24T10:59:14.078489+0000","last_clean":"2024-03-24T10:59:14.078489+0000","last_became_active":"2024-03-24T10:45:00.628712+0000","last_became_peered":"2024-03-24T10:45:00.628712+0000","last_unstale":"2024-03-24T10:59:14.078489+0000","last_undegraded":"2024-03-24T10:59:14.078489+0000","last_fullsized":"2024-03-24T10:59:14.078489+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'2","last_scrub_stamp":"2024-03-24T10:59:14.078396+0000","last_deep_scrub":"21'2","last_deep_scrub_stamp":"2024-03-24T10:59:14.078396+0000","last_clean_scrub_stamp":"2024-03-24T10:59:14.078396+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T17:34:40.082305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":34,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:23.088250+0000","last_change":"2024-03-24T10:59:23.088268+0000","last_active":"2024-03-24T10:59:23.088250+0000","last_peered":"2024-03-24T10:59:23.088250+0000","last_clean":"2024-03-24T10:59:23.088250+0000","last_became_active":"2024-03-24T10:45:01.894362+0000","last_became_peered":"2024-03-24T10:45:01.894362+0000","last_unstale":"2024-03-24T10:59:23.088250+0000","last_undegraded":"2024-03-24T10:59:23.088250+0000","last_fullsized":"2024-03-24T10:59:23.088250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T10:59:23.088183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T10:59:23.088183+0000","last_clean_scrub_stamp":"2024-03-24T10:59:23.088183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T22:30:32.909234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"24'19","reported_seq":54,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-03-24T10:59:12.424975+0000","last_change":"2024-03-24T10:59:12.425030+0000","last_active":"2024-03-24T10:59:12.424975+0000","last_peered":"2024-03-24T10:59:12.424975+0000","last_clean":"2024-03-24T10:59:12.424975+0000","last_became_active":"2024-03-24T10:45:00.638871+0000","last_became_peered":"2024-03-24T10:45:00.638871+0000","last_unstale":"2024-03-24T10:59:12.424975+0000","last_undegraded":"2024-03-24T10:59:12.424975+0000","last_fullsized":"2024-03-24T10:59:12.424975+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"24'19","last_scrub_stamp":"2024-03-24T10:59:12.424858+0000","last_deep_scrub":"24'19","last_deep_scrub_stamp":"2024-03-24T10:59:12.424858+0000","last_clean_scrub_stamp":"2024-03-24T10:59:12.424858+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T19:14:26.074546+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":204,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":281,"ondisk_log_size":281,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":65430,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542327,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":352256,"data_stored":685052,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542327,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31468,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340372,"statfs":{"total":96636764160,"available":96604540928,"internally_reserved":0,"allocated":299008,"data_stored":96516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542326,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":282624,"data_stored":95020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542327,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":282624,"data_stored":94046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542327,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":94116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575029,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":94986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575029,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31444,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340396,"statfs":{"total":96636764160,"available":96604565504,"internally_reserved":0,"allocated":274432,"data_stored":93914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575029,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31512,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340328,"statfs":{"total":96636764160,"available":96604495872,"internally_reserved":0,"allocated":344064,"data_stored":683826,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32715,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T10:59:38.686 DEBUG:teuthology.orchestra.run.smithi005:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-24T10:59:38.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.824+0000 7fc25d75e640 1 Processor -- start 2024-03-24T10:59:38.824 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.824+0000 7fc25d75e640 1 -- start start 2024-03-24T10:59:38.825 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.825+0000 7fc25d75e640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc2580cf340 0x7fc2580cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:38.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.825+0000 7fc25d75e640 1 -- --> v1:172.21.15.5:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2580cc3e0 con 0x7fc2580cc000 2024-03-24T10:59:38.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.825+0000 7fc25d75e640 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc25815a5f0 con 0x7fc2580ccb80 2024-03-24T10:59:38.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.825+0000 7fc25d75e640 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 -- 0x7fc25815a400 con 0x7fc2580cf340 2024-03-24T10:59:38.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.826+0000 7fc2567fc640 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc2580cf340 0x7fc2580cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:38.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.826+0000 7fc256ffd640 1 --1- >> v1:172.21.15.5:6789/0 conn(0x7fc2580cc000 0x7fc25815e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.5:6789/0 says I am v1:172.21.15.5:38506/0 (socket says 172.21.15.5:38506) 2024-03-24T10:59:38.826 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.826+0000 7fc256ffd640 1 -- 172.21.15.5:0/2508758000 learned_addr learned my addr 172.21.15.5:0/2508758000 (peer_addr_for_me v1:172.21.15.5:0/0) 2024-03-24T10:59:38.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.829+0000 7fc255ffb640 1 -- 172.21.15.5:0/2508758000 <== mon.0 v1:172.21.15.5:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3918809840 0 0) 0x7fc2580cc3e0 con 0x7fc2580cc000 2024-03-24T10:59:38.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.829+0000 7fc255ffb640 1 -- 172.21.15.5:0/2508758000 --> v1:172.21.15.5:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc240003520 con 0x7fc2580cc000 2024-03-24T10:59:38.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.829+0000 7fc2567fc640 1 -- 172.21.15.5:0/2508758000 >> v1:172.21.15.57:6789/0 conn(0x7fc2580ccb80 legacy=0x7fc25806bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T10:59:38.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.829+0000 7fc2567fc640 1 -- 172.21.15.5:0/2508758000 >> v1:172.21.15.5:6789/0 conn(0x7fc2580cc000 legacy=0x7fc25815e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:38.829 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.829+0000 7fc2567fc640 1 -- 172.21.15.5:0/2508758000 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc25815a8b0 con 0x7fc2580cf340 2024-03-24T10:59:38.833 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.833+0000 7fc2567fc640 1 --2- 172.21.15.5:0/2508758000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2580cf340 0x7fc2580cc640 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc24c0097a0 tx=0x7fc24c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4671929efadd0dd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:38.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.838+0000 7fc255ffb640 1 -- 172.21.15.5:0/2508758000 <== mon.1 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24c031830 con 0x7fc2580cf340 2024-03-24T10:59:38.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.838+0000 7fc255ffb640 1 -- 172.21.15.5:0/2508758000 <== mon.1 v2:172.21.15.57:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc24c031990 con 0x7fc2580cf340 2024-03-24T10:59:38.838 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.838+0000 7fc255ffb640 1 -- 172.21.15.5:0/2508758000 <== mon.1 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24c031d20 con 0x7fc2580cf340 2024-03-24T10:59:38.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.839+0000 7fc25d75e640 1 -- 172.21.15.5:0/2508758000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2580cf340 msgr2=0x7fc2580cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:38.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.839+0000 7fc25d75e640 1 --2- 172.21.15.5:0/2508758000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2580cf340 0x7fc2580cc640 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc24c0097a0 tx=0x7fc24c02b8d0 comp rx=0 tx=0).stop 2024-03-24T10:59:38.839 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.839+0000 7fc25d75e640 1 -- 172.21.15.5:0/2508758000 shutdown_connections 2024-03-24T10:59:38.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.839+0000 7fc25d75e640 1 --2- 172.21.15.5:0/2508758000 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2580cf340 0x7fc2580cc640 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:38.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.839+0000 7fc25d75e640 1 -- 172.21.15.5:0/2508758000 >> 172.21.15.5:0/2508758000 conn(0x7fc2580cfc30 msgr2=0x7fc2580d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:38.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.839+0000 7fc25d75e640 1 -- 172.21.15.5:0/2508758000 shutdown_connections 2024-03-24T10:59:38.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.839+0000 7fc25d75e640 1 -- 172.21.15.5:0/2508758000 wait complete. 2024-03-24T10:59:38.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.840+0000 7fc25d75e640 1 Processor -- start 2024-03-24T10:59:38.840 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.840+0000 7fc25d75e640 1 -- start start 2024-03-24T10:59:38.841 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.841+0000 7fc25d75e640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc2580cc000 0x7fc25815cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:38.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.841+0000 7fc25d75e640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2580ccb80 0x7fc2581f0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:38.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.841+0000 7fc25d75e640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc2580cf340 0x7fc2581f0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:38.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.841+0000 7fc25d75e640 1 -- --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_getmap magic: 0 -- 0x7fc2581e4af0 con 0x7fc2580cf340 2024-03-24T10:59:38.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.841+0000 7fc25d75e640 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 -- 0x7fc2581f34d0 con 0x7fc2580ccb80 2024-03-24T10:59:38.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.841+0000 7fc25d75e640 1 -- --> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] -- mon_getmap magic: 0 -- 0x7fc2581f3790 con 0x7fc2580cc000 2024-03-24T10:59:38.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.841+0000 7fc2567fc640 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2580ccb80 0x7fc2581f0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:38.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.841+0000 7fc2577fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc2580cf340 0x7fc2581f0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:38.842 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.842+0000 7fc256ffd640 1 --2- >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc2580cc000 0x7fc25815cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:38.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.842+0000 7fc2577fe640 1 --2- >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc2580cf340 0x7fc2581f0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.5:3300/0 says I am v2:172.21.15.5:50222/0 (socket says 172.21.15.5:50222) 2024-03-24T10:59:38.843 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.842+0000 7fc2577fe640 1 -- 172.21.15.5:0/2932894091 learned_addr learned my addr 172.21.15.5:0/2932894091 (peer_addr_for_me v2:172.21.15.5:0/0) 2024-03-24T10:59:38.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.847+0000 7fc2577fe640 1 -- 172.21.15.5:0/2932894091 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc2580cc000 msgr2=0x7fc25815cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:38.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.847+0000 7fc2577fe640 1 --2- 172.21.15.5:0/2932894091 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc2580cc000 0x7fc25815cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:38.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.847+0000 7fc2577fe640 1 -- 172.21.15.5:0/2932894091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2580ccb80 msgr2=0x7fc2581f0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:38.847 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.847+0000 7fc2577fe640 1 --2- 172.21.15.5:0/2932894091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2580ccb80 0x7fc2581f0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:38.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.847+0000 7fc2577fe640 1 -- 172.21.15.5:0/2932894091 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24c009400 con 0x7fc2580cf340 2024-03-24T10:59:38.848 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.847+0000 7fc2567fc640 1 --2- 172.21.15.5:0/2932894091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2580ccb80 0x7fc2581f0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T10:59:38.850 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.850+0000 7fc2577fe640 1 --2- 172.21.15.5:0/2932894091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc2580cf340 0x7fc2581f0780 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc25000c780 tx=0x7fc25000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:38.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.855+0000 7fc23f7fe640 1 -- 172.21.15.5:0/2932894091 <== mon.0 v2:172.21.15.5:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc250004b60 con 0x7fc2580cf340 2024-03-24T10:59:38.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.855+0000 7fc23f7fe640 1 -- 172.21.15.5:0/2932894091 <== mon.0 v2:172.21.15.5:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc25000ce90 con 0x7fc2580cf340 2024-03-24T10:59:38.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.855+0000 7fc23f7fe640 1 -- 172.21.15.5:0/2932894091 <== mon.0 v2:172.21.15.5:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25000a850 con 0x7fc2580cf340 2024-03-24T10:59:38.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.855+0000 7fc25d75e640 1 -- 172.21.15.5:0/2932894091 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2581f3a70 con 0x7fc2580cf340 2024-03-24T10:59:38.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.855+0000 7fc25d75e640 1 -- 172.21.15.5:0/2932894091 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2581f4030 con 0x7fc2580cf340 2024-03-24T10:59:38.855 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.856+0000 7fc25d75e640 1 -- 172.21.15.5:0/2932894091 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2581eff60 con 0x7fc2580cf340 2024-03-24T10:59:38.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.927+0000 7fc23f7fe640 1 -- 172.21.15.5:0/2932894091 <== mon.0 v2:172.21.15.5:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc250006b50 con 0x7fc2580cf340 2024-03-24T10:59:38.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.927+0000 7fc23f7fe640 1 --2- 172.21.15.5:0/2932894091 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc24406b860 0x7fc24406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T10:59:38.927 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.927+0000 7fc256ffd640 1 --2- 172.21.15.5:0/2932894091 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc24406b860 0x7fc24406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T10:59:38.928 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:38.928+0000 7fc256ffd640 1 --2- 172.21.15.5:0/2932894091 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc24406b860 0x7fc24406dd20 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc248005880 tx=0x7fc248009040 comp rx=0 tx=0).ready entity=mgr.4120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T10:59:39.160 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:39.160+0000 7fc23f7fe640 1 -- 172.21.15.5:0/2932894091 <== mon.0 v2:172.21.15.5:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5911+0+0 (secure 0 0 0) 0x7fc250059090 con 0x7fc2580cf340 2024-03-24T10:59:39.161 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:39.160+0000 7fc23f7fe640 1 -- 172.21.15.5:0/2932894091 <== mon.0 v2:172.21.15.5:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc250055330 con 0x7fc2580cf340 2024-03-24T10:59:39.344 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:39.344+0000 7fc25d75e640 1 -- 172.21.15.5:0/2932894091 --> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fc2581eff60 con 0x7fc2580cf340 2024-03-24T10:59:39.440 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:39.440+0000 7fc23f7fe640 1 -- 172.21.15.5:0/2932894091 <== mon.0 v2:172.21.15.5:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) ==== 123+0+0 (secure 0 0 0) 0x7fc2581eff60 con 0x7fc2580cf340 2024-03-24T10:59:39.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:39.442+0000 7fc25d75e640 1 -- 172.21.15.5:0/2932894091 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc24406b860 msgr2=0x7fc24406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:39.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:39.442+0000 7fc25d75e640 1 --2- 172.21.15.5:0/2932894091 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc24406b860 0x7fc24406dd20 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fc248005880 tx=0x7fc248009040 comp rx=0 tx=0).stop 2024-03-24T10:59:39.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:39.442+0000 7fc25d75e640 1 -- 172.21.15.5:0/2932894091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc2580cf340 msgr2=0x7fc2581f0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T10:59:39.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:39.442+0000 7fc25d75e640 1 --2- 172.21.15.5:0/2932894091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc2580cf340 0x7fc2581f0780 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc25000c780 tx=0x7fc25000cc40 comp rx=0 tx=0).stop 2024-03-24T10:59:39.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:39.442+0000 7fc25d75e640 1 -- 172.21.15.5:0/2932894091 shutdown_connections 2024-03-24T10:59:39.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:39.442+0000 7fc25d75e640 1 --2- 172.21.15.5:0/2932894091 >> [v2:172.21.15.5:6832/1102439181,v1:172.21.15.5:6833/1102439181] conn(0x7fc24406b860 0x7fc24406dd20 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:39.442 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:39.442+0000 7fc25d75e640 1 --2- 172.21.15.5:0/2932894091 >> [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] conn(0x7fc2580cf340 0x7fc2581f0780 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:39.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:39.442+0000 7fc25d75e640 1 --2- 172.21.15.5:0/2932894091 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc2580ccb80 0x7fc2581f0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:39.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:39.442+0000 7fc25d75e640 1 --2- 172.21.15.5:0/2932894091 >> [v2:172.21.15.57:3301/0,v1:172.21.15.57:6790/0] conn(0x7fc2580cc000 0x7fc25815cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T10:59:39.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:39.442+0000 7fc25d75e640 1 -- 172.21.15.5:0/2932894091 >> 172.21.15.5:0/2932894091 conn(0x7fc2580cfc30 msgr2=0x7fc25815e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T10:59:39.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:39.442+0000 7fc25d75e640 1 -- 172.21.15.5:0/2932894091 shutdown_connections 2024-03-24T10:59:39.443 INFO:teuthology.orchestra.run.smithi005.stderr:2024-03-24T10:59:39.442+0000 7fc25d75e640 1 -- 172.21.15.5:0/2932894091 wait complete. 2024-03-24T10:59:39.453 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy 2024-03-24T10:59:39.455 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-24T10:59:39.455 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T10:59:39.455 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T11:00:03.336 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:00:03.339 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:00:32.920 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:16:24.421 36946== Warning: unimplemented fcntl command: 1036 2024-03-24T11:00:32.925 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:16:24.426 36946== Warning: unimplemented fcntl command: 1036 2024-03-24T11:00:32.982 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:16:24.446 36958== Warning: unimplemented fcntl command: 1036 2024-03-24T11:00:32.983 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:16:24.448 36958== Warning: unimplemented fcntl command: 1036 2024-03-24T11:00:33.019 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:16:24.483 36957== Warning: unimplemented fcntl command: 1036 2024-03-24T11:00:33.020 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:16:24.521 36946== Warning: unimplemented fcntl command: 1036 2024-03-24T11:00:33.027 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:16:24.491 36957== Warning: unimplemented fcntl command: 1036 2024-03-24T11:00:33.096 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:16:24.559 36958== Warning: unimplemented fcntl command: 1036 2024-03-24T11:00:33.141 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:16:24.605 36957== Warning: unimplemented fcntl command: 1036 2024-03-24T11:00:33.406 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:00:33.408 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:01:03.471 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:01:03.475 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:01:03.831 INFO:tasks.ceph.mds.a.smithi005.stderr:2024-03-24T11:01:03.794+0000 b5e9640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i a (PID: 41003) UID: 0 2024-03-24T11:01:03.831 INFO:tasks.ceph.mds.a.smithi005.stderr:2024-03-24T11:01:03.829+0000 b5e9640 -1 mds.a *** got signal Terminated *** 2024-03-24T11:01:07.878 INFO:tasks.ceph.mds.a.smithi005.stderr:2024-03-24T11:01:07.874+0000 cdec640 -1 mgrc handle_mgr_configure dropping unexpected configure message 2024-03-24T11:01:33.542 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:01:33.545 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:01:36.802 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:17:28.303 36946== Warning: unimplemented fcntl command: 1036 2024-03-24T11:01:36.804 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:17:28.305 36946== Warning: unimplemented fcntl command: 1036 2024-03-24T11:01:36.832 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:17:28.333 36946== Warning: unimplemented fcntl command: 1036 2024-03-24T11:01:36.892 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:17:28.356 36958== Warning: unimplemented fcntl command: 1036 2024-03-24T11:01:36.896 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:17:28.360 36958== Warning: unimplemented fcntl command: 1036 2024-03-24T11:01:36.901 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:17:28.365 36957== Warning: unimplemented fcntl command: 1036 2024-03-24T11:01:36.907 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:17:28.371 36957== Warning: unimplemented fcntl command: 1036 2024-03-24T11:01:36.938 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:17:28.402 36957== Warning: unimplemented fcntl command: 1036 2024-03-24T11:01:36.946 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:17:28.410 36958== Warning: unimplemented fcntl command: 1036 2024-03-24T11:01:41.881 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:17:33.382 36946== Warning: unimplemented fcntl command: 1036 2024-03-24T11:01:41.934 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:17:33.435 36946== Warning: unimplemented fcntl command: 1036 2024-03-24T11:01:41.949 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:17:33.450 36946== Warning: unimplemented fcntl command: 1036 2024-03-24T11:01:41.953 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:17:33.417 36957== Warning: unimplemented fcntl command: 1036 2024-03-24T11:01:41.959 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:17:33.423 36957== Warning: unimplemented fcntl command: 1036 2024-03-24T11:01:41.974 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:17:33.438 36957== Warning: unimplemented fcntl command: 1036 2024-03-24T11:01:42.029 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:17:33.493 36958== Warning: unimplemented fcntl command: 1036 2024-03-24T11:01:42.034 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:17:33.498 36958== Warning: unimplemented fcntl command: 1036 2024-03-24T11:01:42.058 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:17:33.522 36958== Warning: unimplemented fcntl command: 1036 2024-03-24T11:01:59.937 INFO:tasks.ceph.mds.a.smithi005.stderr:2024-03-24T11:01:59.934+0000 a5e7640 -1 monclient: get_auth_request but no auth handler is set up 2024-03-24T11:01:59.947 INFO:tasks.ceph.mds.a.smithi005.stderr:2024-03-24T11:01:59.947+0000 a5e7640 -1 monclient: get_auth_request but no auth handler is set up 2024-03-24T11:02:03.597 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:02:03.600 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:02:33.663 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:02:33.666 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:03:03.726 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:03:03.729 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:03:33.789 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:03:33.792 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:04:03.852 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:04:03.855 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:04:33.916 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:04:33.919 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:04:38.529 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-24T11:04:38.529 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-24T11:04:38.529 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T11:04:38.530 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-03-24T11:04:38.530+0000 7fc3e0518640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 0 (PID: 37224) UID: 0 2024-03-24T11:04:38.531 INFO:tasks.ceph.osd.0.smithi005.stderr:2024-03-24T11:04:38.530+0000 7fc3e0518640 -1 osd.0 25 *** Got signal Terminated *** 2024-03-24T11:04:44.630 INFO:tasks.ceph.osd.0:Stopped 2024-03-24T11:04:44.631 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-24T11:04:44.631 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T11:04:44.632 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-03-24T11:04:44.632+0000 7f017fbe8640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 1 (PID: 37230) UID: 0 2024-03-24T11:04:44.632 INFO:tasks.ceph.osd.1.smithi005.stderr:2024-03-24T11:04:44.632+0000 7f017fbe8640 -1 osd.1 27 *** Got signal Terminated *** 2024-03-24T11:04:48.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-03-24T11:04:50.732 INFO:tasks.ceph.osd.1:Stopped 2024-03-24T11:04:50.733 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-24T11:04:50.733 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T11:04:50.734 INFO:tasks.ceph.osd.2.smithi005.stderr:2024-03-24T11:04:50.734+0000 7fb5dfd68640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 2 (PID: 37237) UID: 0 2024-03-24T11:04:50.734 INFO:tasks.ceph.osd.2.smithi005.stderr:2024-03-24T11:04:50.734+0000 7fb5dfd68640 -1 osd.2 29 *** Got signal Terminated *** 2024-03-24T11:04:55.183 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-03-24T11:04:55.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-03-24T11:04:56.834 INFO:tasks.ceph.osd.2:Stopped 2024-03-24T11:04:56.835 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-24T11:04:56.835 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T11:04:56.835 INFO:tasks.ceph.osd.3.smithi005.stderr:2024-03-24T11:04:56.835+0000 7f00b1292640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 3 (PID: 37239) UID: 0 2024-03-24T11:04:56.836 INFO:tasks.ceph.osd.3.smithi005.stderr:2024-03-24T11:04:56.835+0000 7f00b1292640 -1 osd.3 31 *** Got signal Terminated *** 2024-03-24T11:05:01.698 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-03-24T11:05:01.698 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-03-24T11:05:02.936 INFO:tasks.ceph.osd.3:Stopped 2024-03-24T11:05:02.937 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-24T11:05:02.937 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T11:05:02.937 INFO:tasks.ceph.osd.4.smithi057.stderr:2024-03-24T11:05:02.938+0000 7f7cfffcf640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 4 (PID: 37536) UID: 0 2024-03-24T11:05:02.938 INFO:tasks.ceph.osd.4.smithi057.stderr:2024-03-24T11:05:02.938+0000 7f7cfffcf640 -1 osd.4 33 *** Got signal Terminated *** 2024-03-24T11:05:03.979 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:05:03.981 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:05:08.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-03-24T11:05:08.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-03-24T11:05:09.038 INFO:tasks.ceph.osd.4:Stopped 2024-03-24T11:05:09.039 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-24T11:05:09.039 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T11:05:09.040 INFO:tasks.ceph.osd.5.smithi057.stderr:2024-03-24T11:05:09.040+0000 7f0b6c5e2640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 5 (PID: 37540) UID: 0 2024-03-24T11:05:09.040 INFO:tasks.ceph.osd.5.smithi057.stderr:2024-03-24T11:05:09.040+0000 7f0b6c5e2640 -1 osd.5 35 *** Got signal Terminated *** 2024-03-24T11:05:14.422 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-03-24T11:05:14.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-03-24T11:05:15.140 INFO:tasks.ceph.osd.5:Stopped 2024-03-24T11:05:15.141 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-24T11:05:15.141 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T11:05:15.142 INFO:tasks.ceph.osd.6.smithi057.stderr:2024-03-24T11:05:15.142+0000 7f9b74c46640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 6 (PID: 37543) UID: 0 2024-03-24T11:05:15.142 INFO:tasks.ceph.osd.6.smithi057.stderr:2024-03-24T11:05:15.142+0000 7f9b74c46640 -1 osd.6 37 *** Got signal Terminated *** 2024-03-24T11:05:20.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-03-24T11:05:20.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-03-24T11:05:21.242 INFO:tasks.ceph.osd.6:Stopped 2024-03-24T11:05:21.243 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-24T11:05:21.243 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T11:05:21.243 INFO:tasks.ceph.osd.7.smithi057.stderr:2024-03-24T11:05:21.244+0000 7fb68ee6f640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 7 (PID: 37542) UID: 0 2024-03-24T11:05:21.244 INFO:tasks.ceph.osd.7.smithi057.stderr:2024-03-24T11:05:21.244+0000 7fb68ee6f640 -1 osd.7 37 *** Got signal Terminated *** 2024-03-24T11:05:26.742 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-03-24T11:05:26.743 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-03-24T11:05:27.344 INFO:tasks.ceph.osd.7:Stopped 2024-03-24T11:05:27.344 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-24T11:05:27.345 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-24T11:05:27.345 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T11:05:27.372 INFO:tasks.ceph.mgr.y:Stopped 2024-03-24T11:05:27.372 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-24T11:05:27.372 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T11:05:27.408 INFO:tasks.ceph.mgr.x:Stopped 2024-03-24T11:05:27.408 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-24T11:05:27.409 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-24T11:05:27.409 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T11:05:27.437 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-03-24T11:05:27.426+0000 9f92640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 36920) UID: 0 2024-03-24T11:05:27.440 INFO:tasks.ceph.mon.a.smithi005.stderr:2024-03-24T11:05:27.428+0000 9f92640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T11:05:32.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-03-24T11:05:34.036 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:05:34.039 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:05:58.815 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:21:50.316 36946== 2024-03-24T11:05:58.815 INFO:tasks.ceph.mon.a.smithi005.stderr:==00:00:21:50.316 36946== Exit program on first error (--exit-on-first-error=yes) 2024-03-24T11:05:58.828 INFO:tasks.ceph.mon.a.smithi005.stderr:daemon-helper: command failed with exit status 42 2024-03-24T11:06:02.085 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-03-24T11:06:04.018 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-24T11:06:04.019 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi005 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-24T11:06:04.020 INFO:tasks.ceph.mon.a:Stopped 2024-03-24T11:06:04.021 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-24T11:06:04.021 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T11:06:04.037 INFO:tasks.ceph.mon.b.smithi057.stderr:2024-03-24T11:06:04.025+0000 9f92640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 36950) UID: 0 2024-03-24T11:06:04.037 INFO:tasks.ceph.mon.b.smithi057.stderr:2024-03-24T11:06:04.026+0000 9f92640 -1 mon.b@1(leader) e1 *** Got Signal Terminated *** 2024-03-24T11:06:04.100 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:06:04.103 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:06:04.813 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:21:56.277 36957== 2024-03-24T11:06:04.814 INFO:tasks.ceph.mon.b.smithi057.stderr:==00:00:21:56.277 36957== Exit program on first error (--exit-on-first-error=yes) 2024-03-24T11:06:04.824 INFO:tasks.ceph.mon.b.smithi057.stderr:daemon-helper: command failed with exit status 42 2024-03-24T11:06:07.792 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-03-24T11:06:07.792 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-03-24T11:06:10.122 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-24T11:06:10.122 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi057 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-24T11:06:10.123 INFO:tasks.ceph.mon.b:Stopped 2024-03-24T11:06:10.123 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-24T11:06:10.123 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T11:06:10.139 INFO:tasks.ceph.mon.c.smithi057.stderr:2024-03-24T11:06:10.127+0000 9f92640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 36953) UID: 0 2024-03-24T11:06:10.139 INFO:tasks.ceph.mon.c.smithi057.stderr:2024-03-24T11:06:10.129+0000 9f92640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-24T11:06:13.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-03-24T11:06:34.169 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:06:34.172 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:07:04.233 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:07:04.236 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:07:04.864 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:22:56.328 36958== 2024-03-24T11:07:04.864 INFO:tasks.ceph.mon.c.smithi057.stderr:==00:00:22:56.328 36958== Exit program on first error (--exit-on-first-error=yes) 2024-03-24T11:07:04.875 INFO:tasks.ceph.mon.c.smithi057.stderr:daemon-helper: command failed with exit status 42 2024-03-24T11:07:05.038 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-24T11:07:05.038 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi057 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-24T11:07:05.039 INFO:tasks.ceph.mon.c:Stopped 2024-03-24T11:07:05.039 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-24T11:07:05.039 DEBUG:teuthology.orchestra.run.smithi005:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v MON_DOWN | head -n 1 2024-03-24T11:07:05.075 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi005.front.sepia.ceph.com 2024-03-24T11:07:05.076 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-24T11:07:05.198 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi005.front.sepia.ceph.com 2024-03-24T11:07:05.198 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-24T11:07:05.288 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi005.front.sepia.ceph.com 2024-03-24T11:07:05.288 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-24T11:07:05.340 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi005.front.sepia.ceph.com 2024-03-24T11:07:05.340 DEBUG:teuthology.orchestra.run.smithi005:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-24T11:07:05.406 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi057.front.sepia.ceph.com 2024-03-24T11:07:05.407 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-24T11:07:05.495 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi057.front.sepia.ceph.com 2024-03-24T11:07:05.495 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-24T11:07:05.567 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi057.front.sepia.ceph.com 2024-03-24T11:07:05.568 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-24T11:07:05.625 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi057.front.sepia.ceph.com 2024-03-24T11:07:05.626 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-24T11:07:05.692 INFO:tasks.ceph:Archiving mon data... 2024-03-24T11:07:05.694 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619063/data/mon.a.tgz 2024-03-24T11:07:05.694 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-03-24T11:07:05.708 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.J9O56bUVyO 2024-03-24T11:07:05.708 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.J9O56bUVyO 2024-03-24T11:07:05.844 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.J9O56bUVyO 2024-03-24T11:07:05.929 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.J9O56bUVyO is 285KB 2024-03-24T11:07:05.994 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.J9O56bUVyO 2024-03-24T11:07:06.009 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619063/data/mon.b.tgz 2024-03-24T11:07:06.009 DEBUG:teuthology.orchestra.run.smithi057:> mktemp 2024-03-24T11:07:06.023 INFO:teuthology.orchestra.run.smithi057.stdout:/tmp/tmp.qAccez9ZMq 2024-03-24T11:07:06.023 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.qAccez9ZMq 2024-03-24T11:07:06.174 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0666 /tmp/tmp.qAccez9ZMq 2024-03-24T11:07:06.257 DEBUG:teuthology.orchestra.remote:smithi057:/tmp/tmp.qAccez9ZMq is 301KB 2024-03-24T11:07:06.325 DEBUG:teuthology.orchestra.run.smithi057:> rm -fr /tmp/tmp.qAccez9ZMq 2024-03-24T11:07:06.339 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619063/data/mon.c.tgz 2024-03-24T11:07:06.339 DEBUG:teuthology.orchestra.run.smithi057:> mktemp 2024-03-24T11:07:06.393 INFO:teuthology.orchestra.run.smithi057.stdout:/tmp/tmp.bSRz9uUyv5 2024-03-24T11:07:06.394 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.bSRz9uUyv5 2024-03-24T11:07:06.521 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0666 /tmp/tmp.bSRz9uUyv5 2024-03-24T11:07:06.606 DEBUG:teuthology.orchestra.remote:smithi057:/tmp/tmp.bSRz9uUyv5 is 301KB 2024-03-24T11:07:06.671 DEBUG:teuthology.orchestra.run.smithi057:> rm -fr /tmp/tmp.bSRz9uUyv5 2024-03-24T11:07:06.685 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-24T11:07:06.685 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-24T11:07:06.690 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-24T11:07:06.827 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-24T11:07:06.873 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-24T11:07:06.891 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-24T11:07:06.940 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-24T11:07:07.014 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-24T11:07:07.031 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-24T11:07:07.048 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 50, column 1 (, line 50) 2024-03-24T11:07:07.087 INFO:tasks.ceph:Archiving crash dumps... 2024-03-24T11:07:07.089 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619063/remote/smithi005/crash 2024-03-24T11:07:07.090 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-24T11:07:07.117 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619063/remote/smithi057/crash 2024-03-24T11:07:07.118 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-24T11:07:07.144 INFO:tasks.ceph:Shutting down logrotate 2024-03-24T11:07:07.144 DEBUG:teuthology.orchestra.run.smithi005:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:07:07.161 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T11:07:07.238 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-24T11:07:07.264 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-24T11:07:07.291 INFO:tasks.ceph:Compressing logs... 2024-03-24T11:07:07.291 DEBUG:teuthology.orchestra.run.smithi005:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T11:07:07.306 DEBUG:teuthology.orchestra.run.smithi057:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T11:07:07.390 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-24T11:07:07.390 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-24T11:07:07.390 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.f.log 2024-03-24T11:07:07.390 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.b.log 2024-03-24T11:07:07.390 INFO:teuthology.orchestra.run.smithi057.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-24T11:07:07.391 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/valgrind/mds.f.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-24T11:07:07.391 INFO:teuthology.orchestra.run.smithi057.stderr: 55.3% -- replaced with /var/log/ceph/valgrind/mds.f.log.gz 2024-03-24T11:07:07.391 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/valgrind/mds.b.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.d.log 2024-03-24T11:07:07.391 INFO:teuthology.orchestra.run.smithi057.stderr: 55.2% -- replaced with /var/log/ceph/valgrind/mds.b.log.gz 2024-03-24T11:07:07.391 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37207.log 2024-03-24T11:07:07.391 INFO:teuthology.orchestra.run.smithi057.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-24T11:07:07.391 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/valgrind/mds.d.log: gzip -5 --verbose -- 55.3% -- replaced with /var/log/ceph/valgrind/mds.d.log.gz 2024-03-24T11:07:07.391 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.37287.log 2024-03-24T11:07:07.391 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.37207.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-24T11:07:07.391 INFO:teuthology.orchestra.run.smithi057.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37207.log.gz 2024-03-24T11:07:07.391 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.37287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37367.log 2024-03-24T11:07:07.391 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph.audit.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37287.log.gz 2024-03-24T11:07:07.391 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-24T11:07:07.391 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.37367.log: gzip -5 --verbose -- 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37367.log.gz 2024-03-24T11:07:07.391 INFO:teuthology.orchestra.run.smithi057.stderr: /var/log/ceph/ceph-client.admin.37447.log 2024-03-24T11:07:07.392 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-24T11:07:07.392 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.37447.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-24T11:07:07.392 INFO:teuthology.orchestra.run.smithi057.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.37447.log.gz 2024-03-24T11:07:07.392 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-osd.4.log: 93.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-24T11:07:07.392 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-24T11:07:07.392 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37407.log 2024-03-24T11:07:07.392 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-osd.5.log: gzip: /var/log/ceph/ceph-mds.b.log: file size changed while zipping 2024-03-24T11:07:07.392 INFO:teuthology.orchestra.run.smithi057.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-24T11:07:07.392 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2024-03-24T11:07:07.392 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.37407.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-24T11:07:07.392 INFO:teuthology.orchestra.run.smithi057.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37407.log.gz 2024-03-24T11:07:07.392 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-24T11:07:07.392 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37151.log 2024-03-24T11:07:07.392 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mgr.x.log: 90.2% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-03-24T11:07:07.392 INFO:teuthology.orchestra.run.smithi057.stderr: 93.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-24T11:07:07.392 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-24T11:07:07.393 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.37151.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37151.log.gz 2024-03-24T11:07:07.399 INFO:teuthology.orchestra.run.smithi057.stderr: 94.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-24T11:07:07.399 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37247.log 2024-03-24T11:07:07.399 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-24T11:07:07.399 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.37247.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37247.log.gz 2024-03-24T11:07:07.399 INFO:teuthology.orchestra.run.smithi057.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37327.log 2024-03-24T11:07:07.415 INFO:tasks.ceph.mds.d.smithi057.stderr:2024-03-24T11:07:07.403+0000 139f9640 -1 MDSIOContextBase: failed with -110, restarting... 2024-03-24T11:07:07.416 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-mds.d.log: 90.7%gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-24T11:07:07.416 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-client.admin.37327.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37327.log.gz 2024-03-24T11:07:07.417 INFO:teuthology.orchestra.run.smithi057.stderr: -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-24T11:07:07.508 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47122.log 2024-03-24T11:07:07.509 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48040.log 2024-03-24T11:07:07.509 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55669.log 2024-03-24T11:07:07.509 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48040.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47122.log.gz 2024-03-24T11:07:07.509 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52993.log 2024-03-24T11:07:07.509 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.55669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45427.log 2024-03-24T11:07:07.509 INFO:teuthology.orchestra.run.smithi005.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48040.log.gz 2024-03-24T11:07:07.509 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-24T11:07:07.509 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52993.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55669.log.gz 2024-03-24T11:07:07.509 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45427.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52993.log.gz 2024-03-24T11:07:07.509 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49084.log 2024-03-24T11:07:07.509 INFO:teuthology.orchestra.run.smithi005.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45427.log.gz 2024-03-24T11:07:07.509 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43810.log 2024-03-24T11:07:07.510 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.49084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42615.log 2024-03-24T11:07:07.510 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43810.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49084.log.gz 2024-03-24T11:07:07.510 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42844.log 2024-03-24T11:07:07.510 INFO:teuthology.orchestra.run.smithi005.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.43810.log.gz 2024-03-24T11:07:07.510 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52033.log 2024-03-24T11:07:07.510 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49378.log 2024-03-24T11:07:07.510 INFO:teuthology.orchestra.run.smithi005.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42615.log.gz 2024-03-24T11:07:07.510 INFO:teuthology.orchestra.run.smithi005.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42844.log.gz 2024-03-24T11:07:07.510 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49462.log 2024-03-24T11:07:07.510 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 87.9%/var/log/ceph/ceph-client.admin.49378.log: -- replaced with /var/log/ceph/ceph-client.admin.52033.log.gz 2024-03-24T11:07:07.510 INFO:teuthology.orchestra.run.smithi005.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49210.log 2024-03-24T11:07:07.510 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49462.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49378.log.gz 2024-03-24T11:07:07.510 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43180.log 2024-03-24T11:07:07.511 INFO:teuthology.orchestra.run.smithi005.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49462.log.gz 2024-03-24T11:07:07.511 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41289.log 2024-03-24T11:07:07.511 INFO:teuthology.orchestra.run.smithi005.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49210.log.gz 2024-03-24T11:07:07.511 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49588.log 2024-03-24T11:07:07.511 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41289.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.43180.log.gz 2024-03-24T11:07:07.511 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46639.log 2024-03-24T11:07:07.511 INFO:teuthology.orchestra.run.smithi005.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41289.log.gz 2024-03-24T11:07:07.511 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54625.log 2024-03-24T11:07:07.511 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49588.log: /var/log/ceph/ceph-client.admin.46639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57884.log 2024-03-24T11:07:07.511 INFO:teuthology.orchestra.run.smithi005.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49588.log.gz 2024-03-24T11:07:07.511 INFO:teuthology.orchestra.run.smithi005.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46639.log.gz 2024-03-24T11:07:07.511 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.54625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44905.log 2024-03-24T11:07:07.511 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41373.log 2024-03-24T11:07:07.511 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.57884.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54625.log.gz 2024-03-24T11:07:07.512 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44905.log: gzip -5 88.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.57884.log.gz 2024-03-24T11:07:07.512 INFO:teuthology.orchestra.run.smithi005.stderr: /var/log/ceph/ceph-client.admin.46471.log 2024-03-24T11:07:07.512 INFO:teuthology.orchestra.run.smithi005.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44905.log.gz 2024-03-24T11:07:07.512 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46555.log 2024-03-24T11:07:07.512 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41373.log: /var/log/ceph/ceph-client.admin.46471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44341.log 2024-03-24T11:07:07.512 INFO:teuthology.orchestra.run.smithi005.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41373.log.gz 2024-03-24T11:07:07.512 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46555.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46471.log.gz 2024-03-24T11:07:07.512 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53935.log 2024-03-24T11:07:07.512 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44341.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46555.log.gz 2024-03-24T11:07:07.512 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47518.log 2024-03-24T11:07:07.512 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.53935.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44341.log.gz 2024-03-24T11:07:07.512 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- 89.1% /var/log/ceph/ceph-client.admin.53413.log 2024-03-24T11:07:07.512 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53935.log.gz 2024-03-24T11:07:07.512 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42308.log 2024-03-24T11:07:07.513 INFO:teuthology.orchestra.run.smithi005.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47518.log.gz 2024-03-24T11:07:07.513 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.53413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45907.log 2024-03-24T11:07:07.513 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42308.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53413.log.gz 2024-03-24T11:07:07.513 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55753.log 2024-03-24T11:07:07.513 INFO:teuthology.orchestra.run.smithi005.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42308.log.gz 2024-03-24T11:07:07.513 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51679.log 2024-03-24T11:07:07.513 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50989.log 2024-03-24T11:07:07.513 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.55753.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45907.log.gz 2024-03-24T11:07:07.513 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51679.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55753.log.gz 2024-03-24T11:07:07.513 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48082.log 2024-03-24T11:07:07.513 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50989.log: gzip 88.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.51679.log.gz --verbose 2024-03-24T11:07:07.513 INFO:teuthology.orchestra.run.smithi005.stderr: -- /var/log/ceph/ceph-client.admin.50381.log 2024-03-24T11:07:07.513 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48082.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50989.log.gz 2024-03-24T11:07:07.513 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43264.log 2024-03-24T11:07:07.513 INFO:teuthology.orchestra.run.smithi005.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48082.log.gzgzip 2024-03-24T11:07:07.514 INFO:teuthology.orchestra.run.smithi005.stderr: -5 --verbose -- /var/log/ceph/valgrind/mds.c.log 2024-03-24T11:07:07.514 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50381.log: /var/log/ceph/ceph-client.admin.43264.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.a.log 2024-03-24T11:07:07.514 INFO:teuthology.orchestra.run.smithi005.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50381.log.gz/var/log/ceph/valgrind/mds.c.log: 2024-03-24T11:07:07.514 INFO:teuthology.orchestra.run.smithi005.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-24T11:07:07.514 INFO:teuthology.orchestra.run.smithi005.stderr: 55.3% -- replaced with /var/log/ceph/valgrind/mds.c.log.gz 2024-03-24T11:07:07.514 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43264.log.gz 2024-03-24T11:07:07.514 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/valgrind/mds.a.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.e.log 2024-03-24T11:07:07.514 INFO:teuthology.orchestra.run.smithi005.stderr: 55.3% -- replaced with /var/log/ceph/valgrind/mds.a.log.gz 2024-03-24T11:07:07.514 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43094.log 2024-03-24T11:07:07.514 INFO:teuthology.orchestra.run.smithi005.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-24T11:07:07.514 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/valgrind/mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44737.log 2024-03-24T11:07:07.514 INFO:teuthology.orchestra.run.smithi005.stderr: 55.3% -- replaced with /var/log/ceph/valgrind/mds.e.log.gz 2024-03-24T11:07:07.514 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42085.log 2024-03-24T11:07:07.514 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44737.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43094.log.gz 2024-03-24T11:07:07.515 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46723.log 2024-03-24T11:07:07.515 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45739.log 2024-03-24T11:07:07.515 INFO:teuthology.orchestra.run.smithi005.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44737.log.gz 2024-03-24T11:07:07.515 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42085.log: /var/log/ceph/ceph-client.admin.46723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47476.log 2024-03-24T11:07:07.515 INFO:teuthology.orchestra.run.smithi005.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42085.log.gz 2024-03-24T11:07:07.515 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47998.log 2024-03-24T11:07:07.515 INFO:teuthology.orchestra.run.smithi005.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46723.log.gz 2024-03-24T11:07:07.515 INFO:teuthology.orchestra.run.smithi005.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45739.log.gz 2024-03-24T11:07:07.515 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51157.log 2024-03-24T11:07:07.515 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56275.log 2024-03-24T11:07:07.515 INFO:teuthology.orchestra.run.smithi005.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47476.log.gz 2024-03-24T11:07:07.515 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51157.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47998.log.gz 2024-03-24T11:07:07.515 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52117.log 2024-03-24T11:07:07.516 INFO:teuthology.orchestra.run.smithi005.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51157.log.gz 2024-03-24T11:07:07.516 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42648.log 2024-03-24T11:07:07.516 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56275.log: /var/log/ceph/ceph-client.admin.52117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50508.log 2024-03-24T11:07:07.516 INFO:teuthology.orchestra.run.smithi005.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56275.log.gz 2024-03-24T11:07:07.516 INFO:teuthology.orchestra.run.smithi005.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52117.log.gz 2024-03-24T11:07:07.516 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42884.log 2024-03-24T11:07:07.516 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54937.log 2024-03-24T11:07:07.516 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50508.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42648.log.gz 2024-03-24T11:07:07.516 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42884.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50508.log.gz 2024-03-24T11:07:07.516 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55291.log 2024-03-24T11:07:07.516 INFO:teuthology.orchestra.run.smithi005.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42884.log.gz 2024-03-24T11:07:07.516 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47248.log 2024-03-24T11:07:07.516 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.54937.log: /var/log/ceph/ceph-client.admin.55291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53329.log 2024-03-24T11:07:07.516 INFO:teuthology.orchestra.run.smithi005.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54937.log.gz 2024-03-24T11:07:07.516 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43390.log 2024-03-24T11:07:07.516 INFO:teuthology.orchestra.run.smithi005.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55291.log.gz 2024-03-24T11:07:07.517 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.53329.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47248.log.gz 2024-03-24T11:07:07.517 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50947.log 2024-03-24T11:07:07.517 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43390.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53329.log.gz 2024-03-24T11:07:07.517 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44695.log 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43390.log.gz 2024-03-24T11:07:07.517 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-24T11:07:07.517 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50947.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50947.log.gz 2024-03-24T11:07:07.517 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49420.log 2024-03-24T11:07:07.517 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54163.log 2024-03-24T11:07:07.517 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49420.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44695.log.gz 2024-03-24T11:07:07.517 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42924.log 2024-03-24T11:07:07.517 INFO:teuthology.orchestra.run.smithi005.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49420.log.gz 2024-03-24T11:07:07.517 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.54163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46387.log 2024-03-24T11:07:07.517 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52243.log 2024-03-24T11:07:07.517 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42924.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54163.log.gz 2024-03-24T11:07:07.518 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42944.log 2024-03-24T11:07:07.518 INFO:teuthology.orchestra.run.smithi005.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42924.log.gz 2024-03-24T11:07:07.518 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52243.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46387.log.gz 2024-03-24T11:07:07.518 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43684.log 2024-03-24T11:07:07.518 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42944.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52243.log.gz 2024-03-24T11:07:07.518 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45385.log 2024-03-24T11:07:07.518 INFO:teuthology.orchestra.run.smithi005.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42944.log.gz 2024-03-24T11:07:07.518 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49024.log 2024-03-24T11:07:07.518 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43684.log: /var/log/ceph/ceph-client.admin.45385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47770.log 2024-03-24T11:07:07.518 INFO:teuthology.orchestra.run.smithi005.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43684.log.gz 2024-03-24T11:07:07.518 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49024.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45385.log.gz 2024-03-24T11:07:07.518 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55711.log 2024-03-24T11:07:07.518 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47770.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49024.log.gz 2024-03-24T11:07:07.519 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56881.log 2024-03-24T11:07:07.519 INFO:teuthology.orchestra.run.smithi005.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47770.log.gz 2024-03-24T11:07:07.519 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.55711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42493.log 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55711.log.gz 2024-03-24T11:07:07.519 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-24T11:07:07.519 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56881.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56881.log.gz 2024-03-24T11:07:07.519 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41694.log 2024-03-24T11:07:07.519 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52891.log 2024-03-24T11:07:07.519 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41694.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42493.log.gz 2024-03-24T11:07:07.519 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52075.log 2024-03-24T11:07:07.519 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55813.log 2024-03-24T11:07:07.519 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52891.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41694.log.gz 2024-03-24T11:07:07.519 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52075.log: gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52891.log.gz -5 2024-03-24T11:07:07.519 INFO:teuthology.orchestra.run.smithi005.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47896.log 2024-03-24T11:07:07.519 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.55813.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52075.log.gz 2024-03-24T11:07:07.519 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54373.log 2024-03-24T11:07:07.520 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47896.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55813.log.gz 2024-03-24T11:07:07.520 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30944.log 2024-03-24T11:07:07.520 INFO:teuthology.orchestra.run.smithi005.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47896.log.gz 2024-03-24T11:07:07.520 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47644.log 2024-03-24T11:07:07.520 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.54373.log: /var/log/ceph/ceph.tmp-client.admin.30944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57319.log 2024-03-24T11:07:07.520 INFO:teuthology.orchestra.run.smithi005.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30944.log.gz 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54373.log.gz 2024-03-24T11:07:07.520 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-24T11:07:07.520 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42296.log 2024-03-24T11:07:07.520 INFO:teuthology.orchestra.run.smithi005.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47644.log.gz 2024-03-24T11:07:07.520 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.57319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50466.log 2024-03-24T11:07:07.520 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5/var/log/ceph/ceph-client.admin.42296.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57319.log.gz 2024-03-24T11:07:07.520 INFO:teuthology.orchestra.run.smithi005.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41652.log 2024-03-24T11:07:07.520 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50466.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42296.log.gz 2024-03-24T11:07:07.520 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47332.log 2024-03-24T11:07:07.521 INFO:teuthology.orchestra.run.smithi005.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50466.log.gz 2024-03-24T11:07:07.521 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55585.log 2024-03-24T11:07:07.521 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47332.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41652.log.gz 2024-03-24T11:07:07.521 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46303.log 2024-03-24T11:07:07.521 INFO:teuthology.orchestra.run.smithi005.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47332.log.gz 2024-03-24T11:07:07.521 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.55585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46201.log 2024-03-24T11:07:07.521 INFO:teuthology.orchestra.run.smithi005.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55585.log.gz 2024-03-24T11:07:07.521 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52849.log 2024-03-24T11:07:07.521 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46303.log: /var/log/ceph/ceph-client.admin.46201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48940.log 2024-03-24T11:07:07.521 INFO:teuthology.orchestra.run.smithi005.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46303.log.gz 2024-03-24T11:07:07.521 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52849.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46201.log.gz 2024-03-24T11:07:07.521 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43474.log 2024-03-24T11:07:07.521 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56671.log 88.4% 2024-03-24T11:07:07.521 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52849.log.gz 2024-03-24T11:07:07.521 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48940.log: /var/log/ceph/ceph-client.admin.43474.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48940.log.gz 2024-03-24T11:07:07.522 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48460.log 2024-03-24T11:07:07.522 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49126.log 2024-03-24T11:07:07.522 INFO:teuthology.orchestra.run.smithi005.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43474.log.gz 2024-03-24T11:07:07.522 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56671.log: /var/log/ceph/ceph-client.admin.48460.log: gzip -5 88.4% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.56671.log.gz /var/log/ceph/ceph-client.admin.57193.log 2024-03-24T11:07:07.522 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-24T11:07:07.522 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52201.log 2024-03-24T11:07:07.522 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49126.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48460.log.gz 2024-03-24T11:07:07.522 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.57193.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49126.log.gz 2024-03-24T11:07:07.522 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55543.log 2024-03-24T11:07:07.522 INFO:teuthology.orchestra.run.smithi005.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57193.log.gz 2024-03-24T11:07:07.522 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53599.log 2024-03-24T11:07:07.522 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52201.log: /var/log/ceph/ceph-client.admin.55543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37364.log 2024-03-24T11:07:07.522 INFO:teuthology.orchestra.run.smithi005.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52201.log.gz 2024-03-24T11:07:07.522 INFO:teuthology.orchestra.run.smithi005.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55543.log.gz 2024-03-24T11:07:07.523 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.53599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50593.log 2024-03-24T11:07:07.523 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42559.log 2024-03-24T11:07:07.523 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.37364.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53599.log.gz 2024-03-24T11:07:07.523 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50593.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37364.log.gz 2024-03-24T11:07:07.523 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55021.log 2024-03-24T11:07:07.523 INFO:teuthology.orchestra.run.smithi005.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50593.log.gz 2024-03-24T11:07:07.523 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55063.log 2024-03-24T11:07:07.523 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42559.log: /var/log/ceph/ceph-client.admin.55021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50677.log 2024-03-24T11:07:07.523 INFO:teuthology.orchestra.run.smithi005.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42559.log.gz 2024-03-24T11:07:07.523 INFO:teuthology.orchestra.run.smithi005.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55021.log.gz 2024-03-24T11:07:07.523 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.55063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44425.log 2024-03-24T11:07:07.523 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55939.log 2024-03-24T11:07:07.523 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50677.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55063.log.gz 2024-03-24T11:07:07.523 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51511.log 2024-03-24T11:07:07.524 INFO:teuthology.orchestra.run.smithi005.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50677.log.gz 2024-03-24T11:07:07.524 INFO:teuthology.orchestra.run.smithi005.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44425.log.gz 2024-03-24T11:07:07.524 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.55939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46597.log 2024-03-24T11:07:07.524 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54853.log 2024-03-24T11:07:07.524 INFO:teuthology.orchestra.run.smithi005.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55939.log.gz 2024-03-24T11:07:07.524 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45823.log 2024-03-24T11:07:07.524 INFO:teuthology.orchestra.run.smithi005.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51511.log.gz 2024-03-24T11:07:07.524 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.54853.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46597.log.gz 2024-03-24T11:07:07.524 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57277.log 2024-03-24T11:07:07.524 INFO:teuthology.orchestra.run.smithi005.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54853.log.gz 2024-03-24T11:07:07.524 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57487.log 2024-03-24T11:07:07.524 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.57277.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45823.log.gz 2024-03-24T11:07:07.524 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48604.log 2024-03-24T11:07:07.524 INFO:teuthology.orchestra.run.smithi005.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57277.log.gz 2024-03-24T11:07:07.524 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52597.log 2024-03-24T11:07:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.57487.log: /var/log/ceph/ceph-client.admin.48604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48646.log 2024-03-24T11:07:07.525 INFO:teuthology.orchestra.run.smithi005.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57487.log.gz 2024-03-24T11:07:07.525 INFO:teuthology.orchestra.run.smithi005.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48604.log.gz 2024-03-24T11:07:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47728.log 2024-03-24T11:07:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57739.log 2024-03-24T11:07:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48646.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52597.log.gz 2024-03-24T11:07:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47728.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48646.log.gz 2024-03-24T11:07:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55189.log 2024-03-24T11:07:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49774.log 2024-03-24T11:07:07.525 INFO:teuthology.orchestra.run.smithi005.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47728.log.gz 2024-03-24T11:07:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.57739.log: /var/log/ceph/ceph-client.admin.55189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55459.log 2024-03-24T11:07:07.525 INFO:teuthology.orchestra.run.smithi005.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57739.log.gz 2024-03-24T11:07:07.525 INFO:teuthology.orchestra.run.smithi005.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55189.log.gz 2024-03-24T11:07:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44257.log 2024-03-24T11:07:07.525 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49774.log: /var/log/ceph/ceph-client.admin.55459.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-24T11:07:07.526 INFO:teuthology.orchestra.run.smithi005.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49774.log.gz 2024-03-24T11:07:07.526 INFO:teuthology.orchestra.run.smithi005.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55459.log.gz 2024-03-24T11:07:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51031.log 2024-03-24T11:07:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44257.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54205.log 2024-03-24T11:07:07.526 INFO:teuthology.orchestra.run.smithi005.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44257.log.gz 2024-03-24T11:07:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42127.log 2024-03-24T11:07:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.54205.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51031.log.gz 2024-03-24T11:07:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51343.log 2024-03-24T11:07:07.526 INFO:teuthology.orchestra.run.smithi005.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54205.log.gz 2024-03-24T11:07:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:gzip/var/log/ceph/ceph-client.admin.42127.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.56983.log 2024-03-24T11:07:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51343.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42127.log.gz 2024-03-24T11:07:07.526 INFO:teuthology.orchestra.run.smithi005.stderr: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-24T11:07:07.526 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53683.log 2024-03-24T11:07:07.526 INFO:teuthology.orchestra.run.smithi005.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51343.log.gz 2024-03-24T11:07:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44569.log 2024-03-24T11:07:07.527 INFO:teuthology.orchestra.run.smithi005.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56983.log.gz 2024-03-24T11:07:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.53683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46870.log 2024-03-24T11:07:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44569.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53683.log.gz 2024-03-24T11:07:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42555.log 2024-03-24T11:07:07.527 INFO:teuthology.orchestra.run.smithi005.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44569.log.gz 2024-03-24T11:07:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57235.log 2024-03-24T11:07:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46954.log 2024-03-24T11:07:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42555.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46870.log.gz 2024-03-24T11:07:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.57235.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42555.log.gz 2024-03-24T11:07:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46117.log 2024-03-24T11:07:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55897.log 2024-03-24T11:07:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46954.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57235.log.gz 2024-03-24T11:07:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46117.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46954.log.gz 2024-03-24T11:07:07.527 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48856.log 2024-03-24T11:07:07.528 INFO:teuthology.orchestra.run.smithi005.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46117.log.gz 2024-03-24T11:07:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42305.log 2024-03-24T11:07:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.55897.log: /var/log/ceph/ceph-client.admin.48856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49252.log 2024-03-24T11:07:07.528 INFO:teuthology.orchestra.run.smithi005.stderr: 88.5%/var/log/ceph/ceph-client.admin.42305.log: -- replaced with /var/log/ceph/ceph-client.admin.55897.log.gz 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48856.log.gz 2024-03-24T11:07:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46033.log 2024-03-24T11:07:07.528 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-24T11:07:07.528 INFO:teuthology.orchestra.run.smithi005.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42305.log.gz 2024-03-24T11:07:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45073.log 2024-03-24T11:07:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50110.log 2024-03-24T11:07:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46033.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49252.log.gz 2024-03-24T11:07:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47038.log 2024-03-24T11:07:07.528 INFO:teuthology.orchestra.run.smithi005.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46033.log.gz 2024-03-24T11:07:07.528 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.50110.log: -- /var/log/ceph/ceph-client.admin.50338.log 2024-03-24T11:07:07.528 INFO:teuthology.orchestra.run.smithi005.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45073.log.gz 2024-03-24T11:07:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47038.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50110.log.gz 2024-03-24T11:07:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56233.log 2024-03-24T11:07:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55417.log 2024-03-24T11:07:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50338.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47038.log.gz 2024-03-24T11:07:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56233.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50338.log.gz 2024-03-24T11:07:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51721.log 2024-03-24T11:07:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.55417.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56233.log.gz 2024-03-24T11:07:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41992.log 2024-03-24T11:07:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51721.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55417.log.gz 2024-03-24T11:07:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51847.log 2024-03-24T11:07:07.529 INFO:teuthology.orchestra.run.smithi005.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51721.log.gz 2024-03-24T11:07:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48898.log 2024-03-24T11:07:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:gzip 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41992.log.gz 2024-03-24T11:07:07.529 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51847.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49984.log/var/log/ceph/ceph-client.admin.48898.log: 2024-03-24T11:07:07.530 INFO:teuthology.orchestra.run.smithi005.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51847.log.gz 2024-03-24T11:07:07.530 INFO:teuthology.orchestra.run.smithi005.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48898.log.gz 2024-03-24T11:07:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56545.log 2024-03-24T11:07:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43600.log 2024-03-24T11:07:07.530 INFO:teuthology.orchestra.run.smithi005.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49984.log.gz 2024-03-24T11:07:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50026.log 2024-03-24T11:07:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55501.log 2024-03-24T11:07:07.530 INFO:teuthology.orchestra.run.smithi005.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56545.log.gz 2024-03-24T11:07:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43600.log: /var/log/ceph/ceph-client.admin.50026.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43600.log.gz 2024-03-24T11:07:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56713.log 2024-03-24T11:07:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45511.log 2024-03-24T11:07:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.55501.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50026.log.gz 2024-03-24T11:07:07.530 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51241.log 2024-03-24T11:07:07.530 INFO:teuthology.orchestra.run.smithi005.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55501.log.gz 2024-03-24T11:07:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55627.log 2024-03-24T11:07:07.531 INFO:teuthology.orchestra.run.smithi005.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56713.log.gz 2024-03-24T11:07:07.531 INFO:teuthology.orchestra.run.smithi005.stderr: 88.0%/var/log/ceph/ceph-client.admin.51241.log: gzip -- replaced with /var/log/ceph/ceph-client.admin.45511.log.gz 2024-03-24T11:07:07.531 INFO:teuthology.orchestra.run.smithi005.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47434.log 2024-03-24T11:07:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.55627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43432.log 2024-03-24T11:07:07.531 INFO:teuthology.orchestra.run.smithi005.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51241.log.gz 2024-03-24T11:07:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47434.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55627.log.gz 2024-03-24T11:07:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41908.log 2024-03-24T11:07:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56587.log 2024-03-24T11:07:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43432.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47434.log.gz 2024-03-24T11:07:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41908.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43432.log.gz 2024-03-24T11:07:07.531 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45553.log 2024-03-24T11:07:07.531 INFO:teuthology.orchestra.run.smithi005.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41908.log.gz 2024-03-24T11:07:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42302.log 2024-03-24T11:07:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53161.log 2024-03-24T11:07:07.532 INFO:teuthology.orchestra.run.smithi005.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56587.log.gz 2024-03-24T11:07:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42302.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45553.log.gz 2024-03-24T11:07:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46513.log 2024-03-24T11:07:07.532 INFO:teuthology.orchestra.run.smithi005.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42302.log.gz 2024-03-24T11:07:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.53161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57151.log 2024-03-24T11:07:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36959.log 2024-03-24T11:07:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46513.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53161.log.gz 2024-03-24T11:07:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.57151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50821.log 2024-03-24T11:07:07.532 INFO:teuthology.orchestra.run.smithi005.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46513.log.gz 2024-03-24T11:07:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.36959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53725.log 2024-03-24T11:07:07.532 INFO:teuthology.orchestra.run.smithi005.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.57151.log.gz 2024-03-24T11:07:07.532 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45217.log 2024-03-24T11:07:07.533 INFO:teuthology.orchestra.run.smithi005.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50821.log.gz 2024-03-24T11:07:07.533 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.53725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46261.log 2024-03-24T11:07:07.533 INFO:teuthology.orchestra.run.smithi005.stderr: 93.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41866.log 2024-03-24T11:07:07.533 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36959.log.gz 88.4%/var/log/ceph/ceph-client.admin.45217.log: -- replaced with /var/log/ceph/ceph-client.admin.53725.log.gz 2024-03-24T11:07:07.533 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-24T11:07:07.533 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48124.log 2024-03-24T11:07:07.533 INFO:teuthology.orchestra.run.smithi005.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45217.log.gz 2024-03-24T11:07:07.533 INFO:teuthology.orchestra.run.smithi005.stderr:gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46261.log.gz 2024-03-24T11:07:07.533 INFO:teuthology.orchestra.run.smithi005.stderr: -5/var/log/ceph/ceph-client.admin.41866.log: --verbose -- /var/log/ceph/ceph-client.admin.45031.log 2024-03-24T11:07:07.533 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48124.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41866.log.gz 2024-03-24T11:07:07.533 INFO:teuthology.orchestra.run.smithi005.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48124.log.gz 2024-03-24T11:07:07.533 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56755.log 2024-03-24T11:07:07.533 INFO:teuthology.orchestra.run.smithi005.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45031.log.gz 2024-03-24T11:07:07.533 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42309.log 2024-03-24T11:07:07.533 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54811.log 2024-03-24T11:07:07.534 INFO:teuthology.orchestra.run.smithi005.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56755.log.gz 2024-03-24T11:07:07.534 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46828.log 2024-03-24T11:07:07.534 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.54811.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-24T11:07:07.534 INFO:teuthology.orchestra.run.smithi005.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42309.log.gz 2024-03-24T11:07:07.534 INFO:teuthology.orchestra.run.smithi005.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51907.log 2024-03-24T11:07:07.534 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54811.log.gz 2024-03-24T11:07:07.534 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46828.log: /var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48250.log 2024-03-24T11:07:07.534 INFO:teuthology.orchestra.run.smithi005.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46828.log.gz 2024-03-24T11:07:07.534 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45175.log 2024-03-24T11:07:07.534 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43516.log 88.0% 2024-03-24T11:07:07.534 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51907.log.gz 2024-03-24T11:07:07.534 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53851.log 2024-03-24T11:07:07.534 INFO:teuthology.orchestra.run.smithi005.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.48250.log.gz 2024-03-24T11:07:07.534 INFO:teuthology.orchestra.run.smithi005.stderr:gzip/var/log/ceph/ceph-client.admin.43516.log: -5 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45175.log.gz 2024-03-24T11:07:07.534 INFO:teuthology.orchestra.run.smithi005.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48520.log 2024-03-24T11:07:07.535 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.53851.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43516.log.gz 2024-03-24T11:07:07.535 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49336.log 2024-03-24T11:07:07.535 INFO:teuthology.orchestra.run.smithi005.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53851.log.gz 2024-03-24T11:07:07.535 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41415.log 2024-03-24T11:07:07.535 INFO:teuthology.orchestra.run.smithi005.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48520.log.gz 2024-03-24T11:07:07.535 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54061.log 2024-03-24T11:07:07.535 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49336.log: /var/log/ceph/ceph-client.admin.41415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42298.log 2024-03-24T11:07:07.535 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.54061.log: gzip -5 88.3% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.49336.log.gz /var/log/ceph/ceph-client.admin.53497.log 2024-03-24T11:07:07.535 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-24T11:07:07.535 INFO:teuthology.orchestra.run.smithi005.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41415.log.gz 2024-03-24T11:07:07.535 INFO:teuthology.orchestra.run.smithi005.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54061.log.gz 2024-03-24T11:07:07.535 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53893.log 2024-03-24T11:07:07.535 INFO:teuthology.orchestra.run.smithi005.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42298.log.gz 2024-03-24T11:07:07.535 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.53497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54769.log 2024-03-24T11:07:07.535 INFO:teuthology.orchestra.run.smithi005.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53497.log.gz 2024-03-24T11:07:07.535 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.53893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54019.log 2024-03-24T11:07:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5/var/log/ceph/ceph-client.admin.54769.log: --verbose 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53893.log.gz 2024-03-24T11:07:07.536 INFO:teuthology.orchestra.run.smithi005.stderr: -- /var/log/ceph/ceph-client.admin.49648.log 2024-03-24T11:07:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.54019.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.54769.log.gz 2024-03-24T11:07:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37244.log 2024-03-24T11:07:07.536 INFO:teuthology.orchestra.run.smithi005.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54019.log.gz 2024-03-24T11:07:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52471.log 2024-03-24T11:07:07.536 INFO:teuthology.orchestra.run.smithi005.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49648.log.gz/var/log/ceph/ceph-client.admin.37244.log: 2024-03-24T11:07:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:gzip 88.6% -5 --verbose -- /var/log/ceph/ceph-client.admin.54979.log 2024-03-24T11:07:07.536 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37244.log.gz 2024-03-24T11:07:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57025.log 2024-03-24T11:07:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.54979.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52471.log.gz 2024-03-24T11:07:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50905.log 2024-03-24T11:07:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.57025.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54979.log.gz 2024-03-24T11:07:07.536 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46766.log 2024-03-24T11:07:07.537 INFO:teuthology.orchestra.run.smithi005.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57025.log.gz 2024-03-24T11:07:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45259.log 2024-03-24T11:07:07.537 INFO:teuthology.orchestra.run.smithi005.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50905.log.gz 2024-03-24T11:07:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5/var/log/ceph/ceph-client.admin.46766.log: --verbose -- /var/log/ceph/ceph-client.admin.52327.log 2024-03-24T11:07:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45259.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46766.log.gz 2024-03-24T11:07:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43034.log 2024-03-24T11:07:07.537 INFO:teuthology.orchestra.run.smithi005.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45259.log.gz 2024-03-24T11:07:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57571.log 89.2% 2024-03-24T11:07:07.537 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52327.log.gz 2024-03-24T11:07:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44047.log 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43034.log.gz 2024-03-24T11:07:07.537 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-24T11:07:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.57571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47392.log 2024-03-24T11:07:07.537 INFO:teuthology.orchestra.run.smithi005.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57571.log.gz 2024-03-24T11:07:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57361.log 2024-03-24T11:07:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47392.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44047.log.gz 2024-03-24T11:07:07.537 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41242.log 2024-03-24T11:07:07.538 INFO:teuthology.orchestra.run.smithi005.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47392.log.gz 2024-03-24T11:07:07.538 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.57361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42556.log 2024-03-24T11:07:07.538 INFO:teuthology.orchestra.run.smithi005.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57361.log.gz 2024-03-24T11:07:07.538 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54685.log 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41242.log.gz 2024-03-24T11:07:07.538 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-24T11:07:07.538 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48166.log 2024-03-24T11:07:07.538 INFO:teuthology.orchestra.run.smithi005.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42556.log.gz 2024-03-24T11:07:07.538 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.54685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43558.log 2024-03-24T11:07:07.538 INFO:teuthology.orchestra.run.smithi005.stderr:gzip/var/log/ceph/ceph-client.admin.48166.log: 87.5% -5 --verbose -- /var/log/ceph/ceph-client.admin.53557.log 2024-03-24T11:07:07.538 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54685.log.gz 2024-03-24T11:07:07.538 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43558.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48166.log.gz 2024-03-24T11:07:07.538 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42966.log 2024-03-24T11:07:07.538 INFO:teuthology.orchestra.run.smithi005.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43558.log.gz 2024-03-24T11:07:07.538 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52639.log 2024-03-24T11:07:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.53557.log: /var/log/ceph/ceph-client.admin.42966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50779.log 2024-03-24T11:07:07.539 INFO:teuthology.orchestra.run.smithi005.stderr: 87.8% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53557.log.gz 2024-03-24T11:07:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41499.log 2024-03-24T11:07:07.539 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42966.log.gz 2024-03-24T11:07:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52639.log: /var/log/ceph/ceph-client.admin.50779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49732.log 2024-03-24T11:07:07.539 INFO:teuthology.orchestra.run.smithi005.stderr: 87.0%/var/log/ceph/ceph-client.admin.41499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44989.log 2024-03-24T11:07:07.539 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50779.log.gz 2024-03-24T11:07:07.539 INFO:teuthology.orchestra.run.smithi005.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52639.log.gz 2024-03-24T11:07:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49732.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41499.log.gz 2024-03-24T11:07:07.539 INFO:teuthology.orchestra.run.smithi005.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49732.log.gz 2024-03-24T11:07:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54541.log 2024-03-24T11:07:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44821.log 2024-03-24T11:07:07.539 INFO:teuthology.orchestra.run.smithi005.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44989.log.gz 2024-03-24T11:07:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.54541.log: /var/log/ceph/ceph-client.admin.56839.log 2024-03-24T11:07:07.539 INFO:teuthology.orchestra.run.smithi005.stderr:gzip 88.3% -5 --verbose -- /var/log/ceph/ceph-client.admin.53641.log 2024-03-24T11:07:07.540 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54541.log.gz/var/log/ceph/ceph-client.admin.44821.log: 2024-03-24T11:07:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54499.log 2024-03-24T11:07:07.540 INFO:teuthology.orchestra.run.smithi005.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.44821.log.gz 2024-03-24T11:07:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:gzip 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56839.log.gz -5 2024-03-24T11:07:07.540 INFO:teuthology.orchestra.run.smithi005.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43768.log 2024-03-24T11:07:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.53641.log: /var/log/ceph/ceph-client.admin.54499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53371.log 88.4% 2024-03-24T11:07:07.540 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53641.log.gz 2024-03-24T11:07:07.540 INFO:teuthology.orchestra.run.smithi005.stderr: 89.2%/var/log/ceph/ceph-client.admin.43768.log: -- replaced with /var/log/ceph/ceph-client.admin.54499.log.gz 2024-03-24T11:07:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48208.log 2024-03-24T11:07:07.540 INFO:teuthology.orchestra.run.smithi005.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43768.log.gz 2024-03-24T11:07:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.53371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57781.log 2024-03-24T11:07:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48208.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53371.log.gz 2024-03-24T11:07:07.540 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53977.log 2024-03-24T11:07:07.541 INFO:teuthology.orchestra.run.smithi005.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48208.log.gz 2024-03-24T11:07:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.57781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45781.log 2024-03-24T11:07:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.53977.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57781.log.gz 2024-03-24T11:07:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44215.log 2024-03-24T11:07:07.541 INFO:teuthology.orchestra.run.smithi005.stderr: 88.3%/var/log/ceph/ceph-client.admin.45781.log: -- replaced with /var/log/ceph/ceph-client.admin.53977.log.gz 2024-03-24T11:07:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56377.log 2024-03-24T11:07:07.541 INFO:teuthology.orchestra.run.smithi005.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45781.log.gz 2024-03-24T11:07:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.44215.log: -- /var/log/ceph/ceph-osd.3.log 2024-03-24T11:07:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56377.log: gzip 87.4% -5 --verbose -- /var/log/ceph/ceph-client.admin.56023.log 2024-03-24T11:07:07.541 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44215.log.gz 2024-03-24T11:07:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-osd.3.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56377.log.gz 2024-03-24T11:07:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44299.log 2024-03-24T11:07:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51637.log 2024-03-24T11:07:07.541 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56023.log: /var/log/ceph/ceph-client.admin.44299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52285.log 2024-03-24T11:07:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51637.log: 88.1% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44299.log.gz 2024-03-24T11:07:07.542 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56023.log.gz 2024-03-24T11:07:07.542 INFO:teuthology.orchestra.run.smithi005.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51637.log.gz 2024-03-24T11:07:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53767.log 2024-03-24T11:07:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41144.log 2024-03-24T11:07:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52285.log: /var/log/ceph/ceph-client.admin.53767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42304.log 2024-03-24T11:07:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41144.log: 88.4% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52285.log.gz 2024-03-24T11:07:07.542 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53767.log.gz 2024-03-24T11:07:07.542 INFO:teuthology.orchestra.run.smithi005.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41144.log.gz 2024-03-24T11:07:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45991.log 2024-03-24T11:07:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42304.log: gzip -5 --verbose -- 87.8% /var/log/ceph/ceph-client.admin.56941.log 2024-03-24T11:07:07.542 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42304.log.gz 2024-03-24T11:07:07.542 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53245.log 2024-03-24T11:07:07.543 INFO:teuthology.orchestra.run.smithi005.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45991.log.gz 2024-03-24T11:07:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50424.log 2024-03-24T11:07:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56941.log: /var/log/ceph/ceph-client.admin.53245.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56941.log.gz 2024-03-24T11:07:07.543 INFO:teuthology.orchestra.run.smithi005.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53245.log.gz 2024-03-24T11:07:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52765.log 2024-03-24T11:07:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52369.log 2024-03-24T11:07:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50424.log: gzip/var/log/ceph/ceph-client.admin.52765.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48772.log 2024-03-24T11:07:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52369.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52765.log.gz 2024-03-24T11:07:07.543 INFO:teuthology.orchestra.run.smithi005.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50424.log.gz 2024-03-24T11:07:07.543 INFO:teuthology.orchestra.run.smithi005.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52369.log.gz 2024-03-24T11:07:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52555.log 2024-03-24T11:07:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44653.log 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48772.log.gz 2024-03-24T11:07:07.543 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-24T11:07:07.543 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50296.log 87.2% -- replaced with /var/log/ceph/ceph-client.admin.52555.log.gz 2024-03-24T11:07:07.543 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-24T11:07:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52159.log 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44653.log.gz 2024-03-24T11:07:07.544 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-24T11:07:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44779.log 2024-03-24T11:07:07.544 INFO:teuthology.orchestra.run.smithi005.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50296.log.gz 2024-03-24T11:07:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52159.log: gzip -5 --verbose -- 87.5% /var/log/ceph/ceph-client.admin.55981.log 2024-03-24T11:07:07.544 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52159.log.gz 2024-03-24T11:07:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56191.log 2024-03-24T11:07:07.544 INFO:teuthology.orchestra.run.smithi005.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44779.log.gz 2024-03-24T11:07:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44131.log 2024-03-24T11:07:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.55981.log: /var/log/ceph/ceph-client.admin.56191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49816.log 2024-03-24T11:07:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44131.log: 87.2% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56191.log.gz 2024-03-24T11:07:07.544 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55981.log.gz 2024-03-24T11:07:07.544 INFO:teuthology.orchestra.run.smithi005.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44131.log.gz 2024-03-24T11:07:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54289.log 2024-03-24T11:07:07.544 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47206.log 2024-03-24T11:07:07.545 INFO:teuthology.orchestra.run.smithi005.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49816.log.gz 2024-03-24T11:07:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.54289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48730.log 2024-03-24T11:07:07.545 INFO:teuthology.orchestra.run.smithi005.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54289.log.gz 2024-03-24T11:07:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42634.log 2024-03-24T11:07:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54331.log 2024-03-24T11:07:07.545 INFO:teuthology.orchestra.run.smithi005.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47206.log.gz/var/log/ceph/ceph-client.admin.42634.log: 2024-03-24T11:07:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48730.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51427.log 2024-03-24T11:07:07.545 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42634.log.gz 2024-03-24T11:07:07.545 INFO:teuthology.orchestra.run.smithi005.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48730.log.gz/var/log/ceph/ceph-client.admin.54331.log: 2024-03-24T11:07:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55105.log 2024-03-24T11:07:07.545 INFO:teuthology.orchestra.run.smithi005.stderr: 87.4%/var/log/ceph/ceph-client.admin.51427.log: -- replaced with /var/log/ceph/ceph-client.admin.54331.log.gz 2024-03-24T11:07:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47854.log 2024-03-24T11:07:07.545 INFO:teuthology.orchestra.run.smithi005.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51427.log.gz 2024-03-24T11:07:07.545 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51469.log 2024-03-24T11:07:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.55105.log: /var/log/ceph/ceph-client.admin.47854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47560.log 2024-03-24T11:07:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51469.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55105.log.gz 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47854.log.gz 2024-03-24T11:07:07.546 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-24T11:07:07.546 INFO:teuthology.orchestra.run.smithi005.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51469.log.gz 2024-03-24T11:07:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45637.log 2024-03-24T11:07:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57109.log 2024-03-24T11:07:07.546 INFO:teuthology.orchestra.run.smithi005.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47560.log.gz/var/log/ceph/ceph-client.admin.45637.log: 2024-03-24T11:07:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55855.log 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45637.log.gz 2024-03-24T11:07:07.546 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-24T11:07:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.57109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57823.log 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57109.log.gz 2024-03-24T11:07:07.546 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-24T11:07:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.55855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53035.log 88.1% 2024-03-24T11:07:07.546 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55855.log.gz 2024-03-24T11:07:07.546 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.57823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51553.log 2024-03-24T11:07:07.546 INFO:teuthology.orchestra.run.smithi005.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.57823.log.gz 2024-03-24T11:07:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52681.log 2024-03-24T11:07:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.53035.log: /var/log/ceph/ceph-client.admin.51553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46681.log 2024-03-24T11:07:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52681.log: 87.1% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53035.log.gz 2024-03-24T11:07:07.547 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51553.log.gz 2024-03-24T11:07:07.547 INFO:teuthology.orchestra.run.smithi005.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52681.log.gz 2024-03-24T11:07:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51115.log 2024-03-24T11:07:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-03-24T11:07:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46681.log: gzip/var/log/ceph/ceph-client.admin.51115.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41603.log 2024-03-24T11:07:07.547 INFO:teuthology.orchestra.run.smithi005.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46681.log.gz 2024-03-24T11:07:07.547 INFO:teuthology.orchestra.run.smithi005.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51115.log.gz 2024-03-24T11:07:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54583.log 2024-03-24T11:07:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45595.log 2024-03-24T11:07:07.547 INFO:teuthology.orchestra.run.smithi005.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41603.log.gz 2024-03-24T11:07:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.54583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43064.log 2024-03-24T11:07:07.547 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45595.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54583.log.gz 2024-03-24T11:07:07.548 INFO:teuthology.orchestra.run.smithi005.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45595.log.gz 2024-03-24T11:07:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-24T11:07:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41457.log 2024-03-24T11:07:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43064.log: /var/log/ceph/ceph-mgr.y.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.43064.log.gz 2024-03-24T11:07:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53809.log 2024-03-24T11:07:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41457.log: 89.9% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2024-03-24T11:07:07.548 INFO:teuthology.orchestra.run.smithi005.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41457.log.gz 2024-03-24T11:07:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54121.log 2024-03-24T11:07:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.53809.log: -- /var/log/ceph/ceph-client.admin.48334.log 2024-03-24T11:07:07.548 INFO:teuthology.orchestra.run.smithi005.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53809.log.gz 2024-03-24T11:07:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.54121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45343.log 2024-03-24T11:07:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48334.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54121.log.gz 2024-03-24T11:07:07.548 INFO:teuthology.orchestra.run.smithi005.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48334.log.gz 2024-03-24T11:07:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51199.log 2024-03-24T11:07:07.548 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45469.log 2024-03-24T11:07:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45343.log: /var/log/ceph/ceph-client.admin.51199.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45343.log.gz 2024-03-24T11:07:07.549 INFO:teuthology.orchestra.run.smithi005.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51199.log.gz 2024-03-24T11:07:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52807.log 2024-03-24T11:07:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45697.log 2024-03-24T11:07:07.549 INFO:teuthology.orchestra.run.smithi005.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45469.log.gz 2024-03-24T11:07:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54457.log 2024-03-24T11:07:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45697.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52807.log.gz 2024-03-24T11:07:07.549 INFO:teuthology.orchestra.run.smithi005.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45697.log.gz 2024-03-24T11:07:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49858.log 2024-03-24T11:07:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40899.log 2024-03-24T11:07:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.54457.log: /var/log/ceph/ceph-client.admin.49858.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54457.log.gz 2024-03-24T11:07:07.549 INFO:teuthology.orchestra.run.smithi005.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49858.log.gz 2024-03-24T11:07:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48376.log 2024-03-24T11:07:07.549 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49900.log 2024-03-24T11:07:07.550 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48376.log: /var/log/ceph/ceph-client.admin.40899.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48376.log.gz 2024-03-24T11:07:07.550 INFO:teuthology.orchestra.run.smithi005.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40899.log.gz 2024-03-24T11:07:07.550 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56149.log 2024-03-24T11:07:07.550 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51949.log 2024-03-24T11:07:07.550 INFO:teuthology.orchestra.run.smithi005.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49900.log.gz 2024-03-24T11:07:07.550 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41541.log 2024-03-24T11:07:07.550 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51949.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56149.log.gz 2024-03-24T11:07:07.550 INFO:teuthology.orchestra.run.smithi005.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51949.log.gz 2024-03-24T11:07:07.550 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57403.log 2024-03-24T11:07:07.550 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44173.log 2024-03-24T11:07:07.550 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41541.log: /var/log/ceph/ceph-client.admin.57403.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41541.log.gz 2024-03-24T11:07:07.550 INFO:teuthology.orchestra.run.smithi005.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57403.log.gz 2024-03-24T11:07:07.550 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51805.log 2024-03-24T11:07:07.550 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43726.log 2024-03-24T11:07:07.550 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44173.log: /var/log/ceph/ceph-client.admin.51805.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44173.log.gz 2024-03-24T11:07:07.551 INFO:teuthology.orchestra.run.smithi005.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51805.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47080.log 2024-03-24T11:07:07.551 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-24T11:07:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53455.log 2024-03-24T11:07:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47080.log: /var/log/ceph/ceph-client.admin.43726.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47080.log.gz 2024-03-24T11:07:07.551 INFO:teuthology.orchestra.run.smithi005.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43726.log.gz 2024-03-24T11:07:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56317.log 2024-03-24T11:07:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.53455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46429.log 2024-03-24T11:07:07.551 INFO:teuthology.orchestra.run.smithi005.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53455.log.gz 2024-03-24T11:07:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44509.log 2024-03-24T11:07:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46429.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56317.log.gz 2024-03-24T11:07:07.551 INFO:teuthology.orchestra.run.smithi005.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46429.log.gz 2024-03-24T11:07:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51991.log 2024-03-24T11:07:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54415.log 2024-03-24T11:07:07.551 INFO:teuthology.orchestra.run.smithi005.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44509.log.gz 2024-03-24T11:07:07.551 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56107.log 2024-03-24T11:07:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.54415.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51991.log.gz 2024-03-24T11:07:07.552 INFO:teuthology.orchestra.run.smithi005.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54415.log.gz 2024-03-24T11:07:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52411.log 2024-03-24T11:07:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47686.log 2024-03-24T11:07:07.552 INFO:teuthology.orchestra.run.smithi005.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56107.log.gz 2024-03-24T11:07:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48814.log 2024-03-24T11:07:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47686.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52411.log.gz 2024-03-24T11:07:07.552 INFO:teuthology.orchestra.run.smithi005.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.47686.log.gz 2024-03-24T11:07:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43642.log 2024-03-24T11:07:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48814.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-24T11:07:07.552 INFO:teuthology.orchestra.run.smithi005.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48814.log.gz 2024-03-24T11:07:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44089.log 2024-03-24T11:07:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-osd.0.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43642.log.gz 2024-03-24T11:07:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56503.log 2024-03-24T11:07:07.552 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44089.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44089.log.gz 2024-03-24T11:07:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55249.log 2024-03-24T11:07:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44383.log 2024-03-24T11:07:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.55249.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56503.log.gz 2024-03-24T11:07:07.553 INFO:teuthology.orchestra.run.smithi005.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55249.log.gz 2024-03-24T11:07:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50551.log 2024-03-24T11:07:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44383.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44383.log.gz 2024-03-24T11:07:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56461.log 2024-03-24T11:07:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50551.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50551.log.gz 2024-03-24T11:07:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44863.log 2024-03-24T11:07:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56461.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56461.log.gz 2024-03-24T11:07:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53077.log 2024-03-24T11:07:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44863.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44863.log.gz 2024-03-24T11:07:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56797.log 2024-03-24T11:07:07.553 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.53077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48562.log 2024-03-24T11:07:07.553 INFO:teuthology.orchestra.run.smithi005.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.53077.log.gz 2024-03-24T11:07:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56797.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56797.log.gz 2024-03-24T11:07:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46075.log 2024-03-24T11:07:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48562.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48562.log.gz 2024-03-24T11:07:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51283.log 2024-03-24T11:07:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55333.log 2024-03-24T11:07:07.554 INFO:teuthology.orchestra.run.smithi005.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46075.log.gz 2024-03-24T11:07:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57924.log 2024-03-24T11:07:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.55333.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51283.log.gz 2024-03-24T11:07:07.554 INFO:teuthology.orchestra.run.smithi005.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55333.log.gz 2024-03-24T11:07:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49168.log 2024-03-24T11:07:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51595.log 2024-03-24T11:07:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49168.log: /var/log/ceph/ceph-client.admin.57924.log: 87.6% 87.5% -- replaced with /var/log/ceph/ceph-client.admin.57924.log.gz 2024-03-24T11:07:07.554 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49168.log.gz 2024-03-24T11:07:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41950.log 2024-03-24T11:07:07.554 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51595.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51595.log.gz 2024-03-24T11:07:07.555 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52723.log 2024-03-24T11:07:07.555 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41950.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41950.log.gz 2024-03-24T11:07:07.555 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54895.log 2024-03-24T11:07:07.555 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52723.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52723.log.gz 2024-03-24T11:07:07.555 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46996.log 2024-03-24T11:07:07.555 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.54895.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54895.log.gz 2024-03-24T11:07:07.555 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-24T11:07:07.555 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46996.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46996.log.gz 2024-03-24T11:07:07.555 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41824.log 2024-03-24T11:07:07.555 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph.log: 88.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-24T11:07:07.555 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47812.log 2024-03-24T11:07:07.555 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41824.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41824.log.gz 2024-03-24T11:07:07.555 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41059.log 2024-03-24T11:07:07.555 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47812.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47812.log.gz 2024-03-24T11:07:07.555 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55375.log 2024-03-24T11:07:07.556 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41059.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41059.log.gz 2024-03-24T11:07:07.556 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-24T11:07:07.556 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.55375.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55375.log.gz 2024-03-24T11:07:07.556 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44947.log 2024-03-24T11:07:07.556 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50068.log 2024-03-24T11:07:07.556 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44947.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44947.log.gz 2024-03-24T11:07:07.556 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-24T11:07:07.556 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50068.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50068.log.gz 2024-03-24T11:07:07.556 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56065.log 2024-03-24T11:07:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-mds.c.log: 93.3% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-24T11:07:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56419.log 2024-03-24T11:07:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56065.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56065.log.gz 2024-03-24T11:07:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53203.log 2024-03-24T11:07:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56419.log: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57067.log 2024-03-24T11:07:07.561 INFO:teuthology.orchestra.run.smithi005.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56419.log.gz 2024-03-24T11:07:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.53203.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53203.log.gz 2024-03-24T11:07:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41782.log 2024-03-24T11:07:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.57067.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57067.log.gz 2024-03-24T11:07:07.561 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41014.log 2024-03-24T11:07:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41782.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41782.log.gz 2024-03-24T11:07:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50152.log 2024-03-24T11:07:07.562 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41014.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41014.log.gz 2024-03-24T11:07:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47956.log 2024-03-24T11:07:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50152.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50152.log.gz 2024-03-24T11:07:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57655.log 2024-03-24T11:07:07.564 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47956.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47956.log.gz 2024-03-24T11:07:07.572 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51073.log 2024-03-24T11:07:07.586 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.57655.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57655.log.gz 2024-03-24T11:07:07.599 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42034.log 2024-03-24T11:07:07.599 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51073.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51073.log.gz 2024-03-24T11:07:07.606 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51763.log 2024-03-24T11:07:07.606 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42034.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42034.log.gz 2024-03-24T11:07:07.610 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45133.log 2024-03-24T11:07:07.610 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51763.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51763.log.gz 2024-03-24T11:07:07.613 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42698.log 2024-03-24T11:07:07.613 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45133.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45133.log.gz 2024-03-24T11:07:07.613 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44467.log 2024-03-24T11:07:07.613 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42698.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42698.log.gz 2024-03-24T11:07:07.613 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41740.log 2024-03-24T11:07:07.613 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44467.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44467.log.gz 2024-03-24T11:07:07.625 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47602.log 2024-03-24T11:07:07.626 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41740.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41740.log.gz 2024-03-24T11:07:07.641 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57697.log 2024-03-24T11:07:07.641 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47602.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47602.log.gz 2024-03-24T11:07:07.647 INFO:teuthology.orchestra.run.smithi005.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-24T11:07:07.649 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54247.log 2024-03-24T11:07:07.649 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.57697.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57697.log.gz 2024-03-24T11:07:07.651 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48688.log 2024-03-24T11:07:07.651 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.54247.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54247.log.gz 2024-03-24T11:07:07.651 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56629.log 2024-03-24T11:07:07.651 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48688.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48688.log.gz 2024-03-24T11:07:07.653 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52513.log 2024-03-24T11:07:07.653 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.56629.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56629.log.gz 2024-03-24T11:07:07.654 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41331.log 2024-03-24T11:07:07.654 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52513.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52513.log.gz 2024-03-24T11:07:07.654 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50212.log 2024-03-24T11:07:07.655 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.41331.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41331.log.gz 2024-03-24T11:07:07.655 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47164.log 2024-03-24T11:07:07.656 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50212.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50212.log.gz 2024-03-24T11:07:07.656 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49942.log 2024-03-24T11:07:07.661 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47164.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47164.log.gz 2024-03-24T11:07:07.661 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46345.log 2024-03-24T11:07:07.662 INFO:teuthology.orchestra.run.smithi005.stderr:gzip/var/log/ceph/ceph-client.admin.49942.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44005.log 2024-03-24T11:07:07.662 INFO:teuthology.orchestra.run.smithi005.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49942.log.gz 2024-03-24T11:07:07.662 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46912.log 2024-03-24T11:07:07.662 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44005.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46345.log.gz 2024-03-24T11:07:07.662 INFO:teuthology.orchestra.run.smithi005.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44005.log.gz 2024-03-24T11:07:07.662 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43306.log 2024-03-24T11:07:07.662 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46912.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46912.log.gz 2024-03-24T11:07:07.662 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55147.log 2024-03-24T11:07:07.662 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43306.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43306.log.gz 2024-03-24T11:07:07.662 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50254.log 2024-03-24T11:07:07.663 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.55147.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55147.log.gz 2024-03-24T11:07:07.664 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45301.log 2024-03-24T11:07:07.664 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50254.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50254.log.gz 2024-03-24T11:07:07.668 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57445.log 2024-03-24T11:07:07.668 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45301.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45301.log.gz 2024-03-24T11:07:07.668 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53287.log 2024-03-24T11:07:07.668 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.57445.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57445.log.gz 2024-03-24T11:07:07.668 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51385.log 2024-03-24T11:07:07.668 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.53287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54727.log 2024-03-24T11:07:07.668 INFO:teuthology.orchestra.run.smithi005.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53287.log.gz 2024-03-24T11:07:07.668 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.51385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48418.log 2024-03-24T11:07:07.668 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.54727.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54727.log.gz 2024-03-24T11:07:07.671 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50863.log 2024-03-24T11:07:07.671 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48418.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48418.log.gz 2024-03-24T11:07:07.671 INFO:teuthology.orchestra.run.smithi005.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.51385.log.gz 2024-03-24T11:07:07.671 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43222.log 2024-03-24T11:07:07.671 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50863.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50863.log.gz 2024-03-24T11:07:07.671 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45949.log 2024-03-24T11:07:07.672 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43222.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43222.log.gz 2024-03-24T11:07:07.675 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50719.log 2024-03-24T11:07:07.675 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45949.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45949.log.gz 2024-03-24T11:07:07.675 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53119.log 2024-03-24T11:07:07.676 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50719.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50719.log.gz 2024-03-24T11:07:07.676 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43348.log 2024-03-24T11:07:07.676 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.53119.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53119.log.gz 2024-03-24T11:07:07.676 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49504.log 2024-03-24T11:07:07.676 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43348.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43348.log.gz 2024-03-24T11:07:07.681 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49294.log 2024-03-24T11:07:07.681 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49504.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49504.log.gz 2024-03-24T11:07:07.681 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48292.log 2024-03-24T11:07:07.681 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49294.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49294.log.gz 2024-03-24T11:07:07.681 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49546.log 2024-03-24T11:07:07.681 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48292.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48292.log.gz 2024-03-24T11:07:07.681 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44611.log 2024-03-24T11:07:07.681 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49546.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49546.log.gz 2024-03-24T11:07:07.681 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43945.log 2024-03-24T11:07:07.682 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.44611.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44611.log.gz 2024-03-24T11:07:07.682 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57613.log 2024-03-24T11:07:07.682 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.43945.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43945.log.gz 2024-03-24T11:07:07.687 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48982.log 2024-03-24T11:07:07.687 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.57613.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.57613.log.gz 2024-03-24T11:07:07.687 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50635.log 2024-03-24T11:07:07.687 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.48982.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48982.log.gz 2024-03-24T11:07:07.687 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49690.log 2024-03-24T11:07:07.687 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.50635.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50635.log.gz 2024-03-24T11:07:07.687 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46159.log 2024-03-24T11:07:07.687 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.49690.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49690.log.gz 2024-03-24T11:07:07.687 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45865.log 2024-03-24T11:07:07.687 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.46159.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46159.log.gz 2024-03-24T11:07:07.688 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-24T11:07:07.688 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.45865.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45865.log.gz 2024-03-24T11:07:07.689 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52933.log 2024-03-24T11:07:07.709 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57529.log 2024-03-24T11:07:07.710 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.52933.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52933.log.gz 2024-03-24T11:07:07.720 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47290.log 2024-03-24T11:07:07.720 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.57529.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57529.log.gz 2024-03-24T11:07:07.727 INFO:teuthology.orchestra.run.smithi005.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-24T11:07:07.728 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42303.log 2024-03-24T11:07:07.728 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.47290.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47290.log.gz 2024-03-24T11:07:07.729 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/ceph-client.admin.42303.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42303.log.gz 2024-03-24T11:07:07.811 INFO:teuthology.orchestra.run.smithi057.stderr:/var/log/ceph/ceph-mon.c.log: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-24T11:07:07.929 INFO:teuthology.orchestra.run.smithi057.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-24T11:07:08.312 INFO:teuthology.orchestra.run.smithi005.stderr: 89.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-24T11:07:08.547 INFO:teuthology.orchestra.run.smithi005.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-24T11:07:08.549 INFO:teuthology.orchestra.run.smithi005.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-24T11:07:08.606 INFO:teuthology.orchestra.run.smithi005.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-24T11:07:08.606 INFO:teuthology.orchestra.run.smithi005.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-24T11:07:08.608 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-24T11:07:08.608 INFO:teuthology.orchestra.run.smithi005.stderr:real 0m1.290s 2024-03-24T11:07:08.608 INFO:teuthology.orchestra.run.smithi005.stderr:user 0m5.767s 2024-03-24T11:07:08.608 INFO:teuthology.orchestra.run.smithi005.stderr:sys 0m0.426s 2024-03-24T11:07:08.625 INFO:teuthology.orchestra.run.smithi057.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-24T11:07:08.634 INFO:teuthology.orchestra.run.smithi057.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-24T11:07:08.721 INFO:teuthology.orchestra.run.smithi057.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-24T11:07:08.738 INFO:teuthology.orchestra.run.smithi057.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-24T11:07:08.740 INFO:teuthology.orchestra.run.smithi057.stderr: 2024-03-24T11:07:08.741 INFO:teuthology.orchestra.run.smithi057.stderr:real 0m1.392s 2024-03-24T11:07:08.741 INFO:teuthology.orchestra.run.smithi057.stderr:user 0m5.624s 2024-03-24T11:07:08.741 INFO:teuthology.orchestra.run.smithi057.stderr:sys 0m0.232s 2024-03-24T11:07:08.741 INFO:tasks.ceph:Archiving logs... 2024-03-24T11:07:08.741 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619063/remote/smithi005/log 2024-03-24T11:07:08.742 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T11:07:09.677 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619063/remote/smithi057/log 2024-03-24T11:07:09.677 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T11:07:10.316 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-24T11:07:10.560 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=64b70c7b635249719f74d0188f0e38f2 Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1916, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1483, in healthy fs.wait_for_daemons(timeout=300) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/filesystem.py", line 1241, in wait_for_daemons raise RuntimeError("Timed out waiting for MDS daemons to become healthy") RuntimeError: Timed out waiting for MDS daemons to become healthy During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 1935, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-24T11:07:10.564 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-24T11:07:10.574 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-24T11:07:10.609 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-24T11:07:10.609 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T11:07:10.623 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T11:07:10.699 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T11:07:10.707 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T11:07:10.707 DEBUG:teuthology.orchestra.run.smithi005:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:07:10.709 DEBUG:teuthology.orchestra.run.smithi057:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:07:10.720 INFO:teuthology.orchestra.run.smithi005.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:07:10.722 INFO:teuthology.orchestra.run.smithi057.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:07:10.724 INFO:teuthology.orchestra.run.smithi005.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:07:10.724 INFO:teuthology.orchestra.run.smithi005.stdout:=============================================================================== 2024-03-24T11:07:10.724 INFO:teuthology.orchestra.run.smithi005.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 +588us[ +588us] +/- 69ms 2024-03-24T11:07:10.725 INFO:teuthology.orchestra.run.smithi005.stdout:^* hv02.front.sepia.ceph.com 3 6 377 39 -299us[ -293us] +/- 63ms 2024-03-24T11:07:10.725 INFO:teuthology.orchestra.run.smithi005.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 -260us[ -260us] +/- 70ms 2024-03-24T11:07:10.725 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:07:10.726 INFO:teuthology.orchestra.run.smithi057.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:07:10.727 INFO:teuthology.orchestra.run.smithi057.stdout:=============================================================================== 2024-03-24T11:07:10.727 INFO:teuthology.orchestra.run.smithi057.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 +592us[ +592us] +/- 69ms 2024-03-24T11:07:10.727 INFO:teuthology.orchestra.run.smithi057.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 -282us[ -288us] +/- 63ms 2024-03-24T11:07:10.727 INFO:teuthology.orchestra.run.smithi057.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 -249us[ -249us] +/- 70ms 2024-03-24T11:07:10.727 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:07:10.727 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T11:07:10.738 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T11:07:10.739 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T11:07:10.803 DEBUG:teuthology.orchestra.run.smithi005:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T11:07:10.878 DEBUG:teuthology.orchestra.run.smithi057:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T11:07:10.945 DEBUG:teuthology.orchestra.run.smithi005:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T11:07:10.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:07:10.975 DEBUG:teuthology.orchestra.run.smithi057:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T11:07:11.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:07:11.005 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T11:07:11.013 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T11:07:11.093 INFO:teuthology.task.internal:Duration was 2327.507437 seconds 2024-03-24T11:07:11.093 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T11:07:11.181 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T11:07:11.181 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T11:07:11.183 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T11:07:11.224 INFO:teuthology.orchestra.run.smithi005.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:07:11.227 INFO:teuthology.orchestra.run.smithi057.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:07:11.726 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T11:07:11.727 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi005.front.sepia.ceph.com 2024-03-24T11:07:11.727 DEBUG:teuthology.orchestra.run.smithi005:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T11:07:11.787 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi057.front.sepia.ceph.com 2024-03-24T11:07:11.787 DEBUG:teuthology.orchestra.run.smithi057:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T11:07:11.817 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T11:07:11.817 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T11:07:11.829 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T11:07:11.890 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T11:07:11.890 DEBUG:teuthology.orchestra.run.smithi005:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T11:07:11.897 DEBUG:teuthology.orchestra.run.smithi057:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T11:07:12.038 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T11:07:12.047 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T11:07:12.048 DEBUG:teuthology.orchestra.run.smithi005:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T11:07:12.074 DEBUG:teuthology.orchestra.run.smithi057:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T11:07:12.101 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T11:07:12.111 DEBUG:teuthology.orchestra.run.smithi005:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:07:12.118 DEBUG:teuthology.orchestra.run.smithi057:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:07:12.140 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = core 2024-03-24T11:07:12.168 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = core 2024-03-24T11:07:12.207 DEBUG:teuthology.orchestra.run.smithi005:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:07:12.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:07:12.226 DEBUG:teuthology.orchestra.run.smithi057:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:07:12.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:07:12.262 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T11:07:12.271 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T11:07:12.272 DEBUG:teuthology.misc:Transferring archived files from smithi005:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619063/remote/smithi005 2024-03-24T11:07:12.272 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T11:07:12.309 DEBUG:teuthology.misc:Transferring archived files from smithi057:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619063/remote/smithi057 2024-03-24T11:07:12.310 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T11:07:12.349 INFO:teuthology.task.internal:Removing archive directory... 2024-03-24T11:07:12.350 DEBUG:teuthology.orchestra.run.smithi005:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T11:07:12.352 DEBUG:teuthology.orchestra.run.smithi057:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T11:07:12.398 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T11:07:12.408 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T11:07:12.408 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T11:07:12.455 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T11:07:12.456 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T11:07:12.458 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T11:07:12.470 INFO:teuthology.orchestra.run.smithi005.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 11:07 /home/ubuntu/cephtest 2024-03-24T11:07:12.472 INFO:teuthology.orchestra.run.smithi057.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 11:07 /home/ubuntu/cephtest 2024-03-24T11:07:12.473 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T11:07:12.533 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T11:07:12.555 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T11:07:12.580 DEBUG:teuthology.nuke:shortname: smithi005 2024-03-24T11:07:12.580 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T11:07:12.605 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619063', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:18:53.323605', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFn+kiqlbs5zbTuBBHsHzEb/WYC+FI9IOgEygpTTmkkomwTk+KNf0/B2ea6hscWCvKN0mtgrm8ZyKJLkRdZ3nW8='} 2024-03-24T11:07:12.608 DEBUG:teuthology.nuke:shortname: smithi057 2024-03-24T11:07:12.609 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T11:07:12.635 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619063', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:18:53.324931', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMokMhSL3Mys8idkxCh6J+sEHRTJAMEjyaIKAU1fo3o+pBIjGzyT4BjzlGmoyU6eIQ8lbjbW+2u15K3TYKI+qsI='} 2024-03-24T11:07:12.656 INFO:teuthology.orchestra.console.smithi005:Power off 2024-03-24T11:07:12.656 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T11:07:12.680 DEBUG:teuthology.orchestra.console.smithi005:power off output: Chassis Power Control: Down/Off 2024-03-24T11:07:12.681 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:07:12.699 INFO:teuthology.orchestra.console.smithi057:Power off 2024-03-24T11:07:12.699 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T11:07:12.709 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is on 2024-03-24T11:07:12.724 DEBUG:teuthology.orchestra.console.smithi057:power off output: Chassis Power Control: Down/Off 2024-03-24T11:07:12.725 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:07:12.749 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2024-03-24T11:07:16.053 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2024-03-24T11:07:16.053 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-03-24T11:07:16.054 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~0s 2024-03-24T11:07:16.710 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:07:16.750 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:07:16.820 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is on 2024-03-24T11:07:16.862 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2024-03-24T11:07:20.822 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:07:20.863 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:07:20.934 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is off 2024-03-24T11:07:20.974 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is on 2024-03-24T11:07:21.036 INFO:teuthology.orchestra.console.smithi005:Power off completed 2024-03-24T11:07:21.055 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~5s 2024-03-24T11:07:21.055 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~5s 2024-03-24T11:07:21.055 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.e is failed for ~5s 2024-03-24T11:07:21.055 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-24T11:07:21.055 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-03-24T11:07:21.056 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.f is failed for ~0s 2024-03-24T11:07:24.978 DEBUG:teuthology.orchestra.console.smithi057:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T11:07:25.088 DEBUG:teuthology.orchestra.console.smithi057:check power output: Chassis Power is off 2024-03-24T11:07:25.189 INFO:teuthology.orchestra.console.smithi057:Power off completed 2024-03-24T11:07:25.294 INFO:teuthology.run:Summary data: description: fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/5 tasks/fsstress validater/valgrind} duration: 2327.5074367523193 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=64b70c7b635249719f74d0188f0e38f2 status: fail success: false 2024-03-24T11:07:25.294 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T11:07:25.441 INFO:teuthology.run:FAIL